Skip to content

In-situ muon efficiency method for the Z dilepton and W analyses - #709

Draft
davidwalter2 wants to merge 16 commits into
WMass:mainfrom
davidwalter2:260526_inSituEfficiencies
Draft

In-situ muon efficiency method for the Z dilepton and W analyses#709
davidwalter2 wants to merge 16 commits into
WMass:mainfrom
davidwalter2:260526_inSituEfficiencies

Conversation

@davidwalter2

@davidwalter2 davidwalter2 commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Draft — opened for visibility while the scope is still being split. Not ready to merge as-is, see Scope caveats below.

What this adds

An in-situ determination of the muon ID / HLT / Iso data-MC scale factors, as an alternative to the external tag-and-probe SFs. The three efficiency steps are floated as Chebyshev polynomials in (pT, uT), decorrelated per probe-eta bin, as unconstrained nuisances (not NOIs), so the scale factors are determined by the fit itself rather than taken from an external measurement.

The Z dilepton analysis is split into four categories — nominal, failIso, failHLT, failID — which become four fit channels sharing the same coefficient nuisances, and that cross-category constraint is what determines the efficiencies. Reco and tracking SFs stay external.

The passing-leg scale factor is the polynomial directly, f_pass = 1 + P, which makes it MC-independent; the MC efficiency enters only the failing-leg factor f_fail = (1 - eMC(1+P))/(1 - eMC). Each analysis therefore supplies its own effMC. The method is implemented for both the dilepton (two-leg tag-and-probe) and the W single-muon (single-leg) topologies through a shared C++ base. The coefficients are common to W and Z, so the two can be fitted simultaneously with the scale factors fully correlated between them.

The exact likelihood is linearised around a central point and iterated: make_insitu_effSF.py accumulates theta_new = theta_prev + delta * n_hat, fed back via --insituSFFile.

Workflow

  1. mz_dilepton.py --makeInsituEffMC --makeUTQuantileHists — probe spectra and uT quantile inputs
  2. scripts/corrections/make_insitu_effMC.py — per-step MC efficiencies
  3. mz_dilepton.py --insituEffMCFile ... — templates and coefficient response tensors
  4. setupRabbit.py --muonInsituEfficiency with the four --baseNames
  5. fit, then make_insitu_effSF.py to produce the next iteration's central coefficients

Conditioning of the coefficients

Chebyshev polynomials are orthogonal under the weight 1/sqrt(1-x^2), which is nothing like the probe density, so the coefficients come out strongly correlated — correlation condition number 6.1e4 over the 2112-coefficient block in the 2D fit, 1.3e5 in the 4D one. That is what made the fit slow: trust-krylov's inner solve needs ~sqrt(kappa) Hessian-vector products, and steps get rejected. Measured on the full-statistics 2D fit: 145 minutes, with 20 of 78 iterations returning a bit-identical loss and the worst single iteration burning 27 minutes to make no progress.

This branch previously carried an offline fix (make_insitu_basis.py plus --insituBasisFile) that recombined the coefficients into a basis orthonormal under the probe density. That has been removed in favour of preconditioning the fit parameters inside the fitter, WMass/rabbit#150, which builds the transform from the model's own Hessian, refreshes it as the fit moves, and needs no regenerated inputs upstream. The offline version also required the same basis file to be used across the W and Z histmakers by hand, since the nuisances are correlated between them — a consistency requirement nothing checked.

For reference, on the same full-statistics 2D input: unpreconditioned 145.5 min, the offline basis 179 min, preconditioning in rabbit 14.3 min. All three converge to genuine minima; the preconditioned fits reach EDM ~1e-16 against ~1e-11 unpreconditioned.

Scope caveats

  • The branch carries content merged in from the unmerged 260216_totalPhaseSpace branch (make_pixel_corrections.py, setup.sh, .gitignore). Those are not part of this feature and should be split out or land separately.
  • The in-situ mode is opt-in throughout: without --insituEffMCFile the histmakers follow the existing external tag-and-probe path unchanged.

🤖 Generated with Claude Code

@davidwalter2 @claude
In-situ muon efficiency method for the Z dilepton and W analyses
3dac3e2 
Determine the muon ID/HLT/Iso efficiency scale factors in situ, as
Chebyshev polynomials in (pT, uT) fitted as unconstrained nuisances,
instead of taking them from external tag-and-probe measurements.
Reco and tracking scale factors continue to come from tag-and-probe.
Covers both the Z dilepton path and the single-leg W path.

Squashed from the following commits:

  - WIP: in-situ muon efficiency + local changes before merging origin/260216
  - In-situ Z muon efficiency: finalize dilepton-only path and fix effStat
  - In-situ muon efficiency: (pT,uT) extension, both-legs effMC, uT quantiles
  - In-situ muon efficiency: logit SF, theory-weight fix, pT/uT binning
  - Restore full tag-and-probe smooth SF helpers alongside reco+tracking
  - mw: single-leg in-situ muon efficiency at parity with mz_dilepton
  - In-situ effMC: T&P-aligned probe denominator and tag-unbiased Z effMC
  - Fix HLT-bit operator precedence in effMC trigger flags; clamp uT at fill
  - In-situ effMC: logit-Chebyshev smoothing, correlation-factor plots, pt 26-90
  - In-situ helpers: separate tag-leg effMC (probe conditional, tag unbiased)
  - In-situ SF: linear f_pass = 1 + P (MC-independent passing leg)
  - In-situ effMC plots: per-cell eff(uT) cells and 2D eff(pT,uT) maps
  - plot_insitu_effSF: linear SF = 1 + P parameterization
  - In-situ efficiency: pt window to 65 GeV and basis orthogonalisation
  - Fix stale muon collection names in the dilepton CVH SF block

Co-Authored-By: Claude Opus 5 <[email protected]>
@davidwalter2
davidwalter2 force-pushed the 260526_inSituEfficiencies branch from 8339e1b to 3dac3e2 Compare August 18, 2026 15:33
davidwalter2 and others added 14 commits August 18, 2026 21:02
@davidwalter2 @claude
Drop the in-situ basis orthogonalisation
4ca17be 
The Chebyshev coefficients of the in-situ efficiency parameterisation are
strongly correlated under the probe density, which is what made the fit slow,
and this recombined them into a basis orthonormal under that density to fix the
conditioning. Preconditioning the fit parameters belongs in the fitter, where
the transform can be built from the model's own Hessian, refreshed as the fit
moves, and changed without regenerating anything upstream; rabbit does that now.
Keeping a second mechanism here would mean the histmaker output depends on a
basis file that has to be kept identical across the W and Z runs by hand.

Removes make_insitu_basis.py, --insituBasisFile from both histmakers and from
plot_insitu_effSF.py, the transform plumbing in muon_efficiencies_insitu.py, and
the per-block transform matrices from the C++ helper, which goes back to
evaluating the raw product basis directly.

Co-Authored-By: Claude Opus 5 <[email protected]>
@davidwalter2 @claude
Allow the muon scale and resolution nuisances to be split by analysis
3201f85 
Fitting W and Z together shares every nuisance of the same name, so the
muon momentum scale and resolution uncertainties become common to both and
the Z constrains the W's calibration, and vice versa. They should not:
each measurement carries its own calibration uncertainty.

--decorrelateMuonCalibration prefixes their baseName with 'dilepton_' or
'singlelepton_', giving each analysis its own copies. The prefix is keyed
off the dilepton flag rather than the channel, so the Z's four category
channels stay correlated with each other as they must. The systematic
group names are left alone, so grouped impacts still report the
calibration as one entry.

Off by default; single-analysis fits are unchanged. On the combined W+Z
input the 289 muon calibration nuisances become 578, each name appearing
once per analysis and none left untagged.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HrsrmERvJF2Vafn7wE1B3v
@davidwalter2 @claude
Cap the in-situ linearisation point at the physical boundary
b997996 
Feeding a fitted set of scale factors back as the linearisation point made
the histmakers exit rc=0 having processed a few per-mille of the MC events.
The Z signal kept 2954 of 394 million events and 620 of 51200 cells, with
398 effective entries where the previous iteration had 6e7. Nothing was
logged, and the yields still looked right to within 10% because the xsec
normalisation divides by the bookkeeping weight_sum, which does not know
how many events were processed.

The cause is that the fitted coefficients are unconstrained and leave the
physical region where little data constrains them -- the corners of the
pt/ut window. Where eMC*(1+P) >= 1 the fail factor's numerator turns
negative, central_factor threw, and the exception was swallowed inside
RunGraphs/IMT, ending that dataset's event loop silently. Single threaded
it surfaces:

  unphysical fail factor for step trigger at pt=50.3 eta=-1.49
  effMC=0.860014 P=0.223648 -> data efficiency effMC*(1+P) >= 1

1+P is now capped so eMC*(1+P) <= effData_max, projecting such a point back
onto the boundary instead. Value and gradient are both taken at the capped
point, so accumulate and central_factor still linearise around the same
place. With effData_max < 1 the fail numerator is positive by construction
and the fail-side throw is unreachable, so it goes; 1+P <= 0 still throws.

report_capped_cells logs where this happens, per step. Capping silently
would hide exactly the condition that caused the original bug -- a fit that
has walked outside the physical region -- and the count is the signal that
the scale factors should not be trusted there. On the iteration-1 file: 27
trigger and 172 iso cells for the Z effMC, 23 and 160 for the W.

Verified on 3 files with the same coefficients that previously truncated
the run: every process back to a 1.000 event ratio and effective entries
within 0.3% of the unreweighted reference.

The cap is a guard on the linearisation point, not a fit constraint, and it
bites hard where it fires: at eMC = 0.86 it leaves a fail factor of 7e-4,
turning a 2.4x overprediction into a 2.2x underprediction in the affected
bins. Constraining the fit itself is the better answer.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HrsrmERvJF2Vafn7wE1B3v
@davidwalter2 @claude
Write the in-situ efficiency grid for rabbit's physical-region penalty
c93226e 
The cap on the linearisation point is a guard applied after the fit has
already left the physical region, and it bites hard: at eMC = 0.86 it
leaves a fail factor of 7e-4, so bins holding thousands of data events lose
their prediction almost entirely. Constraining the fit is the better answer,
and rabbit's InSituEfficiencyBound does that -- but it needs the same MC
efficiencies the histmaker used, and the Chebyshev basis evaluated on them.

build_insitu_bound_aux emits one row per live effMC cell: the efficiency,
the basis there, and which coefficients it multiplies. --insituEffMCFile
stores it beside the datacard, so the grid cannot drift away from the
coefficient definitions the nuisances stand for.

It takes several files and merge_insitu_bound_aux stacks them, because a
combined fit shares the coefficients while each analysis keeps its own
effMC, and both evaluate fail factors -- the W through its non-isolated
sideband (category_expr "passIso ? 0 : 1"). Bounding against only one of
them would leave the other unconstrained. Every cell is kept, so the
binding constraint per coefficient is whichever analysis has the larger
eMC there.

The bundle carries coeff_scale = insitu_delta: the helper folds the
variation step into the stored response, so the fitted nuisance and the
polynomial coefficient differ by that factor.

The flat layout is now generated in one place, _insitu_cell_blocks, shared
by the bundle and by report_capped_cells, so the penalty and the report
cannot drift from each other or from the header.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HrsrmERvJF2Vafn7wE1B3v
@davidwalter2
Sample the in-situ bound at the Chebyshev window edges
ce4a668 
eval_leg clamps pt and ut to the Chebyshev window before the transform, so
the polynomial's value at an edge is its value everywhere beyond it. The
bound grid sampled bin centres only, which left the whole extrapolation
unconstrained: the effMC centres stop at pt = 57.5 while the window runs to
65, so the top 7.5 GeV -- and the ut tails beyond +70 -- were never seen by
the penalty. That is where the fitted scale factor curves diverge, and it is
invisible to a scan over bin centres: a fit that looked physical everywhere
it was checked reached f_fail = -22.9 at the edges.

Add the two window edges to the sample points along each axis, reusing the
nearest bin's efficiency, which is what the helper reads there anyway.
@davidwalter2
Compare two in-situ fits and allow per-step y ranges
a94cf51 
Make the covariance optional. A fit run with --noHessian has none, and the
comparison overlay is drawn as a dashed line with no band, so only the
primary input actually needs one.

Add --ylimStep step=lo,hi. The ID, isolation and trigger scale factors sit
on quite different scales, and one shared --ylim either flattens the small
ones or clips the large ones.

Report the largest scale factor difference between the two fits, both
absolutely and in units of the first fit's uncertainty. Two coefficient sets
can differ substantially while describing the same scale factors, so a
by-eye comparison of the curves is not enough to say whether two fits agree.
@davidwalter2
Update the submodules for the in-situ efficiency fits
e6219df 
rabbit moves to the branch carrying both the in-situ efficiency bound and
the bins-sharded multi-device fitter, which is what these fits run on.

narf and wums pick up two independent fixes needed to run at all with the
current container: boost-histogram's storage_type rename, and setAxisFlow
rebuilding axes from plain Python edge values.

wremnants-data picks up new resource files from upstream.
@davidwalter2 @claude
Point rabbit at the integration branch with the in-situ speedup
6d71976 
rabbit moves from 76b5f3d to 3e59c30, which adds the parameter snapshots'
threaded signal path and, more importantly, evaluates the in-situ polynomial
as a dense product instead of an atomic scatter.

That last change is worth the pointer bump on its own: the scatter cost 7.4 s
per Hessian-vector product against 0.29 s for the entire likelihood without
it, so it was ~97% of the wall clock of every in-situ fit run so far. On one
H200 the fit now reaches a lower loss in 7.6 hours than two 4x V100 fits
reached in 9 days 20 hours.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01VUPqWUC6BkC7qWM9SM86LH
@davidwalter2 @claude
Impose the in-situ bound on the total scale factor
32af2c4 
The bound arrays described the scale factor the fit varies, 1 + delta*P(n),
which is the whole scale factor only on iteration 0. From iteration 1 the
histmaker has already reweighted the MC by 1 + P(theta_central), so the total
is 1 + P(theta_central) + delta*P(n), and the penalty was guarding the wrong
quantity -- by up to 0.52 in scale factor where the accumulated correction is
largest.

Write the per-cell P(theta_central) into the bundle as "offset" and thread
the accumulated coefficient file through setupRabbit as --insituSFFile, the
same pkl the histmaker was run with. Iteration 0 passes nothing and the offset
is zero, so that path is unchanged.

The offset is capped exactly as fpass_capped is in the histmaker helper,
min(1+P, effMC_max/e) - 1. That is not cosmetic: where theta_central is
unphysical the MC was reweighted by the *capped* factor, so storing the
uncapped polynomial makes the bound and the histograms disagree about where
theta = 0 is, and the fit starts in a region the model does not represent.
With the T&P-derived central values that left 130 of 58368 cells at u >= 1
with a fail probability of -0.17, and the fit stalled 1700 above the capped
one.

Verified end to end: the offset the setup writes reproduces the previous
iteration's fitted coefficients to 1.9e-16, and the reconstruction from the
theta_central pkl is bit-identical to what the file contains.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_012sxpuabhZaYxUN3erGezFa
@davidwalter2 @claude
Express the external T&P scale factors in the in-situ basis
66065e4 
Gives the iteration a second, independent linearisation point. Instead of
starting from our own fit, start from the scale factors the default mW
analysis applies, and check the iteration reaches the same minimum. The
output is the same label-keyed theta_central pkl make_insitu_effSF.py writes,
so the histmakers and setupRabbit consume it through --insituSFFile unchanged.

The projection turns out to be exact rather than approximate, to ~1e-16.
SF = 1 + P(theta) is linear in theta, and the smoothed 3D scale factors were
themselves produced by smoothing with a 3rd-order polynomial in pt and 2nd
in uT -- the same function space our basis spans -- so the least-squares
solve reproduces them rather than approximating them. Shrinking the basis
to 3x2 breaks it immediately (residuals 3e-2), which is the control showing
the agreement is real and not a degenerate fit.

Sources match what muon_efficiencies_smooth actually applies: trigger and iso
from smoothSF3D_*, 3D in (eta, pt, uT), whose uT axis spans exactly
insitu_ut_range and whose 48 eta bins are ours; idip from allSmooth_*.root,
2D, since idip has no uT dependence and is absent from the 3D file.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_012sxpuabhZaYxUN3erGezFa
@davidwalter2 @claude
Overlay the external 3D scale factors on the in-situ ones
92e9107 
Compares the fitted scale factors against the default T&P ones at matched uT
slices, dashed against solid, so the two are read off the same axes.

Takes them from smoothSF3D_*.pkl.lz4 rather than the 2D allSmooth file: the
default analysis applies the 3D (eta, pt, uT) scale factors for trigger and
iso, so a 2D comparison would be against something the analysis does not use
and would attribute their uT dependence to us.

--sf3dIsoKey selects between smoothSF3D_iso (triggering muon) and
smoothSF3D_isonotrig; our in-situ iso leg is measured on Z dilepton, so which
one is the right comparison is a physics choice and left to the caller.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_012sxpuabhZaYxUN3erGezFa
@davidwalter2 @claude
Update rabbit for the total-scale-factor bound
bfe9ad9 
Picks up the matching rabbit-side change: the penalty now reads the per-cell
offset this repo writes into the bound bundle, so it guards
1 + P(theta_central) + delta*P(n) rather than only the part the fit varies.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_012sxpuabhZaYxUN3erGezFa
@davidwalter2 @claude
Add same-sign dilepton control regions for the in-situ fail categories
e5f359b 
The in-situ efficiencies are measured in the failID/failHLT/failIso legs, so
any nonprompt background there is absorbed into the fitted efficiency rather
than subtracted. Those legs turn out to be strongly fake-enriched: processes
that cannot give two prompt muons (W, W->tau, QG->WQ) are 0.004% of the MC in
the signal region but 1.7% in failID, a factor ~430. QCD multijet, which would
be two fakes, is not in the dilepton model at all.

--dileptonSameSign {plus,minus} rebuilds the pair from same-charge muons, so
the same regions can be produced in a sample that is almost pure nonprompt:
prompt Z->mumu is opposite-sign, and in a 60-file test it gives a same-sign
yield of 3e-6 of the opposite-sign one (charge misassignment), rising to 4e-4
in failID where badly measured muons both flip charge and fail ID.

The two charges are separate runs rather than one axis, because the fake
composition differs between them and merging would hide that.

Implementation note: the opposite-sign selection separates the pair with
"Muon_correctedCharge == {name}_charge0", one collection per sign. That cannot
work when both muons have the same charge -- both collections resolve to the
same muons and select_z_candidate's "exactly one of each" filter is never
satisfied. The same-sign branch therefore builds the collections itself, by pt
rank via the new wrem::nth_by_pt, and returns before the shared charge-based
Define. Nothing downstream is affected by the ordering: the tag/probe
assignment keys off the _tag0 flags and the event parity, not the charge.

The opposite-sign path is untouched -- the diff removes no lines from it -- and
a 60-file A/B gives identical yields through the unchanged branch.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_012sxpuabhZaYxUN3erGezFa
@davidwalter2 @claude
Update rabbit for the global-impacts fix
d313a80 
Picks up the rabbit-side change that lets --globalImpacts run from a
covariance loaded by --externalPostfit instead of silently computing nothing.
Not needed by the in-situ analysis itself, but the pointer should name the
code that actually produced the impact results in this working tree.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_012sxpuabhZaYxUN3erGezFa
github-actions[bot]
github-actions Bot reviewed Sep 18, 2026
View reviewed changes

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issues Found

Important (P2)

  • [P2] Remove leftover personal PATH export from setup.sh (/home/c/cmsmwbot/WRemnantsCI/actions-runner-5/_work/WRemnants/WRemnants/setup.sh:6)
    • setup.sh now appends the hardcoded user-specific directory /home/submit/david_w/.local/bin/ to PATH for every user sourcing this file. On the shared submit cluster that path exists and can shadow the centrally provided binaries as a fallback (e.g. pip --user entry points), and everywhere else it pollutes the environment of every interactive session and batch job. This looks like a development leftover from the author's own machine and should be dropped before merging (the new Singularity/Apptainer LD_LIBRARY_PATH workaround below it is self-contained and unaffected).

Minor (P3)

  • [P3] Per-channel HistoBoost reuses the fixed name "weight" (/home/c/cmsmwbot/WRemnantsCI/actions-runner-5/_work/WRemnants/WRemnants/scripts/histmakers/mz_dilepton.py:2019-2026)
    • In the new per-category channel loop, every MC channel books the nominal_weight control histogram with the fixed name "weight". In in-situ mode four channels (nominal, failIso, failHLT, failID) therefore produce four results with the same name for one dataset; narf's build_and_run keys the per-dataset output dict by histogram name (output[name] = H5PickleProxy(...)), so the first three are silently overwritten and only the failID variant reaches the pickle. T&P mode (single "nominal" channel) is unaffected. Name it f"{channel}_weight" to keep all four.

Summary

Total issues: 0 critical, 1 important, 1 minor.

Overall Verdict

Status: Patch has blocking issues

Explanation: The physics machinery (in-situ Chebyshev SF helpers in C++, effMC production/recombination, per-category histmaker split, and rabbit integration) is internally consistent, and the PR additionally fixes two genuine pre-existing bugs (HLT-string parenthesization and a wrong pixel-hit histogram key). Two issues were introduced: a hardcoded personal PATH export committed to setup.sh that pollutes every user's environment (P2), and the per-channel loop silently overwriting three of the four same-named "weight" debug histograms in in-situ mode (P3).

Review details
  • Model: cern/Kimi-K3-FLATIRON

Review metrics: 86 turns · 89 tool calls · 35m 22s

  • Tokens: 10.30M input · 8.5K output

HaydenTedrake added a commit to HaydenTedrake/WRemnants that referenced this pull request Oct 2, 2026
@HaydenTedrake @claude
Merge PR WMass#709 (in-situ muon efficiencies) into setuprabbit_5tev
9748d14 
Adopt the in-situ muon efficiency workflow from WMass/WRemnants PR WMass#709
(davidwalter2, head d313a80) as the basis for the 5 TeV efficiency
implementation: ID/HLT/Iso floated in-situ as Chebyshev coefficients, with
only reco+tracking taken from external tag-and-probe SFs.

Conflict resolutions:
- scripts/rabbit/setupRabbit.py: kept BOTH independent additions at the same
  spot -- main's SCETlib-NP response-matrix embedding and the PR's in-situ
  efficiency-bound auxiliary; each writes its own auxiliary.
- wremnants-data: kept OUR pointer (94d2ce48). The PR's pointer (124a4aee)
  differs only in five 5 TeV files -- the 5020 GeV pdfas/pdfvars theory
  corrections and three scarekit calibration files -- all older uploads
  (Jun 16 / Jul 2) of files since remade locally (Jul 13 / Jul 22). Taking it
  would silently revert the CT18Z-denominator corrections and the puWeight=1
  calibration. The PR needs nothing else from that bump.

Submodules move to the PR's commits: rabbit 9a84645 (needed for
rabbit.regularization.insitu), narf 57b6656, wums 3a55815. The local lnN
scalar-broadcast fix in rabbit/tensorwriter.py was re-applied on top of the
new rabbit (still not upstream).

Verified regression-free: the 5 TeV production + tensor + blinded-Asimov fit
on this tree reproduces the pre-merge result to 8.4e-14 across all 33 grouped
impact rows (sigma(alpha_s) 1.4933 either way).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
HaydenTedrake added a commit to HaydenTedrake/WRemnants that referenced this pull request Oct 2, 2026
@HaydenTedrake @claude
5 TeV: in-situ muon efficiencies following PR WMass#709
eedc666 
mz_5TeV.py gets the PR WMass#709 in-situ mode, implied by its inputs as in
mz_dilepton.py (--makeInsituEffMC, --insituEffMCFile, --insituSFFile):
tag-and-probe selection from the shared muon_selections helpers, four fit
channels (nominal = 2HLT_passIso, failIso, failHLT, failID) with every
systematic booked per channel, the muonInsituEff Chebyshev variations, the
pass-0 effMCprobe spectra and the iteration >= 1 central reweight. The
default (no in-situ flag) path is bit-identical to before.

setupRabbit_5TeV.py builds one channel per fit category
(--muonInsituEfficiency), registers the 2112 coefficients as unconstrained
mirrored nuisances with the 13 TeV names, and stores the
InSituEfficiencyBound auxiliary (--insituEffMCFile/--insituSFFile).
Default tensor unchanged.

Era_2017G + HLT_HIMu17 trigger matching: the filter bit is a placeholder (0 =
no muon passes) until the 5.02 TeV NanoAOD carries muon trigger objects.
make_insitu_effMC.py accepts the z_lowpu mode.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
HaydenTedrake added a commit to HaydenTedrake/WRemnants that referenced this pull request Oct 6, 2026
@HaydenTedrake @claude
5 TeV: nano v15 inputs, low-PU trigger matching, damping-wall API fix
4f4040a 
Nano v15 (5.02 TeV data and MC reproduced with muon trigger objects and the
standalone-muon branches the in-situ efficiencies need):
- mz_5TeV.py --nanoVersion {v9,v15} selects ONE production for data and MC via
  datasetDict_2017G NANO_PROD_TAGS / NANO_DATA_TAGS (getDatasets honours an
  era's tag maps, so productions are never mixed per sample). Default v9 keeps
  the old paths.
- nanoaod_compat.harmonize_nano_types: v12+ NanoAOD stores 14 integer branches
  in narrower types (e.g. Muon_nTrackerLayers uint8); redefine them to the v9
  types when they differ. No-op on v9.
- In-situ trigger matching uses wrem::goodMuonTriggerCandidateLowPU (muon
  TrigObj with filter bit 0 = "Mu17 for Muon"), as mz_lowPU.py does, passed
  through a new optional goodTrigObjs argument of apply_triggermatching_muon.
  The Era_2017G placeholder in utils.hpp/defines.hpp is removed.

np_damping_wall.py: the rabbit bump that came with PR WMass#709 passes indata= to
regularizers, routes -r key=value tokens to the regularizer instead of the
mapping, and adds set_expectations(parms=). Accept all three and route
smallb/margin through, so margin=0 is not silently replaced by the 5e-3
default; the configuration is printed.

hayden_notes.txt: reorganized; full 13 TeV pipeline description; v15 status.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@davidwalter2 @claude
Merge upstream main into 260526_inSituEfficiencies
329385a 
Conflicts resolved:
- mz_dilepton: keep the branch's per-channel layout and first/second muon
  naming; port upstream's top pt reweighting (+ nominal_topPtNNLO syst as
  <channel>_topPtNNLO), the recoil call (first/secondMuons), jpsi_style_cols
  for the J/psi-style helpers and the v9-only pixel multiplicity uncertainty.
- mw_with_mu_eta_pt, setupRabbit: keep both sides.
- rabbit: rebuilt the in-situ branch as upstream main + the 4 in-situ commits
  (261110_insituEfficiencyBound); the rest of the old branch is upstream.
- narf, wremnants-data: upstream (the narf commit of the branch is upstream).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GdFuNr1QskK393tsRzxHMU
@cmsmwbot

cmsmwbot Bot commented Nov 10, 2026

Copy link
Copy Markdown

CI run attempt 1 (42916136735) finished with a failure: linting failed and all downstream jobs were skipped.

Diagnosis: actions/checkout could not fetch the pinned rabbit submodule commit 213097db75546702e83aefecf94bbabde46d2526 ("not our ref" / exit 128) — that commit doesn't exist on the rabbit remote. Please push the referenced rabbit commit (or re-pin the submodule to a pushed ref). A retry (attempt 2) is already running but will hit the same failure until the ref exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

Assignees

No one assigned

Labels

efficiencies framework muon-calibration plotting w-mass z-mass

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@davidwalter2