Skip to content

Document DiffDock weight bundling, the NVRTC crash fix, and batch resume - #9

Merged
AJPreto merged 44 commits into
mainfrom
addressing_reviewers
Sep 23, 2026
Merged

AJPreto merged 44 commits into
mainfrom
addressing_reviewers

Conversation

@AJPreto

@AJPreto AJPreto commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds three doc-only clarifications to README.md answering a manuscript reviewer's DiffDock setup experience: DiffDock's model weights are downloaded at runtime from GitHub Releases (not bundled at build time, contrary to an earlier assumption — verified against the pinned gcorso/DiffDock commit and inference.py's download_and_extract fallback), the NVRTC/CUDA crash on GPU targets that commit 7f2e607 fixed by extending LD_LIBRARY_PATH on run-guild/run-boltz, and the batch-resume behavior exercised by tests/bulk_run/test_diffdock_resume.py (combinations with an existing *_confidence*.sdf are skipped on re-run without CLEAN=1).
  • No code, test, Dockerfile, or Makefile changes — README.md only.

Test plan

  • git diff --stat confirms only README.md changed
  • uv run ruff check . / uv run pytest -v tests/ — could not run in this sandbox (uv sync fails on a Linux/CUDA-only wheel, cuequivariance-ops-cu12, on Windows); unrelated to this docs-only change

🤖 Generated with Claude Code

Copilot AI lite review requested due to automatic review settings September 16, 2026 14:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Correct the documented batch-resume behavior and cache-location details.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This docs-only PR updates README.md with DiffDock setup, CUDA troubleshooting, model caching, and resume guidance.

Changes:

  • Documents runtime model-weight downloads and caching.
  • Explains the NVRTC/CUDA library-path workaround.
  • Describes batch resume behavior and CLEAN.
File summaries
File Summary
README.md Adds DiffDock installation, troubleshooting, caching, and resume documentation.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread README.md Resolved
Comment thread README.md Resolved
AJPreto and others added 26 commits September 16, 2026 15:38
Clarify behavior of the CLEAN option in README.

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Clarify model checkpoint download locations for DiffDock.

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
…se source

diffdock_score and boltz_score are pose-confidence values, not affinity
estimates, so they now keep their rp_* column but no longer vote in
global_rp_score -- the same status gnina_cnn_score already has. The
remaining tracks are averaged by which engine generated the pose before
averaging across engines, so DiffDock's and Boltz's two auto-added rescores
count as one vote per pose source instead of three. The prior flat mean is
still available via aggregation="flat".

Co-Authored-By: Claude Sonnet 5 <[email protected]>
run_pose_validity_analysis existed and was tested, but was reachable only
from the test suite -- scripts/run_guild.py and the Makefile had no
reference to it, so getting pose validity for anything real meant a
hand-rolled script that re-instantiated BulkRun and risked silently
validating the wrong batch layout.

Mirrors the PLIP flag pattern: --posebusters/--no-posebusters/
--posebusters-only plus --posebusters-config, and the matching Makefile
variables (NO_POSEBUSTERS, POSEBUSTERS_CONFIG) wired into _OPTIONAL_FLAGS,
plus a run-posebusters target. Runs by default after docking + scoring,
same as PLIP -- it's comparably cheap (~400ms/pose) and the two share the
same "always write, header-only when empty" file contract.

Each "-only" flag now means "skip docking + scoring and run only this
step"; combining --plip-only and --posebusters-only re-runs both analyses
over an existing tree without re-docking, since resolve_pipeline_steps
treats each step's own flag independently rather than one clobbering the
other.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
In the small-example-all-methods run, posebusters_validity.tsv and
posebusters_full_report.tsv were both written correctly (1,930 validated
poses), but guild_scores.txt came back with zero pb_ columns despite the
run's own README asserting they were merged. Reimplementing
_merge_posebusters_into_scores verbatim against the two output tables
reproduced the expected columns cleanly, so the merge arithmetic itself
was never the bug -- it was reachability, and the only signal of trouble
was a logger.warning nobody saw.

Two changes close that gap:

* _merge_posebusters_into_scores(required=...) raises instead of warning
  when guild_scores.txt is missing at merge time. run_pose_validity_analysis
  gained expect_existing_scores (default True), forwarded as `required`;
  --posebusters-only passes False, since a tree with no scoring done yet is
  the one case where the absence is genuinely expected rather than a sign
  the two steps resolved different project folders.
* A non-empty posebusters_df that adds zero merged columns now raises.
  groupby silently drops any row whose key is null in either grouping
  column, so a null combination_id or docking_method upstream previously
  degraded all the way to "merge did nothing," indistinguishable from
  "there was nothing to merge."

Co-Authored-By: Claude Sonnet 5 <[email protected]>
…lent

_COMPLEX_PDB_FOLDER_BY_METHOD covers boltz/vina/diffdock/gnina only, so
PoseBusters, PLIP and ProLIF all silently cover four of five methods --
karmadock has no row at all in posebusters_validity.tsv or
plip_interactions.tsv, which reads exactly like a clean bill of health in
any table that lists methods as rows, and nearly reached a reviewer
response undetected as one.

Went looking for route 1 (write generate_karmadock_complex_pdbs, mirroring
the four existing generate_*_complex_pdbs functions): KarmaDock's own
docking script (ligand_docking.py --docking --scoring --correct) is
external to this repo, and nothing in guild/docking/karmadock.py or
guild/constants/karmadock.py defines a predicted-pose output file for it --
only a scores CSV (karmadock_guild_scoring / process_karmadock_files). The
SDF/mol2 files center_karmadock_ligand_on_box touches are pre-docking INPUT
conformers guild builds from SMILES, not KarmaDock's predicted pose.
Extending the dict would mean guessing that external tool's output
convention rather than reading it off a verified path, so this takes route
2 instead: _complex_pdb_coverage() partitions requested methods into
complex-PDB-capable vs. structurally excluded, and both
run_pose_validity_analysis and run_interactions_analysis now log the
excluded set by name instead of letting it disappear. Route 1 stays open
as a real follow-up once KarmaDock's actual output convention is confirmed
against a real run.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
In the large Vina case study (vinarun, 403k pairs): 25.2% of rows have no
score at all, 6.06% of scored rows are non-negative (non-physical for a
Vina-family binding energy), 0.79% exceed 1,000,000 in magnitude (maximum
observed: 43,851,078), and only 93.9% fall within a plausible -20 to 0
kcal/mol. The figure notebook and the reviewer-response analysis code each
independently reimplemented the same `< 0.0` filter -- three copies of one
rule, none of them shared, and none of them visible anywhere a stored value
wasn't already being read.

is_physical_score(value, method) in guild.tools.scores centralizes that
rule: it consults SCORES_DIRECTION_DICTIONARY so a "maximum"-direction score
(karmadock_score, where positive is correct) is never flagged, and only
applies the Vina-family plausible range to methods that are actually a
Vina-family energy -- Nesso shares the "minimum" direction but is a
different physical quantity (log10(IC50/uM), not a docking energy), so it's
deliberately excluded.

run_guild_scoring now logs a per-method non-physical count as a warning,
the same place a failed docking attempt is already counted. Stored values
are unchanged by default -- the published case-study numbers were generated
with these values left in the table, so nulling them retroactively would
change those numbers for anyone reproducing them. exclude_non_physical=True
(--exclude-non-physical / EXCLUDE_NON_PHYSICAL=1) opts a NEW run into
nulling them instead, ranking a non-physical score exactly like a failed
docking attempt.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
… code

Three small things, batched:

* score_distribution.ipynb's markdown header cited
  figure_2_dockwizard_scores.txt; the code cell has always loaded
  figure_2_guild_scores.txt. Fixed the header to match the code.
* Added notebooks/analysis/reviewer_response/ (reproduce_response_analyses.py,
  merge_posebusters_flags.py, README.md), sourced from the reviewer-response
  session rather than reconstructed. reproduce_response_analyses.py
  regenerates every number quoted in the response that isn't read directly
  off a guild output table -- the runtime breakdown, the aggregation-rule
  comparison (which this branch's own commit 20b3c50 pose-source grouping
  feeds into), the normalisation comparison with its mechanism and failure-
  tail analysis, and a size-matched decoy control -- with a self-check
  confirming its local rank-percentile reimplementation matches guild's own
  rp_vina_score to < 1e-9. merge_posebusters_flags.py is the recovery path
  for the merge-didn't-run bug fixed in ba02666: it reproduces
  _merge_posebusters_into_scores against two output tables directly, for a
  tree where the merge needs redoing without a full rerun. Ran both against
  the staged inputs (self-check OK; --only a2 a3 confirmed) before adding.
* Pointed the main README's rank-percentile orientation note at
  guild/support/results/README.md's dockwizard_*/drrp_*/guild_* -> rp_*
  column rename, noting it also affects the Azure result files
  (vinarun/dockwizard_scores.txt, knownbindersvinarun/drrp_scores.txt).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
compute_rank_percentile_scores gains aggregation="pose_source_median" and
makes it the default. On the three-target benchmark (165 pairs, 15 known
binders, 150 decoys) the previous unweighted mean scored 0.781 AUC, below
AutoDock Vina alone (0.790), almost entirely because one track (DiffDock,
0.281 standalone) dragged it down with no defence against a single
aberrant vote. The median scores 0.824 with all five methods still
included. Performance-weighting (0.851) and dropping DiffDock (0.852)
both score higher, but the median needs no fitted parameter (weighting
requires each method's standalone AUC known in advance, circular for a
prospective tool) and makes no engine-specific judgement (dropping
DiffDock hard-codes something that may not hold on other targets).

This is a direction, not a significant result: confidence intervals
overlap heavily at 15 binders. aggregation="pose_source" (the previous
default, a mean) and aggregation="flat" both remain available so scores
produced under either earlier rule stay reproducible.

Within a pose source the combination stays a mean regardless of the
outer mode -- DiffDock and Boltz each have two rescore tracks, where
mean and median are identical anyway -- only the cross-source
combination changes.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
boltz_affinity_score is Boltz-2's own affinity prediction --
log10(IC50/uM), lower is more potent -- read from the same output tree
boltz_guild_scoring already parses, and populated whenever boltz runs.
It was deliberately excluded from every scoring registry as a
side-channel comparator for validating Nesso-1, but that rationale
doesn't survive the criterion the previous commit's exclusion of
diffdock_score/boltz_score actually rests on: it's a genuine affinity
estimate, not a pose confidence, so it qualifies to vote on the same
grounds vina_rescore_boltz and gnina_rescore_boltz do.

Adds BOLTZ_AFFINITY_PREFIX, registers it in SCORES_DIRECTION_DICTIONARY
("minimum", same convention as Nesso-1's head), RANKS_DICTIONARY,
RP_SCORES_DICTIONARY and SCORES_TO_USE_DICTIONARY, and maps it to
BOLTZ_PREFIX in POSE_SOURCE_DICTIONARY -- the affinity head is
co-predicted with the same complex the two rescores judge, so it joins
Boltz's existing vote as a third estimate rather than becoming a sixth
independent one. BulkRun.__init__ auto-adds the prefix to methods_to_run
whenever boltz is requested, the same place the rescore tracks are
added; it needs no docking step of its own (no entry in run_docking's
method_runners, so it's a no-op there like the rescore tracks).

Grepped for BOLTZ_PREFIX substring handling before wiring this up, since
the new prefix starts with it -- every existing check is an exact
membership test (`in self.methods_to_run`, dict key lookup), not a
startswith/split, so there's no collision to fix.

Whether the affinity head's 0.970 standalone AUC on the benchmark
reflects real generalisation or partly training-set recall is
unresolved: the 15 benchmark binders are ChEMBL compounds at pChEMBL
9.15-10.7, and Boltz-2's affinity head is trained on binding-affinity
data. That overlap audit is still outstanding, so this change is
justified on the grounds that an affinity score should be ranked like
one, not on the benchmark number -- no performance claim is added to
the README.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
…as 3

groupby(protein_col, group_keys=False).apply(_score_one_protein, ...)
relied on the grouping column being passed through to the callable and
back out. That stopped being the default on pandas 2.2+ (a
DeprecationWarning) and is gone on pandas 3.x, so the output frame
silently lost protein_config_id -- reproduced against a real pandas
3.0.5 install. Not reachable today (pyproject.toml pins pandas<3), but
every downstream consumer of guild_scores.txt groups by that column, so
it was a landmine for whenever that pin lifts.

protein_col was never used inside _score_one_protein's body, so the fix
is entirely about the frame that comes back out. Replaced the
groupby().apply() with groupby transforms over the whole frame
(.rank(), .transform("size"/"sum")) computed directly on
compute_rank_percentile_scores's result, one method at a time. This
keeps every column -- including the grouping one -- and every row's
original position by construction, so it also needs no
.reset_index(drop=True): a groupby-apply concatenates group results in
sorted-by-key order, which need not match input row order, so the old
reset_index was silently renumbering rows that may already have been
reordered relative to the input.

Numbers are unchanged: the existing orientation, denominator and
aggregation tests pass unmodified, on both pandas 2.3.3 and (checked
manually) pandas 3.0.5. Added a regression test with two proteins
interleaved in the opposite order to their alphabetical sort, so a
positional (rather than index-aligned) re-attachment of the dropped
column would have been caught by the row-order assertions rather than
just a column-presence check.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
…t image

tests/scripts/test_run_guild.py does `import run_guild` after adding
scripts/ to sys.path, which works locally but was failing collection
in CI with ModuleNotFoundError: No module named 'run_guild'.

The test Docker stage only ever copied guild/ and tests/ into the
image, never scripts/, and .dockerignore allowlisted only
scripts/apply_karmadock_patches.py under its "ignore everything by
default" policy -- so scripts/run_guild.py never reached the build
context in the first place. Added it to both, mirroring the existing
single-file pattern already used for apply_karmadock_patches.py;
run_guild.py imports nothing else from scripts/, so no other file is
needed.

Pre-existing since the PoseBusters CLI wiring test was added; unrelated
to the other changes on this branch.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Both notebooks read a results table via a path baked directly into the code,
and both paths are covered by the blanket *.txt rule in .gitignore and were
never committed -- so neither figure could be regenerated from a clean
checkout by anyone, including us.

Replaced the hardcoded paths with a GUILD_FIGURES_DATA_DIR-configurable data
directory in both notebooks (default "data", relative to the notebook's own
directory), mirroring the --data convention already established in
reviewer_response/reproduce_response_analyses.py.

Figure 2 now points at the 3-target rerun (guild_scores.txt) rather than the
many-target case-study file it used to read -- that original file no longer
exists anywhere reachable, including the Zenodo deposit (confirmed by
sampling 0.42% of the deposit's zip index across 120 windows: it holds only
vina_runs/dockwizard_scores.txt and diffdock_runs shards, zero boltz/
karmadock/gnina/decoy/knownbinder entries). The rerun is the deliberate
replacement for Figure 2; how the figure itself changes to use it is a
separate commit. Figure 3's two inputs (vinarun_scores.txt,
knownbinders_scores.txt) are unchanged and load today.

Added notebooks/analysis/README.md documenting, per figure, which file
feeds it, which columns it needs, and its provenance -- including why the
Zenodo deposit cannot reproduce either figure as published.

Added legacy_columns.py: one shared LEGACY_RENAME map for the three
column-naming generations these results files span (published/case-study/
current), imported by both notebooks instead of being duplicated in
score_comparison.ipynb and entirely absent from score_distribution.ipynb.
Also added the one generation LEGACY_RENAME was missing:
global_dockwizard_score, the combined column in vinarun_scores.txt.

Orientation is unaffected: checked directly against all three generations
(knownbinders_scores.txt: raw Vina -12.465 -> guild_vina_score 0.2; +24.668
-> 1.0), 0 = best throughout. Regenerating against current code does not
mirror any figure and nothing here changes a stored score, only where it is
read from.

Verified end-to-end: score_comparison.ipynb executes cleanly against the
staged Figure 3 inputs (133,000 decoys, 655 known binders loaded, matching
the expected counts) via nbconvert. score_distribution.ipynb's data loading
was checked the same way; its plotting code still expects the old
three-method column layout and is reworked in the next commit.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
…hoot

Rows B and C of Figure 2 plotted DiffDock's and Boltz-2's pose confidences
as the ranked quantity -- exactly what R2-3 objects to, and what 20b3c50
already stopped letting vote in global_rp_score. Rebuilt on the 3-target
rerun (guild_scores.txt) to plot what actually enters the score instead:
vina_rescore_diffdock_score and vina_rescore_boltz_score. Also added GNINA
and KarmaDock, so the figure covers all five pose sources its caption
already claimed ("across PLBP algorithms") instead of three.

The native confidences (diffdock_score, boltz_score) are still reported per
panel as a text note, but are no longer ranked or plotted as the voted
quantity -- the same treatment gnina_cnn_score already gets. Boltz-2's
panel also notes that boltz_affinity_score joins its vote (0d36671).

Rank percentiles are recomputed fresh via compute_rank_percentile_scores
rather than trusted from the file, since the rerun predates both
boltz_affinity_score's ranking (0d36671) and the median aggregation default
(0e7c959).

Two coverage problems surfaced once the rescore tracks were actually
plotted, both handled rather than silently plotted through:
- vina_rescore_boltz_score is populated for only 134/165 rows (receptor
  preparation failures).
- vina_rescore_diffdock_score has a 131/165 (79.4%) non-negative block --
  51 rows sitting at exactly 0.0 (a rescoring-failure sentinel distinct from
  guild.tools.scores.is_physical_score's boundary-inclusive [-20, 0] range,
  confirmed absent from vina_score/gnina_score/vina_rescore_boltz_score) plus
  83 genuinely positive rows. Both are nulled before ranking with a local,
  strictly-negative physicality check, consistent with how the reply to
  R2-4 already counts non-negative Vina-family values as failures. Only 34
  rows survive for this track; printed explicitly rather than left to look
  like full coverage.

Also fixes R4 b2: the right column's KDE ran on a grid from -0.06 to 1.06
with the axis merely clipped to (-0.04, 1.04), so the drawn density visibly
crossed the valid [0, 1] boundary -- exactly what the reviewer saw. Replaced
with kde_curve_bounded, a reflection-corrected KDE evaluated only on [0, 1],
so the curve cannot leave the valid range by construction. Verified it still
integrates to ~1.0 under np.trapezoid for every panel with enough points for
that check to be meaningful (below 10 points -- DiffDock's binder rescore
drops to n=3 -- the reflected KDE's own bandwidth estimate is too coarse to
hit the target reliably; reported rather than asserted away, since that is a
real property of how little data survives that track, not a bug in the
function). The left column (raw docking score) is genuinely unbounded and
keeps the original, unreflected KDE.

The figure now covers three targets where the published one covered many
more -- stated on the figure's own face, not left implicit, the same
honesty already applied to the per-method benchmark in the reply to R2-4.

Verified end-to-end via nbconvert against the real rerun data (not just
inspected): all five bounded-KDE panels integrate to within 1e-3 of 1.0
(except the n=3 DiffDock-binder case, flagged above), and the printed
non-physical/coverage counts match hand-checks against the raw file
(131/165 and 79.4% exactly). Cells 4-9 (the Sankey/alluvial plots, out of
scope here) still reference the old guild_vina_score/guild_diffdock_score/
guild_boltz_score column names this rerun file never had, so running the
full notebook top-to-bottom now fails partway through cell 5 -- unchanged
from before this commit's own scope (cells 1-3 only), but worth flagging
since it means the notebook is not yet regenerable end-to-end.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
… print AUCs

R4 b3: the grey and orange marks in Figure 3 are per-molecule rug ticks, not
"per-target values underlying the pooled distributions" as the current draft
reply to b3 says -- that sentence needs correcting in the letter. The decoy
rug is a random 400-molecule subsample, the known-binder rug is complete;
neither fact was visible before. Promoted to named constants
(RUG_SUBSAMPLE_N, RUG_SEED) and printed.

R4 b4: the y-axis was a bare "Density". gaussian_kde integrates to 1 over the
real line, but the panels are drawn on [0, 1] with hidden y-ticks and a
shared kmax, so what was drawn was not a unit-area curve within any one
panel. Relabelled "Probability density" and switched to the same
kde_curve_bounded introduced for Figure 2's b2 fix, so each curve genuinely
does integrate to 1 on its own bounded support. Moved kde_curve/
kde_curve_bounded into a shared notebooks/analysis/kde_helpers.py, imported
by both figure notebooks, rather than a second copy in this one. The shared
y-scale is kept -- cross-panel comparability is the point of the figure --
and is now stated in the printed output instead of only being inferable from
the hidden y-ticks.

The inversion: panel A plots 1 - rp_vina_score, while the stored column is
0 = best. That is the actual source of the Supp. Text 6 discrepancy ("1
indicates the top-ranked molecule") against guild.tools.scores' own
docstring ("0 = best") -- both are right about different things, and nobody
had noticed the notebook flips the stored orientation for this one plotted
axis. Not changed, just made impossible to miss: a comment at the inversion
plus an x-axis label that says the direction ("Normalized score (higher =
better)") instead of "Normalized score" alone -- which is also now correct
for panels B and C (already higher-is-better by construction).

Also prints the three AUCs, computed for the in-panel annotation but never
emitted. On this dataset: rank percentile 0.684, min-max 0.547, z-score
0.597 -- rank percentile beats z-score by a real margin here, unlike the
reviewer-response reproduction over 47 targets (0.675 vs. 0.674,
effectively tied; see reviewer_response/README.md). These are different
analyses: this notebook's own binder/decoy join vs. that script's stricter
"both classes present per target" filter. Reported rather than resolved --
whoever finalises the a3 reply needs to reconcile the two.

pandas 3: _decoy_pct, _zscore and _decoy_fit_unbounded used
groupby(protein_col, group_keys=False).apply(...), the same pattern e9a6fe7
removed from compute_rank_percentile_scores -- confirmed here to emit the
same FutureWarning on pandas 2.2+ that fix described. _zscore is now a
straightforward groupby(...).transform("mean"/"std"), since mu/sigma are
per-group scalars. _decoy_pct and _decoy_fit_unbounded reference a different
set (decoys only) than the group being scored, so neither is expressible as
a single .transform() call, but both are rewritten to stop using
DataFrameGroupBy.apply() entirely: a per-protein decoy-array lookup (built
by iterating the groupby object, not .apply()) plus the same per-value
arithmetic as before.

Verified against the real Figure 3 input data, not just inspected: the new
code's vina_min_max/decoy_pct/zscore_norm/decoy_fit_unbounded are identical
to the old groupby-apply implementation's output to within 8.8e-16 (floating-
point noise), with the same row count and row order (asserted in the
notebook itself). Ran the full notebook end-to-end via nbconvert against the
staged Figure 3 inputs -- 98,596 decoys, 220 known binders after the valid-
Vina-score filter, matching the pre-existing counts, with no change in
behaviour beyond what is described above.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
…path)

The reply to R2-7 promises Figure 3 repeated on a decoy panel matched to
each target's known binders, plus a descriptor-only baseline to separate
binding signal from property bias -- neither existed yet. Building the code
path, not the final render, per the prompt: on the 3-target rerun the
unmatched decoys are not size-matched (actives mean 445 Da vs decoys
353 Da) and a heavy-atom-count baseline alone reaches AUC 0.865, higher
than any docking method; size-matching collapses it to 0.645, so this
control is not academic.

Added guild/tools/decoy_matching.py:
- match_decoys_to_binders: per target, keeps a decoy only if it is within
  tolerance of at least one known binder on molecular weight, cLogP, HBA,
  HBD, rotatable bonds and net charge, AND its ECFP4 Tanimoto similarity to
  every known binder for that target is below 0.35 -- so a "matched" decoy
  is never simply a close structural analogue of the binder it was matched
  to. Reuses assign_properties from guild.tools.ligand_properties for
  MW/logP/HBA/HBD rather than reimplementing them, following
  notebooks/case_study_data/build_decoy.ipynb's own conventions; only
  rotatable-bond count and net formal charge are new, since
  assign_properties does not compute them.
- descriptor_only_auc: binder-vs-decoy AUC from each descriptor alone, no
  docking score at all -- the control itself.
- combined_descriptor_auc: cross-validated logistic-regression AUC across
  all six descriptors together, so the number isn't inflated by fitting and
  scoring the same rows.

18 tests in tests/ligand_properties/test_decoy_matching.py, using real
molecules with hand-verified properties (e.g. ibuprofen vs. its own methyl
ester, Tanimoto 0.656, rejected as a close analogue; vs. 4-phenylbutyric
acid, Tanimoto 0.243 and property-matched, kept) rather than mocks, plus a
constructed pure-size-confound case for the descriptor baseline. Also
smoke-tested directly against a 2-target slice of the real Figure 3 inputs
(249/2,000 decoys matched; descriptor AUCs 0.27-0.62) to confirm the real
column schema works end to end, not just the synthetic test fixtures.

score_comparison.ipynb's decoy-loading cell now takes a DECOY_SUBSET
(explicit ligand_id list, or a predicate), defaulting to None so current
behaviour is unchanged -- verified end-to-end via nbconvert that the
default reproduces Figure 3's three AUCs exactly (0.684 / 0.547 / 0.597,
byte for byte) so the parameterisation is provably inert. Added a
demonstration cell that runs the real matching + both baselines against a
5-target slice of the actual Figure 3 inputs (460/5,000 decoys kept;
combined descriptor AUC 0.778) and explains how to scale it to the full
133,000-decoy pool. Full scale is not run here: assign_properties computes
RDKit descriptors one molecule at a time, and the 5,000-decoy check above
already took several seconds, so the full pool is several minutes per call
and rendering the matched panel itself is left as follow-up work.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
The rank-percentile schematic embedded in Supplementary Text 3
(guild_rank_percentile_figure.png / Revision 1.docx's image4.png) was
hand-built with illustrative histograms, and nothing in the repository
produced it -- so it had already drifted from the scoring code, silently:
it showed four pose sources (Vina, DiffDock, KarmaDock, Boltz-2), missing
GNINA entirely, and labelled its DiffDock panel "confidence (higher =
better)" -- exactly what R2-3 objects to, and what the rescore tracks and
commit 20b3c50 stopped letting vote in global_rp_score.

Added notebooks/analysis/build_rank_percentile_schematic.py, generating the
same four-step layout (inputs -> score with each method -> raw scores ->
rank percentile) from the real 3-target rerun (guild_scores.txt) instead of
invented distributions, so it can't drift out of step with the code again:

- five pose sources (Vina, GNINA, KarmaDock, DiffDock, Boltz-2), not three
  or four;
- DiffDock's and Boltz-2's raw-score panels show the Vina-rescore ΔG that
  actually enters the score (vina_rescore_diffdock_score,
  vina_rescore_boltz_score), with the native confidence (diffdock_score,
  boltz_score) noted as reported but not ranked or voted -- the same
  treatment gnina_cnn_score already gets;
- Boltz-2's Step 4 panel is labelled and computed as ONE vote (the mean of
  its two shown tracks, including boltz_affinity_score per 0d36671), not
  one panel per track;
- the standardize caption states the cross-source combination is a MEDIAN
  (0e7c959), not a mean.

Rank percentiles are recomputed via compute_rank_percentile_scores rather
than trusted from the file, since the rerun predates both
boltz_affinity_score's ranking and the median aggregation default.
vina_rescore_diffdock_score is nulled for non-physical values before
ranking, the same treatment as score_distribution.ipynb's Figure 2 rebuild
and for the same reason (a cluster of exact-0.0 sentinel values).

Keeps it a schematic -- one example ligand highlighted consistently across
every panel, clarity over completeness -- while making it real: run
end-to-end against the actual rerun data (165 rows, 3 targets, 15 binders,
150 decoys) and visually verified the rendered PNG/PDF/SVG.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
CI's lint job runs ruff over .ipynb files too, which I hadn't checked
locally against the actual repo config (only against specific .py files)
before pushing. `ruff check .` flagged four un-sorted import blocks across
score_comparison.ipynb and score_distribution.ipynb, all from this week's
commits: legacy_columns/kde_helpers imports landing after matplotlib/
sklearn ones instead of being alphabetized with them.

Fixed with `ruff check --fix .`, which also normalized those cells' stored
`source` from a single string to the list-of-lines form the rest of the
notebooks already use (a cosmetic nbformat difference, not a content
change). Re-ran both notebooks end-to-end via nbconvert against the real
data to confirm the fix is inert: score_comparison.ipynb completes with no
errors, score_distribution.ipynb fails at exactly the same three
out-of-scope Sankey cells it already did (cells 4-9, which reference column
names the rerun file never had -- unrelated to this fix, called out in
fe148e1).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Tables S1-S6 carry every per-method AUC and enrichment factor quoted in
R2-3/R2-4, and the PoseBusters pose-validity rates in R2-4 -- but the
builder lived only in a session scratchpad, so nobody could regenerate
them. Ported to notebooks/analysis/reviewer_response/build_supplementary_tables.py,
alongside its committed sibling reproduce_response_analyses.py and following
that script's conventions (--data/--out, module docstring, write() helper).

It already produced the right numbers; this is a port, not a fix, and the
five changes are all about fitting the repo:

- hardcoded C:/Users/... paths replaced with --data/--out (default ./data,
  ./output);
- the sys.path insert onto the repo root dropped, along with the now-unused
  `import sys` -- unnecessary once the file lives inside the repo;
- the pandas-3 _pcid_keep save/restore workaround deleted as dead code:
  e9a6fe7 already fixed compute_rank_percentile_scores to keep its grouping
  column at source, so carrying a second copy under a different name to
  survive the old .apply()-based drop no longer does anything. The
  recompute itself is kept, and so is why: guild_scores.txt on disk
  predates both the median aggregation (0e7c959) and boltz_affinity_score's
  ranking (0d36671);
- a stale docstring line rewritten -- Boltz-2's affinity head has had its
  own rp_boltz_affinity_score column, ranked like every other track, since
  0d36671. The code already read that column correctly; only the comment
  claiming otherwise was wrong;
- no column-rename duplication to fix: this script does no column-name
  normalisation of its own (it reads the current-generation rerun file
  directly), so legacy_columns.LEGACY_RENAME does not apply here.

Verified against the real rerun data (guild_scores.txt, posebusters_validity.tsv,
native_ligand_rmsd.tsv) that all four numbers the response letter quotes
verbatim from Table S1 are unchanged: Guild combined score 0.838, AutoDock
Vina 0.788, Boltz-2 affinity head 0.985, GNINA 0.739. Confirmed the script
runs clean both from its own directory and from the repo root, with --data/
--out overriding correctly either way.

Also updates the reviewer_response README: documents the second script,
its three inputs (two new: posebusters_validity.tsv, native_ligand_rmsd.tsv),
its six outputs and which reviewer point each backs, and cross-references
the existing KarmaDock-has-no-complex-PDB note (merge_posebusters_flags.py)
for why S2/S3/S5 cover four of five methods rather than restating it.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
R2-5 is now the whole of the response's answer on data leakage, and it
existed nowhere as code: it was computed ad hoc. Added
r2_5_training_overlap to reproduce_response_analyses.py, registered in
ANALYSES so `--only r2_5_training_overlap` works, following the shape of
the existing analysis_normalisation / analysis_size_control functions.

For each of the three benchmark targets, computes a within-target
binder-vs-decoy AUC per scoring track (boltz_affinity_score, boltz_score,
diffdock_score, vina_score, karmadock_score), on raw scores, ordered by
each target's PDB release date -- because the argument is that any
training cutoff admitting the most recently released structure (8GDC,
2024-01-10) necessarily admits the other two (7V3Z, 2021-11-24; 6OT0,
2019-06-12), so their ordering is informative without knowing any method's
actual cutoff. Boltz-2's affinity head verifies exactly against the
letter's claim: 0.956, 1.000, 1.000 in that order -- the newest structure
ranked best, the oldest worst.

Release dates are a hardcoded, cited constant (RCSB entry endpoint
https://data.rcsb.org/rest/v1/core/entry/<id>, field
rcsb_accession_info.initial_release_date, retrieved 2026-09-17), not a
live network call. An opt-in --verify-dates flag re-fetches and asserts
them instead; confirmed it actually works against the real RCSB API for
all three targets.

Direction is read from guild.constants.bulk.SCORES_DIRECTION_DICTIONARY
rather than hardcoded -- the only place in this otherwise guild-free script
that imports guild, and still no heavy dependency, since that module is
pure Python constants. The direction-aware AUC computation is factored
into its own small function, direction_aware_auc(active, decoy,
direction), so it's ordinary, tested library code rather than inline
logic: 6 tests in tests/reviewer_response/test_reproduce_response_analyses.py,
including a direction-flip case, since getting a direction backwards
silently inverts an AUC and would invert the argument.

Verifying that helper against real data surfaced exactly this failure mode
in an earlier hand-computed draft of this table: boltz_affinity_score,
diffdock_score and vina_score all verified exactly, but boltz_score (ipTM)
and karmadock_score came out as the exact complement (1 - auc) of that
draft's values. Cross-checked independently against sklearn.roc_auc_score,
which confirmed the direction-correct computation (implemented here), not
the draft, is right. Treated as an error in that earlier draft, not in this
analysis; documented in the docstring and the reviewer_response README
rather than silently reconciled either way. boltz_affinity_score -- the
only track R2-5's own argument rests on -- was never in question.

Deliberately does NOT attempt a ligand-level training-set overlap audit:
no ECFP4 similarity against PDBBind ligands, no enumeration of what
Boltz-2 was actually trained on. The letter's argument rests on the
release-date contrast and the scoping logic (any cutoff admitting the
newest target admits the other two), not on an audit, and building one
would be answering a question this reply does not ask.

Also notes, in both the docstring and the README, that this analysis is
deliberately within-target and raw-score while Table S1 is deliberately
pooled and rank-percentile -- they can legitimately disagree (Boltz-2
affinity 0.970 pooled-raw here vs. 0.985 pooled-rank-percentile in S1;
Vina 0.736 vs. 0.788), and that is not an error in either one.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
The median aggregation default (0e7c959) landed after this file was
written, and two spots still describe the old mean-based world:

- the comment above POSE_SOURCE_VOTES said "each engine's rescore tracks
  are averaged before the cross-engine mean" -- still right that
  confidences don't vote, wrong that the cross-engine combination is a
  mean, and silent about boltz_affinity now also joining Boltz's
  pose-source vote (0d36671, deliberately left out of this fixed
  five-vote comparison set, now said explicitly rather than implied);
- the rule labelled "unweighted mean (current)" asserted the mean is
  still the default it no longer is; relabelled to "unweighted mean", and
  "median" to "median (current default, 0e7c959)" so the comparison table
  doesn't assert the wrong default.

Labels only: a2 computes every aggregation rule itself from the raw votes
rather than calling compute_rank_percentile_scores, so no number in
a2_aggregation_rules.tsv moves. Matters beyond tidiness, though -- the
reply to a2 currently quotes the unweighted mean as Guild's default and
argues about its cost, which the median change already falsified. Getting
the label right here is the first step in fixing that paragraph.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
…ls, restore schematic chrome, and render all three figures

Four fixes, verified against the real 3-target rerun and Figure 3 inputs:

- score_distribution.ipynb's Figure 2 right column and the schematic's Step 4 both
  showed a "P = 98%"-style label at x = 0.02 on an axis labelled "rank percentile" --
  the stored rp_* column is 0 = best, and the annotation was flipped for display while
  the axis position was not, asserting both conventions on one panel (the same
  confusion behind the Supp. Text 6 discrepancy). Both now plot 1 - rp and label the
  axis "(higher = better)", matching Figure 3.

- score_distribution.ipynb's Sankey/alluvial cells (4-9) referenced guild_vina_score /
  guild_boltz_score / guild_diffdock_score, column names the rerun never had (fe148e1
  flagged this honestly). Repointed at rp_vina_score / rp_boltz_score /
  rp_diffdock_score; the notebook now runs top to bottom without raising.

- Restored the schematic's chrome (white rounded cards, tinted step bands, separated
  method chips, em dashes) and switched its Step 3/4 distributions to the same KDE
  treatment as Figure 2 (kde_curve / kde_curve_bounded), removing the comb artefact a
  ~55-value 12-bin histogram produced. The KDE is a deliberate improvement over the
  published bar-histogram original, not a restoration of it.

- Rendered all three figures against real data (600 dpi PNG/PDF/SVG, outside the repo;
  guild_publication/images/ untouched) and re-ran both notebooks with outputs saved, so
  the committed .ipynb files show what the code produces. Figure 3's pooled AUCs: rank
  percentile 0.684, min-max 0.547, z-score 0.597 -- rank percentile clearly ahead of
  z-score here, not the near-tie a3_normalisations gives over its own pooled/raw-score
  computation (0.675 vs 0.674); different analyses, not a contradiction. Figure 2:
  vina_rescore_boltz_score covers 134/165 rows (81.2%), 79.4% of
  vina_rescore_diffdock_score is excluded as non-physical. Added GUILD_FIGURES_OUT_DIR
  (mirroring GUILD_FIGURES_DATA_DIR) to both notebooks for a durable render target.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
…py into the test image

tests/reviewer_response/test_reproduce_response_analyses.py imports the
reviewer-response script directly (sys.path.insert + import, since it has no
__init__.py), but .dockerignore's allowlist and the Dockerfile's test stage
only ever copied guild/, tests/ and scripts/run_guild.py -- notebooks/ never
reached the build context, so the test image never had the file and pytest
failed collection with ModuleNotFoundError. Same root cause as 8d6849a's
run_guild.py fix. Verified by reproducing the container's exact test-stage
file set in an isolated directory and confirming the 6 tests collect and
pass with nothing else present.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
…h code volume

Condensed multi-paragraph rationale comments and docstrings down to their
essential "why" (1-2 lines each) across every file this branch touched,
without changing any logic, test assertions, or notebook outputs. Kept
genuinely non-obvious gotchas, bug workarounds and benchmark numbers that
justify a default; cut restatements of what identifiers already make clear
and redundant :param:/:return: blocks for self-explanatory arguments.

Verified: ruff check . clean; full test suite unaffected (100 passed,
excluding tests/bulk_run and tests/single_run, which cannot collect in this
sandbox regardless since the vina package has no prebuilt wheel here); both
notebooks re-executed end to end with identical printed numbers/AUCs;
reproduce_response_analyses.py and build_supplementary_tables.py re-run
against real data with every previously-verified number unchanged.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
…_method.rst)

Makes R3-1's response letter reply true: it promised the repository documents
the integration contract for a new PLBP method, using GNINA as a worked
example, and that this had no code changes. That page didn't exist.

Documents the six-step contract as it actually is (constants module, runner
module, registration in the six guild/constants/bulk.py dictionaries,
orchestration in guild/bulk.py / scripts/run_guild.py / the Makefile,
explicit complex-PDB coverage or exclusion, a test), the median pose-source
aggregation, and why direction is per-track (Boltz-2's two opposite-direction
columns) not per-tool. GNINA (2502502) is the worked example; Nesso is a
second reference, explicitly flagged as GitHub-only and outside the paper's
five methods. Linked from CONTRIBUTING.md and the README's table of
contents; added to the Sphinx toctree between usage and modules.

Added tests/scores/test_scores.py::TestNewMethodRegistrationIsModular, which
registers a throwaway method via monkeypatch and confirms
compute_rank_percentile_scores ranks and votes it in with no change to that
function -- the modularity claim the guide makes checkable.

Docs-only otherwise; no behaviour change. Two gaps noticed while writing
this are left alone as the author's call, not this commit's:
- nesso is registered in SCORES_DIRECTION_DICTIONARY, RANKS_DICTIONARY,
  RP_SCORES_DICTIONARY, POSE_SOURCE_DICTIONARY and ALL_AVAILABLE_METHODS, but
  missing from scripts/run_guild.py's --methods choices, so it's unreachable
  from the CLI/Makefile, only the Python API.
- The Sphinx build was already broken independently of this page:
  docs/conf.py reads guild.__version__, which doesn't exist; docs/readme.rst,
  docs/contributing.rst and docs/authors.rst each include a sibling .rst that
  is actually .md or absent; the toctree references a missing modules.rst.
  Verified the new page itself is unaffected: an isolated scratch build with
  those four gaps worked around (not fixed in the repo) rendered
  adding_a_prediction_method with zero warnings, against a baseline of 50
  pre-existing ones from the other four pages.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
AJPreto and others added 17 commits September 18, 2026 14:31
Drop the reviewer-response framing ("a reviewer asked...") from the intro
and the paper-defensive phrasing around Nesso -- the page should read as
guide documentation for the repository, not as a rebuttal artifact. No
content or claims changed, wording only.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Revision 1.docx carries Supplementary Tables 2 to 10: the manuscript's own
pre-existing GPCR target list keeps the number Supplementary Table 1, so
everything build_supplementary_tables.py generates shifted up by one.
Shifted the tags in build_tables()'s return list accordingly (S1->S2 ...
S6->S7), which drives both the TSV filenames and the HTML heading tags, plus
a comment there and in the README explaining why the numbering starts at S2.
Table S10 (parameter provenance) is out of scope, generated elsewhere.

Verified by regenerating into a scratch directory and diffing against the
staged, renamed copies at review/supplementary/: identical for S2-S5 and S7.

Found and reporting, not fixing: Table S6 (native-ligand redocking RMSD)
differs in *content*, not just name. The staged, submitted version carries
three extra columns -- RMSD after Kabsch superposition, a within-2-A flag,
and receptor Ca-fit RMSD -- reshaped long by method, with an explicit
KarmaDock "n/a" row. native_ligand_rmsd.tsv already has the source columns
for this (kabsch_rmsd, within_2A, kabsch_fit_rmsd_ca), but build_tables()
only reads the raw rmsd column into a wide per-target pivot. Pre-existing,
not introduced by this renumbering; needs its own commit once it's decided
whether the script should be brought in line with the submitted table or
vice versa.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Supplementary Table 9 reports the sensitivity of the rank percentile to
AutoDock Vina's exhaustiveness, and the reply to a1 quotes it, but nothing in
the repo computed it -- the same gap this file has closed twice before for
other reviewer points. Adds analysis_exhaustiveness, registered as "a1",
following the existing --data/--out convention.

Per target, per pairwise comparison (8 vs 16, 8 vs 32, 16 vs 32): Spearman
rho of the within-target ordering, median top-10% Jaccard overlap of the
best-scoring decile, and the raw score shift (mean absolute change, % within
0.5 kcal/mol, largest change). The shift columns exclude rows where either
compared setting scored >= 0 kcal/mol -- a handful of ligands swing by up to
20 kcal/mol there, which would dominate the mean; they stay in rho/Jaccard,
where they're harmless. Missing input files print a skip line each rather
than raising, matching analysis_runtime's convention.

Verified against the staged sweep at review/exsweep/: median rho
0.978/0.973/0.978 and mean absolute shift 0.157/0.158/0.138 kcal/mol for
ex8-vs-ex16/ex8-vs-ex32/ex16-vs-ex32, matching the task's expected-values
table exactly. Pooled across all three comparisons this also reproduces the
response letter's headline (median rho ~0.98, range 0.90-0.99, 93% within
0.5 kcal/mol). Runtime is deliberately not reported: competing processes
shared the sweep's machine, so its batch-log timings measure contention, not
exhaustiveness cost -- noted in the function's docstring so nobody adds one
later. R3-4's runtime figures come from the three-target benchmark instead.

Added spearman_rho and jaccard_overlap as small, independently-tested
helpers (tests/reviewer_response/test_reproduce_response_analyses.py),
including a tied-score case for the rank computation.

Also fixes two stray "Table S1" references (a docstring and a print
statement in analysis_training_overlap) left over from the sibling commit's
S1->S2 renumbering, since this file was already being touched.

Found and reporting, not fixing: the exsweep's own staged README and the
response-letter draft both say its five non-physical rows are all in target
6me6. They are not -- a fifth (7v3z-A-9GF-A / CNP0002880) is non-physical in
all three settings too. Doesn't change any a1 number (the exclusion is
per-row, not per-target), but that sentence needs correcting before the
letter goes out.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
score_distribution.ipynb drew three blocks of explanatory text onto Figure 2
itself: notes under panel D and panel E about diffdock_score/boltz_score
(native confidence) not being ranked or voted, and a footer disclosing the
3-target rerun. A figure isn't where that belongs -- it puts internal column
names on the face of a manuscript figure, and the panel-E note rendered a
git commit hash (0d36671) into a published figure. Panel labels A-E, axis
labels, column titles and the legend are all the figure needs.

Removed the "note" keys from METHOD_SPECS (cell 2) and the ax.text/fig.text
calls that drew them (cell 3). Kept the comment above METHOD_SPECS explaining
why DiffDock/Boltz-2 report a native confidence without voting -- that
reasoning belongs in the code, just not on the canvas. Removed the trailing
per-method "NOTE FOR CAPTION" print loop too, since it read the same
now-removed spec["note"] key and would otherwise raise a KeyError; left the
target-count "NOTE FOR CAPTION" print alone, since it doesn't touch that key.

Left everything else as it was: the bounded KDE, the five methods, the
rescore tracks standing in for the confidences, and the "higher = better"
axis labels.

Re-rendered to review/figures_regenerated/ at 600 dpi and re-ran the
notebook with its output saved. Numbers unchanged (165 rows, 3 targets,
same coverage/non-physical percentages as the last verified render).

Aspect ratio moved: PNG went from 4480x7467 (aspect 0.600) to 4480x7212
(aspect 0.621) -- a 3.5% shift, well past the ~1% threshold, so the
manuscript's wp:extent for Figure 2 needs updating to match the new height.
Revision 1.docx not touched; the caption edit (moving the 3-target
disclosure off the figure and into the caption text) is the author's, on
that side.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Panel D plots vina_rescore_diffdock_score and panel E plots
vina_rescore_boltz_score -- correct and deliberate (20b3c50, R2-3), and
unchanged here. The figure just never disclosed it: METHOD_SPECS carried a
raw_xlabel per row, but cell 3 never read it -- the bottom row gets a
hardcoded "Raw docking score" and every other row is blank -- so rows D and
E showed only "DiffDock"/"Boltz-2", inviting a reader to take them for those
methods' own native confidences. The manuscript text makes that reading
likelier still, introducing DiffDock's confidence two paragraphs earlier.

Added an optional "track" key to METHOD_SPECS for the two rescored rows and
render it as a second, smaller, non-bold line under the panel label:

    D)  DiffDock
        Vina rescore of pose

Left the shared "Raw docking score" x-label alone -- the left column shares
one x-range across all five rows, so a per-row label there would be wrong.
Deleted the now-dead raw_xlabel entries and the cell-3 code that unpacked
and stored them without ever using the value, rather than leaving a key
that looks load-bearing but isn't.

Also switched the rank-alluvial cell's GUILD_COLS from rp_boltz_score /
rp_diffdock_score (native confidences) to the same rescore columns Figure 2
uses, so this internal, non-manuscript figure's title ("rank percentile
score ranking across docking methods") doesn't contradict how Guild's
consensus actually treats these two methods. Checked the ribbon count
before and after switching: unchanged at 15 non-decoy compounds, and
per-column coverage among just those 15 is 15/15 Vina, 15/15 DiffDock
rescore, 14/15 Boltz rescore -- the 79.4% non-physical rate quoted for
vina_rescore_diffdock_score is a dataset-wide figure that falls almost
entirely on decoys, not on this figure's 15 binders, so nothing thins out.

Re-rendered to review/figures_regenerated/ and re-ran the notebook with
output saved. PNG dimensions and aspect ratio unchanged: 4480x7212, aspect
0.621, same as before this change -- a two-line panel label doesn't move
the figure height. Revision 1.docx not touched; the caption sentence is the
author's, on that side.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
score_comparison.ipynb drew RUG_SUBSAMPLE_N = 400 decoy ticks per panel out
of a 98,596-row decoy set (0.4%). That's worse than it sounds: the top
percentile of decoys is roughly 986 molecules and showed as about four
ticks, visually understating exactly the binder-decoy overlap region the
figure exists to characterise, while the KDE curve directly above
represents that region correctly from every point.

It also cost a caption sentence that can't be phrased safely: "a random
subsample of 400" reads as a claim about how the decoy panel was built, and
the panel is filtered, not random (Supp. Text 2).

Deleted the decoy rug loop and the RUG_SUBSAMPLE_N/RUG_SEED constants
(unused elsewhere). Moved the binder rug -- every known binder, n=220,
unchanged -- up into the freed band ([-kmax*0.03, -kmax*0.11], was
[-kmax*0.11, -kmax*0.19]) and tightened set_ylim's bottom from -kmax*0.25 to
-kmax*0.15 so there's no empty gutter left where the decoy rug was; checked
the rendered panel rather than trusting those numbers blind. Updated the
printed rug-composition line to say the binder rug is the complete set and
decoy density is shown by the curve alone -- kept printed, since it's the
check that the caption matches the code. Left the sample-size footer, the
shared kmax cross-panel y-scale, the bounded KDE, the 1 - rp_vina_score
inversion, and the AUC boxes untouched.

The R2-7 matched-decoy path re-runs this same cell via DECOY_SUBSET and
inherits the fix with no separate change -- also why removing rather than
re-tuning the subsample was the right call: on that much smaller matched
panel, a fixed 400 would have quietly become a different fraction of the
set. score_distribution.ipynb's own decoy rug is untouched: its 150 decoys
already fall under that notebook's min(500, n), so every molecule there is
already drawn.

Re-rendered and re-ran the notebook against the large Vina case study data
(vinarun_scores.txt / knownbinders_scores.txt, not the 3-target rerun).
AUCs unchanged: rank percentile 0.684, min-max 0.547, z-score 0.597.

PNG dimensions and aspect ratio unchanged: 1950x2693, aspect 0.7241, same
as before. The panels' pixel size is fixed by figsize/the gridspec, not by
set_ylim, so tightening the internal margin doesn't move bbox_inches="tight"'s
outer crop here -- no wp:extent correction needed.

Revision 1.docx not touched; the caption sentence is the author's, on that
side, applied together with this render.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Two spots named the review point they were addressing rather than the
reason for the code: a cell-2 config comment labelled "R2-7: decoy subset",
and cell 5's section header/paragraph opened with "R2-7: property-matched
decoy panel" / "The reply to R2-7 promises...". The notebook should read as
an independently built analysis, not a reply keyed to a specific reviewer
point, so reworded both around the actual motivation instead: Figure 3's
decoys are not property-matched to the known binders (445 Da actives vs
353 Da decoys; a heavy-atom-count baseline alone reaches AUC 0.865, higher
than any docking method, collapsing to 0.645 once size-matched).

While in there, trimmed several comments that ran longer than the point
they were making needed, condensing to state the non-obvious "why" in fewer
lines: the rank-percentile compute-call comment, the inversion-for-plotting
comment, the kde_curve_bounded/shared-y-scale comment, the xlabel and
y-axis-label comments, and the demo-slice comment. Dropped two references
that meant nothing without cross-notebook/session context: a commit hash
(e9a6fe7) and "Figure 2's b2 fix" shorthand.

No logic changed. Re-ran the notebook to confirm: same AUCs
(0.684/0.547/0.597) and unchanged PNG dimensions (1950x2693, aspect 0.7241).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
d3c9e1b took explanatory text off Figure 2, but that change targeted
score_distribution.ipynb and nobody looked at build_rank_percentile_schematic.py,
which renders Supplementary Text 3's schematic. It had the same problem, one step
worse: the STANDARDIZE caption rendered a git commit hash (0e7c959) into a
manuscript figure, and two Step 3 panel notes put internal column names
(diffdock_score, boltz_affinity_score) on the figure face.

Removed the hash from the STANDARDIZE caption, keeping the rest of that sentence
-- mean-within-source then median-across-sources is the one thing on the figure
a reader genuinely needs. Removed the two panel notes entirely; that reasoning
already lives in Supplementary Text 1. Kept the "why" as a code comment above
POSE_SOURCES, the same treatment d3c9e1b gave score_distribution.ipynb's
METHOD_SPECS.

Also nudged Step 3's red "ligand" label, which sat on top of the curve for
KarmaDock, DiffDock and Boltz-2 whenever the dashed line landed near a peak (a
fixed fraction of the axes only worked when it happened to land somewhere low).
It now floats a fixed offset above the curve's own interpolated height at that
x-position, with headroom added to each panel's y-limit so there's always room.

Left the four steps, the worked ligand (CHEMBL2541637), the footer, the panel
titles, the colour scheme, and the Step 3/4 layout fixes alone.

Re-rendered to review/figures_regenerated/: 7429x8857 (aspect 0.8388), was
7458x8857/0.8420 -- a 0.39% shift, well under the ~1% threshold, so no
wp:extent correction needed.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
R2-4 currently reports pose validity per pose (80.0% Boltz-2, 26.8% AutoDock
Vina, 25.8% GNINA, 11.6% DiffDock, from Table_S3.tsv). We're switching that
reply to the per-pair figure -- "did any validated pose for this method
pass" -- which is the <method>_pb_valid column merge_posebusters_flags.py
already writes into guild_scores.txt.

Added a "Per-pair valid (%)" column to build_supplementary_tables.py's Table
S3, beside the existing per-pose "pb_valid (%)", so both live in one table
and the reply can cite either without a second source.

Per-pair validity over the same 165 pairs:
  AutoDock Vina  77.6% (128/165) -- confirms the 77.6% already in the tracker
  GNINA          81.8% (135/165)
  DiffDock       53.9% (89/165)
  Boltz-2        80.0% (132/165) -- equals its own per-pose rate, since Boltz
                                    emits exactly one pose per pair

Regenerated Table_S3.tsv into a scratch directory to verify -- not written to
review/supplementary/, left for the author to stage. Diffed the other five
tables against the currently staged ones to confirm nothing else moved:
S4/S5/S7 identical; S6 still carries the previously reported content gap
(unrelated, pre-existing). Found one more, unrelated to this change: S2's
"Vina rescore of Boltz-2 pose" non-physical rate is 0.6% by this script's own
formula (1/165) but 0.7% in the staged file, which matches a different
denominator (1/135) used elsewhere in this codebase. Not fixed here.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
score_comparison.ipynb's descriptor-only baseline and property-matched
decoy panel (cell 6) only ran on a 5-target demo slice, so its numbers
could not be quoted for the full analysis. Set N_TARGETS_FOR_DEMO = None
and committed it that way -- the demo default was for cheap iteration, but
a saved output that doesn't match the committed constant is worse than a
slow cell.

Full-pool numbers (131 targets, 655 binders, 133,000 candidate decoys):
9,910 property-matched, structurally-distinct decoys kept; per-descriptor
AUCs 0.453-0.732 (molecular_weight highest at 0.732); combined-descriptor
cross-validated AUC 0.738. These supersede the 5-target demo's 460 kept /
0.631 / 0.778 -- notably, the full-pool numbers sit closer to Figure 3's
own rank-percentile AUC (0.684) than the demo suggested, not farther.
Binders come from knownbindersvinarun, decoys from vinarun, matched only
on protein_config_id -- if the docking box or protein preparation differed
between those runs, this baseline inherits that mismatch.

Rendered Figure 3 on exactly those 9,910 matched decoys (same 220
binders): AUC 0.588 (rank percentile), 0.451 (min-max), 0.588 (z-score),
against the unmatched panel's 0.684/0.547/0.597 over 98,596 decoys. Saved
separately as figure_3_matched_decoys.{png,pdf,svg} in
review/figures_regenerated/ -- figure_3_validation_three_normalisations.*
(embedded in the manuscript) is untouched. DECOY_SUBSET is committed back
to None so the notebook's own saved outputs stay the unmatched panel; the
matched render came from a scratch copy with DECOY_SUBSET set to a
per-(protein_config_id, ligand_id) pair predicate.

Found and fixed a real bug in cell 6's own trailing instructions along the
way: it told the reader to set DECOY_SUBSET to
matched_decoys['ligand_id'].tolist() -- ligand_id alone. The decoy pool
has only 1,000 unique molecules each docked against ~133 targets, so
filtering by ligand_id alone readmits every target-row for any decoy that
matched anywhere: 108,262 of 133,000 rows (81%), not a per-target matched
panel -- confirmed by actually trying it before switching to the
pair-based predicate. The trailing print now describes the pair-based
recipe instead.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
make docs failed at four separate points, each verified against the tree:

1. docs/conf.py read guild.__version__, which guild/__init__.py never
   defined. Added it via importlib.metadata.version("guild"), falling back
   to "0.0.0" if the package isn't installed (e.g. reached only through
   sys.path rather than a real install).
2. The toctree listed modules, but no docs/modules.rst was ever generated.
   Dropped it rather than wiring up sphinx-apidoc: nothing else in this
   docs setup maintains a full API reference today, and generating one now
   would mean committing dozens of new auto-generated pages nobody asked
   for. Also dropped the toctree's :ref:`modindex` link, since it was
   always going to be dead with no autodoc content behind it.
3. docs/readme.rst did .. include:: ../README.rst, which is actually
   README.md.
4. docs/contributing.rst did the same for ../CONTRIBUTING.rst (actually
   .md), and docs/authors.rst included ../AUTHORS.rst, which doesn't exist
   in any extension.

For (3)/(4): added myst-parser and converted the two include-wrapper pages
to docs/readme.md / docs/contributing.md using MyST's {include} directive
(with relative-docs/relative-images) so the real root README.md /
CONTRIBUTING.md render as actual Markdown instead of being fed to the RST
parser as raw text. Dropped authors.rst outright -- there's nothing to
point it at.

Added a "docs" dependency group (sphinx, myst-parser) -- not installed by
uv sync by default, so a normal dev setup stays as light as it is today --
and a top-level `make docs` target. Added docs/_build/ to .gitignore.

Verified with an ephemeral `uv run --with sphinx --with myst-parser
--no-project` environment, since a full project sync fails in this
sandbox on an unrelated pre-existing dependency (fair-esm[esmfold]'s
pinned deepspeed==0.5.9 doesn't build without a working torch here) --
not something this fix touches or needs.

The build succeeds; it is not warning-free. myst_heading_anchors = 4 cut
the warning count from 33 to 7 by resolving most of README.md's own
internal #anchor links. The remaining 7 are relative links to files
outside the doc tree and two unrecognized-lexer notices for a fenced
"csv" code block -- all benign, none fatal. All seven pages render,
including adding_a_prediction_method.rst.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
The current long-tail passage ("Of 212 known binders across 47 targets, 57
(26.9%) fall in the worse-scoring half of their own target and 14 (6.6%) in
the worst decile") had no committed script computing it against Figure 3's
own binder/decoy set. Added analysis_binder_tail to
reproduce_response_analyses.py: same pool score_comparison.ipynb's Figure 3
uses (EXCLUDE_PROTEINS applied, non-null Vina score required), same tail
statistic. Result: 220 binders, 47 targets, 58 (26.4%) worse half, 10
(4.5%) worst decile, heavy atoms 26.6 vs 28.2, Vina -4.39 vs -8.46 kcal/mol.

212 turns out to already be reachable from committed code. The existing
a3_failure_tail_summary.tsv (written by analysis_normalisation, already
committed) reports 212 binders, 57 (26.9%), 14 (6.6%), 25.9 vs 28.3, -6.36
vs -8.88 -- an exact, decimal-for-decimal match to the current passage. It
comes from a genuinely different, also-legitimate pool: _load_case_study()
keeps the physical Vina range and requires both a binder and a decoy
present per protein, with no named-protein exclusion, instead of
EXCLUDE_PROTEINS. Same 47 targets, different binder counts -- not an error
in either derivation. The actual gap is that the two are being quoted
against each other without saying so: the text's 212 against the figure's
own footer, which already prints 220. Left the choice of which pool to
standardize on to the text; both derivations are committed either way.

Also confirmed the one already-committed number this task asked to check:
a3_normalisations.tsv's raw-score pooled AUC is 0.619, not the 0.626
currently quoted (min-max 0.577 and rank percentile 0.674 both already
match). Not fixed here -- reported for the text to correct.

Documented both new/clarified outputs and this finding in
notebooks/analysis/reviewer_response/README.md's outputs table and caveats
list. Also dropped a stray reference to the specific reply point from a
comment added in 89de8d4, in the same spirit as removing that kind of
framing from this branch's other comments.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
analysis_normalisation now recomputes the pooled rank-percentile/min-max/
z-score AUCs on four intermediate pools between score_comparison.ipynb's
own Figure 3 construction and a3's _load_case_study(), written to
a3_pool_reconciliation.tsv. The rank-percentile/z-score near-tie
_load_case_study() finds (0.675 vs 0.674) turns out to be a physical-
Vina-range effect, not a target-set effect: restricting Figure 3's pool to
binder-bearing targets changes almost nothing, but applying the physical
range alone moves z-score from 0.597 onto rank percentile. Also pins down
why _load_case_study() reports 212 binders instead of 211: it never
applies Figure 3's five-protein exclusion, so it keeps one binder from a
target Figure 3 deliberately drops.

Also has score_comparison.ipynb report decoy/binder counts as unique
molecules alongside protein-ligand combinations (974 decoys / 135 binders,
not 98,596 / 220), in the printed output and Figure 3's footer annotation.
Source-only change; not re-rendered, since a full run is dominated by an
unrelated, unchanged cell.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
6493eb6 changed Figure 3's sample-size annotation to report combinations
alongside unique-molecule counts but was source-only; the saved PNG/PDF/SVG
and the notebook's own cached output for that cell still showed the old,
misleading footer. Executed cells 1-4 only (not cell 6, whose full-pool
descriptor matching is unrelated and unchanged) and re-saved
figure_3_validation_three_normalisations.{png,pdf,svg}.

AUCs and the decoy rug are unchanged (0.684 / 0.547 / 0.597). Dimensions
moved from 1950x2693 (aspect 0.7241) to 2273x2694 (aspect 0.8437), a ~16.5%
width increase -- isolated to the longer footer text alone by re-rendering
with the old footer in the same environment, which reproduces the original
size exactly.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
759a50f added a docs dependency group to pyproject.toml but never
regenerated uv.lock, so the CI Docker build's uv sync --locked step
started failing with "the lockfile needs to be updated".

Fixing that surfaced a separate, pre-existing problem: deepspeed (already
pulled in transitively via fair-esm[esmfold]) runs a native compile step
in its own build backend that needs a C++ toolchain, so uv lock can't
resolve it on a machine without one configured. Added a static
tool.uv.dependency-metadata override with its dependencies copied from
its own requirements.txt, so uv lock reads its metadata instead of
building it -- verified this is the actual fix (not a side effect of
having torch installed locally) by re-running uv lock with torch absent.

uv.lock now carries the docs group's real dependencies (sphinx,
myst-parser, and their transitive deps) and passes uv lock --locked
cleanly.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
This should not be part of the test suite. The paper needs to be reproducible, but having tests to replicate specific experiments simply generates overhead
The commit history serves this purpose if someone needs to rollback, no need to context bloat

@AJPreto AJPreto left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Full pipeline is replicable

@AJPreto
AJPreto merged commit 6eeeff2 into main Sep 23, 2026
2 checks passed
@AJPreto
AJPreto deleted the addressing_reviewers branch September 23, 2026 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants