Skip to content

Reuse native workspaces and stabilize multiscale L2-ACLIME paths - #12

Draft
mlakolar wants to merge 10 commits into
codex/direct-doubly-functional-precisionfrom
codex/multiscale-l2-path-workspaces
Draft

mlakolar wants to merge 10 commits into
codex/direct-doubly-functional-precisionfrom
codex/multiscale-l2-path-workspaces

Conversation

@mlakolar

@mlakolar mlakolar commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

The existing JuMP paths could rebuild Clarabel's native solver after data edits. This adds native sparse workspaces that update score right-hand sides across penalties and coefficient data across equal-size bandwidth sets. Clarabel still initializes fresh interior-point iterates; this is workspace reuse, not iterate warm starting.

multiscale_l2_aclime_path supports pilot-based and spectral-calibrated column programs. fit_multiscale_l2_path crosses fixed bandwidth sets, bias constants and penalties for the pilot-based estimator, then applies interval symmetrization and the existing norm cap. Singleton-width and multiscale-intersection programs remain distinct estimators. The solver does not select parameters, repair covariance inputs, or screen estimates for positive definiteness.

The native interface preserves failures as NaNs and records certified infeasibility separately. It does not use the older spectral interface's labeled zero fallback. Existing APIs are unchanged. This PR remains stacked on estimator PR #9 and is not ready for merging.

Numerical robustness

A real-cache canary exposed five pilot fit rejections, involving 78 SOLVED columns. Their score-coordinate residuals exceeded the unchanged 1e-5 guard; objective budgets ranged from 656 to 2,515. Auxiliary cone/bias-ball constraints were satisfied. Fresh standard-tolerance native solves reproduced every rejection, so retained workspaces were not responsible. All five independent JuMP fits passed feasibility checks. An additional fixed objective-comparison gate found four otherwise guard-passing native columns with normalized objective discrepancies above 1e-4. Frozen-pilot diagnostics identified insufficient native accuracy at 1e-7; native 1e-8 brought these objectives into agreement with stricter JuMP references. Pilot LP and final symmetrization tolerances remain 1e-7. Instrumented and public baseline results were bitwise identical.

Initial native gap/feasibility tolerances are tightened globally from 1e-7 to 1e-8. Pilot LP and final symmetrization defaults remain 1e-7. A numerically rejected reused solve first rebuilds cold at the same 1e-8 accuracy. If native recovery fails, an independent JuMP/Clarabel model solves the identical column program in the original bias-vector units, with the pilot, penalties and spectral bounds fixed. It uses 1e-8, then stricter 1e-9/1e-10 only if needed. Native SOLVED or JuMP OPTIMAL plus every original 1e-5 acceptance guard is mandatory. ALMOST_SOLVED is never accepted. Certified infeasibility is never retried or converted into an estimate. All attempt settings, statuses, backends, residuals, rejected candidate vectors, errors and timings are preserved.

An intermediate confirmation exposed 103 non-SOLVED native columns beyond the historical points. Fresh native 1e-8 recovered 54. All remaining 49 recovered with the equivalent unscaled formulation at 1e-8 and passed the original guards and independent feasibility checks. The failed first confirmation remains preserved. This diagnostic motivates the production fallback; it is not a relaxation or a covariance/precision projection. Regression tests include single-problem accuracy and a reused path that exercises recovery. The expanded replay gate checks every computed pilot fit, in addition to the five historical objective comparisons.

The isolated diagnostic recovered all 78 historical columns at 1e-8, with SOLVED status and maximum original-guard residual 3.76364e-6. At 1e-9, three columns reported ALMOST_SOLVED despite smaller residuals; these remain rejected diagnostic outcomes. The checked-in simulated covariance fixtures exercise full-column feasibility, retry receipts and objective agreement with independent JuMP. No statistical parameter was retuned.

Analytic zero certificate

For either calibration, when every bandwidth's diagonal score allowance is at least one, beta=g=t=0 is a feasible global optimum because the objective is nonnegative. The path returns exact zero columns with ANALYTIC_ZERO termination and no conic solve. This exact certificate adds no numerical slack and is distinct from an infeasibility fallback. Lazy matrix refreshes remain correct when skipped points precede later nonzero solves, including nonmonotone paths and changing covariance problems.

Validation

  • Full integration CI passes on all four platforms at 63b24a45589ef5fc3839cc7b7c0db8a57742e2c8: 5,605 checks on current Linux, 5,607 on Linux LTS and Windows, and 5,609 on macOS. Counts vary because recovery branches add checks.
  • Focused solver CI passes 1,001 checks each on current Julia and LTS. It covers both saved-data objective fixtures and the reused-path recovery case.
  • Discovery gate 13611407 passed 1,001 focused checks, all 49 production fallback replays, all 256 computed pilot fits and all five independent whole-column objective comparisons. The largest relative objective discrepancy was 3.32064e-5 against the unchanged 1e-4 gate.
  • Confirmation 13611439 completed in 10:13: all 16 datasets, 2,048 attempted fits, 102,400 column records, and all 4,044 independent checks passed, including 32 selected cold comparisons. Read-only audit 13611482 verified all 21,610 analytic-zero columns and their exact zero arrays/budgets. All 16 input hashes and 176 output-file hashes were verified.
Method Returned Numerical failure Solver-certified infeasible
Pilot-based 1,024 0 0
Cached-grid spectral 844 0 180

All 1,868 returned estimates completed final symmetrization. There were 208 recovered columns: 105 through fresh native solves and 103 through the equivalent unscaled formulation, using 260 additional attempts. Every final numerical solve used 1e-8; no stricter recovery levels were needed. All 400 all-column-zero spectral fits and 857 non-PD returns remain visible; no norm cap activated. Maximum independent primal violation was 1.84569e-6, symmetry residual 1.20235e-9, and saved-array loss-audit discrepancy 5.33e-15. No non-SOLVED native candidate or non-OPTIMAL fallback was accepted. Infeasible fits remain NaN and analytic-zero optima are explicitly labeled; backend and recovery histories remain available per column.

The fixed confirmation design uses 16 saved datasets: four models, 8 and 120 visits, replicates 1 and 2, n=200 and p=50. Both methods retain the original five bandwidth sets, four bias constants on the full eight-width set, all score penalties, and common calibration width 0.15. The spectral arm uses explicitly empirical cached-grid calibration, not the exact theorem grid. Covariance estimates are reused without refitting. All successful estimates receive final interval symmetrization and the existing norm cap; non-PD and zero-optimal returns remain visible.

The earlier original canary at bcfa63d returned 507/512 pilot fits and 422/512 spectral fits, with five numerical rejections and 90 certified-infeasible spectral fits. Its 1,024 attempts and all failed outcomes remain preserved. The new confirmation is separate and does not overwrite that evidence.

Historical timing canary at bcfa63d: median native reuse/cold times were 13.179/14.908 seconds at p=50 and 140.932/162.909 seconds at p=100, over 18 pilot-based configurations per call and three timed repetitions after compilation. Paired speed ratios were 1.126x and 1.154x; allocation reductions were 42.4% and 36.1%. These shared-node timings exclude covariance fitting and final symmetrization and predate the numerical recovery/zero shortcut. They do not establish universal or end-to-end speedups. Clarabel was used for pilot LPs because the separate initial HiGHS-simplex attempt failed before native workspaces were built; all those diagnostics remain preserved.

Project records: notes/20261030/solver_canary/, notes/20261030/multiscale_study/, and notes/20261030/multiscale_confirmation/. They retain exact revisions, fixed designs, input/output hashes, raw arrays, diagnostics and recreation scripts. The full 192,000-fit campaign remains disabled pending user review. No merge or full simulation campaign is authorized.

@codecov

codecov Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.10726% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.39%. Comparing base (ec5baf7) to head (550aaec).

Files with missing lines Patch % Lines
src/multiscale_path.jl 98.10% 6 Missing ⚠️
Additional details and impacted files
@@                             Coverage Diff                              @@
##           codex/direct-doubly-functional-precision      #12      +/-   ##
============================================================================
+ Coverage                                     94.83%   95.39%   +0.56%     
============================================================================
  Files                                            12       13       +1     
  Lines                                          1529     1846     +317     
============================================================================
+ Hits                                           1450     1761     +311     
- Misses                                           79       85       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mlakolar mlakolar changed the title Reuse native conic workspaces across multiscale L2 ACLIME paths Reuse native workspaces and stabilize multiscale L2-ACLIME paths Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant