Second dispatch solve to spread activity - #1469
Conversation
80191d7 to
ea1b3b8
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds an optional second (lexicographic) HiGHS solve in the dispatch optimisation to reduce “corner solutions” by spreading utilisation more evenly (across identical assets and across time slices at the balance level), while keeping the primary cost objective near-optimal.
Changes:
- Capture and reuse the original cost coefficients to add a cost-bound constraint for a second solve, then add L1 “equalisation” constraints/variables to minimise utilisation spread.
- Introduce new model parameters to enable/disable the equalisation solve and configure its cost tolerance.
- Add integration-style tests using the
simpleexample model to validate equalised dispatch behaviour via debug output CSVs.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 8 comments.
| File | Description |
|---|---|
| src/simulation/optimisation.rs | Implements the second lexicographic dispatch solve and adds equalisation constraints plus integration tests. |
| src/model/parameters.rs | Adds parameters controlling dispatch equalisation and its tolerance (with defaults). |
| schemas/input/model.yaml | Exposes the new model parameters in the input schema with notes/defaults. |
Suppressed comments (2)
src/simulation/optimisation.rs:1112
- This test’s relative error uses
diff = |rate - max| / maxwith only amax <= 0.0guard. Ifmaxis tiny but positive, the division can amplify numerical noise and cause flaky failures. Consider using a small-denominator threshold (or abs+rel tolerance) before computing relative error.
let max = rates.iter().copied().fold(0.0_f64, f64::max);
if max <= 0.0 {
continue;
}
for &rate in rates {
let diff = (rate - max).abs() / max;
assert!(
diff < 1e-6,
"GASDRV utilisation rate not equal across time slices in season {season}: {rates:?}"
src/model/parameters.rs:93
- The docstring formula
Z* * (1 + tolerance)has the same sign issue as the implementation: ifZ*is negative, it tightens the bound. To match the intended “fractional tolerance” meaning, describe the bound in terms of|Z*|(e.g.Z* + |Z*| * tolerance).
/// Fractional tolerance on the primary cost for the lexicographic second solve.
///
/// The second solve constrains total cost to at most `Z* * (1 + tolerance)`, where `Z*` is
/// the optimal cost from the first solve.
#[serde(deserialize_with = "deserialise_finite_non_negative")]
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Constrain total cost to be no worse than z_star * (1 + tolerance) | ||
| model.add_row(..=(z_star * (1.0 + tolerance)), cost_terms); |
There was a problem hiding this comment.
I think this is highly unlikely (/maybe impossible), unless the user is trying to do something stupid
| let solution = Solution { | ||
| solution: solved2.get_solution(), | ||
| variables, | ||
| time_slice_info: &self.model.time_slice_info, | ||
| constraint_keys, | ||
| objective_value: Money(solution.objective_value()), | ||
| }) | ||
| // Report the primary cost objective, not the spreading objective value | ||
| objective_value: Money(z_star), | ||
| }; |
There was a problem hiding this comment.
I guess, but I don't think it's a huge deal. We don't use these objective values for anything, and in reality the tolerance is going to be effectively zero (really just enough to allow for floating point differences)
There was a problem hiding this comment.
I think ideally we'd report the objective value both before and after equalisation, to see the effect that it has on the primary cost. It might just be a bit cumbersome to compute this after the second solve
| for i in 0..terms.len() { | ||
| for j in (i + 1)..terms.len() { | ||
| let (act_a, inv_cap_a) = terms[i]; | ||
| let (act_b, inv_cap_b) = terms[j]; | ||
| let d = model.add_col(1.0, 0.0.., []); | ||
| model.add_row(0.0.., [(d, 1.0), (act_a, -inv_cap_a), (act_b, inv_cap_b)]); | ||
| model.add_row(0.0.., [(d, 1.0), (act_a, inv_cap_a), (act_b, -inv_cap_b)]); | ||
| } |
There was a problem hiding this comment.
I think performance needs to be investigated. The group utilisation variable idea is similar to what I tried in #1463, and it works ok IF the optimum solution is for all variables in the group to be equal, but that won't always necessarily be the case. If there's real variation in the group then this approach stops working because there's no longer an incentive for any variables that should equalise to do so.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dispatch_docs #1469 +/- ##
=================================================
+ Coverage 89.78% 90.03% +0.25%
=================================================
Files 60 60
Lines 8439 8631 +192
Branches 8439 8631 +192
=================================================
+ Hits 7577 7771 +194
+ Misses 545 541 -4
- Partials 317 319 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
tsmbland
changed the title
4e477bd to
ac05803
Compare
Description
Add's a second solve to encourage spreading of utilisation where there's no harm to the main objective:
This hot starts off the back of the first solve, which is a bit more efficient than setting up a new solve from scratch (and I guess should be more efficient than a single-solve approach like in #1463, but I haven't done a direct comparison). There is a significant performance cost revealed by benchmarking - somewhere between 15-40% for the example models, but not the end of the world. I imagine this could be even worse for larger models though - ideally we'd do some targeted benchmarking for the dispatch optimisation, including much bigger problems.
Situations where the performance cost may be particularly big:
Added some tests to verify that this works. Due to the complexity of the input data for the dispatch function, it was easiest just to run integration tests on variants of the
simplemodel and check the results.Could be ways to make this more efficient, like only balancing variables that are truly degenerate in the first solve, which you can probably figure out based on reduced costs/duals (I think anything with a "column dual" > 0 will never be selected so can be safely excluded)
I've found that, while the results are qualitatively identical on different platforms (which is the important thing), I did have to increase the tolerance in the regression tests as we get larger floating point differences compared to before (1e-10 to 1e-6). This is still tiny, and I guess it's just because the problem is larger than before. The results are also ~identical with the new version of highs (see #1471) - just a few changes above the tolerance, which isn't too surprising, but no qualitative differences which we were certainly seeing before.
Fixes #1399
Fixes #1174 (No more qualitative differences, with the caveat that floating point differences are actually larger. It's possible that there may still be degeneracy in the problem that could cause results to differ qualitatively between platforms - difficult to rule this out entirely - but I think this is far less likely than before)
Type of change
Key checklist
$ cargo test$ cargo docpresent in the previous release
Further checks