Problem
scan_windows_via_exact (asap-query-engine/src/engines/simple_engine/mod.rs, added in #616) walks the aggregation's window grid using window_size_ms/slide_interval_ms read from the current AggregationConfig (sc.get_aggregation_config(params.aggregation_id)), and probes the store with query_precomputed_output_exact for entries shaped exactly [window_start, window_start + window_size_ms) on that grid.
update_streaming_config (called by main.rs whenever the planner fires — a normal, periodic event, not an edge case) swaps the engine's and store's config by reference with no purge:
// stores/simple_map_store/per_key.rs
/// Replace the streaming config at runtime. Called by the applier task after
/// the planner fires. Stale precomputed data for dropped aggregations expires
/// naturally via the existing cleanup policy.
pub fn update_streaming_config(&self, new_config: StreamingConfig) {
*self.streaming_config.write().unwrap() = Arc::new(new_config);
}
That comment only covers aggregation IDs that get dropped entirely. If an existing aggregation_id has its window_size_ms/slide_interval_ms changed in place (same ID, new shape), old data written under the previous shape/grid stays in the store — mixed with new data under the new shape — under the same aggregation_id, until count-based retention eventually evicts it (unrelated to shape).
scan_windows_via_exact has no way to know this happened. It always computes its probe shape/grid from whatever config is current right now, so any query whose time range still includes pre-reconfig data gets nothing back for that portion — silently, no error.
Why this is worse than a normal edge case
The old Store::query_precomputed_output tolerant scan never had this problem: it doesn't care about window_size_ms at all, it just returns whatever raw (start,end) entries exist via containment, regardless of shape or which config produced them. This bug is specific to the new exact-lookup grid-walk and hits the common case (any Tumbling query, any time after a reconfig) — not a rare shape.
Suggested fix directions (not decided)
- Version the stored shape per epoch/aggregation so a query can look up the shape that was actually in effect for the time range it's reading, not just the current one.
- Or: purge/reshape old data on reconfig (bigger behavior change, may not be desirable).
- Or: detect "this time range might predate the current config" and fall back to the tolerant scan for it.
Scope
asap-query-engine/src/engines/simple_engine/mod.rs (scan_windows_via_exact), possibly asap-query-engine/src/stores/simple_map_store/*.rs (update_streaming_config) depending on which fix direction is chosen.
Found during review of #616.
Problem
scan_windows_via_exact(asap-query-engine/src/engines/simple_engine/mod.rs, added in #616) walks the aggregation's window grid usingwindow_size_ms/slide_interval_msread from the currentAggregationConfig(sc.get_aggregation_config(params.aggregation_id)), and probes the store withquery_precomputed_output_exactfor entries shaped exactly[window_start, window_start + window_size_ms)on that grid.update_streaming_config(called bymain.rswhenever the planner fires — a normal, periodic event, not an edge case) swaps the engine's and store's config by reference with no purge:That comment only covers aggregation IDs that get dropped entirely. If an existing
aggregation_idhas itswindow_size_ms/slide_interval_mschanged in place (same ID, new shape), old data written under the previous shape/grid stays in the store — mixed with new data under the new shape — under the sameaggregation_id, until count-based retention eventually evicts it (unrelated to shape).scan_windows_via_exacthas no way to know this happened. It always computes its probe shape/grid from whatever config is current right now, so any query whose time range still includes pre-reconfig data gets nothing back for that portion — silently, no error.Why this is worse than a normal edge case
The old
Store::query_precomputed_outputtolerant scan never had this problem: it doesn't care aboutwindow_size_msat all, it just returns whatever raw(start,end)entries exist via containment, regardless of shape or which config produced them. This bug is specific to the new exact-lookup grid-walk and hits the common case (any Tumbling query, any time after a reconfig) — not a rare shape.Suggested fix directions (not decided)
Scope
asap-query-engine/src/engines/simple_engine/mod.rs(scan_windows_via_exact), possiblyasap-query-engine/src/stores/simple_map_store/*.rs(update_streaming_config) depending on which fix direction is chosen.Found during review of #616.