Skip to content

scan_windows_via_exact assumes stored windows match the CURRENT config's shape/grid, but update_streaming_config doesn't purge old data #618

Description

@milindsrivastava1997

Problem

scan_windows_via_exact (asap-query-engine/src/engines/simple_engine/mod.rs, added in #616) walks the aggregation's window grid using window_size_ms/slide_interval_ms read from the current AggregationConfig (sc.get_aggregation_config(params.aggregation_id)), and probes the store with query_precomputed_output_exact for entries shaped exactly [window_start, window_start + window_size_ms) on that grid.

update_streaming_config (called by main.rs whenever the planner fires — a normal, periodic event, not an edge case) swaps the engine's and store's config by reference with no purge:

// stores/simple_map_store/per_key.rs
/// Replace the streaming config at runtime. Called by the applier task after
/// the planner fires. Stale precomputed data for dropped aggregations expires
/// naturally via the existing cleanup policy.
pub fn update_streaming_config(&self, new_config: StreamingConfig) {
    *self.streaming_config.write().unwrap() = Arc::new(new_config);
}

That comment only covers aggregation IDs that get dropped entirely. If an existing aggregation_id has its window_size_ms/slide_interval_ms changed in place (same ID, new shape), old data written under the previous shape/grid stays in the store — mixed with new data under the new shape — under the same aggregation_id, until count-based retention eventually evicts it (unrelated to shape).

scan_windows_via_exact has no way to know this happened. It always computes its probe shape/grid from whatever config is current right now, so any query whose time range still includes pre-reconfig data gets nothing back for that portion — silently, no error.

Why this is worse than a normal edge case

The old Store::query_precomputed_output tolerant scan never had this problem: it doesn't care about window_size_ms at all, it just returns whatever raw (start,end) entries exist via containment, regardless of shape or which config produced them. This bug is specific to the new exact-lookup grid-walk and hits the common case (any Tumbling query, any time after a reconfig) — not a rare shape.

Suggested fix directions (not decided)

  • Version the stored shape per epoch/aggregation so a query can look up the shape that was actually in effect for the time range it's reading, not just the current one.
  • Or: purge/reshape old data on reconfig (bigger behavior change, may not be desirable).
  • Or: detect "this time range might predate the current config" and fall back to the tolerant scan for it.

Scope

asap-query-engine/src/engines/simple_engine/mod.rs (scan_windows_via_exact), possibly asap-query-engine/src/stores/simple_map_store/*.rs (update_streaming_config) depending on which fix direction is chosen.

Found during review of #616.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions