Skip to content

Add download simulation dataset feature - #187

Open
remy-rabideau wants to merge 15 commits into
NASA-AMMOS:developfrom
remy-rabideau:feature/download-simulation-dataset
Open

Add download simulation dataset feature#187
remy-rabideau wants to merge 15 commits into
NASA-AMMOS:developfrom
remy-rabideau:feature/download-simulation-dataset

Conversation

@remy-rabideau

@remy-rabideau remy-rabideau commented Jul 14, 2026

Copy link
Copy Markdown

Summary

Adds a new aerie-cli plans download-simulation-full-results command that downloads simulated activities, resource timelines, simulation configuration arguments, and simulation events for a simulation dataset, converts them into the PlanDev SimulationResultsWriter upload format, and writes the result to a single JSON file — all in one step.

Motivation

Previously, producing a simulation upload JSON required three manual steps:

  1. aerie-cli plans download-simulationsim_results.json
  2. aerie-cli plans download-resourcesres_results.json
  3. Running a standalone conversion script to merge and reformat both outputs

This PR integrates all three steps into a single CLI command. The output JSON matches the format expected by PlanDev's uploadSimulationDataset endpoint. See the PlanDev repository for more information.

Changes

  • src/aerie_cli/utils/simulation_convert.py (new) — conversion logic extracted into a proper importable module:

    • build_simulation_upload(activities, resources, resource_schemas, profile_types, ...) — top-level entry point
    • convert_activities — renames fields, derives childIds from parent_id back-references, normalizes durations to 6 fractional digits, and folds the day component of Postgres intervals (e.g. 9 days 21:55:18.272000) into hours, since Duration.fromString on the upload side only accepts HH:MM:SS.ssssss
    • convert_resources — classifies profiles as real ({initial, rate} dynamics) or discrete (raw value dynamics), reconstructs segments from aerie-cli's flattened sample points
    • infer_window — infers simulation start/end from activity timestamps and resource sample offsets
    • _convert_topics / _convert_events — converts simulation event topics and event rows into the writer format
    • Time helpers: iso_to_doy, micros_to_extent, dt_to_doy
  • src/aerie_cli/commands/plans.py — adds download-simulation-full-results command:

    aerie-cli plans download-simulation-full-results --sim-id <ID> --output <FILE>
    
  • src/aerie_cli/aerie_client.py — three new client methods, plus one new optional argument:

    • get_profile_types(simulation_dataset_id, state_names=None) (new) — returns a mapping of resource name to "real" or "discrete" as recorded in the database. This distinction cannot be recovered from sample points alone: a real profile whose values never change is indistinguishable from a discrete one, which caused profiles to be misclassified on re-upload.
    • get_simulation_dataset_arguments(sim_dataset_id) (new) — returns the simulation configuration arguments snapshot.
    • get_simulation_events(sim_dataset_id) (new) — returns {"topics": [...], "events": [...]}, flattening the nested topic→events GraphQL response into a flat event list.
    • get_resource_samples(...) gains an optional deduplicate argument, defaulting to True. The default preserves existing behavior of collapsing points at segment boundaries when the value is unchanged, which is correct for plotting but merges adjacent segments and loses one segment per continuous boundary. The round-trip path passes False to keep segment counts intact (e.g. 573→573 rather than 573→572). The return shape is unchanged — existing callers are unaffected.
  • tests/unit_tests/test_simulation_convert.py (new) — 65 unit tests covering all helpers and the full build_simulation_upload pipeline, including simulation events, day-component intervals, zero-extent segments, and same-value boundaries. All pass without a running PlanDev instance.

  • tests/unit_tests/test_aerie_client.py — adds test_get_profile_types, with new fixtures under files/expected_results/ and files/mock_responses/. Existing tests and fixtures are unmodified.

  • pyproject.toml — constrains python to >=3.6.8,<4.0, migrates [tool.poetry.dev-dependencies] to the [tool.poetry.group.dev.dependencies] form, and scopes flake8 ^6.0.0 to python >=3.8.1. Required for poetry lock to resolve; poetry.lock is regenerated accordingly.

  • .gitignore — ignores config.json and out/.

Design note

An earlier revision of this branch returned profile types as a second "profileTypes" key from get_resource_samples(). That changed the method's return shape for every caller in order to serve one of them, and it broke get_resource_timelines()ApiResourceSampleResults declares only resourceSamples, so the extra key raised a TypeError in from_dict(). That method has no unit test, so the suite stayed green while it was broken.

Profile types now live in their own get_profile_types() method, which queries name and type only — no segments, no plan-duration lookup. get_resource_samples() keeps its original return shape. The deduplicate flag stays on get_resource_samples(), since it changes how sampling itself works and belongs there.

Output format

The JSON produced matches the SimulationResultsWriter format expected by PlanDev's uploadSimulationDataset endpoint. topics and events are omitted when the simulation dataset has no events. simulationArguments is omitted when empty.

Testing

Unit tests only (no live server required):

pytest tests/unit_tests/ -v

End-to-end, with an active session:

aerie-cli plans download-simulation-full-results --sim-id <ID> --output simulation_upload.json

…rofile type classification in simulation data conversion
…sults

Resources with value type "real" (floating-point) but stored as discrete
profiles in the database were incorrectly classified as real profiles,
producing segments with rate: 0.0 instead of discrete dynamics. This caused
resource curves to appear staggered/stepped after download and re-upload.

Propagate the DB profile type ("discrete"/"real") from get_resource_samples
through to convert_resources, using it as the authoritative source for
real-vs-discrete classification instead of the resource value schema.

Also clean up unused variables and an extraneous f-string prefix in
aerie_client.py.
…sults output

Fetch the simulation dataset's configuration arguments (simulation_dataset.arguments) and include them in the converted upload format as "simulationArguments". This preserves the original simulation configuration when the dataset is re-uploaded.

Add get_simulation_dataset_arguments method to AerieClient to retrieve the arguments snapshot. Update build_simulation_upload to accept and conditionally include simulation_arguments in
Fixed two bugs causing profile segment counts to decrease when downloading
a simulation dataset and re-uploading it:

1. **Boundary deduplication bug**: The download step in get_resource_samples()
   was deduplicating points at segment boundaries when adjacent segments had
   the same value at the boundary. This caused the converter to merge segments,
   losing one segment per continuous boundary.

2. **Zero-extent last segment bug**: When the last segment started at the plan
   end time, it had zero extent (start == end). The converter was skipping
   these zero-extent pairs instead of emitting them as valid segments.

Changes:
- aerie_client.py: Removed point deduplication logic. Every segment now always
  emits both its start and end points, even when values match at boundaries.
- simulation_convert.py: Rewrote _real_segments() and _discrete_segments() to
  use stride-2 indexing instead of scanning all consecutive pairs. This correctly
  handles boundary markers (odd-indexed pairs) and zero-extent segments.
- Updated test fixtures and added tests for zero-extent segments and same-value
  boundaries.

Result: Segment counts are now preserved exactly through download/upload cycle
(e.g., 573→573, 5761→5761 instead of 573→572, 5761→5760).
Fixed three bugs causing data loss when downloading a simulation dataset
and re-uploading it:

1. Boundary deduplication: get_resource_samples() was merging adjacent
   segments that shared the same boundary value, losing segment
   boundaries. Now always emits start+end point pairs per segment.

2. Zero-extent last segment: When the last segment started at the plan
   end time, _real_segments() and _discrete_segments() skipped it.
   Switched from scanning all consecutive pairs to stride-2 indexing
   so boundary markers are skipped structurally.

3. Simulation events: Added get_simulation_events() to download topics
   and events via GraphQL, and included them in the upload JSON via
   _convert_topics() and _convert_events() in simulation_convert.py.

Segment counts and event counts now match exactly through the
download/upload cycle.
… groups

- Add upper bound to Python version constraint (<4.0)
- Migrate from deprecated dev-dependencies to Poetry group syntax
- Upgrade flake8 to ^6.0.0 with Python >=3.8.1 requirement
…imulation data

Postgres intervals can include a day component (e.g. "9 days 21:55:18.272000"), but Duration.fromString on the upload side only accepts HH:MM:SS.ssssss format. Parse the day component with regex and fold it into the hours field before padding fractional seconds to 6 digits.
…esults with optional deduplication and profile type distinction
get_resource_samples() had been changed to return a second "profileTypes"
key alongside "resourceSamples" so the download-simulation-full-results
path could tell real profiles from discrete ones. That changed the return
shape for every caller to serve one of them, and it broke
get_resource_timelines(): ApiResourceSampleResults declares only
resourceSamples, so the extra key raised a TypeError in from_dict(). The
method has no unit test, so the suite stayed green.

Move profile types to a dedicated get_profile_types() method, which
queries name and type only -- no segments, no plan duration lookup -- and
supports the same optional state_names filtering. Revert
get_resource_samples() to its original return shape.

The deduplicate flag stays on get_resource_samples(), since it changes how
sampling itself works and belongs there.

This also reverts the changes made to the get_resource_samples_*.json
expected-result fixtures, which had been edited to accommodate the extra
key. Those fixtures and test_get_resource_samples() are now byte-identical
to their pre-branch state.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant