Add download simulation dataset feature - #187
Open
remy-rabideau wants to merge 15 commits into
Open
Conversation
…ta to PlanDev upload format
…rofile type classification in simulation data conversion
…sults
Resources with value type "real" (floating-point) but stored as discrete
profiles in the database were incorrectly classified as real profiles,
producing segments with rate: 0.0 instead of discrete dynamics. This caused
resource curves to appear staggered/stepped after download and re-upload.
Propagate the DB profile type ("discrete"/"real") from get_resource_samples
through to convert_resources, using it as the authoritative source for
real-vs-discrete classification instead of the resource value schema.
Also clean up unused variables and an extraneous f-string prefix in
aerie_client.py.
…sults output Fetch the simulation dataset's configuration arguments (simulation_dataset.arguments) and include them in the converted upload format as "simulationArguments". This preserves the original simulation configuration when the dataset is re-uploaded. Add get_simulation_dataset_arguments method to AerieClient to retrieve the arguments snapshot. Update build_simulation_upload to accept and conditionally include simulation_arguments in
Fixed two bugs causing profile segment counts to decrease when downloading a simulation dataset and re-uploading it: 1. **Boundary deduplication bug**: The download step in get_resource_samples() was deduplicating points at segment boundaries when adjacent segments had the same value at the boundary. This caused the converter to merge segments, losing one segment per continuous boundary. 2. **Zero-extent last segment bug**: When the last segment started at the plan end time, it had zero extent (start == end). The converter was skipping these zero-extent pairs instead of emitting them as valid segments. Changes: - aerie_client.py: Removed point deduplication logic. Every segment now always emits both its start and end points, even when values match at boundaries. - simulation_convert.py: Rewrote _real_segments() and _discrete_segments() to use stride-2 indexing instead of scanning all consecutive pairs. This correctly handles boundary markers (odd-indexed pairs) and zero-extent segments. - Updated test fixtures and added tests for zero-extent segments and same-value boundaries. Result: Segment counts are now preserved exactly through download/upload cycle (e.g., 573→573, 5761→5761 instead of 573→572, 5761→5760).
Fixed three bugs causing data loss when downloading a simulation dataset and re-uploading it: 1. Boundary deduplication: get_resource_samples() was merging adjacent segments that shared the same boundary value, losing segment boundaries. Now always emits start+end point pairs per segment. 2. Zero-extent last segment: When the last segment started at the plan end time, _real_segments() and _discrete_segments() skipped it. Switched from scanning all consecutive pairs to stride-2 indexing so boundary markers are skipped structurally. 3. Simulation events: Added get_simulation_events() to download topics and events via GraphQL, and included them in the upload JSON via _convert_topics() and _convert_events() in simulation_convert.py. Segment counts and event counts now match exactly through the download/upload cycle.
…for roundtrip path
… groups - Add upper bound to Python version constraint (<4.0) - Migrate from deprecated dev-dependencies to Poetry group syntax - Upgrade flake8 to ^6.0.0 with Python >=3.8.1 requirement
…imulation data Postgres intervals can include a day component (e.g. "9 days 21:55:18.272000"), but Duration.fromString on the upload side only accepts HH:MM:SS.ssssss format. Parse the day component with regex and fold it into the hours field before padding fractional seconds to 6 digits.
…esults with optional deduplication and profile type distinction
get_resource_samples() had been changed to return a second "profileTypes" key alongside "resourceSamples" so the download-simulation-full-results path could tell real profiles from discrete ones. That changed the return shape for every caller to serve one of them, and it broke get_resource_timelines(): ApiResourceSampleResults declares only resourceSamples, so the extra key raised a TypeError in from_dict(). The method has no unit test, so the suite stayed green. Move profile types to a dedicated get_profile_types() method, which queries name and type only -- no segments, no plan duration lookup -- and supports the same optional state_names filtering. Revert get_resource_samples() to its original return shape. The deduplicate flag stays on get_resource_samples(), since it changes how sampling itself works and belongs there. This also reverts the changes made to the get_resource_samples_*.json expected-result fixtures, which had been edited to accommodate the extra key. Those fixtures and test_get_resource_samples() are now byte-identical to their pre-branch state.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
aerie-cli plans download-simulation-full-resultscommand that downloads simulated activities, resource timelines, simulation configuration arguments, and simulation events for a simulation dataset, converts them into the PlanDevSimulationResultsWriterupload format, and writes the result to a single JSON file — all in one step.Motivation
Previously, producing a simulation upload JSON required three manual steps:
aerie-cli plans download-simulation→sim_results.jsonaerie-cli plans download-resources→res_results.jsonThis PR integrates all three steps into a single CLI command. The output JSON matches the format expected by PlanDev's
uploadSimulationDatasetendpoint. See the PlanDev repository for more information.Changes
src/aerie_cli/utils/simulation_convert.py (new) — conversion logic extracted into a proper importable module:
build_simulation_upload(activities, resources, resource_schemas, profile_types, ...)— top-level entry pointconvert_activities— renames fields, deriveschildIdsfromparent_idback-references, normalizes durations to 6 fractional digits, and folds the day component of Postgres intervals (e.g.9 days 21:55:18.272000) into hours, sinceDuration.fromStringon the upload side only acceptsHH:MM:SS.ssssssconvert_resources— classifies profiles as real ({initial, rate}dynamics) or discrete (raw value dynamics), reconstructs segments from aerie-cli's flattened sample pointsinfer_window— infers simulation start/end from activity timestamps and resource sample offsets_convert_topics/_convert_events— converts simulation event topics and event rows into the writer formatiso_to_doy,micros_to_extent,dt_to_doysrc/aerie_cli/commands/plans.py — adds
download-simulation-full-resultscommand:src/aerie_cli/aerie_client.py — three new client methods, plus one new optional argument:
get_profile_types(simulation_dataset_id, state_names=None)(new) — returns a mapping of resource name to"real"or"discrete"as recorded in the database. This distinction cannot be recovered from sample points alone: a real profile whose values never change is indistinguishable from a discrete one, which caused profiles to be misclassified on re-upload.get_simulation_dataset_arguments(sim_dataset_id)(new) — returns the simulation configuration arguments snapshot.get_simulation_events(sim_dataset_id)(new) — returns{"topics": [...], "events": [...]}, flattening the nested topic→events GraphQL response into a flat event list.get_resource_samples(...)gains an optionaldeduplicateargument, defaulting toTrue. The default preserves existing behavior of collapsing points at segment boundaries when the value is unchanged, which is correct for plotting but merges adjacent segments and loses one segment per continuous boundary. The round-trip path passesFalseto keep segment counts intact (e.g. 573→573 rather than 573→572). The return shape is unchanged — existing callers are unaffected.tests/unit_tests/test_simulation_convert.py (new) — 65 unit tests covering all helpers and the full
build_simulation_uploadpipeline, including simulation events, day-component intervals, zero-extent segments, and same-value boundaries. All pass without a running PlanDev instance.tests/unit_tests/test_aerie_client.py — adds
test_get_profile_types, with new fixtures underfiles/expected_results/andfiles/mock_responses/. Existing tests and fixtures are unmodified.pyproject.toml — constrains
pythonto>=3.6.8,<4.0, migrates[tool.poetry.dev-dependencies]to the[tool.poetry.group.dev.dependencies]form, and scopesflake8 ^6.0.0topython >=3.8.1. Required forpoetry lockto resolve;poetry.lockis regenerated accordingly..gitignore — ignores
config.jsonandout/.Design note
An earlier revision of this branch returned profile types as a second
"profileTypes"key fromget_resource_samples(). That changed the method's return shape for every caller in order to serve one of them, and it brokeget_resource_timelines()—ApiResourceSampleResultsdeclares onlyresourceSamples, so the extra key raised aTypeErrorinfrom_dict(). That method has no unit test, so the suite stayed green while it was broken.Profile types now live in their own
get_profile_types()method, which queriesnameandtypeonly — no segments, no plan-duration lookup.get_resource_samples()keeps its original return shape. Thededuplicateflag stays onget_resource_samples(), since it changes how sampling itself works and belongs there.Output format
The JSON produced matches the
SimulationResultsWriterformat expected by PlanDev'suploadSimulationDatasetendpoint.topicsandeventsare omitted when the simulation dataset has no events.simulationArgumentsis omitted when empty.Testing
Unit tests only (no live server required):
End-to-end, with an active session: