[ghukill]

Note

This is the first of three stacked PRs.

Purpose and background context

This PR updates Transmogrifier to accept parquet files as input and output. This builds on work from TIMX-700 that introduced the TIMDEX dataset as an input source.

Before that work, the only input source for Transmogrifier was individual files either locally or in S3. Dataset as input unlocked running a transformation job against a subset of records in the dataset (e.g. all current records for a source) without the need to re-harvest. Parquet as input allows transformation of a subset of records, where the subset is most likely created by Timmy.

The updated README shows the new CLI arguments, whichi also introduce an --output-file to provide some parity with --input-file.

It is worth observing that in TIMDEX, Transmogrifier is the step/application that creates an ETL run in the datatset. Therefore, using parquet files as input extends the idea of a "patch" run -- as opposed to a "daily" or "full" -- because we are transforming source metadata records already in the dataset; they may not have come directly from the dataset like --input-dataset, but the parquet files are likely a direct export of those records.

There are two primary things we get from these new input/output options:

1. Creation of a subset in Timmy, exported to parquet, becomes a true "patch" ETL job of a subset of records.

2. Timmy exports a subset, Transmogrifier transforms and writes back to parquet, Timmy imports and analysis is performed showing what the transformation did to the records compared to the previous versions.

It is # 2 that drove this PR, as we now can run huge alma transformations jobs using parquet as input and output, with the goal being analysis in Timmy. Once we are satisfied with the results of the transformation logic, we will likely use --input-dataset for a "real" ETL job that also writes back to the TIMDEX dataset.

How can a reviewer manually see the effects of these changes?

1- Set TIMDEX Dev1 AWS credentials

2- Perform a Transmogrifier run using parquet files from S3, and write them locally for quick analysis:

uv run transform \
--verbose \
--source alma \
--input-parquet s3://timdex-extract-dev-222053980223/timmy/exports/alma-subject-criticism-and-interpretation/ \
--output-parquet /tmp/timmy/exports/alma-subject-criticism-and-interpretation

You should see some similar output:

2026-11-03 09:27:18,862 INFO transmogrifier.cli.main(): Logger 'root' configured with level=INFO
2026-11-03 09:27:18,862 INFO transmogrifier.cli.main(): No Sentry DSN found, exceptions will not be sent to Sentry
2026-11-03 09:27:18,862 INFO transmogrifier.cli.main(): Running transform for source alma
2026-11-03 09:27:20,062 INFO transmogrifier.sources.transformer.parse_source_records_from_parquet(): Reading source records from 1 parquet file(s)
2026-11-03 09:32:14,901 INFO transmogrifier.sources.transformer.write_to_parquet_files(): Parquet files write complete - files: 1, rows: 13941
2026-11-03 09:32:14,902 INFO transmogrifier.cli.main(): Completed transform, total records processed: 13941, transformed records: 13941, skipped records: 0, deleted records: 0
2026-11-03 09:32:14,902 INFO transmogrifier.cli.main(): Total time to complete transform: 0:04:56.040478

3- Quick analysis of the output parquet using DuckDB. First, start a shell:

duckdb

Then select some rows from the parquet data output by Transmogrifier:

select *
from read_parquet('/tmp/timmy/exports/alma-subject-criticism-and-interpretation')
limit 3;

You should see rows that contain columns [timdex_record_id, source_record, transformed_record, source, run_date, run_type, action, run_id, run_record_offset, run_timestamp]. By no accident or coincidence, these are precisely what Transmogrifier would otherwise write to the TIMDEX dataset.

We might start to consider this the most minimal "Submission Information Package" (SIP) for TIMDEX, as it includes the source + transformed record and metadata about the ETL run that produced it.

4- Lastly, we can demonstrate the roundtrip-ability of this data by using what Transmogrifier just output as input for another transformation:

uv run transform \
--source alma \
--input-parquet /tmp/timmy/exports/alma-subject-criticism-and-interpretation \
--output-parquet /tmp/timmy/exports/alma-subject-criticism-and-interpretation-v2

Obviously the outputs will be the same, but it demonstrates the input + output schema is identical. This demonstrates how Timmy can either produce or consume the parquet "SIPs" as well.

Includes new or updated dependencies?

NO

Changes expectations for external applications?

YES: this opens the door to a "patch" ETL run consisting of a subset of records that came from the TIMDEX dataset.

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/TIMX-738

Code review

• Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

[ghukill]

@jonavellecuerdo - here is the (re)introduction of Transmogrifier also supporting writing to a simple file. Again, may not get used much, but we almost get it for free and it provides nice parity I think at the input + output formats.

[ehanson8]

Looks good and works as expected!

[ehanson8]

Interesting way for the agent to address this! 😆

[ghukill]

I thought the same! It's not what I would have written... but I'm growing to like it.