[ghukill]

Note

This PR is stacked on #282.

Purpose and background context

This is the 3rd and final PR that seeks to improve Transmogrifier transform performance, specifically for alma.

This converts the alma transformer to use lxml instead of BeautifulSoup4 (bs4) for transformations.

This work was POC'ed quite some time ago and was confirmed to be a massive speedup. When combined with the parallel workers from [the previous PR], this pushes transformation speeds into ~4-5k records/second. Before any speed improvements a full transform of alma at ~4.5m records was roughly 14-15 hours. If we assume 4.5k records/second after parallel workers + lxml based transformation, we're looking at ~16.6 minutes.

This fundamentally changes how we can develop in Transmogrifier. Now, we are unblocked to make meaningful and substantive changes and perform an entire run to see how they play out in alma. Timmy is helpful for the analyzing the new transformed records, but without these Transmogrifier speed improvements, we would wait too long between those analysis.

This shift from bs4 --> lxml takes the following approach:

• create some helper methods like Marc.get_element_string() and Marc.find_all_elements() that mimic the ergonomics of bs4, but using lxml under the hood
• updates to all fields methods are still required, but very minimal by leaning on these helper methods

The effect is a shift in the transformation libraries used, but very little change ergonomically in code and zero change to the records themselves. This is confirmed by some analysis in Timmy: see the following Timmy report that analyzes all alma records before any of the work in these stacked PRs and after, timx-738-alma-full-before-and-after.pdf. The finding is that two fields have changes, identifiers and holdings, but these are related to TIMX-692 and not these PRs.

Supporting the case for all this work, the code changes from TIMX-692 have not been applied to the records in production because it's so slow! With the work complete in TIMX-700 it's arguablly much easier, as we could kickoff a "transform all current alma records" style ETL run, but this would still take ~14-15 hours.

Lastly, note that this bs4 --> lxml conversion is only for alma at this time. All other XML sources would benefit from this treatment as well, but were not included in this PR.

How can a reviewer manually see the effects of these changes?

Tempory Timmy Instance

To facilitate seeing the before and after, a temporary Timmy instance has been created on our linux box for review.

The following commands were used to generate this temporary Timmy instance and ingest all alma records (recalling that they predate the TIMX-692 changes):

# create temp timmy
timmy \
instance \
create-temp ~/.timmy/instances/temp/timx-738 \
--name timx-738 \
--aws-profile timmy-prod

# ingest
timmy \
--config ~/.timmy/instances/temp/timx-738/timmy.toml \
timmylake import --sync s3://timdex-extract-prod-300442551476/timmy/exports/alma-current-full-2026-11-02/

Next, Transmogrifier was run on that machine transforming the full alma corpus with the new parallel workers (note that we're using 7 workers because this machine has 8 CPU cores) + lxml code, writing the output to S3 as well:

uv run transform -s alma \
--input-parquet s3://timdex-extract-prod-300442551476/timmy/exports/alma-current-full-2026-11-02/ \
--output-parquet s3://timdex-extract-prod-300442551476/timmy/exports/alma-current-full-timx-738-transformed/ \
--run-id alma-current-full-timx-738-transformed \
--workers 7

The entire process took 22 minutes to complete:

2026-11-03 11:38:50,486 INFO transmogrifier.cli.main(): Running transform for source alma
...
2026-11-03 12:01:05,136 INFO transmogrifier.sources.transformer.write_to_parquet_files(): Parquet files write complete - files: 43, rows: 4229248
2026-11-03 12:01:05,144 INFO transmogrifier.cli.main(): Completed transform, total records processed: 4229248, transformed records: 4229248, skipped records: 0, deleted records: 0
2026-11-03 12:01:05,144 INFO transmogrifier.cli.main(): Total time to complete transform: 0:22:14.658843

Finally, the transformed records are ingested into Timmy for analysis:

# ingest
timmy \
--config ~/.timmy/instances/temp/timx-738/timmy.toml \
timmylake import --sync s3://timdex-extract-prod-300442551476/timmy/exports/alma-current-full-timx-738-transformed/

# serve webapp
uv run timmy --config ~/.timmy/instances/temp/timx-738/timmy.toml serve -d

With VPN connected, access Timmy at: http://10.51.1.241:5100/.

1- Navigate to runs: http://10.51.1.241:5100/runs/.

2- Locate the run alma-current-full-timx-738-transformed: http://10.51.1.241:5100/runs/alma-current-full-timx-738-transformed

3- Click into "Analyze field changes": http://10.51.1.241:5100/runs/alma-current-full-timx-738-transformed/changes

The data here shows what the report summarizes above. Most fields are unchanged when getting transformed via lxml and new transform code. The fields that do change, are unrelated from these PRs.

Includes new or updated dependencies?

NO

Changes expectations for external applications?

NO

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/TIMX-738

Code review

• Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

[Copilot]

🟡 Changes recommended

Element serialization can include tail text and produce invalid standalone XML, while changed whitespace normalization alters transformed records.

2 open issues

• Exclude element tails when serializing standalone XML records · New
• Preserve BeautifulSoup whitespace normalization · New

What changed in this PR

This PR moves Alma’s MARC transformations from BeautifulSoup to lxml, complementing the parallel-worker changes in #282 to accelerate full-corpus processing while aiming to preserve transformed output.

Changes:

• Adds lxml-based MARC parsing and lookup helpers, and migrates field extraction to use them.
• Supports serialization of parsed lxml records and makes contributor-kind ordering deterministic.
• Updates MARC tests and adds coverage for helper behavior, parsing, error handling, and serialization.

File	Description
transmogrifier/​sources/​xml/​marc.py	Migrates MARC parsing and field extraction to lxml.
transmogrifier/​sources/​transformer.py	Adds serialization support for lxml elements.
tests/​sources/​xml/​test_marc.py	Adapts MARC tests and covers new helpers and ordering behavior.
tests/​sources/​test_transformer.py	Tests serialization of parsed lxml records.

_{Review effort: Balanced}

[jonavellecuerdo]

Approving based on the Timmy report comparing records in Alma before and after any of the changes in the last three (stacked) PRs!