Use new machine eflomal implementation by Enkidu93 · Pull Request #336 · sillsdev/machine.py

Enkidu93 · 2026-07-23T15:34:31Z

Also:

Train on training parallel data as well as pretranslation data when running alignments during an MT job
Fix a few copy-paste errors floating in word alignment build job classes

This update is a little difficult to test. I have tested it E2E and the Serval E2E tests do pass. The scores are populated properly. I compared the alignments with the previous version for the E2E test data and they seem mostly similar - maybe the old ones are a little better 😬 - but it's difficult to judge. I'm going to trust that, as long as I am using the API properly, switching to the new implementation and training the aligner on the training data is an improvement. My outstanding questions are:

Am I using the alignment API correctly? Am I preparing the training and inference data correctly? I followed what was in the word alignment build job but I honestly don't have a lot of confidence in that since I noticed a couple things that seem to be bugs/typos. I do wonder as well about the call to lowercase() - see my TODO comments.
Is this the right way to configure things? Or do we want a separate NMT word alignment configuration section?

This change is

codecov-commenter · 2026-07-23T15:48:06Z

Codecov Report

❌ Patch coverage is 93.57430% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.10%. Comparing base (d2ae939) to head (a4f1cf2).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
machine/jobs/word_alignment_file_service.py	33.33%	6 Missing ⚠️
machine/jobs/nmt_engine_build_job.py	94.93%	4 Missing ⚠️
machine/jobs/word_alignment_build_job.py	95.45%	2 Missing ⚠️
machine/corpora/corpora_utils.py	0.00%	1 Missing ⚠️
machine/corpora/flatten.py	85.71%	1 Missing ⚠️
...ine/jobs/thot/thot_word_alignment_model_factory.py	0.00%	1 Missing ⚠️
...nslation/thot/thot_word_alignment_model_trainer.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #336      +/-   ##
==========================================
+ Coverage   92.02%   92.10%   +0.07%     
==========================================
  Files         389      389              
  Lines       24485    24559      +74     
==========================================
+ Hits        22532    22619      +87     
+ Misses       1953     1940      -13

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ddaspit

@ddaspit made 2 comments.
Reviewable status: 0 of 10 files reviewed, 2 unresolved discussions (waiting on Enkidu93).

machine/jobs/nmt_engine_build_job.py line 182 at r1 (raw file):

                parallel_training_corpus.get_rows(),
            )
        )  # TODO .lowercase()?

We should perform Unicode and casing normalization.

machine/jobs/nmt_engine_build_job.py line 196 at r1 (raw file):

        parallel_pretranslation_rows = list(parallel_pretranslation_corpus)  # TODO .lowercase()?
        alignments = alignment_model.align_batch(parallel_pretranslation_rows)

This will work, but it would be better to use the transductive API, i.e. get_training_alignment. align_batch will be more expensive.

Enkidu93

@Enkidu93 made 2 comments.
Reviewable status: 0 of 10 files reviewed, 2 unresolved discussions (waiting on ddaspit).

machine/jobs/nmt_engine_build_job.py line 182 at r1 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

We should perform Unicode and casing normalization.

OK, thank you 👍, but when we actually go to populate the source and target tokens in the pretranslation data structure, we will want them to be exactly as they were in the original text, right? Otherwise, the marker placement won't work properly. Or should we update the code in the marker placement handler to only look for normalized substrings?

machine/jobs/nmt_engine_build_job.py line 196 at r1 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

This will work, but it would be better to use the transductive API, i.e. get_training_alignment. align_batch will be more expensive.

Would you like me only allow transductive models or just use the transductive API if the model supports it and otherwise default to this?

ddaspit

@ddaspit reviewed 14 files and all commit messages, made 3 comments, and resolved 2 discussions.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on Enkidu93).

machine/jobs/nmt_engine_build_job.py line 182 at r1 (raw file):

Previously, Enkidu93 (Eli C. Lowry) wrote…

OK, thank you 👍, but when we actually go to populate the source and target tokens in the pretranslation data structure, we will want them to be exactly as they were in the original text, right? Otherwise, the marker placement won't work properly. Or should we update the code in the marker placement handler to only look for normalized substrings?

You are correct. I added a commit that inferences on the normalized tokens but returns the unnormalized tokens. It also added transductive API support, a full streaming pipeline, and fixed a bug in flatten.

machine/jobs/nmt_engine_build_job.py line 196 at r1 (raw file):

Previously, Enkidu93 (Eli C. Lowry) wrote…

Would you like me only allow transductive models or just use the transductive API if the model supports it and otherwise default to this?

I added support for the transductive API.

machine/jobs/word_alignment_build_job.py line 86 at r2 (raw file):

        return train_corpus_size

    def _batch_inference(

This should be updated to work similar to NmtEngineBuildJob._align.

Enkidu93

@Enkidu93 partially reviewed 1 file and made 3 comments.
Reviewable status: 11 of 14 files reviewed, 1 unresolved discussion (waiting on ddaspit).

machine/jobs/nmt_engine_build_job.py line 182 at r1 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

You are correct. I added a commit that inferences on the normalized tokens but returns the unnormalized tokens. It also added transductive API support, a full streaming pipeline, and fixed a bug in flatten.

OK, great 👍

machine/jobs/nmt_engine_build_job.py line 196 at r1 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

I added support for the transductive API.

Great, thank you

machine/jobs/word_alignment_build_job.py line 86 at r2 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

This should be updated to work similar to NmtEngineBuildJob._align.

Done. Of course, the word alignment API isn't really set up yet to take advantage of the transductive model. For now, I've just had it use align_batch, but maybe we should make an issue to update how training/inferencing work for word alignment in general to take advantage of this. Even if we allow for training and inferencing on the same data (which we probably just should for word alignment in general?), we'd still need a way to index the training data so we can retrieve the alignments.

ddaspit

@ddaspit reviewed 4 files and all commit messages, made 2 comments, and resolved 1 discussion.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on Enkidu93).

machine/jobs/word_alignment_build_job.py line 97 at r4 (raw file):

    ) -> None:

        inference_inputs = self._word_alignment_file_service.get_word_alignment_inputs()

In order to make it worthwhile to spool the data to disk, get_word_alignment_inputs should return a generator instead of a list.

machine/jobs/word_alignment_build_job.py line 135 at r4 (raw file):

                    check_canceled()
                segments = [(lowercase(row.source_segment), lowercase(row.target_segment)) for _, row in wa_batch]
                alignments = alignment_model.align_batch(segments)

Progress is never advanced while inferencing.

Enkidu93

@Enkidu93 reviewed 14 files and all commit messages, and made 2 comments.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on ddaspit).

machine/jobs/word_alignment_build_job.py line 97 at r4 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

In order to make it worthwhile to spool the data to disk, get_word_alignment_inputs should return a generator instead of a list.

Done.

machine/jobs/word_alignment_build_job.py line 135 at r4 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

Progress is never advanced while inferencing.

Done.

ddaspit

@ddaspit reviewed 3 files and all commit messages, made 1 comment, and resolved 2 discussions.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on Enkidu93).

Enkidu93 · 2026-08-10T20:28:27Z

Decided to test this E2E - and I'm glad I did because I hit an error that I wasn't hitting before:

Traceback (most recent call last):
  File "/root/.clearml/venvs-builds/code/untitled.py", line 13, in <module>
    run(args)
  File "/usr/local/lib/python3.12/site-packages/machine/jobs/build_nmt_engine.py", line 74, in run
    raise e
  File "/usr/local/lib/python3.12/site-packages/machine/jobs/build_nmt_engine.py", line 64, in run
    train_corpus_size, _ = job.run(progress, check_canceled)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/machine/jobs/translation_engine_build_job.py", line 45, in run
    self._batch_inference(parallel_corpus, progress_reporter, check_canceled)
  File "/usr/local/lib/python3.12/site-packages/machine/jobs/nmt_engine_build_job.py", line 134, in _batch_inference
    for pt_info in results:
                   ^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/machine/jobs/nmt_engine_build_job.py", line 231, in _align
    transductive_model.get_training_alignment(index + i) for i in range(len(pt_batch))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/machine/translation/thot/thot_symmetrized_word_alignment_model.py", line 74, in get_training_alignment
    _, matrix = self._aligner.get_training_alignment(n)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
MemoryError: std::bad_array_new_length

Not my top priority right now but just so you know why I haven't merged this yet.

Enkidu93 · 2026-08-12T19:31:04Z

This looks to be caused by having empty pretranslations. Peter is going to take a look on the thot side since he's already making updates to that code.

- full streaming pipeline for inferencing - fall back to inductive API - fix text id handling in "flatten"

Enkidu93

Alright, the slow-down was from progress-reporting. I've reduced the reporting significantly and now the whole alignment phase takes a few minutes rather than twenty. We could reduce it further or eliminate it altogether if you prefer. I've also updated thot to the new version that fixes the error above. @ddaspit, I think this is ready for re-review.

@Enkidu93 made 1 comment.
Reviewable status: 14 of 18 files reviewed, all discussions resolved (waiting on ddaspit).

ddaspit

Good job tracking this down. What kind of an impact on speed does it have if we remove progress reporting during alignment inferencing altogether?

@ddaspit reviewed 4 files and all commit messages, and made 2 comments.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on Enkidu93).

machine/translation/thot/thot_word_alignment_model_trainer.py line 195 at r12 (raw file):

        )
        cur_step = 0
        update_frequency = num_steps // 10 if num_steps and num_steps > 10 else None

I remember us running into a similar problem a while ago. We do progress throttling in the get_clearml_progress_caller function. Was that not doing a good enough job? Should that function be updated instead of this? At the very least, we should add a comment explaining the issue.

Enkidu93

It's about comparable if we remove it altogether actually - I just confirmed. It's still taking 2-5 minutes.

@Enkidu93 made 2 comments.
Reviewable status: 17 of 18 files reviewed, 1 unresolved discussion (waiting on ddaspit).

machine/translation/thot/thot_word_alignment_model_trainer.py line 195 at r12 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

I remember us running into a similar problem a while ago. We do progress throttling in the get_clearml_progress_caller function. Was that not doing a good enough job? Should that function be updated instead of this? At the very least, we should add a comment explaining the issue.

Yes, the throttling there doesn't seem to do much. The iteration reporting in ClearML is very slow. We are already throttling NMT training in the trainer similarly: #227. If you'd prefer, we could try to have a uniform way of throttling them in the cleaml progress function. We could also pass the update_frequency to the constructor if you prefer. Part of the reason unifying them and throttling them in one place might be difficult is that you may genuinely want the frequency to differ task to task.

ddaspit

@ddaspit reviewed 1 file and all commit messages, made 2 comments, and resolved 1 discussion.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on Enkidu93).

machine/translation/thot/thot_word_alignment_model_trainer.py line 195 at r12 (raw file):

Previously, Enkidu93 (Eli C. Lowry) wrote…

Yes, the throttling there doesn't seem to do much. The iteration reporting in ClearML is very slow. We are already throttling NMT training in the trainer similarly: #227. If you'd prefer, we could try to have a uniform way of throttling them in the cleaml progress function. We could also pass the update_frequency to the constructor if you prefer. Part of the reason unifying them and throttling them in one place might be difficult is that you may genuinely want the frequency to differ task to task.

I'm good with tailoring the throttling for each task.

Enkidu93 requested a review from ddaspit July 23, 2026 15:34

Enkidu93 mentioned this pull request Jul 23, 2026

Calculate basic alignment score for word pairs from the sentence_score #316

Closed

ddaspit requested changes Jul 23, 2026

View reviewed changes

Enkidu93 commented Jul 23, 2026

View reviewed changes

ddaspit reviewed Jul 24, 2026

View reviewed changes

Enkidu93 force-pushed the use_new_eflomal_implementation branch from 68cff34 to 9cdec1a Compare July 28, 2026 17:08

Enkidu93 commented Jul 29, 2026

View reviewed changes

Enkidu93 requested a review from ddaspit August 5, 2026 14:01

ddaspit reviewed Aug 6, 2026

View reviewed changes

Enkidu93 commented Aug 10, 2026

View reviewed changes

ddaspit approved these changes Aug 10, 2026

View reviewed changes

pmachapman mentioned this pull request Aug 13, 2026

Fix crash when initializing a WordAlignmentMatrix with one dimension sillsdev/thot#16

Merged

Enkidu93 and others added 15 commits August 20, 2026 15:13

Use native eflomal implementation

4b4bba0

Remove redundant alignment score

718b067

Fix api usage; adjust test

a65e946

Remove unused import

a87817c

Properly mock word alignment; revert settings

c211c6e

Fix logging; do not include NULL alignments

883f37d

Make naming more consistent

d1cd61d

Revert settings.yml change

7e96370

Use transductive API for getting alignments

dc5fba1

- full streaming pipeline for inferencing - fall back to inductive API - fix text id handling in "flatten"

Update word alignment build job to properly use alignment API and batch

586deda

Address reviewer comments

8257dc4

Update word alignment job test mock to return correct type

f512b49

Add logging for debugging

9ccbf07

Revert debugging changes

4f7ef98

Debug commit

b247d13

Enkidu93 added 6 commits August 20, 2026 15:13

Update thot; report steps (debug)

6599e83

Debug steps and reporting

963ee7c

Remove debug logging; add proper align phase

01ddc0e

Debug logging

41a5f2b

Fix update frequency setting

7946322

Remove debug logging; make update frequency dynamic

ab22a84

Enkidu93 force-pushed the use_new_eflomal_implementation branch from 98b9e6f to ab22a84 Compare August 20, 2026 19:14

Enkidu93 commented Aug 20, 2026

View reviewed changes

ddaspit requested changes Aug 20, 2026

View reviewed changes

Enkidu93 added 3 commits August 20, 2026 15:55

Remove reporting (debug commit)

0a98932

Fix typo (debug)

33c5844

Remove debug changes; add comment

a4f1cf2

Enkidu93 commented Aug 20, 2026

View reviewed changes

ddaspit approved these changes Aug 20, 2026

View reviewed changes

Uh oh!

Uh oh!

Conversation

Enkidu93 commented Jul 23, 2026 • edited by ddaspit Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ddaspit left a comment

Choose a reason for hiding this comment

Uh oh!

Enkidu93 left a comment

Choose a reason for hiding this comment

Uh oh!

ddaspit left a comment

Choose a reason for hiding this comment

Uh oh!

Enkidu93 left a comment

Choose a reason for hiding this comment

Uh oh!

ddaspit left a comment

Choose a reason for hiding this comment

Uh oh!

Enkidu93 left a comment

Choose a reason for hiding this comment

Uh oh!

ddaspit left a comment

Choose a reason for hiding this comment

Uh oh!

Enkidu93 commented Aug 10, 2026

Uh oh!

Enkidu93 commented Aug 12, 2026

Uh oh!

Enkidu93 left a comment

Choose a reason for hiding this comment

Uh oh!

ddaspit left a comment

Choose a reason for hiding this comment

Uh oh!

Enkidu93 left a comment

Choose a reason for hiding this comment

Uh oh!

ddaspit left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Enkidu93 commented Jul 23, 2026 •

edited by ddaspit

Loading

codecov-commenter commented Jul 23, 2026 •

edited

Loading