fix: propagate LightGBM iteration failures

[Rana Singh (ranadeepsingh)]

Related issues and prior work

Supersedes #2684, which was closed because its author could not sign the CLA.
Related to #569 and #728.

This fixes the error-handling defect exposed by those reports. It does not claim
to fix every executor, native-library, resource, or network failure that can
produce a LightGBM socket error.

What changed

TrainUtils.updateOneIteration previously caught a native iteration exception,
logged it as early termination, set state.isFinished = true, and allowed model
construction to continue.

The updated behavior:

• logs the failed iteration and rethrows the original exception;
• leaves the training state unfinished after failure;
• lets Spark fail the task or job instead of returning a partial model;
• keeps legitimate LightGBM completion and early stopping unchanged.

The regression test uses a booster that fails during its native update. Before
the fix, the test failed because no exception escaped. After the fix, it verifies
that the same exception object propagates and training is not marked complete.

Validation

• sbt scalastyle "Test / scalastyle" compile test:compile
• sbt "lightgbm/testOnly com.microsoft.azure.synapse.ml.lightgbm.split1.TrainUtilsSuite"
• Fabric lightgbm-streaming scenario using jars built from this branch:
  • commit: 0df0110857334c4443be71b174a82bcbc2e8ca0b
  • runtime: Spark 3.5.5.5.4.20260807.1
  • fixed two-node Medium pool with autoscaling disabled
  • 100,000 rows, 8 partitions, 10 repeated distributed fits
  • exact core, lightgbmlib-3.3.510, and LightGBM jar provenance verified
  • application: application_1788541423736_0001
  • result: passed

Repository-wide Black currently reports 11 pre-existing formatting differences
in unchanged Python files. This PR does not modify Python.

Broader LightGBM issue assessment

The customer-reported instability is genuine, but Connection refused and
native socket code 104 are usually secondary symptoms. Current master already
contains fixes for worker-port reservation, driver socket cleanup, retry
diagnostics, barrier topology handling, IPv6 endpoints, and validation-data
transfer.

Separate evidence points to a possible native lightgbmlib-3.3.510 distributed
categorical crash family in #2302, #2438, #2467, and #2674. That needs its own
multi-executor native regression and should not be hidden inside this
error-propagation fix.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Rana Singh (ranadeepsingh)]

/azp run

[Rana Singh (ranadeepsingh)]

/review

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🟢 Approval recommended

The behavior change is narrowly scoped, aligns with the stated defect/acceptance criteria, and is covered by a targeted regression test asserting exception identity and state invariants.

Review tier: Lite
Findings: None

What changed in this PR

This PR fixes LightGBM training error-handling so that native iteration failures are logged and propagated, allowing Spark to fail the task/job rather than returning a partial model and incorrectly marking training as completed.

Changes:

• Update TrainUtils.updateOneIteration to log iteration failures and rethrow the original exception instead of converting failures into “early termination”.
• Add a regression test that injects a failing LightGBMBooster and asserts the same exception instance propagates and state.isFinished remains false.

File	Description
lightgbm/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​lightgbm/​TrainUtils.scala	Changes iteration failure handling to error-log and rethrow, preserving correct task/job failure semantics.
lightgbm/​src/​test/​scala/​com/​microsoft/​azure/​synapse/​ml/​lightgbm/​split1/​TrainUtilsSuite.scala	Adds a deterministic regression test covering exception propagation and training-state completion behavior.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Rana Singh (ranadeepsingh)]

/azp run

[Rana Singh (ranadeepsingh)]

/review

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🟢 Approval recommended

The change is narrowly scoped to exception propagation, matches the stated intent, and includes a focused regression test covering the previously incorrect behavior.

Review tier: Lite
Findings: None

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.41%. Comparing base (b3b5239) to head (0df0110).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2695   +/-   ##
=======================================
  Coverage   87.41%   87.41%           
=======================================
  Files         341      341           
  Lines       20928    20926    -2     
  Branches     2217     2218    +1     
=======================================
- Hits        18294    18293    -1     
+ Misses       2634     2633    -1     

Files with missing lines	Coverage Δ
...crosoft/azure/synapse/ml/lightgbm/TrainUtils.scala	87.39% <100.00%> (-0.21%)	⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.