ci: make SynapseML releases approved and resumable

[Rana Singh (ranadeepsingh)]

What changed

This PR turns a calculated release checklist into an approved, resumable
release process. One sealed plan records the reviewed commits, selected
repositories and runtimes, package versions, rebuild counters, and feeds.
The driver queues only the selected work and saves its build IDs.

Internal can release against an existing OSS base without rebuilding or
republishing OSS. New-base releases publish the OSS dependencies first, so
Internal CI no longer has to pass before its dependency exists.

Track	Selection	OSS publication
New OSS base	full, repositories=oss	Selected OSS artifacts only
Initial Internal release	full, repositories=internal, patch 0	None
Internal hotfix	internal-only, nonzero patch	None

Important safeguards

• preflight, status, and a plain resume make no remote writes. Queueing
  requires --apply and the exact human-approved --approve-plan.
• A locked, atomically written ledger retains pending, failed, and unknown
  submissions. A persistent same-directory claim rejects a second state
  filename. Explicit pip/UPack retries preserve history and require complete
  absence for the original failed group. Maven retries remain unsupported;
  a failed Maven publication needs a new coordinate and approved plan.
• Ordinary CI stays snapshot-only, even at a tagged commit. Release jobs check
  the exact source and refuse immutable-package overwrites. Public PyPI
  collisions remain failures, and primary Maven evidence includes the wheel.
• Inventory is diagnostic, not approval. Notes and BBC-VHD writes require
  complete, fresh producer evidence with matching source, request, and output
  hashes. GitHub receives only bounded public Maven evidence, never Internal
  bindings or Azure credentials.
• Separate approved OSS and Internal UPack plans can update both BBC-VHD pins
  using their original evidence. Internal hotfixes preserve the exact OSS pin.
• Rehearsal uses explicit nonproduction pip/UPack feeds. A package-name change
  inside a production feed is not a rehearsal.

Release workflows also enforce the full target policy, push tag families
atomically, and preserve the reviewed merge commit. ESRP staging is explicit
and nondestructive for both supported Scala lines.

Commands and documentation

Start with the release command guide. It includes
plan generation, no-write previews, approved execution, status, adoption,
guarded retries, read-only lock inspection, evidence, notes, and BBC-VHD handoff.

The release skill links focused
preflight, recovery, and approval-boundary references. AGENTS.md links the
skill and requires review reports under reviews/pr-<number>/ or a defined
feature directory.

Companion changes cover Internal publication, the official publisher, and the
canonical release guide.
These must land before operators use the complete cross-repository procedure.

Validation and remaining gates

Review findings and resolutions are retained in reviews/pr-2628.
All six implementation review rounds are complete.
Follow-up fixes make SBT probes portable, add blob-inspection failure context,
and resolve each ESRP module directory once. Both follow-up review threads are
resolved.

On head 4c166c4928, GitHub run 33965780135
passed all 584 release cases and compilation, Scala style, and Python formatting
checks. Separate Linux/JDK11 runs exercised the native SBT probes. The publisher's
four real-driver consumer cases also passed. The current-head Copilot Lite
review reports no findings but still calls for human review of release-critical
changes.

Azure run 234655017
hit a hostname/certificate mismatch during task authentication, before the NN
test and artifact-publication scripts ran.
Retry 234665132
finished on the same merge commit and exact reviewed source. Publish succeeded,
all 40 NN tests passed, and every other job except the advisory Spark 4.1 replay
succeeded, including Databricks GPU E2E. Azure reports partiallySucceeded;
the advisory replay is still a failed check, not a fully green run.

The advisory Spark 4.1 replay conflicts only in tools/esrp/prepare_jar.py,
where the new explicit-Scala implementation replaces the old hardcoded suffix.
In isolated full checkouts of port 06897e5b27, both the baseline with the
three CI prerequisites and the PR replay with that one manual resolution passed
sbt test:compile on JDK17, Scala 2.13.17, and Spark 4.1.1. An independent replay
matched every staged source byte. No shared port branch was changed.
The automatic replay still needs that resolution; compilation is not runtime
validation.

No production release, approval click, merge, or Fabric rollout has been
performed. ESRP/SAW approval, PR merges, White-Glove, and release-train selection
remain with authorized people.

Does this PR change any dependencies?

• No.
• Yes.

Does this PR add a user-facing library feature?

• No. It changes release infrastructure and workflows.
• Yes.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR adds release-engineering automation to make SynapseML-to-Fabric releases more deterministic and reviewable, including a GitHub Actions “prepare → tag merged commit → publish notes” flow plus supporting Python tooling to derive/verify release identifiers and safely bump BBC-VHD pins.

Changes:

• Introduces a release matrix generator (release_matrix.py) and an end-to-end verifier (verify_release.py) with accompanying tests.
• Adds bump_bbcvhd.py (plus tests) to safely update BBC-VHD component pins while preserving line endings and rolling back on failure.
• Adds/updates GitHub workflows for release prepare/tagging and manual, artifact-gated release notes; enables workflow_dispatch for PR validation so release PRs created via GITHUB_TOKEN can be validated.

Show a summary per file

File	Description
scripts/test_bump_version.py	Extends bump-version tests for denylisted repo-relative paths; normalizes path keys to POSIX.
scripts/bump-version.py	Improves Windows console encoding robustness; adds path-based denylist support; normalizes path handling via as_posix().
scripts/release/release_matrix.py	Adds a single-source-of-truth release matrix for tags and artifact versions + CLI rendering.
scripts/release/verify_release.py	Adds live verification of GitHub/ADO tags and Maven/PyPI/UPack/Azure Artifacts presence.
scripts/release/bump_bbcvhd.py	Adds deterministic, rollback-safe BBC-VHD component pin updater with CRLF/LF preservation.
scripts/release/README.md	Documents how to use the new release tooling and where it fits in the guide.
scripts/release/test_release_matrix.py	Adds regression/contract tests for the release matrix derivations and CLI validation.
scripts/release/test_verify_release.py	Adds unit tests for verifier networking/error handling and run plan behavior.
scripts/release/test_bump_bbcvhd.py	Adds tests for BBC-VHD bump idempotency, rollback, and newline preservation.
scripts/release/test_release_workflows.py	Adds tests asserting key workflow contract properties (manual gating, dispatch behavior).
scripts/release/test_prev_tag.sh	Adds a repo-taglist regression script for “previous primary tag” selection logic.
.github/workflows/release-prepare.yml	Adds a reviewed release PR generator and a merged-commit tagger that dispatches downstream orchestration.
.github/workflows/release-notes.yml	Adds a manual, artifact-gated GitHub Release publisher with explicit previous-tag selection.
.github/workflows/pr-validation.yml	Enables workflow_dispatch so validations can be dispatched for bot-opened release PR branches.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Suppressed comments (1)

scripts/release/verify_release.py:260

• --skip internal does not currently skip the internal PyPI feed checks (the synapseml_internal package); it only skips internal git tag checks. This makes --skip internal behave inconsistently compared with --skip public.

    def pip(self, package: str, version: str) -> str:
        if "pip" in self.skip or "ado" in self.skip:
            return SKIPPED
        # Azure Artifacts normalises pypi names: synapseml_internal -> synapseml-internal
        return (
            OK
            if version
            in self._feed_versions("Synapse-Conda", "pypi", package.replace("_", "-"))
            else MISSING
        )

• Files reviewed: 14/14 changed files
• Comments generated: 2
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/release/verify_release.py:75

• On Windows, subprocess.run(..., shell=True) expects a command string, but this code passes a list. That combination can fail (or behave differently) on win32, which would break the fallback ADO auth path even when az is installed. Build a platform-appropriate command (string for shell=True, list otherwise).

    out = subprocess.run(
        [
            "az",
            "account",
            "get-access-token",
            "--resource",
            ADO_RESOURCE,
            "--query",
            "accessToken",
            "-o",
            "tsv",
        ],
        capture_output=True,
        text=True,
        shell=(sys.platform == "win32"),
    )

• Files reviewed: 14/14 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Addressed the current-head suppressed Windows auth finding in 6e80f27.

• Windows now gives subprocess.run(..., shell=True) a command string; non-Windows retains an argument list with shell=False.
• Parameterized regression coverage locks both command shapes.
• A token-less Windows fallback replay authenticated through �z and found all 12 selected v1.1.3 master artifacts; the complete release-tooling suite is 279 passed.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/release/verify_release.py:313

• The Maven publication gate here verifies only synapseml-core_{scala} (via public_maven()), but the workflow-generated release notes advertise com.microsoft.azure:synapseml_2.12:${VERSION} (see .github/workflows/release-notes.yml:137) and the docs use the synapseml_2.12 coordinate (website/doctest.py:25). This can let verify_release.py report COMPLETE even if the primary install coordinate is missing (or vice versa). Consider aligning the verification to check the same Maven artifact(s) that users are instructed to consume (e.g., verify both synapseml_{scala} and synapseml-core_{scala}, or switch to the canonical one).

            "maven",
            tp.key,
            f"synapseml-core_{tp.scala}",
            tp.oss_maven_version,
            c.public_maven(tp.scala, tp.oss_maven_version),

• Files reviewed: 14/14 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🔵 Needs a closer look

It introduces and modifies high-impact release workflows and end-to-end publication gates across GitHub and Azure DevOps, which warrants final human review of the automation boundaries and operational safety.

Review tier: Lite
Findings: None

Issues resolved since last review (2)

Severity	Finding
	scripts/​release/​verify_release.py — GitHub API calls will not authenticate because the Authorization header is currently set to the… View resolved comment
	scripts/​release/​verify_release.py — ADO-backed checks will never authenticate because the Authorization header is currently the literal… View resolved comment

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/review

[Copilot]

Copilot review overview

🔵 Needs a closer look

It makes broad, high-impact changes to release workflows/policy gates and publication tooling that warrant final human validation beyond automated review.

Review tier: Lite
Findings: 1 · 1

New issues introduced by this change (2)

Severity	Finding
	project/​build.scala — refuseExistingReleaseBlob() uses Process(command).!!, which throws on any Azure CLI error…
	tools/​esrp/​prepare_jar.py — collect_artifacts() calls source.resolve() inside the per-module loop and again inside the…

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

[Rana Singh (ranadeepsingh)]

/review

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🔵 Needs a closer look

It changes release-critical CI/workflows and publication/immutability guards where mistakes can directly impact tagging and package publication safety.

Review tier: Lite
Findings: None

Issues resolved since last review (2)

Severity	Finding
	tools/​esrp/​prepare_jar.py — collect_artifacts() calls source.resolve() inside the per-module loop and again inside the… View resolved comment
	project/​build.scala — refuseExistingReleaseBlob() uses Process(command).!!, which throws on any Azure CLI error… View resolved comment

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.48%. Comparing base (8c71438) to head (4c166c4).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2628      +/-   ##
==========================================
- Coverage   87.52%   87.48%   -0.04%     
==========================================
  Files         341      341              
  Lines       21035    21035              
  Branches     2232     2232              
==========================================
- Hits        18410    18403       -7     
- Misses       2625     2632       +7     

see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.