[prateeksinghalgit]

Implements Azure Blob Storage backup repository as discussed in SOLR-17949.

Description

Adds a new BackupRepository implementation for Azure Blob Storage, enabling Solr collections to be backed up to and restored from Azure. The module follows the structure and patterns of the existing S3 and GCS repository modules.

Highlights

• Full and incremental backup/restore, including the Install Shard API.
• Four authentication methods: connection string, account key, SAS token, and Azure Identity (Managed Identity / Service Principal / Azure CLI).
• Checksum verification on index-file copy, consistent with the S3/GCS modules.
• Uses azure-core-http-okhttp instead of the default Netty transport, for Security Manager compatibility and a smaller dependency footprint.
• Secrets can be supplied via system properties / environment variables instead of solr.xml (documented in the Reference Guide).

Main classes

• AzureBlobBackupRepository — the BackupRepository implementation.
• AzureBlobStorageClient — Azure SDK wrapper.
• AzureBlobIndexInput / AzureBlobOutputStream — Lucene BufferedIndexInput and block-blob upload stream.

All dependencies are Apache 2.0 or MIT licensed (Azure SDK and msal4j are MIT).

Tests

The module runs the same shared SolrCloud test suites used by the S3 and GCS modules, backed by an Azurite emulator via Testcontainers (tests skip cleanly when Docker is unavailable):

• AbstractBackupRepositoryTest — repository contract compliance, exercised through the real init() / configuration path.
• AbstractIncrementalBackupTest — full incremental backup/restore against a SolrCloud cluster.
• AbstractInstallShardTest — Install Shard API flows.

Plus module-specific tests for the storage client, output stream (block staging/commit, failure handling), index input (slices, clones, connection-loss resume via ResumableInputStream), and path handling.

Connection-string authentication is exercised in CI against Azurite. Account key, SAS token, and Azure Identity (Service Principal) were verified manually against a real Azure Blob Storage account.

Testing Instructions:
Can be tested locally with Azurite emulator (no Azure account needed) or with real Azure Blob Storage. See solr/modules/azure-blob-repository/README.md for detailed setup instructions.

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
• I have developed this patch against the main branch.
• I have run ./gradlew :solr:modules:azure-blob-repository:check (module-specific check passed).
• I have added tests for my changes.
• I have added documentation for the Reference Guide

[pratsgit]

Quick update: following up on the dev@ thread.

To clarify scope — although the diff is large, the actual implementation is centered in 8 main files and 8 test files; the rest are license header updates.

Happy to split the PR into smaller logical chunks (core module / tests / docs) if that helps with review.

Thanks everyone!

[janhoy]

Don’t have time to review but asked copilot for an opinion 😉

[janhoy]

Only immediate comment is naming. «Blob-repository» is too generic. Should contain word «azure»?

[Copilot]

Pull Request Overview

This PR implements Azure Blob Storage backup repository support for Apache Solr, enabling collections to be backed up to and restored from Microsoft Azure. The implementation follows established patterns from existing S3 and GCS modules, providing 4 authentication methods (Connection String, Account Key, SAS Token, Azure Identity), incremental backup support, and comprehensive documentation with 76 passing unit tests.

Reviewed Changes

Copilot reviewed 70 out of 70 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
backup-restore.adoc	Adds comprehensive documentation for BlobBackupRepository with authentication methods and configuration
BlobBackupRepository.java	Main repository implementation following Solr's BackupRepository interface
BlobStorageClient.java	Azure SDK wrapper providing blob storage operations
BlobOutputStream.java	Custom output stream for block-based blob uploads
BlobIndexInput.java	Lucene IndexInput implementation with page caching
Test files	8 test classes with 76 passing tests covering all functionality
build.gradle	Module build configuration with Azure SDK dependencies
License files	License and notice files for Azure and Netty dependencies

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[prateeksinghalgit]

Only immediate comment is naming. «Blob-repository» is too generic. Should contain word «azure»?

made the change to azure-blob-repository

[janhoy]

I'll leave the rest of the review to others more proficient in Azure Blob than me (have never used it). I'd love to test it with a read AzBlob but I'll leave that to someone who already have an active account.

I have a concern that the PR feels largely AI generated(?), lacking the care to details that we require for contributions. Have me excused @prateeksinghalgit if this is not correct, it was just a hunch I got while reviewing. I'd rather have a short well thought out README with useful advice for users than three pages of detailed step by step instructions for testing and developing the feature.

For other reviewers to pick up where I left, consider in partifular the HTTP client choice, correct licensing and avoiding hard coded ports in tests. I have not done a complete review, not read the ref-guide part at all, just commented on things me and Copilot saw immediately.

[prateeksinghalgit]

@janhoy could you please approve workflow runs on this fork PR again. I had to make updates to the pr. Also I did run the ci jobs locally with all the flags to make sure .

[prateeksinghalgit]

@janhoy could you please approve the workflow runs on this PR again? The previous CI failure was because my branch hadn't merged the latest main

[prateeksinghalgit]

@janhoy could you please trigger a CI re-run, seems like a unrelated flaky test in the CI run. I ran CI locally, it did run successfully.

[janhoy]

Skimmed through and found some questions and suggestsions.

[prateeksinghalgit]

@janhoy thank you for the review. I have addressed all review feedback.

[janhoy]

Thanks for the latest iteration, to me this starts to look quite mature. But as said earlier I'll defer to someone who have an actual AZ environment at hand to test it and do the last blessing.

[prateeksinghalgit]

Thanks @janhoy. I've addressed all your points (dependency trim, lockfile cleanup, README, solr.xml, and the verified security.policy example). If you are ok with the updates could you please clear the "changes requested" status so it's not blocking the final reviewer.

[prateeksinghalgit]

To make the final review easier: testing doesn't actually require an Azure account — the integration tests run against the Azurite emulator (Docker, free)- https://hub.docker.com/r/microsoft/azure-storage-azurite. I've separately verified backup/restore against a real Azure Storage account, including under the Java Security Manager.

[github-actions]

This PR has had no activity for 60 days and is now labeled as stale. Any new activity will remove the stale label. To attract more reviewers, please tag people who might be familiar with the code area and/or notify the [email protected] mailing list. To exempt this PR from being marked as stale, make it a draft PR or add the label "exempt-stale". If left unattended, this PR will be closed after another 60 days of inactivity. Thank you for your contribution!

[prateeksinghalgit]

@janhoy, any pointers on who in the Solr community might have access to an Azure environment and could help with the final review/testing of this PR? Thanks!

[elangelo]

Heads-up from the S3 side: this module copies the directory-marker design of the S3 repository, which turned out to break restores of copied backups. I'm fixing that for S3 in #5128.

The problem:

• createDirectory() (and pushStream() for missing parents) writes a zero-byte hdi_isfolder=true blob for every directory.
• pathExists() only checks for that exact blob, so a directory without its marker "doesn't exist". The javadoc already mentions this for azcopy.
• Core relies on exists() for directories during restore. Examples are the backup path in BackupManager.forRestore(), the configset dir in BackupManager.uploadConfigDir(), and the location check in BackupAPIBase. A backup that was copied to another container or account without its marker blobs therefore can't be restored. As far as I know, azcopy skips these directory stubs unless you pass --include-directory-stub.

isDirectory() already falls back to a prefix listing; pathExists() doesn't. In #5128 I went further and dropped the markers entirely:

• createDirectory() is a no-op.
• exists() on a directory path and isDirectory() are a single prefix listing (max 1 result). That also finds old marker blobs, so existing backups keep working, and deleteDirectory() cleans them up.
• Writes no longer check or create parent directories, which saves a request or more per uploaded file.
• To run the shared contract tests without empty directories, I added a supportsEmptyDirectories() hook to AbstractBackupRepositoryTest.

It may be worth doing the same here, or at least making pathExists() fall back to a prefix listing for directory paths. Hierarchical-namespace (ADLS Gen2) accounts have real directories and probably aren't affected.