ATLAS-5317: Resilient bulk purge with pre-validation, worker batches,…

[sheetalshah1007]

… partial-success responses, and runId-correlated admin audit APIs

Fix addEntity() duplicate check in EntityMutationResponse.

Attachments

1. Design specification:ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-DESIGN.pdf`
2. Implementation notes:
   ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-IMPLEMENTATION_NOTES.md

Summary

This PR implements resilient bulk purge according to the Resilient Bulk Purge in Atlas Using Transaction Batching design spec. It replaces the old, single-transaction REST bulk purge with a pre-validated, worker-batch processing model that is now shared by both REST and scheduled (cron) purges.

---

What changes were proposed in this pull request?

Key changes

• Pre-validation — REST GUIDs checked before writes (format, existence, DELETED state, type). Invalid GUIDs return in failedEntities.
• Request limit — atlas.purge.api.max.request.size (default 1000).
• Shared worker path — REST and cron use PurgeBatchOrchestrator / WorkItemManager (atlas.purge.worker.batch.size=100, atlas.purge.workers.count=2).
• Batch isolation — Each batch is its own transaction. Lock conflicts retry at batch level (3 attempts, 500/1000/1500 ms). Other batch failures do not roll back committed batches.
• Partial success — Committed batches stay committed when a later batch fails.
• Enriched response — failedEntities + PurgeSummary (runId, dependency counts, executionFailed).
• EntityMutationResponse bugfix — Replace O(n) list scans per addEntity() with an O(1) GUID Set per operation; rebuild on setMutatedEntities().
• Run-correlated audit — Each run writes one SUMMARY row, BATCH rows per worker batch, and failures to purgefailure.log, linked by indexed runId and auditRowKind.
• Admin audit APIs — /purgedEntities, /batches, and /details drill-down on summary and batch rows.
• Cron alignment — Same orchestration as REST; no monolithic AUTO_PURGE audit row.
• Parallel safety — accumulateDeletionCandidates() returns GUIDs; DeleteHandlerV1 skips already-removed vertices (ATLAS-4766).
• Schema — AUTO_PURGE, runId, and audit_row_kind on __AtlasAuditEntry (typedef patches 009/010).

Compatibility notes

• accumulateDeletionCandidates() returns Set<String> (not Set<AtlasVertex>).
• Ineligible GUIDs appear in failedEntities, not silently dropped.
• Completed purges return HTTP 200 with outcome in the body (400 for empty input, size limit, auth, bad GUID length).
• REST and cron can overlap; cron-to-cron runs are serialized.

See ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-IMPLEMENTATION_NOTES.md for flow, config, audit details, and design deviations.

---

How was this patch tested?

Area	Tests
End-to-end / cron audit	PurgeTest
Workers / retry	PurgeBatchOrchestratorTest, PurgeBatchExecutorTest
Graph	AtlasEntityStoreV2Test, DeleteHandlerV1Test
REST	AdminResourceTest
Client	AtlasClientV2Test

---

Performance testing

Load: 1,000 hive_table entities (2 columns each) — 5,000 entities total (tables + columns + storage-desc + ddl ).

Config	Avg purge_ms	Time	vs pre-5317
Pre-5317 (single transaction)	273,318	4m 33s	baseline
Post-5317, workers.count=2	~58,541	~59s	~79% faster (~4.7×)
Post-5317, workers.count=1	~109,471	~1m 49s	~60% faster (~2.5×)

Regarding atlas.purge.workers.count

Default = 2 — Cron purge historically used 2 workers; the new design keeps that. One batch can commit while another runs — solid speed-up without overwhelming the database.

Setting	Behavior	When to use
2 (default)	Two parallel batches	Regular large purges; best balance of speed and load on this test
1	Sequential (one batch at a time)	Prefer legacy-like behavior; still ~60% faster than pre-5317
> 2 (e.g. 5)	More parallel batches	Only after cluster testing — may not scale linearly