Skip to content

Private PJNZ archive: classifier + Azure infra (issue 16) - #386

Open
r-ash wants to merge 1 commit into
mainfrom
validator-4
Open

Private PJNZ archive: classifier + Azure infra (issue 16)#386
r-ash wants to merge 1 commit into
mainfrom
validator-4

Conversation

@r-ash

@r-ash r-ash commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Completes ticket 16 (.scratch/leapfrog-validation/issues/16-private-pjnz-corpus-hosting-tagging.md) of the leapfrog-validation wayfinder map: the PJNZ classifier plus the Azure infra it now runs against.

  • leapfrog_validate/classify.pyshape_tags (cheap zip-content peek: goals/aim_only from .HV member presence) and domain_tags (imports via leapfrog::process_pjnz(), checks whether the PMTCT/cotrimoxazole input arrays are all-zero)
  • leapfrog_validate/manifest.py — small JSON manifest for tags that aren't derivable from a PJNZ's contents at all (provenance/purpose), per ticket 09
  • leapfrog_validate/r_scripts/classify_pjnz.R — new Rscript backing domain_tags
  • New leapfrog-validate classify <ref> <pjnz> [--manifest] CLI command
  • leapfrog-validate/infra/ — Bicep for a private Azure Blob container + a GitHub-OIDC-federated CI reader identity (no static secret), replacing the git-LFS-submodule mechanism ticket 09 originally chose (GitHub LFS bandwidth quota risk on this org's actual plan — see ticket 16's Comments for the full amendment trail)

Notable finding recorded on the ticket: age-stratification turned out not to be a PJNZ-intrinsic property — process_pjnz() takes use_coarse_age_groups as a caller-supplied argument, not something read from the file — so it's dropped from shape tags rather than faked. Also discovered a pre-existing leapfrog::process_pjnz() limitation on Goals-enabled PJNZ (process_pjnz_ha errors on SouthAfrica.PJNZ), unrelated to this ticket; domain_tags surfaces it as ClassifyError and a test locks down that it fails loudly rather than guessing.

A code review pass (medium effort) found and fixed several real bugs before this was pushed: a silent has_cotrim=NA-gets-dropped bug (cotrim wasn't NA-scrubbed the way PMTCT is upstream), a manifest entry that could silently explode a bare string into per-character tags, a misleading fork-PR safety claim in the infra README (GitHub's OIDC subject claim doesn't actually distinguish fork PRs — corrected to point at the real boundary), allowSharedKeyAccess left open on the storage account alongside the OIDC/RBAC setup, and deploy.sh hardcoding the repo instead of using the actual deployed values. All covered by new regression tests.

Test plan

  • ruff check . clean
  • Full test suite: 68 passing (59 fast + 9 R-gated integration tests against real, non-sensitive PJNZ fixtures already in this repo)
  • Empirically verified domain_tags against all 3 real AIM-only fixtures (france_default, both bwa_aim-*) and confirmed the Goals-fixture failure mode
  • leapfrog-validate/infra/'s Bicep not yet compile-checked against a live Azure subscription from this sandbox (no az/bicep CLI here) — was reviewed manually and the role/resource types verified against Microsoft's docs; the Azure resources it describes are already deployed and the real corpus already uploaded (per ticket 16's comments)

🤖 Generated with Claude Code

Adds the shape/domain-tag classifier and manifest support ticket 16 asks
for, plus the Azure Blob infra it now runs against instead of a git-LFS
submodule (GitHub LFS bandwidth quota risk on this org's plan made that
mechanism a bad fit -- see ticket 16's comments).

- classify.shape_tags: zip-content peek, goals/aim_only from .HV presence
- classify.domain_tags: leapfrog::process_pjnz() import + all-zero check
  for has_pmtct/has_cotrim, via new r_scripts/classify_pjnz.R
- manifest.py: small JSON manifest for tags that aren't derivable at all
  (provenance/purpose), per ticket 09
- new `leapfrog-validate classify` CLI command
- leapfrog-validate/infra/: Bicep for the private container + a
  GitHub-OIDC-federated CI reader identity, no static secret

Age-stratification dropped from shape tags after finding it's a
process_pjnz() runtime choice, not a PJNZ-intrinsic property -- see
ticket 16's comments for the full amendment trail.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant