da_build: check PDF templates for PDF/UA-1 accessibility - #95
nonprofittechy wants to merge 6 commits into
Conversation
Runs veraPDF over the PDF templates users receive and reports failures as GitHub annotations plus a job summary, with per-rule severity classification so metadata-only findings do not read like real blockers. pdf-validation-mode: off|warning|error mirrors docx-validation-mode exactly. It defaults to warning, so adopting the check annotates a build rather than breaking it. pdf-strict keeps tab-order and form-annotation rules off by default, since AL forms are usually flattened before users see them. Only docassemble/*/data/templates is scanned. An earlier rglob over the whole repository swept in PDFs that are not output documents -- reference material in data/static, ALKiln fixtures in data/sources, and other packages' templates from a local .venv -- and reported them as this repo's failures. Over the local AL checkouts that narrows the scanned set from 272 to 96. Rebased onto main so it keeps the uv migration and the env-var input handling rather than reverting to the old pip-based steps.
There was a problem hiding this comment.
🟡 Changes recommended
The new checker has a reliability bug (unhandled veraPDF timeout) and a user-facing configuration mismatch in the job summary text.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a new PDF accessibility validation step to the da_build composite action by running veraPDF (PDF/UA-1) against Assembly Line PDF templates and reporting results via GitHub annotations and the job summary, with severity bucketing and an optional strict mode for form/tab-order rules.
Changes:
- Introduces
check_pdf_accessibility.pyto locate template PDFs, run veraPDF, parse XML output, classify rule severities, and emit annotations + a Markdown job summary. - Extends
da_build/action.ymlwithpdf-validation-modeandpdf-strictinputs, installs veraPDF when enabled, and runs the new checker.
File summaries
| File | Description |
|---|---|
da_build/check_pdf_accessibility.py |
Implements PDF discovery, veraPDF execution, XML parsing, severity classification, annotations, and job summary rendering. |
da_build/action.yml |
Adds inputs and workflow steps to install veraPDF and run the PDF accessibility checker (with opt-out). |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Working on resolving conflicts; will mark ready soon |
…mplates # Conflicts: # da_build/action.yml
There was a problem hiding this comment.
🟡 Changes recommended
There are correctness and security gaps in the veraPDF integration (version enforcement/exit-code handling and unverified installer download) that should be addressed before relying on this in CI.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 3
- Review effort level: Lite
VerdictThe findings are valid as PDF/UA-1 conformance findings, but the corpus shows they are not yet a practical pass/fail
The strongest findings are genuine accessibility defects:
The warning classifications are sensible, but language checks are noisy: missing language metadata affected 53 PDFs Strict mode made no difference to pass/fail status for this corpus because every failing file already had an Scope assessmentThe path filter in da_build/check_pdf_accessibility.py:188 is mostly correct. Of 126 non-hidden, non-virtualenv PDFs There are still likely scope-noise candidates inside data/templates, especially:
These are plausibly test fixtures rather than user-facing templates and may warrant relocation or an exclusion Recommended follow-ups
No repository files were changed; the existing untracked URL report was preserved. |
Expanded Massachusetts corpus analysisI reran the PDF checker against the published Massachusetts interview corpus represented by the Court Forms Online snapshot.
This is a more targeted corpus than the earlier all-local scan: it covers the PDF templates associated with currently published Massachusetts interviews. Dominant blocking findings
The checker continues to treat the observed unmapped rules 7.21.7.2 and 7.21.8.1 as failures through the fail-conservative fallback. Remediation thresholdsFor this calculation, a “fix” means eliminating one failure family across the affected templates. A PDF counts as passing only when all of its blocker families have been fixed.
The four-family set is:
For the 50% threshold, add:
Mechanical-fix-only resultIf we deliberately avoid adding a bare StructTreeRoot and only perform the mechanical/content decisions—field TU labels, font embedding, ToUnicode mappings, and correct Artifact/Figure treatment—only three currently failing PDFs have no remaining structural/table/annotation blockers:
Together with the one PDF that already passes, that would produce 4/54 passing (7.4%). To reach a strict 10% threshold, five additional PDFs would be needed. Image recognition could be useful for triage and drafting classifications/alt text, but it should not make unreviewed accessibility decisions. The model should distinguish decorative content from meaningful images in context, and generated alt text should be reviewed for accuracy. veraPDF can check that structural entries exist, but it cannot determine whether an image was incorrectly marked decorative or whether its alt text is substantively correct. The full expanded-corpus report is available locally at:
|
Runs veraPDF over the PDF templates users receive and reports failures as GitHub annotations plus a job summary, with per-rule severity classification so metadata-only findings do not read like real blockers.
Input, matching the DOCX one exactly
pdf-strictkeeps tab-order and form-annotation rules off by default, since AL forms are usually flattened before users see them.Naming the format rather than the tool (
pdf-notverapdf-) keeps the input stable if veraPDF is ever swapped out, and makes it read the same asdocx-validation-modein #94.Scoped to templates
Only
docassemble/*/data/templatesis scanned. An earlierrglobover the whole repository swept in PDFs that are not output documents — reference material indata/static, ALKiln fixtures indata/sources, and other packages' templates from a local.venv— and reported them as this repo's failures. Over the local AL checkouts that narrows the scanned set from 272 PDFs to 96.Note on history
This supersedes
add-pac-to-build, which was 24 commits behind and predated the uv migration — merging it would have reverted uv and the env-var input handling that fixed the command-injection issue. Rebuilt on current main.