Skip to content

Option to validate additional standalone markdown files through the extras pipeline (e.g. validate_additional_docs) #2272

Closed

Description

@lukegalea

Context: usage_rules validates usage-rules markdown files (AGENTS.md-style docs embedded in projects) for broken docs references (ash-project/usage_rules#88). To avoid reimplementing reference resolution, we drive ex_doc's own pipeline over those files, which works well today with minimal glue.

What already exists (verified against 0.40.3):

  • ExDoc.Extras.build/2 parses arbitrary .md files into %ExDoc.ExtraNode{}
  • ExDoc.Formatter.autolink/5 runs the autolink phase over extras and emits warnings via ExDoc.warn/2 (file/line, native format)
  • ExDoc.warn/2 sets the {ExDoc, :warned?} persistent_term flag; ExDoc.generate_docs/4 already returns warned?: per formatter and the CLI exits nonzero under warnings_as_errors: true

So a config option roughly like:

validate_additional_docs: [AGENTS.md, skills/**/*.md]

would mostly be plumbing: accept the paths, feed them through ExDoc.Extras.build alongside extras, and let the existing warning and exit semantics apply. Our out-of-tree glue for this is ~15 lines (Extras.build + Formatter.autolink + reading the warned flag), so ex_doc-side support would be small.

Native gaps we noticed while doing this — candidates for a stricter mode in a validation context, listed only as observations:

  • bare undefined module mentions in code spans (`NoSuch.Module.Here`) are silent, since spans autolink in :regular_link mode
  • mix task mentions in plain code spans are silent; only explicit [x](mix task) links warn
  • fenced code blocks are skipped by walk_doc/2
  • warnings emitted from {:a, ...} nodes carry no line number for extras (line is only propagated from inline code span meta)

Happy to hear whether this fits ex_doc's roadmap; happy to help test or benchmark against our use case.

Activity

  1. josevalim commented on Sep 21, 2026

    @josevalim
    Member

    Thank you @lukegalea. Why not have a flag such as EXTRA_DOCS and then you do:

    extras: [...usual values...] ++ if(System.get_env("EXTRA_DOCS"), do: ["AGENTS.md", ...], else: [])
    

    It could be neatly encapsulated in a function too.

  2. lukegalea commented on Sep 22, 2026

    @lukegalea
    Author

    Closing this — prototyped your suggestion today and it works great, so no ex_doc change needed. Thank you!

    For anyone finding this later, the encapsulated version we landed on:

    defp docs do
      [
        # ...
        extras:
          [
            {"README.md", title: "Home"},
            "CHANGELOG.md"
          ] ++ extra_docs()
      ]
    end
    
    # EXTRA_DOCS=1 mix docs routes standalone agent docs (AGENTS.md-style
    # files) through the same extras pipeline, so broken refs warn like any
    # other doc. In CI we run it with warnings_as_errors: true.
    defp extra_docs do
      if glob = System.get_env("EXTRA_DOCS"), do: Path.wildcard(glob), else: []
    end

    With a file containing planted breakages, EXTRA_DOCS=1 mix docs reports:

    warning: documentation references module "UsageRules.NoSuchModule" but it is undefined
    └─ EXTRA_DOCS_PROTOTYPE.md: (file)
    warning: documentation references function "UsageRules.Validator.validate/9" but it is undefined or private
    └─ EXTRA_DOCS_PROTOTYPE.md: (file)
    warning: documentation references "mix no_such.task" but it is undefined
    └─ EXTRA_DOCS_PROTOTYPE.md: (file)
    

    and with warnings_as_errors: true exits 1 (generation for html, epub formats failed due to warnings while using the --warnings-as-errors option). Without EXTRA_DOCS set, mix docs is unchanged.

    Two observations from the run, both consistent with how moduledocs behave, so we'd treat them as fine rather than blockers:

    • bare undefined module mentions in plain code spans (`NoSuch.Module.Here`) stay silent — only explicit [x](`Mod`)-style links warn, as in moduledocs
    • extras warnings carry the file but no line number

    This covers our CI use case (rotting refs in agent-facing markdown break the build). Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions