Multi-Content Markdown files (like YAML's multi-documents)

[ek9r]

Description

I'd like something similar to what the YAML world calls "multi-documents": being able to keep several independent documents, each with their own frontmatter, inside a single Markdown file.

The motivating case: a file describing several related things (e.g. a handful of small records that reference each other) where splitting into one-file-per-record feels premature for small sets, but a single growing file becomes messy without any internal structure.

Rough idea of how I'd approach it, happy to be told this is a bad fit:

---
type: application
name: storefront
status: up
composed_of: [web-service]
---
The customer-facing storefront application.
Handles browsing, checkout, and order history for end users.

<!-- next: web-1 -->
---
type: host
name: web-1
status: up
---
Primary application server, running in the eu-central region.
Serves all incoming HTTP traffic for the storefront.

<!-- next: web-service -->
---
type: service
name: web-service
runs_on: [web-1]
---
Handles incoming HTTP traffic and renders the storefront pages.
Depends on the primary database service for product and order data.

$chunks = $markdown->parseMany($content);
$chunks[0]->frontmatter['type'];    // application
$chunks[2]->frontmatter['runs_on']; // ['web-1']

• parseMany() alongside the existing parse(), additive only
• a <!-- next --> / <!-- next: name --> marker as the document separator, recognized before tokenization (so each document still gets its own frontmatter parsed from position 0)
• ParsedMarkdown gains an optional $name, defaulting to null

I've got a rough, tested implementation on a branch here, mostly to validate the idea works end-to-end rather than as a finished PR: https://github.com/ek9r/markdown_dev/tree/Multi-Content

[brendt]

Sounds like a very interesting idea. A couple of things worth discussing:

• Separator syntax, is there any precedence in other Markdown parsers that you know of?
• API: not sure yet about parseMany/parse, but this is something we'll figure out. In your experimentation, did you consider other ways of implementing it?

[ek9r]

Not that I know of in the PHP world, though from my old Perl days, Mojolicious' Mojo::Loader does something similar with named @@ sections in a __DATA__ block. That's roughly what inspired the idea.

My thinking on the separators was that the simplest case shouldn't need frontmatter at all:

First document.
<!-- next -->
Second document.

while a mixed file, with frontmatter optional per document, should work just as naturally:

---
title: A
---
First document, with frontmatter.
<!-- next -->
Second document, no frontmatter needed.

--- is already a separator of sorts, but it's already used for multiple purposes here (frontmatter boundary and thematic break). A third meaning wouldn't just fail, it would silently render as the wrong element. @@ is already reserved for raw blocks (RawRule); I confirmed reusing it breaks an existing raw block. <!-- next --> fits well precisely because it's a silent companion if it ever ends up unprocessed in the HTML output, rather than visible garbage or an incorrect HTML element.

On the API side, I've so far only explored the explicit opt-in path. parse() and parseMany() are two separate methods to avoid any risk of breaking existing code. The one thing I did revisit was upgrading the return value from a plain array to a small collection object, mirroring TokenCollection, which already exists in this package. parseMany() now returns it directly, just as parse() already returns ParsedMarkdown directly (see the latest commit on my branch).

[kchaot]

Forking a FlatFile blog out of a Markdown parser. That's delightfully nerdy. And I love the hidden poem. 😉