Skip to content

Add a reflection back end - #278

Draft
AaronWebster wants to merge 4 commits into
masterfrom
emboss/reflection-backend
Draft

Add a reflection back end#278
AaronWebster wants to merge 4 commits into
masterfrom
emboss/reflection-backend

Conversation

@AaronWebster

Copy link
Copy Markdown
Collaborator

Adds a reflection back end: embossc --generate reflection foo.emb writes a JSON
description of the module's types instead of C++ code — every field with its bit
offset and bit size, every enum member with its value, every let value, plus
documentation, requires clauses, existence conditions, byte order, and runtime
parameters.

The point is to let a program consume an .emb as data. The case that motivated
it: a host tool that has to speak two revisions of one FPGA register interface at
once cannot #include both generated headers, because the two declare the same
fully-qualified names. Reading the registers as data sidesteps the collision
entirely, and the same output serves documentation generators and register-map
catalogs.

No new attributes, so no .emb needs an expected_back_ends update and the
front end is untouched.

Commits

  1. Make embossc honor --generate, and add an ir back end. --generate
    was declared with choices=["cc"] but main() never read it — the C++ back
    end was imported and run unconditionally. Restructured around
    _generate_code_and_log_errors() / _default_file_suffix() / _format_code()
    so adding a back end is a matter of extending three match statements, and
    dropped the nargs=1 that made --generate/--output-path/--output-file
    lists while their defaults were bare strings. --generate ir writes the
    compiler's own serialized IR, byte-identical to the front end's --output-file.
  2. Add compiler/util/expression_printer. Renders an ir_data.Expression
    back to Emboss source text, the inverse of expression_parser.parse. The only
    renderer in the tree today is header_generator's, which emits C++ helper
    names (Sum, LessThanOrEqual) rather than infix.
  3. Add the reflection back end — generator, CLI, tests, Bazel rules, goldens.
  4. Report structure size bounds, so a dynamically-sized structure reports
    min_bit_size/max_bit_size rather than nothing at all.

Notes on a few decisions

  • Offsets are reported relative to the type a field is declared in. Anonymous
    bits blocks are the exception — they have no name for an offset to be
    relative to — so their fields are lifted into the enclosing type at absolute
    bit offsets, which is where the compiler's own alias virtuals already point.
  • A field whose placement is not a compile-time constant reports a null offset
    or size plus the Emboss expression that determines it, and, where the front end
    knows one, the modulus it is aligned to.
  • Mixed byte order within one structure is legal Emboss, so a structure whose
    fields disagree reports a null byte_order rather than failing. Every field
    reports its own order regardless, so nothing is lost.
  • is_signed comes from the enum's is_signed attribute for enum-typed fields
    and from the expression's computed bounds for let fields, rather than from a
    type-name test, which would miss both.
  • The compiler's own virtuals — the $-prefixed size fields and the alias fields
    synthesized for anonymous bits members — are not reported.

Testing

  • New unit tests: //compiler/util:expression_printer_test (round-trips through
    expression_parser.parse), //compiler/back_end/reflection:reflection_generator_test,
    //:embossc_test.
  • Nine goldens under testdata/golden_reflection/, with a REFLECTION_GOLDENS
    list in scripts/regenerate_goldens.py.
  • scripts/regenerate_goldens.py reproduces all 34 existing C++ goldens
    byte-identically, which is what establishes commit 1 as behavior-preserving.
  • Checked against a real set of 24 FPGA register maps by diffing this back end
    against the ad-hoc IR walker it was seeded from: 398 types / 991 fields agree
    exactly on bit_offset, bit_size, enum_ref, is_array and documentation;
    every remaining difference is one of the intended fixes (named sub-structs no
    longer inlined away, synthetic aliases de-duplicated, dynamic placement
    reported rather than dropped).

Draft: opening it for CI. Happy to split commit 1 out into its own PR if that
reads better.

`--generate` was declared with `choices=["cc"]` but `main()` never read it:
the C++ back end was imported and run unconditionally.  Restructure the
driver around `_generate_code_and_log_errors()`, `_default_file_suffix()`
and `_format_code()` so that adding a back end is a matter of extending
three `match` statements.

Add `--generate ir`, which writes the compiler's own serialized IR as JSON.
That is the format back ends already consume -- the front end writes it via
`--output-file`, the `emboss_library` Bazel rule produces `<src>.emb.ir`
with it, and `emboss_codegen_cpp` reads it -- but `embossc` had no spelling
for it, so anything wanting IR had to import `compiler.front_end.glue`
directly.  The bytes are identical to the front end's own output.

Also drop `nargs=1` from `--generate`, `--output-path` and `--output-file`.
All three were indexed as lists while defaulting to bare strings; the
`--output-path` default survived only because `"."[0] == "."`.

Verified with scripts/regenerate_goldens.py: all 34 C++ goldens regenerate
byte-identically.
Renders an ir_data.Expression back to Emboss source text, the inverse of
expression_parser.parse.  The only expression renderer in the tree today is
header_generator's, which emits C++ helper names (Sum, LessThanOrEqual)
rather than infix, and format_emb.py works on the parse tree rather than on
ir_data.Expression.  A back end that reports expressions to a human -- a
`requires` clause in documentation, a non-constant field offset in reflection
output -- needs the Emboss spelling.

Parentheses are emitted only where the grammar needs them, which is a little
subtler than a precedence table:

  * `&&` and `||` are grammar siblings, not tiers, so mixing them without
    parentheses does not parse.  They get equal precedence plus a rule that
    parenthesizes a boolean child of a different boolean parent.
  * `?:` takes logical-expressions, so a nested `?:` is always parenthesized.
  * Comparisons do not re-associate, so a comparison inside a comparison is
    always parenthesized.
  * Unary minus parses to a binary SUBTRACTION against a synthesized
    zero-width `0`; that form is detected and printed back as `-x`.
`compiler/back_end/reflection` emits a JSON description of a module's types
instead of code: every field with its bit offset and bit size, every enum
member with its value, every `let` value, plus documentation, `requires`
clauses, existence conditions, byte order, and runtime parameters.

The point is to let a program consume an `.emb` as data.  The case that
motivated it: a host tool that has to speak two revisions of one FPGA register
interface at once cannot `#include` both generated headers, because the two
declare the same fully-qualified names.  Reading the registers as data sidesteps
the collision entirely, and the same output serves documentation generators and
register-map catalogs.

Offsets are reported relative to the type a field is declared in.  Anonymous
`bits` blocks are the exception -- they have no name for an offset to be
relative to -- so their fields are lifted into the enclosing type at absolute
bit offsets, which is where the compiler's own alias virtuals already point.
A field whose placement is not a compile-time constant reports a null offset or
size plus the Emboss expression that determines it, and, where the front end
knows one, the modulus it is aligned to.

Notes on a few decisions:

  * Mixed byte order within one structure is legal Emboss, so a structure whose
    fields disagree reports a null `byte_order` rather than failing.  Every
    field reports its own order regardless, so nothing is lost.
  * `is_signed` comes from the enum's `is_signed` attribute for enum-typed
    fields and from the expression's computed bounds for `let` fields, rather
    than from a type-name test, which would miss both.
  * The compiler's own virtuals -- the `$`-prefixed size fields and the alias
    fields it synthesizes for anonymous `bits` members -- are not reported.

No new attributes, so no `.emb` needs an `expected_back_ends` update and the
front end is untouched.

Also adds `embossc --generate reflection` (output suffix `.json`), the
`emboss_reflection_library` Bazel macro, goldens under
`testdata/golden_reflection/`, and their entries in
`scripts/regenerate_goldens.py`.
A dynamically-sized structure has no one size, so `bit_size` is null for it --
which left a consumer with nothing at all.  The compiler already knows the
range: it computes bounds on the `$size_in_bytes` virtual whether or not the
value folds to a constant.  Report them as `min_bit_size` / `max_bit_size`, and
null out a bound the compiler could not establish.

Found by diffing this back end against the walker it was seeded from, over a
real set of register maps: the seed reported a size for six structures where
this one reported none.  In one case the size was genuinely fixed and the
bounds now pin it; in the other five the seed was quietly reporting the offset
of the last fixed-position field, having dropped the variable-length tail.
@jasongraffius

Copy link
Copy Markdown
Collaborator

Potentially bikeshed-y, but could we consider a different name for the backend, like descriptor or just json? The backend currently doesn't enable any reflection in the common use of the term.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants