Nuklai Nexus Compact Metadata Standard (CMeta)

CMeta is a compact metadata format for describing data lakes, tables, columns, and how tables join, in a way that is:

• 🪶 Token-efficient: compresses schema/metadata for LLMs with narrow context windows
• 🧑‍💻 Human-readable: easy to read and edit manually
• 🔄 Reversible: can be converted to and from JSON (compact or verbose)

This package provides Python utilities to:

• Parse CMeta text ⇄ JSON
• Convert to/from a compact JSON structure
• Convert to/from a flat, extended JSON structure (very verbose for some specific Nexus use cases)
• Normalize the type names a data source reports (utf8, int64, ...) to SQL ones

---

✨ Installation

uv add nuklai-cmeta

Uses uv. You can also install with pip install nuklai-cmeta if you prefer.

---

📐 CMeta v1.1 Format

• Hierarchy:

  • Lake → Tables → Columns
  • : denotes containers (lakes and tables)
  • * denotes columns
  • ~ denotes joins (see Joins)

• Descriptions: in [ ... ]

• Types: in < ... > using SQL types (string, int, boolean, date, timestamp, etc.). Nested types keep their own angle brackets: <list<item: string>>

• Nested fields: use dot notation (car.engine.horsepower)

Example:

Webshop[Contains all webshop data]:
  users[Contains all registered users]:
    * user_id<int>[Unique ID of a user]
    * name<string>[Full name of a user]
    * email<string>[Email address]
  orders[Customer orders]:
    * id<int>[Order id]
    * total<double>[Total amount]
    * created_at<timestamp>[When created]
    * user_id<int>[The user who placed the order]
  ~ orders.user_id -> users.user_id<many:1>

Escapes

Inside a description ([ ... ]):

Write	For
\], \<, \>	the literal ], <, >
\\	a literal backslash
\n, \r	a line break (CMeta is line based, a description always stays on one line)

A backslash followed by anything else is kept as written, so \d+ reads as \d+.

Names

Lake and table names are written as they are, spaces included (Demo Shop, Order Details). Column names too, as long as they hold no whitespace (user_id, car.engine.hp).

A name that would otherwise read back differently is written in double quotes, like a SQL identifier:

"Demo Shop"[..]:            <- only if it had e.g. a ':' or '[' in it, otherwise Demo Shop
  "sales[2024]"[..]:
    * "Order Date"<timestamp>[When it was placed]
    * "say \"hi\""<string>

Inside quotes, \", \\, \n and \r are escapes. A lake or table needs quotes when its name contains [, ], :, " or \, starts with * or ~, is empty, or has whitespace other than single spaces or at either end. A column needs them when its name contains whitespace, <, >, [, ], " or \, or is empty.

🔗 Joins

A join says a column of one table holds values of a column of another table in the same lake. It is a line inside the lake, indented like a table, starting with ~:

  ~ orders.user_id -> users.user_id<many:1>[unconfirmed: 60% of values match]

• from.column -> to.column: from is conventionally the referencing (many) side, to the referenced (one) side
• <many:1> (optional): the cardinality, one token such as many:1, 1:1 or many:many
• [ ... ] (optional): a description, escaped like any other
• Names follow the quoting rules above, except that a . also needs quotes here, because it separates table from column: "Support Tickets".customer_id -> Customers.customer_id, Cars."engine.hp" -> ...

Joins can be anywhere directly under their lake, and they are written after its tables.

Compatibility

v1.1 only adds to v1. Text that is valid v1 reads exactly the same, and a model that only uses what v1 can express is written byte for byte as before. New in v1.1: the \\, \n and \r escapes, quoted names, nested types, and joins. One thing to know: a v1 name that started with a " used to be read literally, and is now read as a quoted name.

---

🔧 Usage

Parse CMeta text → Model

from cmeta import parse_cmeta

text = """
Webshop[Contains all webshop data]:
  users[Contains all registered users]:
    * user_id<int>[Unique ID]
    * name<string>[Full name]
"""

model = parse_cmeta(text)
print(model.lakes[0].tables[0].columns[0].name)
# "user_id"

Text that isn't valid CMeta raises CMetaParseError, with the line and what is wrong (and never anything else, whatever it is given).

---

Model → CMeta text

from cmeta import to_cmeta

cmeta_str = to_cmeta(model)
print(cmeta_str)

Whatever a model holds comes back the same when its text is parsed, with these exceptions: an empty description is left out and reads back as None; a type is written on one line with single spaces; and no type at all is written as unknown, not silently as string. A model holding something CMeta cannot express (a type with unbalanced </>, a join cardinality that isn't a single token) raises CMetaFormatError.

---

Joins

from cmeta import Join, Lake, Model, Table, to_cmeta

lake = Lake("Webshop", tables=[Table("users"), Table("orders")])
lake.joins.append(Join("orders", "user_id", "users", "user_id", cardinality="many:1"))
print(to_cmeta(Model([lake])))

---

Compact JSON

from cmeta import model_to_compact_json, compact_json_to_model

cj = model_to_compact_json(model)
print(cj[0]["tables"][0]["columns"][0])
# {'name': 'user_id', 'type': 'int', 'description': 'Unique ID'}

m2 = compact_json_to_model(cj)
assert to_cmeta(m2) == to_cmeta(model)

Compact JSON format:

[
  {
    "name": "Webshop",
    "description": "Contains all webshop data",
    "tables": [
      {
        "name": "users",
        "description": "Contains all registered users",
        "columns": [
          {"name": "user_id", "type": "int", "description": "Unique ID"}
        ]
      }
    ],
    "joins": [
      {
        "fromTable": "orders", "fromColumn": "user_id",
        "toTable": "users", "toColumn": "user_id",
        "cardinality": "many:1", "description": null
      }
    ]
  }
]

joins is only there for lakes that have some.

---

Extended JSON (flat, very verbose)

from cmeta import model_to_extended_json, extended_json_to_model, model_to_join_rows

ej = model_to_extended_json(model)
print(ej[0])
# {
#   'columnName': 'user_id',
#   'columnDescription': 'Unique ID',
#   'dataType': 'int',
#   'sourceDescription': 'Contains all webshop data',
#   'sourceName': 'Webshop',
#   'tableDescription': 'Contains all registered users',
#   'tableName': 'users'
# }

m3 = extended_json_to_model(ej)

# Joins don't fit one row per column, so they get rows of their own
joins = model_to_join_rows(model)  # [{'sourceName': 'Webshop', 'fromTable': 'orders', ...}]
m4 = extended_json_to_model(ej, joins)

Extended JSON format:

[
  {
    "columnName": "user_id",
    "columnDescription": "Unique ID",
    "dataType": "int",
    "sourceDescription": "Contains all webshop data",
    "sourceName": "Webshop",
    "tableDescription": "Contains all registered users",
    "tableName": "users"
  }
]

---

Normalizing types

Data sources report types in their own words. normalize_type turns them into the SQL names CMeta documents:

from cmeta import normalize_type

normalize_type("utf8")                  # "string"
normalize_type("int64")                 # "bigint"
normalize_type("timestamp[ns, tz=UTC]") # "timestamp"
normalize_type("decimal128(10, 2)")     # "decimal(10, 2)"
normalize_type("list<item: string>")    # "array"
normalize_type(None)                    # "unknown"

---

✅ Supported Data Types

CMeta supports common datatypes:

• string
• int, bigint
• float, double, decimal
• boolean
• date, timestamp
• json, array, map

The type is free text, so any other name works too.

---

🧪 Development

Clone and set up:

uv venv && source .venv/bin/activate
make install

Run checks:

make ci     # lint + typecheck + test
make lint   # ruff
make format # autoformat
make test   # pytest

The tests include seeded random ones: random models, full of the characters that tend to break line-based formats, must survive being written and read back, and random garbage must only ever raise CMetaParseError.

---

🚀 Publishing

Dev workflow for trusted contributors:

1. Bump the version in pyproject.toml (and add it to CHANGELOG.md).

2. Commit & push to main.

3. Tag the release:

   git tag v0.2.0 && git push origin v0.2.0

4. The GitHub Actions workflow will:

   • run tests
   • build wheels/sdist
   • publish to PyPI

---

📄 License

MIT — free for personal and commercial use.