Skip to content

refactor: remove legacy conversion subsystem #109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SkyeAv merged 5 commits into from
Aug 21, 2026
Merged

refactor: remove legacy conversion subsystem #109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6772173
test: [US-001] remove legacy conversion tests
SkyeAv Aug 21, 2026
28c1f9b
refactor: [US-002] remove legacy converter
SkyeAv Aug 21, 2026
9773976
docs: [US-003] remove legacy converter runbook
SkyeAv Aug 21, 2026
5e956b0
docs: [US-005] remove stale converter reference
SkyeAv Aug 21, 2026
374b356
Merge origin/main into delete-legacy-converter
SkyeAv Aug 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 4 additions & 8 deletions docs/agent.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -451,8 +451,8 @@ intentionally-impoverished single-section no-`explode_by` negative control that
uv run --extra qc --extra agent pytest tests/test_agent_edgecount.py -q
```

**Real runbook (PMC10766526):** run the agent over the downloaded payload, convert the hand-written
legacy reference config against the same downloads, build one-table graphs for each, and compare the
**Real runbook (PMC10766526):** run the agent over the downloaded payload, build one-table graphs
for the agent config and the already-converted v12 reference config, and compare the
`<name>_<version>.edges.ndjson` line counts from `rig.artifact_base_path`:

```bash
Expand All @@ -461,15 +461,11 @@ tablassert agent PMC10766526 --configuration-file ./graph.yaml \
--local PMC10766526=./downloads/PMC10766526
# accepted config: .tablassert/agent/configs/PMC10766526.yaml

# 2. Convert the hand-written legacy reference config over the same downloads
tablassert convert-legacy ./legacy/PMC10766526.yaml --downloads ./downloads
# -> ./legacy/PMC10766526.v12.yaml

# 3. Build each as a one-table graph (same fullmap, same graph name/version conventions)
# 2. Build each as a one-table graph (same fullmap, same graph name/version conventions)
tablassert build-kg -f ./agent_graph.yaml # tables: [.tablassert/agent/configs/PMC10766526.yaml]
tablassert build-kg -f ./reference_graph.yaml # tables: [./legacy/PMC10766526.v12.yaml]

# 4. Count edges; acceptance: agent >= 0.5 * reference
# 3. Count edges; acceptance: agent >= 0.5 * reference
wc -l <agent-graph-name>_<version>.edges.ndjson <reference-graph-name>_<version>.edges.ndjson
```

Expand Down
81 changes: 1 addition & 80 deletions docs/cli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# CLI Reference

Tablassert extracts knowledge assertions from tabular data into KGX NDJSON. The `tablassert` app
exposes **six subcommands**: `agent`, `build-fullmap`, `build-kg`, `convert-legacy`, `validate`,
exposes **five subcommands**: `agent`, `build-fullmap`, `build-kg`, `validate`,
and `validate-kgx`, plus an app-level `--version` flag. Run `tablassert --help` (or `<command> --help`)
for the live surface.

Expand All @@ -12,7 +12,6 @@ for the live surface.
| [`agent`](#agent) | Autonomously derive, build, audit, and improve KG configs from PMC articles |
| [`build-fullmap`](#build-fullmap) | Build the embedded fullmap redb used for entity resolution |
| [`build-kg`](#build-kg) | Build a KGX NDJSON knowledge graph from a YAML configuration |
| [`convert-legacy`](#convert-legacy) | Convert legacy table configs into the v12 config shape |
| [`validate`](#validate) | Validate a graph or table configuration without executing it |
| [`validate-kgx`](#validate-kgx) | Validate built KGX NDJSON against the Biolink Model |

Expand Down Expand Up @@ -172,84 +171,6 @@ or incomplete RIG fails the build with `[rig-validation-failed]` and nothing is

---

## convert-legacy

Use this to migrate a legacy table config (or a whole directory of them) into the current v12
`{template, sections}` shape: duplicate mapping keys are merged (never silently dropped),
exact-duplicate entries the template/section overlay produced are dropped and same-key
`qualifiers` entries merged into one per key, the removed `relationship_strength` annotation
is renamed to `effect_size`, every `source.reindex` entry is validated against the v12 model,
and each `source.local` is resolved onto the real downloaded payload, never left pointing at
a stale `./DATALAKE` path.

```bash
tablassert convert-legacy LEGACY-PATH [ARGS]
```

`LEGACY-PATH` is a single legacy YAML file or a directory of them (non-recursive). Each converted
config is written as `<stem>.v12.yaml`, beside its input by default, or under `--out` when given.

| Option | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `LEGACY-PATH` | Path | Yes | n/a | Legacy YAML file, or a directory of `*.yaml` legacy configs |
| `--downloads`, `-d` | Path | No | `None` | Directory holding the downloaded article payloads (`PMC<n>/PMC<n>.<v>/...`) that each `source.local` is resolved against: the `local` basename is tried first, then each `source.url` entry's basename (the urls hold the real payload filenames), recursively, preferring a hit under the section's own publication directory; an unresolved source lists every basename tried |
| `--fetch` | Flag | No | `False` | When no local payload match exists, download the article from PMC open access (via `provenance.publication`) into `--downloads` instead of failing the source as unresolved |
| `--out`, `-o` | Path | No | `None` (beside each input) | Directory the `<stem>.v12.yaml` outputs are written into (created when missing); defaults to each input's own directory |

Directory mode converts every `*.yaml` file (skipping `*.v12.yaml` outputs from earlier runs, so
a rerun never re-converts its own output) and prints one status line per file:
`CONVERTED <input> -> <output>` or `FAILED <input> (<code>)`. One failure never aborts the batch,
but the command exits non-zero when ANY file failed (the full coded message for every failure is
printed on stderr). `--out` is optional in directory mode too: without it each `<stem>.v12.yaml`
lands beside its input. The glob is `*.yaml` ONLY: `.yml` files and every other extension are
never picked up.

Exit codes: `0` everything converted; `1` any conversion failed; `2` usage error (a missing input
or `--downloads` path, an `--out` that is not a directory, or a directory holding no `*.yaml`
files). Conversion is all-or-nothing per file: a construct v12 cannot express fails the file with
`legacy-unsupported-syntax`, and an unresolvable `source.local` fails it with
`legacy-source-unresolved`, the same coded error `--fetch` ends in when the network is down or
the article is not open access (a bounded fetch, never a hang).

```bash
# One file: writes my-table.v12.yaml beside it
tablassert convert-legacy my-table.yaml --downloads ./downloads
# One file into an explicit directory (created when missing)
tablassert convert-legacy my-table.yaml --downloads ./downloads --out ./v12
# A whole directory in place; fetch any payload not already downloaded
tablassert convert-legacy ./legacy-configs --downloads ./downloads --fetch
```

### Runbook: the full MOKG corpus (26 files)

The MOKG corpus (26 legacy table configs) has an executable ingestability acceptance:
`tests/test_legacy.py::test_corpus_mokg_convert_or_fail_unresolved`, gated on
`TABLASSERT_MOKG_DIR`. It skips with a printed reason when the variable is unset; when set,
every corpus file must either convert and validate against the downloads directory
(`TABLASSERT_MOKG_DOWNLOADS`, recursive basename match with the `source.url` basenames as
fallback) or fail loudly with exactly `legacy-source-unresolved`: no other error class, no
silent skip, and at least 15 of the 26 must convert (with the full downloads tree below,
24 convert; only QIN9 and WAINBERG3 stay unresolved because their payloads are absent):

```bash
TABLASSERT_MOKG_DIR=/home/skyeav/Code/ISB/TableConfigs/TABLE/MOKG \
TABLASSERT_MOKG_DOWNLOADS=/home/skyeav/Code/ISB/MultiomicsNext/.tablassert \
uv run pytest tests/test_legacy.py -q -k corpus
```

For the remaining files, run the batch with `--fetch` so every open-access article payload
missing from the downloads directory is downloaded from PMC open access first (network
required); `--out` keeps the outputs out of the curated corpus directory. QIN9 stays
unresolved either way: its payload is hosted on figshare, not in the PMC article bundle:

```bash
tablassert convert-legacy /home/skyeav/Code/ISB/TableConfigs/TABLE/MOKG \
--downloads /home/skyeav/Code/ISB/MultiomicsNext/.tablassert \
--fetch --out /tmp/mokg-v12
```

---

## validate

Use this to validate a configuration against a schema without running the build, ideal for CI and
Expand Down
97 changes: 1 addition & 96 deletions src/tablassert/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

from tablassert import extras
from tablassert._lazy import LazyModule
from tablassert.errors import BabelDownloadError, GraphValidationError, SectionValidationError, TablassertError
from tablassert.errors import BabelDownloadError, GraphValidationError, SectionValidationError
from tablassert.log import cat

if TYPE_CHECKING:
Expand Down Expand Up @@ -735,101 +735,6 @@ def validate_kgx_command(
print("KGX output is Biolink-compliant.", file=sys.stderr)


@APP.command(name="convert-legacy")
def convert_legacy_command(
legacy_path: Annotated[Path, cyclopts.Parameter(allow_leading_hyphen=False)],
downloads: Annotated[Path | None, cyclopts.Parameter(name=["--downloads", "-d"])] = None,
fetch: Annotated[bool, cyclopts.Parameter(name=["--fetch"], negative="")] = False,
out: Annotated[Path | None, cyclopts.Parameter(name=["--out", "-o"])] = None,
) -> None:
"""Convert legacy table configs into the v12 ``{template, sections}`` shape.

``LEGACY-PATH`` is one legacy YAML file or a directory of them. Each converted config is
written as ``<stem>.v12.yaml`` — beside its input by default, or under ``--out`` when
given (created when missing). Directory mode converts every ``*.yaml`` file — skipping
``*.v12.yaml`` outputs from earlier runs so a rerun never re-converts its own output —
and prints one status line per file (``CONVERTED`` / ``FAILED (<code>)``); one failure
never aborts the batch, but the command exits non-zero when ANY file failed.

``--downloads`` points at the directory holding the articles' downloaded payloads
(``PMC<n>/PMC<n>.<v>/...``) so each ``source.local`` resolves onto a real file; with
``--fetch``, a payload missing there is downloaded from PMC open access instead. A
source that still cannot resolve fails its file with ``legacy-source-unresolved`` — a
coded error, never a hang, whether the network is down or the article is not open access.

Exit codes: ``0`` everything converted; ``1`` any conversion failed; ``2`` usage error
(missing input or ``--downloads`` path, an ``--out`` that is not a directory, or a
directory holding no ``*.yaml`` files).

Args:
legacy_path: Legacy YAML file, or a directory of ``*.yaml`` legacy configs.
downloads: Directory holding the downloaded article payloads each ``source.local``
is resolved against.
fetch: Download a missing payload from PMC open access (via ``provenance.publication``)
into ``--downloads`` instead of failing the source as unresolved.
out: Directory the ``<stem>.v12.yaml`` outputs are written into; defaults to each
input's own directory.
"""
# Lazy imports: the legacy converter stays out of the base CLI import surface, exactly
# like the agent module (convert_legacy itself lazy-imports agent on the fetch path).
from tablassert.ingests import to_yaml
from tablassert.legacy import convert_legacy

if not legacy_path.exists():
print(f"tablassert convert-legacy: input does not exist: {legacy_path}", file=sys.stderr)
raise SystemExit(2)
if not legacy_path.is_file() and not legacy_path.is_dir():
print(f"tablassert convert-legacy: input is neither a file nor a directory: {legacy_path}", file=sys.stderr)
raise SystemExit(2)
if downloads is not None and not downloads.is_dir():
print(f"tablassert convert-legacy: --downloads directory does not exist: {downloads}", file=sys.stderr)
raise SystemExit(2)
if out is not None:
if out.exists() and not out.is_dir():
print(f"tablassert convert-legacy: --out is not a directory: {out}", file=sys.stderr)
raise SystemExit(2)
out.mkdir(parents=True, exist_ok=True)

def convert_one(legacy_file: Path) -> Path:
converted: dict[str, Any] = convert_legacy(legacy_file, downloads, fetch)
target: Path = (out if out is not None else legacy_file.parent) / f"{legacy_file.stem}.v12.yaml"
to_yaml(target, converted)
return target

if legacy_path.is_file():
try:
target = convert_one(legacy_path)
except TablassertError as exc:
print(str(exc), file=sys.stderr)
raise SystemExit(1) from exc
except Exception as exc: # malformed YAML etc: one-line failure, never a raw traceback
print(f"FAILED {legacy_path} (error): {type(exc).__name__}: {exc}", file=sys.stderr)
raise SystemExit(1) from exc
print(f"CONVERTED {legacy_path} -> {target}")
return

legacy_files: list[Path] = sorted(p for p in legacy_path.glob("*.yaml") if not p.name.endswith(".v12.yaml"))
if not legacy_files:
print(f"tablassert convert-legacy: no *.yaml files found in {legacy_path}", file=sys.stderr)
raise SystemExit(2)
failures: int = 0
for legacy_file in legacy_files:
try:
target = convert_one(legacy_file)
except TablassertError as exc:
failures += 1
print(f"FAILED {legacy_file} ({exc.code})")
print(str(exc), file=sys.stderr)
except Exception as exc: # malformed YAML etc: one bad file must never abort the batch
failures += 1
print(f"FAILED {legacy_file} (error)")
print(f"{legacy_file}: {type(exc).__name__}: {exc}", file=sys.stderr)
else:
print(f"CONVERTED {legacy_file} -> {target}")
if failures:
raise SystemExit(1)


@APP.command(name="agent")
def agent(
pmc_ids: Annotated[list[str], cyclopts.Parameter(allow_leading_hyphen=False)],
Expand Down
45 changes: 0 additions & 45 deletions src/tablassert/errors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,6 @@
"rig-terms-empty",
"rig-legacy-keys",
"rig-validation-failed",
"legacy-unsupported-syntax",
"legacy-source-unresolved",
]


Expand Down Expand Up @@ -99,19 +97,6 @@ class UnpairedEffectAnnotationWarning(UserWarning):
"""


class LegacyDuplicateKeyWarning(UserWarning):
"""A legacy YAML config declared the same mapping key more than once.

PyYAML's default construction silently keeps only the LAST occurrence of a duplicate key,
dropping everything earlier occurrences contributed — unnoticed data loss in human-curated
legacy configs that accumulated duplicates over years. The legacy loader instead MERGES the
occurrences (dict+dict deep-merged, list+list extended, later value wins otherwise) and fires
this warning naming the key and line so the curator can fix the source file. Its own category
so callers can silence or assert on duplicate merges without touching the relocation or
deprecation scaffolds.
"""


def format_missing_extra(extra: str, problem: str) -> str:
"""Append the install instructions for ``extra`` to a one-sentence ``problem``.

Expand Down Expand Up @@ -190,36 +175,6 @@ def __init__(self, config: Path, section_hash: str, detail: str) -> None:
super().__init__(f"Section validation failed: {config} (hash {section_hash[:8]})\n{detail}", code="section-validation-failed")


class LegacyUnsupportedSyntaxError(TablassertError):
"""A legacy config carries a construct the v12 converter cannot translate.

Conversion is all-or-nothing: raising instead of skipping keeps a
half-understood legacy file from ever producing a partial v12 config that
silently drops whatever the converter did not recognize.
"""

def __init__(self, config: Path, detail: str) -> None:
super().__init__(f"Legacy config {config} uses a construct unsupported by the v12 conversion: {detail}", code="legacy-unsupported-syntax")


class LegacySourceUnresolvedError(TablassertError):
"""A legacy ``source.local`` could not be mapped onto a real payload file.

The converter NEVER keeps a stale legacy path (``./DATALAKE/...``): a config
pointing at a file that does not exist would only fail deep inside a build.
Resolution needs the downloads directory holding the fetched payload — and,
when it is missing, ``fetch=True`` to download the article from PMC open access.
"""

def __init__(self, config: Path, local: str, tried: Sequence[str]) -> None:
locations: str = "; ".join(tried)
super().__init__(
f"Legacy config {config}: cannot resolve `source.local` {local!r} to a downloaded payload — tried: {locations}. "
"Supply the downloads directory holding the article payload, or pass fetch=True to download it from PMC open access.",
code="legacy-source-unresolved",
)


class BabelDownloadError(TablassertError):
def __init__(self, url: str, retries: int, last_error: BaseException) -> None:
super().__init__(
Expand Down
Loading
Loading