Skip to content

chore(tracker): sync Step 1.3 → ✅ and log PRs #56, #57 #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
officialCodeWork merged 1 commit into from
May 25, 2026
Merged

chore(tracker): sync Step 1.3 → ✅ and log PRs #56, #57 #58

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions TRACKER.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

**Last updated:** 2026-05-24
**Current phase:** Phase 1 — Ingestion + Knowledge Store
**Next action:** Phase 1 Step 1.4 — OCR pipeline: Tesseract + PaddleOCR plugins, image region extraction, confidence scoring. (Step 1.3 — Document parsers — is in progress: see PR for `build/phase-1/step-1.3-document-parsers`.)
**Next action:** Phase 1 Step 1.4 — OCR pipeline: Tesseract + PaddleOCR plugins, image region extraction, confidence scoring.

> **Refactor window (Steps 1.1a–1.1f):** Before resuming the connectors framework (1.2), we insert a six-step refactor that locks in architecture + optimization decisions which are very expensive to retrofit later (PolicyEngine PDP, RequestContext-threaded SPIs, split Retrieval/Index backends, bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split, hot-path discipline). See [docs/adr/ADR-0005…0009] and [docs/architecture/policy-engine.md], [request-context.md], [caching.md], [performance.md].

Expand All @@ -32,14 +32,14 @@
| Phase | Title | Steps | ✅ Done | Remaining |
|-------|-------|------:|-------:|----------:|
| 0 | Foundation | 13 | **13** | 0 |
| 1 | Ingestion + Knowledge Store | 16 | **10** | 6 |
| 1 | Ingestion + Knowledge Store | 16 | **11** | 5 |
| 2 | Retrieval Engine | 11 | 0 | 11 |
| 3 | Gateway & Agent Runtime | 11 | 0 | 11 |
| 4 | Reliability | 6 | 0 | 6 |
| 5 | Eval & Observability | 7 | 0 | 7 |
| 6 | Governance & Tenancy | 10 | 0 | 10 |
| 7 | Pilot, Harden, GA | 10 | 0 | 10 |
| **Total** | | **84** | **23** | **61** |
| **Total** | | **84** | **24** | **60** |

---

Expand Down Expand Up @@ -75,7 +75,7 @@
| 1.1e | Cache SPI split + perf discipline + async telemetry | ✅ | `build/phase-1/step-1.1e-cache-split-perf-telemetry` | [#51](https://github.com/officialCodeWork/AgentContextOS/pull/51) | `EmbeddingCache` (`rag_core.spi.embedding_cache`, key tenant+model_id+model_version+text_hash, `invalidate_model`), `RetrievalCache` (`rag_core.spi.retrieval_cache`, key tenant+plan_hash+corpus_version, `invalidate_corpus`), `AnswerCache` (`rag_core.spi.answer_cache`, key tenant+plan_hash+corpus_version+policy_version, `invalidate_policy`). All ctx-first; signature linter extended. Noop in-memory impls + 18 conformance tests covering hit/miss/version-partition/invalidate/tenant-isolation. `AsyncTelemetrySink` in `rag-observability` — bounded buffer (default 10_000), non-blocking `submit()`, per-kind drop + exporter-error counters, idempotent start/stop, drain-timeout-cancel; 9 unit tests. Hot-path discipline doc in `performance.md` expanded with concrete hot/cold call-site table. New `docs/reference/rag-observability.md`. `caching.md` SPI shapes updated with ctx-first signatures. `rag-core` 0.6.0 → 0.7.0. |
| 1.1f | ADRs 0005–0009 + reviewer checklist | ✅ | `build/phase-1/step-1.1f-adrs-reviewer-checklist` | [#52](https://github.com/officialCodeWork/AgentContextOS/pull/52) | ADR-0005 (PolicyEngine PDP), ADR-0006 (two-stage reranker default), ADR-0007 (tiered storage with BlobRef), ADR-0008 (cost-aware planner replacing reactive fallback), ADR-0009 (vector index strategy + quantization) all promoted to Accepted with implementation-path backlinks. Reviewer checklist in `docs/architecture/performance.md` extended with explicit items for plan/budget awareness, IndexHint+dtype handling, two-stage rerank shape, and BlobRef-safe consumers. ADR backlinks added to `rag_core.spi.reranker` and `rag_core.spi.storage` (the two missing module-level pointers). |
| 1.2 | Connectors framework | ✅ | `build/phase-1/step-1.2-connectors-framework` | [#55](https://github.com/officialCodeWork/AgentContextOS/pull/55) | `Connector` SPI evolved to `(Document, ConnectorState)` async-iterator with resumable watermark (`ConnectorState` frozen model: connector_id, tenant_id, cursor, last_seen_at, extra). Internal `Crawler[Raw]` base class in `rag_backends.connectors._base` (iteration scaffold, error policy, monotonic watermark). Built-in connectors: `FilesystemConnector` (cross-platform pathlib, sha256 hashing, sorted resume cursor), `S3Connector` (aioboto3, ListObjectsV2 continuation token, MinIO-compatible), `GCSConnector` (gcloud-aio-storage, REST nextPageToken, `[gcs]` extra). Policy boundary documented — connectors are upstream of `PolicyEngine`; the ingest pipeline (Step 1.10) is the enforcement point. 9 contract tests (NoopConnector resume + watermark + tenant mismatch), 9 unit tests (FilesystemConnector), MinIO + GCS integration tests skip-if-no-service. `ConnectorState` schema added to `dist/schemas/`. New docs: [reference/connectors.md](docs/reference/connectors.md), [architecture/connectors.md](docs/architecture/connectors.md). |
| 1.3 | Document parsers | 🚧 | `build/phase-1/step-1.3-document-parsers` | [#57](https://github.com/officialCodeWork/AgentContextOS/pull/57) | New `rag-parsers` package (`packages/parsers/`, v0.1.0) shipping `PlainTextParser`, `MarkdownParser` (markdown-it-py), `HtmlParser` (BeautifulSoup + stdlib `html.parser`), `JsonParser`, `CsvParser`, `YamlParser` (PyYAML `safe_load`), `PdfParser` (pypdf), `DocxParser` (python-docx), `PptxParser` (python-pptx), `XlsxParser` (openpyxl). `Parser` SPI evolved: `parse(...)` now returns `ParsedDocument` (`Document` + extracted text + structural `Block` list with offsets + heading levels + parser-specific `attributes`). New types in `rag_core`: `Block`, `BlockType` (8-value enum), `ParsedDocument`. `rag_parsers.mime.detect_mime()` (puremagic + OOXML ZIP central-directory disambiguation + extension shortcut + hint fallback). `ParserRegistry` + `default_registry()` (first-match-wins; specific MIMEs before generics). `ragctl parse <path>` smoke-test command (MIME detect → parser select → print summary + first-N blocks). 35 parser unit tests + 6 ragctl tests + extended contract tests (148 contract tests total pass). Schemas regenerated (`Block.json`, `ParsedDocument.json`). `rag-core` 0.7.0 → 0.8.0. Cross-platform: every parser library is pure-Python or wheel-distributed; no `libmagic` / `lxml` system deps. New docs: [reference/parsers.md](docs/reference/parsers.md), [architecture/parsers.md](docs/architecture/parsers.md). |
| 1.3 | Document parsers | ✅ | `build/phase-1/step-1.3-document-parsers` | [#57](https://github.com/officialCodeWork/AgentContextOS/pull/57) | New `rag-parsers` package (`packages/parsers/`, v0.1.0) shipping `PlainTextParser`, `MarkdownParser` (markdown-it-py), `HtmlParser` (BeautifulSoup + stdlib `html.parser`), `JsonParser`, `CsvParser`, `YamlParser` (PyYAML `safe_load`), `PdfParser` (pypdf), `DocxParser` (python-docx), `PptxParser` (python-pptx), `XlsxParser` (openpyxl). `Parser` SPI evolved: `parse(...)` now returns `ParsedDocument` (`Document` + extracted text + structural `Block` list with offsets + heading levels + parser-specific `attributes`). New types in `rag_core`: `Block`, `BlockType` (8-value enum), `ParsedDocument`. `rag_parsers.mime.detect_mime()` (puremagic + OOXML ZIP central-directory disambiguation + extension shortcut + hint fallback). `ParserRegistry` + `default_registry()` (first-match-wins; specific MIMEs before generics). `ragctl parse <path>` smoke-test command (MIME detect → parser select → print summary + first-N blocks). 35 parser unit tests + 6 ragctl tests + extended contract tests (148 contract tests total pass). Schemas regenerated (`Block.json`, `ParsedDocument.json`). `rag-core` 0.7.0 → 0.8.0. Cross-platform: every parser library is pure-Python or wheel-distributed; no `libmagic` / `lxml` system deps. New docs: [reference/parsers.md](docs/reference/parsers.md), [architecture/parsers.md](docs/architecture/parsers.md). |
| 1.4 | OCR pipeline | ⏳ | — | — | Tesseract + PaddleOCR plugins; image region extraction; confidence scoring |
| 1.5 | Structure-aware chunker | ⏳ | — | — | Heading-based chunking, parent-child `Chunk.parent_id` links, sentence-boundary, overlap, size normalization |
| 1.6 | Metadata enricher | ⏳ | — | — | Auto-tag: language, doc type, created/modified dates, author, section path, reading level |
Expand Down Expand Up @@ -223,6 +223,8 @@
| [#52](https://github.com/officialCodeWork/AgentContextOS/pull/52) | docs(adr,architecture): finalize ADRs 0005–0009 + extend reviewer checklist (Step 1.1f) | `build/phase-1/step-1.1f-adrs-reviewer-checklist` | ✅ Merged | 2026-05-24 |
| [#53](https://github.com/officialCodeWork/AgentContextOS/pull/53) | chore(tracker): sync Step 1.1f → ✅ and log PR #52 | `chore/tracker-sync-pr52` | ✅ Merged | 2026-05-24 |
| [#55](https://github.com/officialCodeWork/AgentContextOS/pull/55) | feat(core,backends): connectors framework — SPI evolution + filesystem/S3/GCS (Step 1.2) | `build/phase-1/step-1.2-connectors-framework` | ✅ Merged | 2026-05-24 |
| [#56](https://github.com/officialCodeWork/AgentContextOS/pull/56) | chore(tracker): sync Step 1.2 → ✅ and log PR #55 | `chore/tracker-sync-pr55` | ✅ Merged | 2026-05-24 |
| [#57](https://github.com/officialCodeWork/AgentContextOS/pull/57) | feat(core,parsers): document parsers — PDF/DOCX/PPTX/XLSX/HTML/MD/text/JSON/CSV/YAML (Step 1.3) | `build/phase-1/step-1.3-document-parsers` | ✅ Merged | 2026-05-24 |

---

Expand Down
Loading