Skip to content

Deduplicate DataProvider, LQL, Migration and Sync below a 15% Deslop budget - #122

Merged
MelbourneDeveloper merged 128 commits into
mainfrom
refactor/deslop-ten-percent
Oct 13, 2026
Merged

MelbourneDeveloper merged 128 commits into
mainfrom
refactor/deslop-ten-percent

Conversation

@MelbourneDeveloper

Copy link
Copy Markdown
Collaborator

TLDR

Code deduplication across DataProvider, LQL, Migration and Sync: measured duplication is now 14.76%, and the Deslop budget drops from 39.44% on main to 15%.

What Was Added?

  • Shared production helpers that replace copy-pasted logic:
    • DataProvider codegen: CodeGenerationGuards, DataAccessGenerator partials (Bulk, BulkMethods, KeyWrites, SqlMethods).
    • CLI: CliOptions, CliFileSystem, CliErrorWriter, SqlServerCli, Other/Nimblesite.Cli.Shared/CommandDispatch.
    • LQL: StatementRendering, StatementBuilderOperations, SelectContextBase, CommonFunctionMappings.
    • Migration: SchemaInspectionSupport, ForeignKeyActionSql, SqlIdentifierRewriter, SqlSelectionExtractor, IndexExpressionSplitter, FunctionBodyValidation, SqliteColumnChecks, SqlServerExpressionNormalizer, PostgresCatalogQuery.
    • Sync: SyncClientStorage, SyncCommandExecution, SyncLogEntryFactory, TriggerGenerationWorkflow, SqliteCommandExecution, RelationalSyncCommands.
  • Nimblesite.Sync.SqlServer: change applier, sync schema, store and trigger generator built on the shared relational sync commands.
  • Tests.Shared/: shared test infrastructure (ResultAssertions, MigrationAssertions, MappingAssertions, DatabaseCommands, PostgresDatabaseTest, GeneratedCodeCompiler, CodeGenerationTestDoubles, ExtensionTestCases, PersonCommandParameters).
  • Data-only files split out of algorithm files: *.Fixtures.cs / *.fixtures.rs scenario data, *.SqlSpecifications.cs / *.CatalogQueryText.cs SQL text, and *.EmissionText.cs code templates.
  • Docs/reports/deslop-resources-2026-10-04*: duplication audit evidence.

What Was Changed or Deleted?

  • Duplicated generators, inspectors, sync repositories, LQL renderers and test setups now call the shared helpers above.
  • The DataProvider SQL Server generator (SqlServerCodeGenerator, SqlServerParser, SqlServerTableMetadata) is consolidated onto the shared codegen and uses Microsoft's ScriptDom T-SQL parser. PlaceholderMarker.cs is removed.
  • Repeated test methods are collapsed into data-driven theories over shared scenario runners.
  • Lql/Nimblesite.Lql.Core/DetailedLexerErrorListener.cs is merged into the generic DetailedParserErrorListener<TSymbol>, which now handles lexer and parser errors.
  • .deslop.toml:
    • max_duplication_percent is lowered from 39.44 to 15.
    • Data-only files are excluded: scenario fixtures, SQL text specifications, emission templates, public DTO and model contracts, and the standalone example projects. Every exclusion has a stated reason, and every runner, algorithm and assertion is still analysed.
  • .github/workflows/ci.yml: Deslop is bumped from 0.36.0 to 0.37.0.
  • CustomCodeGenerationTests: fixed the test-local repository generator. It emitted "" around its SQL after the template moved out of a verbatim string. The expected fixture is unchanged.
  • SqlModelCoverageTests.BuilderWhere.cs (new): runs SelectStatementBuilder.Where<T> predicates with AND, OR (wrapped in parentheses) and != against SQLite and asserts the exact rows returned. Moving codegen into new DataProvider.Core partials had pushed DataProvider.Example coverage to 39.44%, under its 40% gate; it is now 40.72%.
  • RlsPredicateTranspiler: fixed a regression from moving bare-identifier quoting into the shared SqlIdentifierRewriter. The shouldQuote callback returned the keep-bare condition without negating it, so policies emitted tenant_id = "current_setting"(…)::"uuid". It now quotes columns and leaves functions, casts, keywords and qualified names bare, as on main. This change brought 127 failing Migration tests (RLS, trigger and policy round-trip) back to passing.
  • SyncCoordinatorTests.Mapped.cs (new): the first tests for MappedSyncCoordinator. They run mapped pull and push between real SQLite replicas, with mapping state and record hashes stored by the SQLite MappingStateRepository. Covered: version tracking (applied once, then skipped), hash tracking (pushed once, then skipped while unchanged), echo prevention, strict unmapped tables, and apply, send and suppression failures. After the later Deslop commits Sync.Core coverage had fallen to 69.47%, under its 74% gate; it is now 88.79%.
  • coverage-thresholds.json: the coverage gate ratcheted every threshold up to measured coverage: DataProvider.Core 92→93, Lql.Core 74→75, Migration.Core 88→89, Sync.Core 74→88, Sync.SQLite 43→46, Sync.Postgres 28→29, Sync.Integration 11→13, Sync.Http 43→44. No threshold was lowered.

How Do The Automated Tests Prove It Works?

make ci (lint, then every test project with its coverage gate, then build) exits 0 locally. make fmt CHECK=1 and deslop . also pass.

Project Coverage Gate
DataProvider/Nimblesite.DataProvider.Core 93.03% 93%
DataProvider/Nimblesite.DataProvider.Example 40.72% 40%
Lql/Nimblesite.Lql.Core 75.62% 75%
Lql/Nimblesite.Lql.TypeProvider.FSharp 49.16% 49%
Migration/Nimblesite.DataProvider.Migration.Core 89.27% 89%
Sync/Nimblesite.Sync.Core 88.79% 88%
Sync/Nimblesite.Sync.SQLite 46.62% 46%
Sync/Nimblesite.Sync.Postgres 29.39% 29%
Sync/Nimblesite.Sync.Integration 13.24% 13%
Sync/Nimblesite.Sync.Http 44.22% 44%
Reporting/Nimblesite.Reporting 78.82% 78%
Reporting/Nimblesite.Reporting.Integration 76.38% 76%
Lql/lql-lsp-rust 98.57% 98%
Lql/LqlExtension 54.48% 54%
  • .NET: 2163 tests passed across 12 projects; 3 skipped (the Native AOT smoke tests, which need a published AOT binary).
  • Rust (lql-lsp-rust): 442 tests passed.
  • VS Code extension: 70 passing.
  • Regression tests that caught the fixes: RlsCurrentSettingLqlTests, RlsPredicateTranspilerTests, RlsLqlExhaustiveTests and the Postgres/SQLite RLS E2E suites (127 failures before the quoting fix), and CustomCodeGenerationTests.CustomTableOperationGenerator_GeneratesRepositoryPattern.

Spec / Doc Changes

  • docs/specs/*: adds IDs for each shared helper and the rules for keeping independent data, for example [CI-TEST-SHARED-SCENARIOS], [CI-TEST-SHARED-RESULT], [CI-DESLOP-INDEPENDENT-DATA], [CI-DESLOP-CONTRACT-DATA], [CI-INDEPENDENT-SQL-SPECIFICATIONS], [DP-CODEGEN-SHARED-BULK-METHODS], [DP-CODEGEN-SHARED-SQL-METHODS], [DP-CLI-SHARED-EMISSION-TEXT] and [LQL-VSCODE-SHARED-DIAGNOSTIC-SCENARIOS].
  • Agent instructions move into AGENTS.md, and CLAUDE.md now points to it.

Breaking Changes

  • None

MelbourneDeveloper and others added 30 commits December 28, 2025 19:30
- Create Samples/ICD10AM folder structure for new microservice
- Add comprehensive SPEC.md with:
  - RAG search feature using MedEmbed-Large-v1 medical embedding model
  - Basic lookup with JSON and FHIR response formats
  - Mermaid ER diagram for database schema
  - API endpoint documentation
  - PostgreSQL with pgvector for vector similarity search
  - RLS (Row Level Security) via user impersonation
- Add icd10am-schema.yaml with DataProvider YAML migrations for:
  - ICD-10-AM chapters, blocks, categories, and codes
  - ACHI procedure blocks and codes
  - Embedding tables for vector storage
  - Coding standards and user search history
- Add Python import script (import_icd10am.py) to:
  - Parse IHACPA data files
  - Generate medical embeddings with MedEmbed
  - Bulk import into PostgreSQL
- Remove "Too Many Cooks" multi-agent section from CLAUDE.md

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Replace all SQL query files with LQL equivalents
- Add ICD10AM.Api.csproj with LQL transpilation support
- Add DataProvider.json configuration
- Add DatabaseSetup.cs and GlobalUsings.cs
- All queries now use pipeline syntax: filter, join, select, order_by

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Hierarchical browse: chapters, blocks, categories, codes
- Code lookup with JSON and FHIR format support
- ACHI procedure endpoints
- RAG search with embedding service integration
- Cosine similarity ranking for semantic search
- LQL transpilation enabled in csproj
- Updated DataProvider.json to use .generated.sql files

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Dockerfile using MedEmbed-Small-v0.1 (384 dims, ~500MB)
- FastAPI service with /embed and /embed/batch endpoints
- docker-compose.yml for easy deployment
- Health checks and resource limits configured
- Model downloaded at build time for fast startup

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- ICD10AMApiFactory with seeded test data
- ChapterEndpointTests: hierarchical browse tests
- CodeLookupTests: code search and FHIR format tests
- AchiEndpointTests: procedure code tests
- HealthEndpointTests: health check tests
- Real database, zero mocking

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- generate_sample_data.py creates test SQLite database
- Includes common ICD-10-AM codes (infectious, diabetes, cardiac, respiratory, etc.)
- Includes sample ACHI procedures (angiography, appendicectomy, hip replacement, etc.)
- Note: Full ICD-10-AM data requires licensing from IHACPA

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Update SPEC.md with IHACPA licensing requirements
- Add .gitignore for generated files, databases, Python cache
- Note: Full ICD-10-AM data requires purchase from IHACPA

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
BREAKING: No more licensed IHACPA data!

- Add import_icd10cm.py that downloads FREE data from CMS.gov
- Successfully imports 71,704 diagnosis codes
- 19 chapters, 1,910 blocks, 1,910 categories
- Update SPEC.md to document free data sources
- Remove licensing requirements (CMS data is public domain)

Data sources:
- Primary: https://www.cms.gov/medicare/coding-billing/icd-10-codes
- Mirror: GitHub JSON gist for faster downloads

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Fixed syntax error in try/except blocks around IHACPA download
- Added CDC ICD-10-CM as fallback when IHACPA returns 503 errors
- Uses free US Government CDC data (74,260 codes) which shares
  WHO ICD-10 base with Australian ICD-10-AM
- Script now successfully imports codes when IHACPA is unavailable

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Renamed folder from ICD10AM to ICD10CM (honest about data source)
- Simplified import script - CDC data only, no fallback bullshit
- 74,260 ICD-10-CM codes from CDC (public domain)
- Clean database schema with icd10cm_ table prefix

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Fix DI registration for Func<HttpClient> embedding service
- Fix DatabaseSetup to skip initialization if tables exist (for tests)
- Remove unsupported 'unique' property from schema indexes
- Remove SearchCodes/SearchAchiCodes LQL files (LIKE not supported)
- Implement manual SQL search endpoints in Program.cs
- Disable AOT in LqlCli.SQLite to avoid missing ILCompiler packages
- Update GlobalUsings to remove unused Search result types
- Disable NuGet audit in Directory.Build.props for proxy issues

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- generate_embeddings.py: Populates icd10cm_code_embedding table
  using MedEmbed-small-v0.1 model (384 dimensions)
- embedding_service.py: Runtime service for encoding user queries
- SPEC.md: Document the 3-step setup process:
  1. Import codes (import_icd10cm.py)
  2. Generate embeddings (generate_embeddings.py)
  3. Start embedding service (embedding_service.py)

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
Updated LQL query to reference icd10cm_code_embedding and icd10cm_code
tables where the 74,260 embeddings are stored. Added icd10cm_code and
icd10cm_code_embedding table definitions to schema and DataProvider.json.

30 E2E tests passing, RAG semantic search working with MedEmbed model.

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
Replaced Python embedding service with native C# ONNX Runtime:
- Added Microsoft.ML.OnnxRuntime and BERTTokenizers NuGet packages
- EncodeWithOnnx helper performs tokenization and mean pooling
- Updated SPEC.md with download instructions for ONNX model
- Model (127MB) excluded from git - download with optimum-cli

30 E2E tests passing, RAG search works without any Python dependency.

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
The BERTTokenizers library requires vocabulary files in a Vocabularies
directory. These files are needed for tokenizing query text before
encoding with the ONNX model.

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
Documents how to:
- Setup database and generate embeddings (one-time Python)
- Export ONNX model for C# runtime
- Run the API
- Run E2E tests
- Troubleshoot common issues

https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
…xtension/multi-57637e47fe' into refactor/deslop-ten-percent
…xtension/undici-7.30.0' into refactor/deslop-ten-percent
…ebsite-Eleventy/brace-expansion-1.1.21' into refactor/deslop-ten-percent
…markdown-it-14.3.2' into refactor/deslop-ten-percent
…ctor

Retain the current refactored source tree and preserve the older local branch commits as merge ancestry. Active workspace changes and stashes remain in place.
RLS predicate quoting regressed when it moved into the shared
SqlIdentifierRewriter: shouldQuote returned the keep-bare condition, so
functions and casts were quoted and columns were not. Negate it.

The test-local repository generator in CustomCodeGenerationTests emitted
doubled quotes after its SQL template left a verbatim string.

Cover MappedSyncCoordinator end to end against real SQLite replicas and
the SelectStatementBuilder expression Where path, bringing Sync.Core and
DataProvider.Example back above their coverage gates. Thresholds ratchet
up to measured coverage.
@MelbourneDeveloper
MelbourneDeveloper enabled auto-merge (squash) October 13, 2026 10:31
@MelbourneDeveloper
MelbourneDeveloper merged commit b99ebf3 into main Oct 13, 2026
11 checks passed
@MelbourneDeveloper
MelbourneDeveloper deleted the refactor/deslop-ten-percent branch October 13, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants