Repository navigation
Deduplicate DataProvider, LQL, Migration and Sync below a 15% Deslop budget - #122
Merged
Merged
Conversation
- Create Samples/ICD10AM folder structure for new microservice - Add comprehensive SPEC.md with: - RAG search feature using MedEmbed-Large-v1 medical embedding model - Basic lookup with JSON and FHIR response formats - Mermaid ER diagram for database schema - API endpoint documentation - PostgreSQL with pgvector for vector similarity search - RLS (Row Level Security) via user impersonation - Add icd10am-schema.yaml with DataProvider YAML migrations for: - ICD-10-AM chapters, blocks, categories, and codes - ACHI procedure blocks and codes - Embedding tables for vector storage - Coding standards and user search history - Add Python import script (import_icd10am.py) to: - Parse IHACPA data files - Generate medical embeddings with MedEmbed - Bulk import into PostgreSQL - Remove "Too Many Cooks" multi-agent section from CLAUDE.md https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Replace all SQL query files with LQL equivalents - Add ICD10AM.Api.csproj with LQL transpilation support - Add DataProvider.json configuration - Add DatabaseSetup.cs and GlobalUsings.cs - All queries now use pipeline syntax: filter, join, select, order_by https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Hierarchical browse: chapters, blocks, categories, codes - Code lookup with JSON and FHIR format support - ACHI procedure endpoints - RAG search with embedding service integration - Cosine similarity ranking for semantic search - LQL transpilation enabled in csproj - Updated DataProvider.json to use .generated.sql files https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Dockerfile using MedEmbed-Small-v0.1 (384 dims, ~500MB) - FastAPI service with /embed and /embed/batch endpoints - docker-compose.yml for easy deployment - Health checks and resource limits configured - Model downloaded at build time for fast startup https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- ICD10AMApiFactory with seeded test data - ChapterEndpointTests: hierarchical browse tests - CodeLookupTests: code search and FHIR format tests - AchiEndpointTests: procedure code tests - HealthEndpointTests: health check tests - Real database, zero mocking https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- generate_sample_data.py creates test SQLite database - Includes common ICD-10-AM codes (infectious, diabetes, cardiac, respiratory, etc.) - Includes sample ACHI procedures (angiography, appendicectomy, hip replacement, etc.) - Note: Full ICD-10-AM data requires licensing from IHACPA https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Update SPEC.md with IHACPA licensing requirements - Add .gitignore for generated files, databases, Python cache - Note: Full ICD-10-AM data requires purchase from IHACPA https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
BREAKING: No more licensed IHACPA data! - Add import_icd10cm.py that downloads FREE data from CMS.gov - Successfully imports 71,704 diagnosis codes - 19 chapters, 1,910 blocks, 1,910 categories - Update SPEC.md to document free data sources - Remove licensing requirements (CMS data is public domain) Data sources: - Primary: https://www.cms.gov/medicare/coding-billing/icd-10-codes - Mirror: GitHub JSON gist for faster downloads https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Fixed syntax error in try/except blocks around IHACPA download - Added CDC ICD-10-CM as fallback when IHACPA returns 503 errors - Uses free US Government CDC data (74,260 codes) which shares WHO ICD-10 base with Australian ICD-10-AM - Script now successfully imports codes when IHACPA is unavailable https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Renamed folder from ICD10AM to ICD10CM (honest about data source) - Simplified import script - CDC data only, no fallback bullshit - 74,260 ICD-10-CM codes from CDC (public domain) - Clean database schema with icd10cm_ table prefix https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- Fix DI registration for Func<HttpClient> embedding service - Fix DatabaseSetup to skip initialization if tables exist (for tests) - Remove unsupported 'unique' property from schema indexes - Remove SearchCodes/SearchAchiCodes LQL files (LIKE not supported) - Implement manual SQL search endpoints in Program.cs - Disable AOT in LqlCli.SQLite to avoid missing ILCompiler packages - Update GlobalUsings to remove unused Search result types - Disable NuGet audit in Directory.Build.props for proxy issues https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
- generate_embeddings.py: Populates icd10cm_code_embedding table using MedEmbed-small-v0.1 model (384 dimensions) - embedding_service.py: Runtime service for encoding user queries - SPEC.md: Document the 3-step setup process: 1. Import codes (import_icd10cm.py) 2. Generate embeddings (generate_embeddings.py) 3. Start embedding service (embedding_service.py) https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
Updated LQL query to reference icd10cm_code_embedding and icd10cm_code tables where the 74,260 embeddings are stored. Added icd10cm_code and icd10cm_code_embedding table definitions to schema and DataProvider.json. 30 E2E tests passing, RAG semantic search working with MedEmbed model. https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
Replaced Python embedding service with native C# ONNX Runtime: - Added Microsoft.ML.OnnxRuntime and BERTTokenizers NuGet packages - EncodeWithOnnx helper performs tokenization and mean pooling - Updated SPEC.md with download instructions for ONNX model - Model (127MB) excluded from git - download with optimum-cli 30 E2E tests passing, RAG search works without any Python dependency. https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
The BERTTokenizers library requires vocabulary files in a Vocabularies directory. These files are needed for tokenizing query text before encoding with the ONNX model. https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
Documents how to: - Setup database and generate embeddings (one-time Python) - Export ONNX model for C# runtime - Run the API - Run E2E tests - Troubleshoot common issues https://claude.ai/code/session_01Pxhuxx7gqh8RrTCEGJ2QU4
…xtension/multi-57637e47fe' into refactor/deslop-ten-percent
…xtension/undici-7.30.0' into refactor/deslop-ten-percent
…ebsite-Eleventy/brace-expansion-1.1.21' into refactor/deslop-ten-percent
…markdown-it-14.3.2' into refactor/deslop-ten-percent
…ctor Retain the current refactored source tree and preserve the older local branch commits as merge ancestry. Active workspace changes and stashes remain in place.
…o refactor/deslop-ten-percent
RLS predicate quoting regressed when it moved into the shared SqlIdentifierRewriter: shouldQuote returned the keep-bare condition, so functions and casts were quoted and columns were not. Negate it. The test-local repository generator in CustomCodeGenerationTests emitted doubled quotes after its SQL template left a verbatim string. Cover MappedSyncCoordinator end to end against real SQLite replicas and the SelectStatementBuilder expression Where path, bringing Sync.Core and DataProvider.Example back above their coverage gates. Thresholds ratchet up to measured coverage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Code deduplication across DataProvider, LQL, Migration and Sync: measured duplication is now 14.76%, and the Deslop budget drops from 39.44% on
mainto 15%.What Was Added?
CodeGenerationGuards,DataAccessGeneratorpartials (Bulk,BulkMethods,KeyWrites,SqlMethods).CliOptions,CliFileSystem,CliErrorWriter,SqlServerCli,Other/Nimblesite.Cli.Shared/CommandDispatch.StatementRendering,StatementBuilderOperations,SelectContextBase,CommonFunctionMappings.SchemaInspectionSupport,ForeignKeyActionSql,SqlIdentifierRewriter,SqlSelectionExtractor,IndexExpressionSplitter,FunctionBodyValidation,SqliteColumnChecks,SqlServerExpressionNormalizer,PostgresCatalogQuery.SyncClientStorage,SyncCommandExecution,SyncLogEntryFactory,TriggerGenerationWorkflow,SqliteCommandExecution,RelationalSyncCommands.Nimblesite.Sync.SqlServer: change applier, sync schema, store and trigger generator built on the shared relational sync commands.Tests.Shared/: shared test infrastructure (ResultAssertions,MigrationAssertions,MappingAssertions,DatabaseCommands,PostgresDatabaseTest,GeneratedCodeCompiler,CodeGenerationTestDoubles,ExtensionTestCases,PersonCommandParameters).*.Fixtures.cs/*.fixtures.rsscenario data,*.SqlSpecifications.cs/*.CatalogQueryText.csSQL text, and*.EmissionText.cscode templates.Docs/reports/deslop-resources-2026-10-04*: duplication audit evidence.What Was Changed or Deleted?
SqlServerCodeGenerator,SqlServerParser,SqlServerTableMetadata) is consolidated onto the shared codegen and uses Microsoft's ScriptDom T-SQL parser.PlaceholderMarker.csis removed.Lql/Nimblesite.Lql.Core/DetailedLexerErrorListener.csis merged into the genericDetailedParserErrorListener<TSymbol>, which now handles lexer and parser errors..deslop.toml:max_duplication_percentis lowered from 39.44 to 15..github/workflows/ci.yml: Deslop is bumped from 0.36.0 to 0.37.0.CustomCodeGenerationTests: fixed the test-local repository generator. It emitted""around its SQL after the template moved out of a verbatim string. The expected fixture is unchanged.SqlModelCoverageTests.BuilderWhere.cs(new): runsSelectStatementBuilder.Where<T>predicates with AND, OR (wrapped in parentheses) and!=against SQLite and asserts the exact rows returned. Moving codegen into newDataProvider.Corepartials had pushedDataProvider.Examplecoverage to 39.44%, under its 40% gate; it is now 40.72%.RlsPredicateTranspiler: fixed a regression from moving bare-identifier quoting into the sharedSqlIdentifierRewriter. TheshouldQuotecallback returned the keep-bare condition without negating it, so policies emittedtenant_id = "current_setting"(…)::"uuid". It now quotes columns and leaves functions, casts, keywords and qualified names bare, as onmain. This change brought 127 failing Migration tests (RLS, trigger and policy round-trip) back to passing.SyncCoordinatorTests.Mapped.cs(new): the first tests forMappedSyncCoordinator. They run mapped pull and push between real SQLite replicas, with mapping state and record hashes stored by the SQLiteMappingStateRepository. Covered: version tracking (applied once, then skipped), hash tracking (pushed once, then skipped while unchanged), echo prevention, strict unmapped tables, and apply, send and suppression failures. After the later Deslop commitsSync.Corecoverage had fallen to 69.47%, under its 74% gate; it is now 88.79%.coverage-thresholds.json: the coverage gate ratcheted every threshold up to measured coverage:DataProvider.Core92→93,Lql.Core74→75,Migration.Core88→89,Sync.Core74→88,Sync.SQLite43→46,Sync.Postgres28→29,Sync.Integration11→13,Sync.Http43→44. No threshold was lowered.How Do The Automated Tests Prove It Works?
make ci(lint, then every test project with its coverage gate, then build) exits 0 locally.make fmt CHECK=1anddeslop .also pass.DataProvider/Nimblesite.DataProvider.CoreDataProvider/Nimblesite.DataProvider.ExampleLql/Nimblesite.Lql.CoreLql/Nimblesite.Lql.TypeProvider.FSharpMigration/Nimblesite.DataProvider.Migration.CoreSync/Nimblesite.Sync.CoreSync/Nimblesite.Sync.SQLiteSync/Nimblesite.Sync.PostgresSync/Nimblesite.Sync.IntegrationSync/Nimblesite.Sync.HttpReporting/Nimblesite.ReportingReporting/Nimblesite.Reporting.IntegrationLql/lql-lsp-rustLql/LqlExtensionlql-lsp-rust): 442 tests passed.RlsCurrentSettingLqlTests,RlsPredicateTranspilerTests,RlsLqlExhaustiveTestsand the Postgres/SQLite RLS E2E suites (127 failures before the quoting fix), andCustomCodeGenerationTests.CustomTableOperationGenerator_GeneratesRepositoryPattern.Spec / Doc Changes
docs/specs/*: adds IDs for each shared helper and the rules for keeping independent data, for example[CI-TEST-SHARED-SCENARIOS],[CI-TEST-SHARED-RESULT],[CI-DESLOP-INDEPENDENT-DATA],[CI-DESLOP-CONTRACT-DATA],[CI-INDEPENDENT-SQL-SPECIFICATIONS],[DP-CODEGEN-SHARED-BULK-METHODS],[DP-CODEGEN-SHARED-SQL-METHODS],[DP-CLI-SHARED-EMISSION-TEXT]and[LQL-VSCODE-SHARED-DIAGNOSTIC-SCENARIOS].AGENTS.md, andCLAUDE.mdnow points to it.Breaking Changes