diff --git a/LICENSE b/LICENSE index 261eeb9..784ced0 100644 --- a/LICENSE +++ b/LICENSE @@ -1,201 +1,15 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. + AFFERO GENERAL PUBLIC LICENSE + Version 2, November 2007 + + Copyright (C) 2007 Affero Inc. + 510 Third Street - Suite 225, San Francisco, CA 94107, USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + +This is version 2 of the Affero General Public License. It gives each +licensee permission to distribute the Program or a work based on the Program +(as defined in version 1 of the Affero GPL) under the GNU Affero General +Public License, version 3 or any later version. If the Program was licensed +under version 1 of the Affero GPL "or any later version", no additional +obligations are imposed on any author or copyright holder of the Program as +a result of a licensee's choice to follow this version 2 of the Affero GPL. diff --git a/README.md b/README.md index 4c45820..fbfa098 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,122 @@ # LineageLens -A deterministic code graph for coding agents. Control flow, data flow and -cross-service contracts across six languages, in one SQLite file. +> **Real-Time Code Lineage & Visual Analytics** +> A deterministic code graph for coding agents. Control flow, data flow, and cross-service contracts across seven languages, in one SQLite file. -The problem it solves: an agent asked to fix a bug or plan a feature reads -whole files to reconstruct what calls what, where a value comes from, and what -a change will break. That is slow, expensive, and gets worse as the repository -grows. LineageLens answers those questions directly β€” with the source you need -and the honest statement of what it could not determine. +The problem it solves: an agent asked to fix a bug or plan a feature reads whole files to reconstruct what calls what, where a value comes from, and what a change will break. That is slow, expensive, and gets worse as the repository grows. LineageLens answers those questions directly β€” with the source you need and an honest statement of what it could not determine. + +> **100% Local & Offline**: LineageLens requires no account, no login, and no API key. All indexing, call graph generation, and MCP queries run entirely inside your local machine in SQLite (`.lineagelens/graph.db`). Zero telemetry by default. + +--- + +## ⚑ Quick Start + +### 1. Recommended Setup: Docker Compose Sidecar (Cursor & VS Code) + +Keep a warm LineageLens container running in the background for sub-millisecond MCP query latency without polluting your host toolchain: + +Create `docker-compose.mcp.yml` in your project root: + +```yaml +services: + lineagelens: + image: ghcr.io/contextserve/lineagelens:latest + container_name: lineagelens + restart: unless-stopped + working_dir: /workspace + entrypoint: sleep + command: infinity + volumes: + - .:/workspace + # environment: + # - LINEAGELENS_API_TOKEN=${LINEAGELENS_API_TOKEN} +``` ```bash -pip install lineagelens +# Start the persistent MCP sidecar +docker compose -f docker-compose.mcp.yml up -d + +# Index your project instantly inside the running sidecar +docker exec lineagelens lineagelens index . + +# Query the graph +docker exec lineagelens lineagelens query . impact src/orders/service.py:412 +``` + +--- + +### Alternative: Single Docker Run Sidecar + +```bash +# Start container in detached mode (persistent execution sidecar) +docker run --rm -d --name lineagelens \ + -v $(pwd):/workspace \ + -w /workspace \ + --entrypoint sleep \ + ghcr.io/contextserve/lineagelens:latest infinity +``` + +--- + +### 2. Shell Alias (Feels 100% Native) + +Add this alias to your shell profile (`~/.zshrc` or `~/.bashrc`) to run LineageLens with native CLI speed: + +```bash +alias lineagelens='docker exec lineagelens lineagelens' +``` + +```bash +# Reload profile +source ~/.zshrc + +# Now run commands directly: lineagelens index . lineagelens query . impact src/orders/service.py:412 +lineagelens coverage . +``` + +--- + +### 3. One-Off Execution (Ephemeral `docker run`) -# Or run with Docker (zero install, zero host toolchain pollution): +Run without keeping a container running in the background: + +```bash +# One-off index docker run --rm -v $(pwd):/workspace -w /workspace ghcr.io/contextserve/lineagelens:latest lineagelens index . + +# One-off query docker run --rm -v $(pwd):/workspace -w /workspace ghcr.io/contextserve/lineagelens:latest lineagelens query . impact src/orders/service.py:412 ``` -> **100% Local & Offline**: LineageLens requires no account, no login, and no API key. All indexing, call graph generation, and MCP queries run entirely inside your local machine in SQLite (`.lineagelens/graph.db`). Zero telemetry by default. +--- +### 4. Native Installation (Python 3.10+) -## What it answers +```bash +pip install lineagelens +lineagelens index . +lineagelens query . impact src/orders/service.py:412 +``` + +--- + +### Available Docker Images + +| Tag / Target | Runtimes Included | Best For | +|---|---|---| +| `:latest` / `:monolith` | All 7 runtimes + all SCIP indexers | Multi-language & polyglot codebases | +| `:node` | Python + Node.js LTS & scip-typescript | TypeScript and JavaScript projects | +| `:java` | Python + OpenJDK 17 & scip-java | Java, Kotlin, Scala / Maven & Gradle | +| `:go` | Python + Go runtime & scip-go | Go services & modules | +| `:rust` | Python + Rust & rust-analyzer | Rust crates & workspaces | +| `:clang` | Python + LLVM / Clang & scip-clang | C and C++ projects | +| `:ruby` | Python + Ruby & scip-ruby | Ruby and Rails projects | + +--- + +## πŸ” What It Answers | Question | Command | |---|---| @@ -35,16 +129,13 @@ docker run --rm -v $(pwd):/workspace -w /workspace ghcr.io/contextserve/lineagel | What does this stack trace touch? | `query . stacktrace error.log` | | What does the index *not* cover? | `coverage` | -Every answer carries a **coverage envelope**: which boundaries the walk -crossed, which files were skipped, whether data flow was computed or deferred. -A negative answer tells you whether it means "nothing calls this" or "we could -not see". +Every answer carries a **coverage envelope**: which boundaries the walk crossed, which files were skipped, and whether data flow was computed or deferred. A negative answer tells you whether it means "nothing calls this" or "we could not see". -## Cross-service flow +--- + +## 🌐 Cross-Service Flow -The capability no call graph has. A TypeScript `fetch` and a Python route -handler never reference each other β€” but both name the same route, so both -attach to one contract node: +The capability traditional call graphs lack. A TypeScript `fetch` and a Python route handler never reference each other directly β€” but both name the same route, so both attach to one contract node: ``` web/client.ts placeOrder ──CONSUMES──▢ POST /api/orders ◀──EXPOSES── create_order api/routes.py @@ -60,13 +151,9 @@ contract_map 1 of 1 cross_service: true ``` -Thirty framework adapters ship with it: FastAPI, Flask, Django, Spring, -Express, NestJS, ASP.NET, Go mux, fetch/axios/requests/RestTemplate, Dubbo -`@DubboService`/`@DubboReference`, gRPC, Kafka, Celery, `click`, env vars, and -Java SPI (`META-INF/services`). +Thirty framework adapters ship out of the box: FastAPI, Flask, Django, Spring, Express, NestJS, ASP.NET, Go mux, fetch/axios/requests/RestTemplate, Dubbo `@DubboService`/`@DubboReference`, gRPC, Kafka, Celery, `click`, env vars, and Java SPI (`META-INF/services`). -**A framework with no adapter is reported, not ignored.** Anything -route-shaped that no adapter claims comes back as a ranked work list: +**Unmodelled frameworks are reported, not ignored.** Anything route-shaped that no adapter claims comes back as a ranked work list: ``` frameworks present but unmodelled (no contract adapter): @@ -74,10 +161,11 @@ frameworks present but unmodelled (no contract adapter): add an adapter under .lineagelens/adapters/ to link these ``` -Adapters are YAML, so adding your in-house framework does not mean patching the -package. +Adapters are simple YAML files, so adding your in-house framework does not require patching the package. -## Line-level impact +--- + +## 🎯 Line-Level Impact ```console $ lineagelens query . impact api/routes.py:4 @@ -87,17 +175,18 @@ operations : EXPOSES -> POST /api/orders cross_service: placeOrder via POST /api/orders ``` -`change_kind` matters because the blast radii differ by orders of magnitude. A -**body** change is local. A **signature** change breaks every caller. A string -literal that happens to be a route key is a **contract** change with -cross-service reach β€” and impact separates in-process consequences (a build or -test failure, one deploy) from cross-service ones (wire compatibility and -deploy ordering). +`change_kind` matters because blast radii differ by orders of magnitude: +- A **body** change is local. +- A **signature** change breaks every caller. +- A string literal that happens to be a route key is a **contract** change with cross-service reach. + +LineageLens separates in-process consequences (a build or test failure, one deploy) from cross-service ones (wire compatibility and deploy ordering). -## Data flow +--- + +## 🌊 Data Flow -Follows `READS`, `WRITES`, `PARAM_BINDS` and `RETURNS`, so a walk crosses call -boundaries instead of stopping at them: +Follows `READS`, `WRITES`, `PARAM_BINDS`, and `RETURNS`, so a walk crosses call boundaries instead of stopping at them: ```console $ lineagelens query . dataflow submit/amount @@ -105,11 +194,11 @@ $ lineagelens query . dataflow submit/amount upstream READS amount -> submit ``` -Scope is deterministic: intraprocedural def-use, positional argument binding, -field read/write, return flow. Where a chain needs alias analysis or dynamic -dispatch it reports a **boundary with its candidate set** rather than guessing. +Scope is deterministic: intraprocedural def-use, positional argument binding, field read/write, and return flow. Where a chain needs alias analysis or dynamic dispatch, it reports a **boundary with its candidate set** rather than guessing. + +--- -## Intent: pay for the detail you need +## πŸ’‘ Intent: Pay for the Detail You Need A bug fix needs line-exact detail. A feature plan does not. @@ -118,79 +207,184 @@ lineagelens query . impact svc.py:412 --intent precise # spans, data flow, sou lineagelens query . impact OrderService --intent plan # shape only, ~10x cheaper ``` -`plan` computes no data flow and returns no source; the envelope says so. This -is the main cost control, and the saving comes from *not computing* detail the -question skips β€” never from truncating detail it needed. +`plan` computes no data flow and returns no source; the envelope says so. This is the main cost control: savings come from *not computing* detail the question skips β€” never from truncating detail it needed. -## Languages +--- -| | nodes | calls | inherits | implements | data flow | contracts | -|---|---|---|---|---|---|---| -| Python | βœ“ | βœ“ | βœ“ | n/a | βœ“ | βœ“ | -| TypeScript / TSX | βœ“ | βœ“ | βœ“ | βœ“ | βœ“ | βœ“ | -| JavaScript | βœ“ | βœ“ | βœ“ | n/a | βœ“ | βœ“ | -| Java | βœ“ | βœ“ | βœ“ | βœ“ | βœ“ | βœ“ | -| Go | βœ“ | βœ“ | βœ“ | partial | βœ“ | β€” | -| Rust | βœ“ | βœ“ | n/a | partial | βœ“ | β€” | -| C# | βœ“ | βœ“ | βœ“ | βœ“ | βœ“ | β€” | +## πŸ€– MCP Integration & Real-Time IDE Setup -Generated by the conformance suite, not written by hand. The two `partial` -entries are honest: Go satisfies interfaces structurally and Rust has blanket -impls, so neither has a single resolvable target. +LineageLens exposes 15 tools via the Model Context Protocol (MCP). Agents query graph structure instead of reading raw files β€” typically reducing context consumption by 10Γ—. + +### Claude / Claude Code + +Configure `.claude/settings.json` or Claude Desktop: + +```json +{ + "mcpServers": { + "lineagelens": { + "command": "docker", + "args": [ + "run", + "--rm", + "-i", + "-v", + "${workspaceFolder}:/workspace", + "-w", + "/workspace", + "ghcr.io/contextserve/lineagelens:latest", + "lineagelens", + "mcp", + "." + ] + } + } +} +``` -Everything above works with no external toolchain β€” tree-sitter grammars are -pinned dependencies and install with the package. An optional **type resolver** -raises resolution quality further: Python has one vendored (jedi), and Java, -Go, Rust and C# can use a detected system toolchain. Without one a language -still indexes; more references are recorded as *ambiguous* rather than -resolved, and none are guessed. `lineagelens ontology` reports what your -machine has. +### Real-Time IDE Setup (Cursor & VS Code) -A polyglot repository produces **one** graph. Language is a property of a file, -not of a repository. +#### Step 1: Run Persistent Sidecar in Docker -## Honesty by construction +```bash +docker run --rm -d --name lineagelens \ + -v $(pwd):/workspace \ + -w /workspace \ + --entrypoint sleep \ + ghcr.io/contextserve/lineagelens:latest infinity +``` -Four properties, each enforced by a test rather than intended: +#### Step 2: Configure IDE with `mcp.json` -**Never guesses.** When several candidates match a reference, no edge is -emitted and the ambiguity is recorded with its candidate set. An ambiguity is a -useful answer; a wrong edge is not. +Add the server to `.cursor/mcp.json` (Cursor) or `.vscode/mcp.json` (VS Code): -**Never silently degrades.** A missing *grammar* skips the file with a recorded -reason β€” there is no regex fallback. A missing *type resolver* lowers resolution -rather than blocking the language, and the tier used is reported per language on -every answer. Either way the degradation is in the output, not hidden. +**Option A: Recommended β€” Connect to Running Sidecar (`docker exec`)** -**Never loses a reference.** Every observed reference becomes an edge or an -`unresolved_refs` row, and the per-file totals must balance β€” enforced by a -database constraint. +```json +{ + "mcpServers": { + "lineagelens": { + "command": "docker", + "args": [ + "exec", + "-i", + "lineagelens", + "lineagelens", + "mcp", + "/workspace" + ] + } + } +} +``` -**Reproducible.** `lineagelens verify` builds twice and compares digests. Same -commit, same graph. +> **Note for VS Code**: VS Code native MCP in `.vscode/mcp.json` uses the top-level key `"servers"` (e.g. `{"servers": {"lineagelens": {...}}}`), whereas Claude Desktop and Cursor use `"mcpServers"`. -## MCP +**Option B: On-Demand Docker Execution (`docker run`)** + +```json +{ + "mcpServers": { + "lineagelens": { + "command": "docker", + "args": [ + "run", + "--rm", + "-i", + "-v", + "${workspaceFolder}:/workspace", + "-w", + "/workspace", + "ghcr.io/contextserve/lineagelens:latest", + "lineagelens", + "mcp", + "/workspace" + ] + } + } +} +``` + +**Option C: Native Host CLI** ```json { "mcpServers": { "lineagelens": { "command": "lineagelens-mcp", - "env": {"LINEAGELENS_PROJECT": "/path/to/repo"} + "env": { + "LINEAGELENS_PROJECT": "${workspaceFolder}" + } } } } ``` -Fifteen tools: `find_paths`, `trace_flow`, `callers_of`, `callees_of`, -`impact_of`, `impact_of_diff`, `dataflow_of`, `similar_flows`, `contract_map`, -`list_entry_points`, `search`, `get_symbol`, `explain`, `map_stacktrace`, -`coverage_report`, `get_ontology`. +#### Step 3: Run the Real-Time Watcher + +Keep the graph continuously synchronized as you edit code: + +```bash +# Run the watcher inside the running MCP container +docker exec -it lineagelens python -c "from lineagelens.watcher import IndexWatcher; IndexWatcher('.').run()" + +# Or run as a standalone detached watcher container +docker run --rm -d --name lineagelens-watcher \ + -v $(pwd):/workspace \ + -w /workspace \ + ghcr.io/contextserve/lineagelens:latest \ + python -c "from lineagelens.watcher import IndexWatcher; IndexWatcher('.').run()" +``` + +--- + +### MCP Tools Reference + +| Tool | Description | +|---|---| +| `find_paths` | All execution paths between two symbols. | +| `trace_flow` | Trace a value through call boundaries. | +| `callers_of` | Transitive callers of a symbol (with depth cap). | +| `callees_of` | Transitive callees of a symbol. | +| `impact_of` | Blast radius of changing a symbol or line. ⭐ Most useful. | +| `impact_of_diff` | Blast radius from a diff/patch, not a single symbol. | +| `dataflow_of` | Upstream + downstream data flow for a value. | +| `similar_flows` | Find structurally similar code patterns. | +| `contract_map` | All cross-service contracts (HTTP, RPC, Kafka, etc.). | +| `list_entry_points` | All API routes, CLI commands, and event consumers. | +| `search` | Fuzzy symbol search across the codebase. | +| `get_symbol` | Full metadata for one symbol. | +| `explain` | Compiler evidence for an edge (why two components connect). | +| `map_stacktrace` | Resolve a crash log or stack trace to graph nodes. | +| `get_ontology` | Active capabilities and index levels for this install. | + +Responses carry verbatim line-numbered source at `intent=precise`, so there is usually no follow-up file read. Every response is budgeted and reports truncation explicitly. + +--- + +## πŸ“Š Local Token Savings & MCP Metrics (ROI Reporter) + +Every MCP tool invocation computes the ground-truth counterfactual baseline (source tokens an LLM would have read without LineageLens) versus the optimized response payload, saving metrics locally to `.lineagelens/metrics.sqlite`: + +```bash +lineagelens report # Net tokens saved, context reduction %, and ROI +lineagelens report --by-tool # Tool-by-tool breakdown +lineagelens report --period 7d # Filter by time window (today, 7d, 30d, all) +lineagelens report --model o1 # Price against target LLMs (gpt-4o, claude-3-5-sonnet, deepseek-r1, o1, o3-mini) +lineagelens report --sync # Sync metrics to ContextServe.ai dashboard +lineagelens report --json # Machine-readable JSON output +``` + +### Deterministic Counterfactual Calculation + +1. **Baseline**: For each tool call, LineageLens calculates the exact tokens in the full files that an agent would have needed to read sequentially to answer the question without a graph. +2. **Actual**: Measures the compact JSON payload returned by LineageLens. +3. **Net Saved**: $\text{Baseline Tokens} - \text{Actual Tokens}$. +4. **ROI**: Value calculated using live input/output token pricing for selected LLMs. -Responses carry verbatim line-numbered source at `intent=precise`, so there is -usually no follow-up file read. Every one is budgeted and reports truncation. +--- -## Commands +## πŸ› οΈ CLI Commands ``` lineagelens index [path] build the graph (all languages, one graph) @@ -201,9 +395,23 @@ lineagelens index [path] build the graph (all languages, one graph) --upload publish the graph to ContextServe (opt-in) --upload-required make a publish failure fatal --dry-run print the exact payload, send nothing + lineagelens query [path] ... the twelve primitives; --json for machine output + impact blast radius analysis + paths execution paths + callers transitive callers + callees transitive callees + dataflow upstream/downstream value flow + contracts cross-service HTTP/RPC contract map + similar structurally similar flows + stacktrace map stack trace to graph + search fuzzy symbol search + get_symbol symbol metadata + explain compiler evidence for an edge + map_stacktrace resolve crash log + lineagelens coverage [path] what the index does and does not cover -lineagelens verify [path] rebuild twice, confirm identical +lineagelens verify [path] rebuild twice, confirm identical output lineagelens ontology [path] measured capability for this installation lineagelens mcp [path] run the MCP server over stdio lineagelens serve [path] HTTP surface for the dashboard (loopback default) @@ -212,245 +420,247 @@ lineagelens telemetry ... enable, disable, status -- off until you enable i lineagelens auth ... login, logout, status, token, switch-env ``` -## Local Token Savings & MCP Metrics - -Every MCP tool invocation computes the ground-truth counterfactual baseline (source tokens an LLM would have read without LineageLens) versus the optimized response payload, saving the metrics locally to `.lineagelens/metrics.sqlite`: - -```bash -lineagelens report # View net tokens saved, context reduction %, and ROI -lineagelens report --by-tool # Tool-by-tool breakdown -lineagelens report --period 7d # Filter by time window (today, 7d, 30d, all) -lineagelens report --model o1 # Price against target LLMs (gpt-4o, claude-3-5-sonnet, deepseek-r1, o1, o3-mini) -lineagelens report --sync # Sync metrics to ContextServe.ai dashboard -lineagelens report --json # Machine-readable JSON output -``` - -## Telemetry - -Off until you turn it on. `DO_NOT_TRACK` and `CI` suppress it permanently and -cannot be overridden by configuration. +--- -``` -lineagelens telemetry status prints the exact event, not a description -``` +## 🌐 Languages & Capability Levels -Anonymous events are counts, versions and digests. Never a path, symbol, -repository name, branch or line of source β€” enforced by an assertion over the -event keys, not by review. Full field list and reasoning in -[docs/TELEMETRY.md](docs/TELEMETRY.md). +Breadth and depth are independent axes. A language enters at a **level** derived from what extraction specs loaded, never declared: -## Languages +| Level | Delivers | Languages | +|---|---|---| +| **L2** flow | Data flow, contracts, call graph, symbol inventory | Python, Java, TypeScript, JavaScript, Go, Rust, C# | +| **L1** graph | Call graph, `impact_of`, `callers_of` | β€” | +| **L0** inventory | Symbols, `search`, `get_symbol` | C, Ruby, Bash, Kotlin | -Breadth and depth are independent axes, so a language enters at a **level** -rather than being claimed or not claimed: +### Feature Matrix -| Level | Delivers | Languages | -| --- | --- | --- | -| **L2** flow | data flow, contracts, everything below | python, java, typescript, javascript, go, rust, csharp | -| **L1** graph | call graph, `impact_of`, `callers_of` | β€” | -| **L0** inventory | symbols, `search`, `get_symbol` | c, ruby, bash, kotlin | +| Language | Nodes | Calls | Inherits | Implements | Data Flow | Contracts | +|---|---|---|---|---|---|---| +| **Python** | βœ“ | βœ“ | βœ“ | n/a | βœ“ | βœ“ | +| **TypeScript / TSX** | βœ“ | βœ“ | βœ“ | βœ“ | βœ“ | βœ“ | +| **JavaScript** | βœ“ | βœ“ | βœ“ | n/a | βœ“ | βœ“ | +| **Java** | βœ“ | βœ“ | βœ“ | βœ“ | βœ“ | βœ“ | +| **Go** | βœ“ | βœ“ | βœ“ | partial | βœ“ | β€” | +| **Rust** | βœ“ | βœ“ | n/a | partial | βœ“ | β€” | +| **C#** | βœ“ | βœ“ | βœ“ | βœ“ | βœ“ | β€” | -The level is *derived* from which extraction specs loaded, never declared, and -the conformance run fails if a claimed level is not corroborated by what the -corpus produced. `lineagelens ontology` prints it; every query response carries -it, so an agent knows a missing call answer on an L0 language is *unavailable* -rather than *absent*. +*Generated by the conformance suite. Go satisfies interfaces structurally and Rust has blanket impls, so both report `partial`.* -The core seven install by default. Breadth grammars are extras: +The core seven install by default. Breadth grammars are available via extras: -``` +```bash pip install 'lineagelens[wave1]' # c, ruby, bash, kotlin pip install 'lineagelens[all]' ``` -A file whose grammar is not installed is skipped with -`skip_reason = 'missing_grammar'` and reported in the coverage envelope β€” never -silently ignored. +A file whose grammar is not installed is skipped with `skip_reason = 'missing_grammar'` and reported in the coverage envelope β€” never silently ignored. Adding a new language is documented in [docs/ADDING-A-LANGUAGE.md](docs/ADDING-A-LANGUAGE.md). -Adding one is documented end to end in -[docs/ADDING-A-LANGUAGE.md](docs/ADDING-A-LANGUAGE.md). +--- -## Compiler-grade resolution +## πŸ”¬ Compiler-Grade Resolution (SCIP) -Tier A is tree-sitter and always runs. Tier B is a type resolver, and a SCIP -index is the one that covers more than Python: +- **Tier A**: Tree-sitter AST queries (always active). +- **Tier B**: Type resolvers (Python uses vendored `jedi`; Java, Go, Rust, and C# use detected toolchains). +- **SCIP**: Compiler-verified facts from external SCIP indexers (`scip-java`, `scip-typescript`, `scip-python`, `scip-go`). Outranks static-analysis inferences. -``` +```bash +# Generate SCIP index and build graph scip-java index # or scip-typescript / scip-python / scip-go lineagelens index --scip # discovers ./index.scip -lineagelens index --scip=path.scip # or point at one +lineagelens index --scip=path.scip # or specify an explicit path + +# Strict mode for CI: skip or fail on languages without Tier B/SCIP +lineagelens index . --require-tier-b ``` -A SCIP answer is `evidence: fact` with label `scip_resolve`, and it outranks -jedi β€” a compiler-verified fact beats a static-analysis inference. +- **LineageLens never runs indexers for you**: Doing so would make indexing non-deterministic and break `verify --determinism`. +- **Stale index protection**: Each SCIP document's text is hashed against disk; any file that differs falls back to Tier A with the count reported. -**LineageLens never runs the indexer for you.** Subprocessing `scip-java` -inside `index` would make indexing network-dependent, slow and -non-deterministic, and `verify --determinism` could not hold. When a toolchain -is present and no index is, the CLI prints the command to generate one. +--- -**A stale index is refused per file.** Each SCIP document's text is hashed -against what is on disk; a file that differs gets no SCIP answers and falls -back to Tier A, with the count reported. An index built against a different -commit would otherwise produce confident edges that do not match the code β€” -the exact failure the rewrite was written to eliminate. +## πŸ”’ Honesty by Construction -## Publishing a graph +Four invariant properties, each enforced by tests: -Indexing is local and offline. `--upload` is the only thing that sends -anything, and a test fails on any outbound socket during a plain `index`. +1. **Never guesses**: When multiple candidates match a reference, no edge is emitted and the ambiguity is recorded with its candidate set. +2. **Never silently degrades**: A missing grammar or type resolver is explicitly reported in the coverage envelope. +3. **Never loses a reference**: Every observed reference becomes an edge or an `unresolved_refs` row, and per-file totals must balance by database constraint. +4. **Reproducible**: `lineagelens verify` builds twice and compares digests. Same commit = same graph. -``` -lineagelens auth login once, per environment -lineagelens index --upload build, then publish -LINEAGELENS_TOKEN=… lineagelens index --upload for CI -``` +--- -Two properties worth knowing before you point it at private code. +## ☁️ Cloud Sync & Team Publishing (Optional) -**You can read the payload before it leaves.** `--dry-run` prints the exact -bytes to stdout and sends nothing: +Indexing and querying are 100% offline. Publishing is strictly opt-in: -``` -lineagelens index --dry-run | jq '.graph_data | keys' -``` +```bash +# 1. Login or provide token via environment variable +lineagelens auth login +# Or: export LINEAGELENS_API_TOKEN="ll_sk_..." -**Source text does not go.** Nodes carry a signature and a docstring β€” that is -the interface, and it is structure. File contents do not: `usage_sites` holds -verbatim lines (`context_before` / `context_line` / `context_after`) and those -columns are excluded, with an assertion over the payload keys that raises -rather than silently strips. Uploading source into a context product is a -materially different proposition from uploading its structure, and the -difference is visible in `src/lineagelens/upload.py`. +# 2. Build and publish +lineagelens index --upload -**A failed upload is not a failed build.** Offline, expired token, plan limit, -server outage β€” all print a diagnostic and exit 0 if the index succeeded. The -graph is the product; publication is a side effect. Use `--upload-required` -when you want CI to disagree. +# 3. For CI/CD (fail build if upload fails) +lineagelens index --upload --upload-required +``` -## Scale +### Privacy & Zero Source Code Leakage -Apache Dubbo β€” 4,046 Java files, 120 Maven modules: +- **Inspect payload before sending**: `--dry-run` prints the exact bytes to stdout: + ```bash + lineagelens index --dry-run | jq '.graph_data | keys' + ``` +- **Source text is never uploaded**: Nodes carry only signatures and docstrings. Verbatim source lines in `usage_sites` are excluded via assertions in `src/lineagelens/upload.py` that raise rather than silently strip. +- **Failed uploads do not fail local builds**: Offline or expired tokens print a diagnostic and exit 0 unless `--upload-required` is set. -| | | -|---|---| -| nodes | 99,649 | -| edges | 411,445 | -| index time | 21s | -| dangling endpoints | 0 | -| contracts found | 378 | +--- -## Requirements +## 🐳 Docker Sandbox & Distribution -Python 3.10+, and nothing else. All eight tree-sitter grammars are pinned core -dependencies, so every language indexes out of the box. +The `Dockerfile` uses a multi-stage architecture providing isolated environments with clean APT caches: -Optional type resolvers improve resolution but are not required. Python's -(jedi) is vendored. Java, Go, Rust and C# use a detected system toolchain -(`javac`, `gopls`, `rust-analyzer`, `dotnet`); without one, those languages -index at Tier A and the coverage envelope says so on every query. Use -`--require-tier-b` in CI if a partial graph should be an error. +### Multi-Stage Build Targets -## Documentation +```bash +# Build language-specific slim image +docker build --target node -t ghcr.io/contextserve/lineagelens:node . +docker build --target java -t ghcr.io/contextserve/lineagelens:java . +docker build --target go -t ghcr.io/contextserve/lineagelens:go . -- [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) β€” architecture, adding a language, - writing an adapter, the invariants and where they are enforced -- [docs/ADDING-A-LANGUAGE.md](docs/ADDING-A-LANGUAGE.md) β€” the capability - ladder and the exact artifacts a new language needs -- [docs/TELEMETRY.md](docs/TELEMETRY.md) β€” every field sent, what is never - sent, and how to switch it off -- Design rationale and measured baselines: issue #51 +# Build all-in-one monolith image +docker build --target monolith -t ghcr.io/contextserve/lineagelens:latest . +``` +### Publishing to GHCR +Images are hosted on GitHub Container Registry at `ghcr.io/contextserve/lineagelens`. +- Automated builds via [`.github/workflows/docker-publish.yml`](.github/workflows/docker-publish.yml) on push to `main` and version tags (`v*`). +- Language targets: `ghcr.io/contextserve/lineagelens:` (`node`, `java`, `go`, `rust`, `clang`, `ruby`, `latest`). -## Dockerfile build +### Multi-Language Polyglot Pipelines -Refactored the root Dockerfile to use the Multi-Stage Build Target architecture. -This allowed us to define a shared base layer and then branch off into 8 -isolated environments (node, java, go, ruby, rust, clang, and the monolith) -within a single, highly maintainable file. +```bash +# Pattern 1: Monolith (one container, all languages) +docker run --rm -v $(pwd):/workspace -w /workspace \ + ghcr.io/contextserve/lineagelens:latest \ + bash -c "scip-java index && npm install && scip-typescript index && lineagelens index ." + +# Pattern 2: Multi-stage pipeline (shared volume) +docker run --rm -v my-ws:/workspace -w /workspace \ + ghcr.io/contextserve/lineagelens:node \ + bash -c "npm install && scip-typescript index -o index-ts.scip" + +docker run --rm -v my-ws:/workspace -w /workspace \ + ghcr.io/contextserve/lineagelens:java \ + bash -c "scip-java index -o index-java.scip" + +docker run --rm -v my-ws:/workspace -w /workspace \ + ghcr.io/contextserve/lineagelens:latest \ + lineagelens index . +``` -To build the Node variant, for example, your CI pipeline just runs: +### Enterprise Edge Cases ```bash -docker build --target node -t registry.contextserve.ai/lineagelens/cli:node . +# Shadow node_modules for cross-OS native binary compatibility +docker run --rm \ + -v $(pwd):/workspace \ + -v /workspace/node_modules \ + -w /workspace \ + ghcr.io/contextserve/lineagelens:node \ + bash -c "npm install && scip-typescript index && lineagelens index ." + +# Mount private npm / SSH keys for registry access +docker run --rm \ + -v $(pwd):/workspace \ + -v ~/.npmrc:/root/.npmrc:ro \ + -v ~/.ssh:/root/.ssh:ro \ + -w /workspace \ + ghcr.io/contextserve/lineagelens:node \ + bash -c "npm install && scip-typescript index" ``` -To build the massive all-in-one image: +--- + +## πŸ›‘οΈ Telemetry + +Off by default. `DO_NOT_TRACK` and `CI` environment variables suppress it permanently. ```bash -docker build --target monolith -t registry.contextserve.ai/lineagelens/cli:latest . +lineagelens telemetry status # Inspect the exact event +lineagelens telemetry enable # Opt in +lineagelens telemetry disable # Opt out ``` -All stages have been optimized to clean their APT caches to keep the image sizes as small as possible! +Anonymous events contain only counts, execution duration, and OS versions β€” never a path, symbol, repository name, branch, or line of code. Full specifications are documented in [docs/TELEMETRY.md](docs/TELEMETRY.md). -Build the index +--- -``` -docker run --rm -v $(pwd):/workspace ghcr.io/contextserve/lineagelens:latest lineagelens index -``` +## πŸ“ˆ Scale -## Publishing to GHCR +Benchmark on Apache Dubbo (4,046 Java files, 120 Maven modules): -Images are hosted on the GitHub Container Registry at `ghcr.io//lineagelens`. +| Metric | Result | +|---|---| +| **Nodes** | 99,649 | +| **Edges** | 411,445 | +| **Index Time** | 21s | +| **Dangling Endpoints** | 0 | +| **Contracts Found** | 378 | -### Automated via GitHub Actions (recommended) +--- -The workflow [`.github/workflows/docker-publish.yml`](.github/workflows/docker-publish.yml) -builds all 7 targets in parallel and pushes to GHCR automatically. +## ❓ FAQ & Troubleshooting -**Triggers** +
+Graph not found / MCP server fails to start -| Event | Behaviour | -|---|---| -| Push to `main` | Builds every target, pushes `ghcr.io/…/lineagelens:`. The `monolith` target also gets `:latest`. | -| Push of a `v*` tag | Same as above **plus** versioned tags, e.g. `:node-1.2.3` and `:node-1.2`. | -| Pull request | Build only β€” no push. Validates the Dockerfile without touching the registry. | +Run `docker exec lineagelens lineagelens index .` (or `lineagelens index .`) first. The SQLite database is created at `.lineagelens/graph.db` in your project root. +
-**One-time repo setup** +
+Symbol not found -Enable the workflow to publish packages with the built-in `GITHUB_TOKEN` (no -extra secret required): +Symbols are fully qualified (e.g. `myapp.module.Class.method`). Use `lineagelens query . search ` to find the exact symbol ID. +
-> **Settings β†’ Actions β†’ General β†’ Workflow permissions** -> β†’ select **Read and write permissions** β†’ Save. +
+Impact analysis shows 0 affected symbols -Then push to `main` (or create a `v*` tag) and the workflow runs automatically. +The symbol might be dead code, or callers are in an unindexed language. Run `lineagelens query . callers ` and inspect `lineagelens coverage .` output. +
-**Pulling a published image** +
+How do I add a custom framework adapter? -```bash -docker pull ghcr.io/ContextServe/lineagelens:node -docker pull ghcr.io/ContextServe/lineagelens:latest # monolith -``` +Adapters are simple YAML files placed under `.lineagelens/adapters/`. See [docs/ADDING-A-LANGUAGE.md](docs/ADDING-A-LANGUAGE.md) for schema details. No Python code modifications required. +
---- +
+Does LineageLens send my source code anywhere? -### Manual push (local machine) +No. Source text is never included in uploads. Run `lineagelens index --dry-run` to inspect the exact payload before anything leaves your machine. +
-```bash -# 1. Authenticate (once per machine) -echo $CR_PAT | docker login ghcr.io -u --password-stdin -# CR_PAT = GitHub PAT with write:packages scope - -# 2. Build + tag in one step -TARGET=node # or java | go | rust | clang | ruby | monolith -docker build --target $TARGET \ - -t ghcr.io//lineagelens:$TARGET \ - . - -# 3. Push -docker push ghcr.io//lineagelens:$TARGET -``` +
+How do I verify the graph is reproducible? -Or tag an image that was already built locally: +Run `lineagelens verify .`. It rebuilds the graph twice and asserts identical digests. Same commit = same graph. +
-```bash -docker tag registry.contextserve.ai/lineagelens/cli:node ghcr.io//lineagelens:node -docker push ghcr.io//lineagelens:node -``` +--- + +## πŸ“š Documentation + +- [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) β€” Architecture, writing adapters, and core invariants. +- [docs/ADDING-A-LANGUAGE.md](docs/ADDING-A-LANGUAGE.md) β€” Capability ladder and steps to add a language. +- [docs/TELEMETRY.md](docs/TELEMETRY.md) β€” Telemetry payload schema and privacy controls. +- [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md) β€” Contribution workflow and guidelines. + +--- -## Licence +## πŸ“„ Licence -Apache-2.0 +[AGPL-2.0](LICENSE) (Affero General Public License Version 2) diff --git a/pyproject.toml b/pyproject.toml index df1f21c..4716a94 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -8,7 +8,7 @@ version = "3.1.0" description = "Deterministic multi-language code graph: control flow, data flow and cross-service contracts, for humans and coding agents." readme = "README.md" requires-python = ">=3.10" -license = { text = "Apache-2.0" } +license = { text = "AGPL-2.0" } authors = [{ name = "LineageLens contributors" }] classifiers = [ "Programming Language :: Python :: 3",