diff --git a/README.md b/README.md index ec780dc..b77f92a 100644 --- a/README.md +++ b/README.md @@ -395,10 +395,16 @@ the GitHub URL). | Code | [github.com/lukefwalton/answer-engine](https://github.com/lukefwalton/answer-engine) | | About | [lukefwalton.com/ask/about/](https://lukefwalton.com/ask/about/) | -**Artifact note:** cite [10.5281/zenodo.20710897](https://doi.org/10.5281/zenodo.20710897) -for v1.2 of the formal write-up ([`docs/ARTIFACT-NOTE-v1.2.md`](./docs/ARTIFACT-NOTE-v1.2.md)). -Its concept DOI, [10.5281/zenodo.20686053](https://doi.org/10.5281/zenodo.20686053), -is separate from the software archive above and resolves to the latest version. +**Software paper:** [`paper.md`](./paper.md) (references in [`paper.bib`](./paper.bib)) +describes the v3 substrate: the disclosure-typed contract, its design, and its +evaluation. It is written in the Journal of Open Source Software form and is not yet +submitted; until it has a venue, cite the software DOI above. + +**Technical note:** cite [10.5281/zenodo.20710897](https://doi.org/10.5281/zenodo.20710897) +for v1.2 of the earlier technical note on the v1 engine +([`docs/ARTIFACT-NOTE-v1.2.md`](./docs/ARTIFACT-NOTE-v1.2.md)). Its concept DOI, +[10.5281/zenodo.20686053](https://doi.org/10.5281/zenodo.20686053), is separate from +the software archive above and resolves to the latest version. To pin a specific archived snapshot, pick that release's version DOI on the [Zenodo versions page](https://zenodo.org/records/20676773) — no README update @@ -442,7 +448,8 @@ the latest tag already advanced. ## Related writing -Formal description of this implementation: +The software paper for the v3 substrate: [`paper.md`](./paper.md). The earlier +technical note on the v1 engine: [`docs/ARTIFACT-NOTE-v1.2.md`](./docs/ARTIFACT-NOTE-v1.2.md) — [DOI](https://doi.org/10.5281/zenodo.20710897) (CC BY-NC-ND 4.0). diff --git a/paper.bib b/paper.bib new file mode 100644 index 0000000..2c6b2fc --- /dev/null +++ b/paper.bib @@ -0,0 +1,186 @@ +@inproceedings{lewis2020, + author = {Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K{\"u}ttler, Heinrich and Lewis, Mike and Yih, Wen-tau and Rockt{\"a}schel, Tim and Riedel, Sebastian and Kiela, Douwe}, + title = {Retrieval-Augmented Generation for Knowledge-Intensive {NLP} Tasks}, + booktitle = {Advances in Neural Information Processing Systems 33 (NeurIPS 2020)}, + year = {2020}, + doi = {10.48550/arXiv.2005.11401} +} + +@article{gao2023, + author = {Gao, Yunfan and Xiong, Yun and Gao, Xinyu and Jia, Kangxiang and Pan, Jinliu and Bi, Yuxi and Dai, Yi and Sun, Jiawei and Wang, Meng and Wang, Haofen}, + title = {Retrieval-Augmented Generation for Large Language Models: A Survey}, + journal = {arXiv preprint}, + year = {2023}, + doi = {10.48550/arXiv.2312.10997} +} + +@article{rashkin2023, + author = {Rashkin, Hannah and Nikolaev, Vitaly and Lamm, Matthew and Aroyo, Lora and Collins, Michael and Das, Dipanjan and Petrov, Slav and Tomar, Gaurav Singh and Turc, Iulia and Reitter, David}, + title = {Measuring Attribution in Natural Language Generation Models}, + journal = {Computational Linguistics}, + volume = {49}, + number = {4}, + pages = {777--840}, + year = {2023}, + doi = {10.1162/coli_a_00486} +} + +@inproceedings{liu2023, + author = {Liu, Nelson F. and Zhang, Tianyi and Liang, Percy}, + title = {Evaluating Verifiability in Generative Search Engines}, + booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023}, + year = {2023}, + pages = {7001--7025}, + doi = {10.18653/v1/2023.findings-emnlp.467} +} + +@article{bohnet2022, + author = {Bohnet, Bernd and Tran, Vinh Q. and Verga, Pat and Aharoni, Roee and Andor, Daniel and Soares, Livio Baldini and Ciaramita, Massimiliano and Eisenstein, Jacob and Ganchev, Kuzman and Herzig, Jonathan and Hui, Kai and Kwiatkowski, Tom and Ma, Ji and Ni, Jianmo and Saralegui, Lierni Sestorain and Schuster, Tal and Cohen, William W. and Collins, Michael and Das, Dipanjan and Metzler, Donald and Petrov, Slav and Webster, Kellie}, + title = {Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models}, + journal = {arXiv preprint}, + year = {2022}, + doi = {10.48550/arXiv.2212.08037} +} + +@inproceedings{rebedea2023, + author = {Rebedea, Traian and Dinu, Razvan and Sreedhar, Makesh and Parisien, Christopher and Cohen, Jonathan}, + title = {{NeMo} Guardrails: A Toolkit for Controllable and Safe {LLM} Applications with Programmable Rails}, + booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations}, + year = {2023}, + pages = {431--445}, + doi = {10.18653/v1/2023.emnlp-demo.40} +} + +@article{shoemaker2011, + author = {Shoemaker, David}, + title = {Attributability, Answerability, and Accountability: Toward a Wider Theory of Moral Responsibility}, + journal = {Ethics}, + volume = {121}, + number = {3}, + pages = {602--632}, + year = {2011}, + doi = {10.1086/659003} +} + +@article{sabelfeld2003, + author = {Sabelfeld, Andrei and Myers, Andrew C.}, + title = {Language-Based Information-Flow Security}, + journal = {IEEE Journal on Selected Areas in Communications}, + volume = {21}, + number = {1}, + pages = {5--19}, + year = {2003}, + doi = {10.1109/JSAC.2002.806121} +} + +@inproceedings{goguen1982, + author = {Goguen, Joseph A. and Meseguer, Jos{\'e}}, + title = {Security Policies and Security Models}, + booktitle = {Proceedings of the 1982 IEEE Symposium on Security and Privacy}, + year = {1982}, + pages = {11--20}, + doi = {10.1109/SP.1982.10014} +} + +@misc{mcp2026, + author = {{Model Context Protocol project}}, + title = {Model Context Protocol specification, revision 2026-07-28}, + year = {2026}, + url = {https://modelcontextprotocol.io/specification/2026-07-28} +} + +@misc{langchain, + author = {Chase, Harrison}, + title = {{LangChain}}, + year = {2022}, + url = {https://github.com/langchain-ai/langchain} +} + +@misc{llamaindex, + author = {Liu, Jerry}, + title = {{LlamaIndex}}, + year = {2022}, + url = {https://github.com/run-llama/llama_index} +} + +@misc{walton2026a, + author = {Walton, Luke F.}, + title = {The Decision No One Authored: The Answerability Gap in Generative {AI}}, + year = {2026}, + publisher = {Zenodo}, + doi = {10.5281/zenodo.20614374} +} + +@misc{walton2026b, + author = {Walton, Luke F.}, + title = {The Captured Oracle: Authorship and Agency in the Ethics of Answer-Engine Optimization}, + year = {2026}, + publisher = {Zenodo}, + doi = {10.5281/zenodo.20676327} +} + +@misc{walton2026c, + author = {Walton, Luke F.}, + title = {The Invariant of Answerability}, + year = {2026}, + publisher = {Zenodo}, + doi = {10.5281/zenodo.20606492} +} + +@misc{walton2026d, + author = {Walton, Luke F.}, + title = {Building Answerable {AI}: Why Automation Needs Owned Error}, + year = {2026}, + publisher = {Zenodo}, + doi = {10.5281/zenodo.20682306} +} + +@misc{walton2026note, + author = {Walton, Luke F.}, + title = {Answer Engine: A Small Reference Implementation for Citation-Grounded Answerability}, + year = {2026}, + publisher = {Zenodo}, + doi = {10.5281/zenodo.20686053} +} + +@software{walton2026software, + author = {Walton, Luke F.}, + title = {Answer Engine}, + year = {2026}, + publisher = {Zenodo}, + doi = {10.5281/zenodo.20676773}, + url = {https://github.com/lukefwalton/answer-engine} +} + +@article{almasoud2026, + author = {Al Masoud, Aiman and Arazzi, Marco and Nocera, Antonino}, + title = {{SD-RAG}: A Prompt-Injection-Resilient Framework for Selective Disclosure in Retrieval-Augmented Generation}, + journal = {arXiv preprint}, + year = {2026}, + doi = {10.48550/arXiv.2601.11199} +} + +@inproceedings{rasul2026, + author = {Rasul, Md Fazle and Aghayarzadeh, Hamed and Ray, Indrakshi}, + title = {Information Flow Control for Retrieval-Augmented {LLM} Chatbots}, + booktitle = {Data and Applications Security and Privacy XL: 40th IFIP WG 11.3 Annual Conference, DBSec 2026}, + publisher = {Springer}, + year = {2026}, + doi = {10.1007/978-3-032-33260-8_17} +} + +@article{siddiqui2025, + author = {Siddiqui, Shoaib Ahmed and Gaonkar, Radhika and K{\"o}pf, Boris and Krueger, David and Paverd, Andrew and Salem, Ahmed and Tople, Shruti and Wutschitz, Lukas and Xia, Menglin and Zanella-B{\'e}guelin, Santiago}, + title = {Permissive Information-Flow Analysis for Large Language Models}, + journal = {Transactions on Machine Learning Research}, + year = {2025}, + url = {https://arxiv.org/abs/2410.03055} +} + +@article{xu2026, + author = {Xu, Jinghan and Fan, Longze and Wang, Zeyuan and Li, Xinjin and Liu, Hankai}, + title = {{MNC}: Scope-Bound Semantic Declassification for Private {LLM}-Agent Communication}, + journal = {arXiv preprint}, + year = {2026}, + doi = {10.48550/arXiv.2608.01719} +} diff --git a/paper.md b/paper.md new file mode 100644 index 0000000..1012e66 --- /dev/null +++ b/paper.md @@ -0,0 +1,63 @@ +--- +title: 'Answer Engine: a disclosure-typed retrieval substrate for an author-owned archive' +tags: + - TypeScript + - retrieval-augmented generation + - information disclosure + - provenance + - citation grounding + - Model Context Protocol +authors: + - name: Luke F. Walton + orcid: 0009-0005-9263-1954 + affiliation: 1 +affiliations: + - name: Independent Researcher + index: 1 +date: 7 October 2026 +bibliography: paper.bib +--- + +# Summary + +`answer-engine` is a small TypeScript package for searching a body of work its author owns, some of it published and some not, and for handing the results to a language model or an external caller without any text of the unpublished part leaving the server. Every retrievable fragment carries a disclosure policy, resolved and stored before any query: which layer its text is in (`public` or `private`) and how much of it may travel (`text`; a short `semantic` description the author released; a `locator` that says only where the moment is; or `none`). One function, `project()`, is the single crossing from retrieval to the wire type a consumer may serialize or prompt from, a union whose private variants have no field for a fragment's text; `private + text` is not a member of the policy type, so the compiler refuses it in code and the index validator refuses it in data. A private fragment can therefore still do work: it is found, it returns as a locator or an authorized description, and every hit carries provenance (creators, date, identifiers, and a locator such as `p. 184` or `12:30–14:05`) a caller can cite with no synthesis on the server. The package provides the contract types, ingest, retrieval, the two index formats, a reference consumer that cites or refuses, and an evaluation harness whose gold suite and canary sweep gate every change. Tests and demo run keyless; the package is on npm and in production. + +# Statement of need + +Retrieval-augmented generation [@lewis2020; @gao2023] assumes that what retrieval returns may be shown to the model, and the Model Context Protocol [@mcp2026] now lets a model the operator does not control call a search tool directly. For an author's archive it is often false: an unpublished podcast transcript, a manuscript under contract, a notebook. The operator's options have been to leave such material out of retrieval, to let it in and filter or redact what comes back, or to trust a prompt. The first two decide whether a chunk may be seen at all; neither lets a private fragment contribute to an answer by being found, located, and described while none of its text travels. The third is a promise made to the model, not a property of the data. + +The research question the software tests is whether that disclosure can be made a property of the data: fixed per fragment before any query, carried on the wire as the only representation a consumer can receive, and unrepresentable in the type when forbidden. The question comes from work on answerability in generative systems [@walton2026a; @walton2026b; @walton2026c; @walton2026d], Shoemaker's standing to be asked for the judgment behind an act [@shoemaker2011]: the frame a system acts under must have an owner who can be reached [@walton2026b], and an operation no one answers for should be unavailable as an ordinary operation of the system [@walton2026d]. What may travel about a retrieved private fragment is one such operation. `answer-engine` is that argument's materials, the runnable link between its design theory and a live deployment, offered to the conversation on selective disclosure in retrieval-augmented systems. + +# State of the field + +Retrieval frameworks such as LangChain [@langchain] and LlamaIndex [@llamaindex] return chunks with free-form metadata and leave what may be shown to the model to the application; their access controls decide which whole chunks a caller may retrieve. Attributed and verifiable generation [@bohnet2022; @rashkin2023; @liu2023] measures whether an answer is supported by its citations after the fact, and runtime guardrails [@rebedea2023] filter inputs and outputs with programmable rules. + +Two recent lines of work come closest. SD-RAG [@almasoud2026] shares the aim, selective disclosure in retrieval-augmented generation, and the move of taking the decision away from the answering model: it carries constraints in the data model and sanitizes retrieved material before it reaches the generator. MNC [@xu2026] shares the vocabulary: a typed semantic-declassification protocol between agents that selects a task-sufficient disclosure from supplied candidates and binds it to recipient, purpose, and lifetime under a reference monitor; it governs what an agent may say onward, per message, not how a retrieval result is represented. Nearby, @rasul2026 add an information-flow monitor to a retrieval-augmented chatbot's serving path, and @siddiqui2025 propagate confidentiality labels through a model's output. In each, the permitted representation of a retrieved item is computed in response to the query, by a filter, a monitor, or a model, from text present at that moment. Here it is fixed before any query, nothing is generated in response to one, and the served artifact holds nothing else. The older lineage is language-based information-flow control [@goguen1982; @sabelfeld2003], where a forbidden flow is made inexpressible rather than filtered; the package borrows that move for the form of a hit, and only there. To the author's knowledge, no retrieval package combines these concerns in a disclosure-typed retrieval contract: one representation per fragment fixed before any query, forbidden representations uninhabitable in the consumer-facing type, and the drafting of a released description separated from the authority to release it. + +# Software design + +The contract (`docs/CONTRACT.md`) is the design of record. An **entity** (a song, a transcript, a book) declares a default disclosure; its **fragments** (a chapter, a transcript window) may narrow the exposure but never change the layer. Disclosure is resolved at build time by a function that only downgrades: a requested `semantic` becomes `locator` when its description is missing, failed the lint, was vetoed, went stale, or awaits a required review. + +**One crossing.** `retrieve()` returns scored fragments to the process that holds the index; `project()` is the only path from them to `EvidenceHit`, a discriminated union on `exposure` whose `locator` variant carries provenance and nothing else and whose `semantic` variant carries a branded `LintedGist` only the lint can construct. The private index's validator refuses `private + text` in data, where types do not reach. Scores on private hits are rounded and carry no per-plugin breakdown, because a full-precision similarity against a private vector is a measurement of it. The type fixes the form of a hit; what a sequence of hits reveals (which fragment answered, and roughly how well) is a residue the contract names and assigns to the consumer's rate limiting and the author's policy, `none` included. + +**Authorship and authorization are separate.** A description is drafted by a model at ingest, stored beside the index as private material, and released only by the entity's policy, a per-fragment override, or the author's edit, veto, or review. Release by policy authorizes a class of machine drafts the author need not have read; release by edit or review authorizes text the author has seen. The build-time lint is a tripwire, not a classifier: it refuses any run of five words shared with the fragment or its whole entity and catches nothing shorter. Close paraphrase, names, and spoilers are owned by policy, veto, and the evaluation's canaries. + +**Two artifacts, one validator.** The private index holds text and vectors and re-earns every lint at each load; the served index is the same file after a strip that blanks text and drops every projection a policy did not release, and a load-time check verifies the strip. The lint cannot run where the text is blank, so a served index is trusted to descend from a validated private one. In deployment the strip keeps text off the wire; the type states and checks the contract and binds a consumer holding the private index. + +**Evaluation is the admission gate.** A fixed gold set with required behaviours (answer, route to private material without quoting it, refuse) runs keyless at the retrieval tier, and a canary sweep checks every served gist against private phrases the author lists, so an exposure change is gated like a scoring change. + +Retrieval is brute-force cosine over an in-memory index; HTTP, transport, and site-specific ranking live in consumers. + +# Research impact statement + +The package is the substrate under Ask the Archive (lukefwalton.com/ask), a production deployment over the author's archive of roughly ten thousand embedded fragments, where published passages are public and unpublished transcript text is private: embedded for search, served as where to listen, never as the text itself. The same deployment exposes a rate-limited retrieval-only endpoint and an MCP server through which an external model receives hits at the exposure the author set, with no synthesis on the server, and it carries an unpublished novel as a private entity whose chapters are served as locators. The `semantic` exposure is exercised by the package's tests and demo and reaches the deployment only as a policy change and a rebuild. A gold suite of 311 queries, 28 of which must refuse, every row carrying hard-fail guards against leakage and misattribution, gates every change to that deployment. The software is cited as the materials of the answerability working paper [@walton2026d], documented in a technical note [@walton2026note], and archived under a concept DOI [@walton2026software]. What it shows is the feasibility of one architectural claim in those papers: that a class of normative juncture, here what may travel about a retrieved private fragment, can be moved out of model behaviour into representable operations. It does not bear on the working paper's empirical claim about how correction compounds into capability, which that paper leaves unmeasured. No use by other research groups is claimed. + +# AI usage disclosure + +Generative AI was used in the software and in this paper. Of the 169 commits on the default branch at the time of writing, 69 are authored as Claude (Claude Code, with Claude Opus 4.8, Claude Fable 5, and Claude Fable 5.1, Anthropic), working from the author's written specifications; every change was reviewed in a pull request, with an automated review (Surmado Code Review), an offline test suite, and the gold evaluation as the admission gate. Documentation and this paper were drafted with the same tools under the author's direction and edited by the author. OpenAI models are runtime components of the software (`text-embedding-3-large` for embedding; a Responses API model for gist drafting and for the reference consumer's synthesis); no evaluation result reported here was generated by an AI system. The author reviewed, edited, and validated all AI-assisted output and is responsible for it. + +# Acknowledgements + +This work received no funding. The author is the founder of Surmado, Inc., which builds managed AI systems of the kind this software describes; the paper is written in a personal capacity. + +# References