Production-resilient TypeScript engine for PDF document text chunking, Google Gemini vector embeddings, and local ChromaDB HNSW semantic vector search.
-
Recursive Sentence-Aware Chunking: Hierarchical text splitting (
\n\n$\rightarrow$ \n$\rightarrow$ .$\rightarrow$ -
Resilient API Processing: Exponential backoff with delay jitter retries to handle Gemini API rate limits (
HTTP 429) and network dropouts. -
Memory Efficient Ingestion:
$O(1)$ streaming batch flushes to prevent heap memory exhaustion on large documents. - Relevance Guardrails: Bounded similarity scoring with minimum threshold filtering.
Install as a dependency in your Node.js or TypeScript project:
npm install git+https://github.com/DileepWick/vector-doc-engine.gitOr via GitHub Packages registry:
npm install @dileepwick/vector-doc-engineCreate a .env file in your root directory:
CHROMA_URL=http://localhost:8000
GEMINI_API_KEY=your_gemini_api_key
GEMINI_EMBED_MODEL=gemini-embedding-2-previewStart a local ChromaDB instance:
docker run -p 8000:8000 chromadb/chromaView Programmatic Code Examples
Ingest PDF Document
import { ingestPdfDocument } from "@dileepwick/vector-doc-engine";
const result = await ingestPdfDocument({
filePath: "./data/documents/sem-reg.pdf",
});
console.log(`Ingested ${result.totalChunks} chunks.`);Perform Vector Search Query
import { queryVectorSearch } from "@dileepwick/vector-doc-engine";
const matches = await queryVectorSearch({
query: "What are the main key takeaways?",
topK: 3,
minSimilarity: 0.35,
});
matches.forEach((match, idx) => {
console.log(`[${idx + 1}] Score: ${match.score.toFixed(4)} | Excerpt: "${match.doc}"`);
});View CLI Execution Commands
Build Package
npm run buildIngest Document via CLI
npx ts-node src/ingest.ts sem-reg.pdfQuery Vector Search via CLI
npx ts-node src/ask.ts "What are the key takeaways?"vector-doc-engine/
├── src/
│ ├── index.ts # Public API exports
│ ├── ingest.ts # Document ingestion API
│ ├── ask.ts # Vector search query API
│ ├── chunker.ts # Recursive text chunking utility
│ └── embedder.ts # Gemini API embedder with retries
├── data/documents/ # Default document storage
├── dist/ # Built JavaScript binaries & declaration files
├── docs/ # Technical documentation & failure mode audits
└── tests/ # Unit test suites (Jest)