Repository navigation
Adding Cacheing - #48
MGoradeCodes wants to merge 3 commits into
Conversation
|
Warning Some commit messages in this PR do not follow the Conventional Commits specification. Expected format: 📋 Invalid commit messages
Tip Valid types: Note This is just a friendly reminder and will not block the PR from being merged. |
Summary by BeetleThis PR introduces a semantic caching layer to the RAEL Discord bot's AI command system. The implementation adds intelligent query caching using vector embeddings and SQLite, allowing the bot to recognize and reuse answers for semantically similar questions without making redundant API calls. This optimization reduces latency and API costs for frequently asked or similar questions. 📁 File Changes Summary (Consolidated across all commits):
Total Changes: 5 files changed, +2132 additions, -0 deletions 🗺️ Walkthrough:graph TD
UserQuery["User Asks Question"]
CheckHistory{"Has Chat History?"}
CheckCache["Query Semantic Cache"]
CacheHit{"Similar Answer Found?"}
APICall["Call AI API"]
GetEmbedding["Generate Query Embedding OpenAI text-embedding-3-small"]
VectorSearch["Search SQLite Vector DB Cosine Distance < 0.15"]
StoreCache["Store Question + Answer + Embedding Vector"]
ReturnAnswer["Return Answer to User"]
UserQuery --> CheckHistory
CheckHistory -->|Yes| APICall
CheckHistory -->|No| CheckCache
CheckCache --> GetEmbedding
GetEmbedding --> VectorSearch
VectorSearch --> CacheHit
CacheHit -->|Yes| ReturnAnswer
CacheHit -->|No| APICall
APICall --> StoreCache
StoreCache --> ReturnAnswer
🎯 Key Changes:
📊 Impact Assessment:
⚙️ SettingsSeverity Threshold: 📖 User Guide
|
![beetle-ai[bot]](https://avatars.githubusercontent.com/in/1636950?s=60&v=4){kind=small}
| const history = getContext(userId); | ||
|
|
||
| const hasHistory = Array.isArray(history) && history.length > 0; | ||
| if (!hasHistory && !ctx) { |
There was a problem hiding this comment.
Undefined variable ctx in condition. Line 75 checks !ctx but ctx is not defined in the function scope. This will cause a ReferenceError at runtime. The condition logic appears incomplete or references a parameter that doesn't exist.
Confidence: 4/5
Suggested Fix
Verify what this condition should actually check. Based on the context, it seems like you want to check if there's no existing context for the user. The condition should likely be:
| if (!hasHistory && !ctx) { | |
| if (!hasHistory) { | |
Or if you need to check for a specific parameter, ensure that parameter is defined in the function signature and passed correctly.
| @@ -84,6 +104,12 @@ export default { | |||
| return; | |||
| } | |||
|
|
|||
|
|
|||
| if (!hasHistory && !ctx) { | |||
| await semanticCache.set(question, result.text); | |||
There was a problem hiding this comment.
Race condition in cache operations. Multiple concurrent requests for the same question could all miss the cache (line 76) and make duplicate API calls, then all attempt to store results (line 109). Without locking or deduplication, this defeats the purpose of caching and wastes API calls.
Confidence: 4/5
Suggested Fix
Implement request deduplication by tracking in-flight requests:
| const inFlightRequests = new Map<string, Promise<string>>(); | |
| // In cache retrieval section: | |
| if (!hasHistory) { | |
| const cachedAnswer = await semanticCache.get(question); | |
| if (cachedAnswer) { | |
| addToContext(userId, "user", question); | |
| addToContext(userId, "assistant", cachedAnswer); | |
| await message.reply({ content: pretty(cachedAnswer) }); | |
| return; | |
| } | |
| // Check if request is already in flight | |
| if (inFlightRequests.has(question)) { | |
| const result = await inFlightRequests.get(question); | |
| addToContext(userId, "user", question); | |
| addToContext(userId, "assistant", result); | |
| await message.reply({ content: pretty(result) }); | |
| return; | |
| } | |
This prevents duplicate API calls for the same question when multiple requests arrive simultaneously.
📍 This suggestion applies to lines 74-109
| constructor() { | ||
| this.db = new Database('ai_cache.db'); | ||
| sqliteVec.load(this.db); | ||
|
|
||
| // Simplification: One single table for everything (text data + vector fields combined) | ||
| this.db.prepare(` | ||
| CREATE VIRTUAL TABLE IF NOT EXISTS semantic_cache USING vec0( | ||
| question TEXT, | ||
| answer TEXT, | ||
| question_vector float[1536] | ||
| ); | ||
| `).run(); |
There was a problem hiding this comment.
Database connection is never closed, causing resource leaks. The constructor initializes a SQLite database connection but there's no destructor, close method, or cleanup mechanism. In a long-running Discord bot, this connection will accumulate and eventually exhaust system resources.
Confidence: 4/5
Suggested Fix
Add a close method and ensure it's called when the bot shuts down:
| constructor() { | |
| this.db = new Database('ai_cache.db'); | |
| sqliteVec.load(this.db); | |
| // Simplification: One single table for everything (text data + vector fields combined) | |
| this.db.prepare(` | |
| CREATE VIRTUAL TABLE IF NOT EXISTS semantic_cache USING vec0( | |
| question TEXT, | |
| answer TEXT, | |
| question_vector float[1536] | |
| ); | |
| `).run(); | |
| export class Cache { | |
| private db: Database.Database; | |
| private openai = new OpenAI(); | |
| constructor() { | |
| this.db = new Database('ai_cache.db'); | |
| sqliteVec.load(this.db); | |
| this.db.prepare(` | |
| CREATE VIRTUAL TABLE IF NOT EXISTS semantic_cache USING vec0( | |
| question TEXT, | |
| answer TEXT, | |
| question_vector float[1536] | |
| ); | |
| `).run(); | |
| } | |
| public close(): void { | |
| this.db.close(); | |
| } | |
Then ensure cache.close() is called in your bot's shutdown handler.
Prompt for AI
Copy this prompt to your AI IDE to fix this issue locally:
In src/utils/Cache.ts, add a public close() method that calls this.db.close() to properly clean up the database connection. Then update the bot's shutdown handler to call cache.close() when the bot is shutting down to prevent resource leaks.
📍 This suggestion applies to lines 9-20
| const match = this.db.prepare(` | ||
| SELECT answer FROM semantic_cache | ||
| WHERE vec_distance_cosine(question_vector, ?) < 0.15 | ||
| ORDER BY vec_distance_cosine(question_vector, ?) ASC LIMIT 1; | ||
| `).get(Buffer.from(vector.buffer), Buffer.from(vector.buffer)) as { answer: string } | undefined; |
There was a problem hiding this comment.
Redundant vector distance calculation reduces performance. The query calculates vec_distance_cosine(question_vector, ?) twice - once in the WHERE clause (line 44) and again in the ORDER BY clause (line 45). This causes the distance to be computed twice for every row, wasting CPU cycles.
Confidence: 4/5
Suggested Fix
Restructure the query to calculate distance only once using a subquery or CTE:
| const match = this.db.prepare(` | |
| SELECT answer FROM semantic_cache | |
| WHERE vec_distance_cosine(question_vector, ?) < 0.15 | |
| ORDER BY vec_distance_cosine(question_vector, ?) ASC LIMIT 1; | |
| `).get(Buffer.from(vector.buffer), Buffer.from(vector.buffer)) as { answer: string } | undefined; | |
| const match = this.db.prepare(` | |
| SELECT answer FROM semantic_cache | |
| WHERE vec_distance_cosine(question_vector, ?) < 0.15 | |
| ORDER BY vec_distance_cosine(question_vector, ?) ASC LIMIT 1; | |
| `).get(Buffer.from(vector.buffer), Buffer.from(vector.buffer)) as { answer: string } | undefined; | |
Actually, better-sqlite3 doesn't support CTEs efficiently. Instead, accept that SQLite will optimize this, but consider if the threshold check is necessary - you could remove the WHERE clause and just ORDER BY distance, then check the result:
| const match = this.db.prepare(` | |
| SELECT answer FROM semantic_cache | |
| WHERE vec_distance_cosine(question_vector, ?) < 0.15 | |
| ORDER BY vec_distance_cosine(question_vector, ?) ASC LIMIT 1; | |
| `).get(Buffer.from(vector.buffer), Buffer.from(vector.buffer)) as { answer: string } | undefined; | |
| const match = this.db.prepare(` | |
| SELECT answer FROM semantic_cache | |
| ORDER BY vec_distance_cosine(question_vector, ?) ASC LIMIT 1; | |
| `).get(Buffer.from(vector.buffer)) as { answer: string } | undefined; | |
| if (match && this.calculateDistance(vector) < 0.15) { | |
| return match.answer; | |
| } | |
| return null; | |
Or keep the current approach if SQLite's query optimizer handles it well. Monitor query performance in production.
Prompt for AI
Copy this prompt to your AI IDE to fix this issue locally:
In src/utils/Cache.ts line 42-46, the vec_distance_cosine function is called twice in the same query (WHERE and ORDER BY clauses). Optimize this by either removing the WHERE clause threshold check and validating the distance in code, or by restructuring the query to calculate distance only once. Benchmark the performance impact.
📍 This suggestion applies to lines 42-46
| public async get(userQuestion: string): Promise<string | null> { | ||
| try { | ||
| const vector = await this.getEmbedding(userQuestion); | ||
| const match = this.db.prepare(` | ||
| SELECT answer FROM semantic_cache | ||
| WHERE vec_distance_cosine(question_vector, ?) < 0.15 | ||
| ORDER BY vec_distance_cosine(question_vector, ?) ASC LIMIT 1; | ||
| `).get(Buffer.from(vector.buffer), Buffer.from(vector.buffer)) as { answer: string } | undefined; | ||
|
|
||
| return match ? match.answer : null; | ||
| } catch { | ||
| return null; | ||
| } | ||
| } |
There was a problem hiding this comment.
Cache lacks user isolation, enabling cross-user data leakage. The get() method retrieves cached answers globally without any user context. Combined with the global cache instance in askai.ts, this means cached answers from one user could be served to another user, creating a privacy/security vulnerability where sensitive information leaks between users.
Confidence: 5/5
Suggested Fix
Modify the cache to accept and store user context:
| public async get(userQuestion: string): Promise<string | null> { | |
| try { | |
| const vector = await this.getEmbedding(userQuestion); | |
| const match = this.db.prepare(` | |
| SELECT answer FROM semantic_cache | |
| WHERE vec_distance_cosine(question_vector, ?) < 0.15 | |
| ORDER BY vec_distance_cosine(question_vector, ?) ASC LIMIT 1; | |
| `).get(Buffer.from(vector.buffer), Buffer.from(vector.buffer)) as { answer: string } | undefined; | |
| return match ? match.answer : null; | |
| } catch { | |
| return null; | |
| } | |
| } | |
| public async get(userId: string, userQuestion: string): Promise<string | null> { | |
| try { | |
| const vector = await this.getEmbedding(userQuestion); | |
| const match = this.db.prepare(` | |
| SELECT answer FROM semantic_cache | |
| WHERE user_id = ? AND vec_distance_cosine(question_vector, ?) < 0.15 | |
| ORDER BY vec_distance_cosine(question_vector, ?) ASC LIMIT 1; | |
| `).get(userId, Buffer.from(vector.buffer), Buffer.from(vector.buffer)) as { answer: string } | undefined; | |
| return match ? match.answer : null; | |
| } catch { | |
| return null; | |
| } | |
Also update the table schema to include user_id and the set() method to accept and store userId.
Prompt for AI
Copy this prompt to your AI IDE to fix this issue locally:
In src/utils/Cache.ts, add user isolation to the cache. Modify the semantic_cache table schema to include a user_id column, update both get() and set() methods to accept userId as a parameter, and filter all queries by user_id to prevent cross-user data leakage.
📍 This suggestion applies to lines 39-52
| } catch { | ||
| return null; | ||
| } |
There was a problem hiding this comment.
Silent error suppression hides critical failures. The get() method catches all errors and returns null without logging. If the database is corrupted, the embedding API fails, or other critical errors occur, they're silently ignored, making debugging difficult and potentially causing the cache to fail silently in production.
Confidence: 4/5
Suggested Fix
Log errors appropriately while still handling them gracefully:
| } catch { | |
| return null; | |
| } | |
| } catch (err) { | |
| console.error('[Cache Get Failed]:', err); | |
| return null; | |
| } | |
This matches the error handling pattern already used in the set() method (line 62).
Prompt for AI
Copy this prompt to your AI IDE to fix this issue locally:
In src/utils/Cache.ts line 49-51, add error logging to the catch block in the get() method, similar to the logging in the set() method at line 62. This will help with debugging cache failures in production.
📍 This suggestion applies to lines 49-51
|
hey gng |
|
beetle-ai did some checking then it like found some stuff lol it gives us prompts to tell ai to fix mind if you fix |
|
super linter :sus |
|
Tip Make sure to check the following before pushing:
Then fix the issues, commit, and push again. Note This is just a friendly reminder and will not block the PR from being merged. |
|
linting failed ic |
|
What changes to do you want me to make? |
Beetle is reviewing the changes — Let’s see what you’ve done!🔍 Under Review Commits (1)
Files Changed (2)
⚙️ SettingsSeverity Threshold: 📖 User Guide
|
Beetle is reviewing the changes — Let’s see what you’ve done!🔍 Under Review Commits (1)
Files Changed (1)
⚙️ SettingsSeverity Threshold: 📖 User Guide
|
|
Tip Make sure to check the following before pushing:
Then fix the issues, commit, and push again. Note This is just a friendly reminder and will not block the PR from being merged. |
🚀 Feature: Added SQLite-Powered Semantic Caching for AI Answers
I have added a few comments to code if its easier to understand
Logic is kinda good and bad
since I stripped it off an another project : )
This PR implements a localized, lightweight Semantic Cache for the
askaiengine to drastically optimize latency, token usage, and API billing overhead.IMPORTANT LOG:
Beetle AI is failing with 403: team is blocked: missing payment method. This appears to be an issue with the Beetle AI integration rather than the PR. (Look Into This Please)
💡 How It Works
Instead of checking for exact word-by-word matches, incoming single prompts are mapped using OpenAI's
text-embedding-3-smallvectors. We leverage a high-speed, local SQLite file embedded with thesqlite-vecbinary extension to compute mathematical vector distance math locally.>90%semantic similarity (distance < 0.15), the engine serves the stored response instantly.ctx), the cache bypasses entirely to keep interaction flows coherent.🛠️ Dependencies Added
better-sqlite3: Standard node driver utility wrapper.sqlite-vec: Ultra-lightweight native vector math addition for SQLite virtual tables (vec0).🔒 Safety Precautions Took
ai_cache.dband its relative workspace journal strings to.gitignoreto prevent any local data logs from accidental upstream tracking.READ MEEE.!!!!!!!!!