fix: revalidate a stale cache and skip broken photos (#79)

.dockerignore

@@ -5,6 +5,7 @@ node_modules
 extension
 test
 test-live
+test-support
 benchmark
 webstore
 tasks

.github/workflows/deploy-verify.yml

@@ -81,4 +81,43 @@ jobs:
           curl -fsS -o /dev/null -w "status=%{http_code} size=%{size_download}\n" \
             "$PROD_URL/api/photo-share?url=$(node -e "console.log(encodeURIComponent(process.argv[1]))" "$IMAGE_URL")"
 
+          # Issue #79: the extension revalidates a long-idle cache through this
+          # route. It fails open (a broken route just means stale cards keep
+          # getting served), so nothing else would notice it breaking in
+          # production. Ask about ids /api/nearby-cats just listed plus one
+          # that cannot exist (12 digits is the validator's max length): the
+          # impossible one must never come back, and at least one of the real
+          # ones must (not all -- a cat can be adopted between the two calls).
+          echo "-- POST /api/validate-cats --"
+          VALIDATE_BODY=$(echo "$NEARBY" | jq -c '{ids: ([.cards[0:3][].id] + ["999999999999"])}')
+          VALIDATED=$(curl -fsS -X POST "$PROD_URL/api/validate-cats" \
+            -H "Content-Type: application/json" \
+            -d "$VALIDATE_BODY")
+          echo "$VALIDATED" | jq .
+          node -e '
+            const asked = JSON.parse(process.argv[1]).ids;
+            const got = JSON.parse(process.argv[2]).availableIds;
+            const fake = asked.at(-1);
+            const real = asked.slice(0, -1);
+            if (got.includes(fake)) {
+              console.error("::error::/api/validate-cats reported an impossible id as available.");
+              process.exit(1);
+            }
+            if (!real.some((id) => got.includes(id))) {
+              console.error("::error::/api/validate-cats returned none of the ids /api/nearby-cats had just listed -- the by-id filter may have stopped working.");
+              process.exit(1);
+            }
+            console.log(`ok: ${got.length} of ${asked.length} ids reported available`);
+          ' "$VALIDATE_BODY" "$VALIDATED"
+
+          echo "-- POST /api/validate-cats rejects invalid ids --"
+          STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$PROD_URL/api/validate-cats" \
+            -H "Content-Type: application/json" \
+            -d '{"ids":["not-an-id"]}')
+          echo "status=$STATUS"
+          if [ "$STATUS" != "400" ]; then
+            echo "::error::/api/validate-cats returned $STATUS for an invalid id (expected 400) -- the route may be missing or its input validation broken."
+            exit 1
+          fi
+
           echo "All smoke checks passed -- safe to proceed with store submission."

PRIVACY.md

@@ -1,6 +1,6 @@
 # Privacy Policy for Tabby
 
-_Last updated: 2026-09-22_
+_Last updated: 2026-09-29_
 
 Tabby ("the extension") replaces your new tab page with one real, adoptable cat sourced from [RescueGroups.org](https://rescuegroups.org). It's the same extension package on both the Chrome Web Store and Edge Add-ons, and this policy applies to both. This policy explains what data Tabby collects, how it's used, and how it's stored.
 
@@ -14,17 +14,17 @@ Tabby does not collect your name, email address, browsing history, or any other
 
 ## How Data Is Used
 
-Your location or ZIP code is sent to Tabby's own backend server, which uses it to search RescueGroups.org for adoptable cats near that location. Nothing else is sent — no browsing history, no device identifiers, no data from other tabs or sites.
+Your location or ZIP code is sent to Tabby's own backend server, which uses it to search RescueGroups.org for adoptable cats near that location. Separately, at most about once a week, Tabby sends the public listing IDs of the cats it has cached on your device (not your location, and nothing that identifies you) to the same backend, which passes them to RescueGroups.org to check whether those listings are still available so removed ones can be dropped. The backend does not store them. Nothing else is sent — no browsing history, no device identifiers, no data from other tabs or sites.
 
 ## How Data Is Stored
 
-Your location/ZIP code and the most recently fetched batch of cat listings are stored locally on your device, using the browser's local extension storage (`chrome.storage.local` — the same API on both Chrome and Edge, since Edge is Chromium-based). This data is **not** synced to Google's, Microsoft's, or any other cloud service, and never leaves your device except for the single search request described above.
+Your location/ZIP code and the most recently fetched batch of cat listings are stored locally on your device, using the browser's local extension storage (`chrome.storage.local` — the same API on both Chrome and Edge, since Edge is Chromium-based). This data is **not** synced to Google's, Microsoft's, or any other cloud service, and never leaves your device except for the search request and the periodic availability check described above.
 
 On the server side, Tabby's backend keeps a short-lived (a few minutes) cache of search results, keyed only by a rounded location and page number — never by anything that identifies you personally, such as an IP address, account, or device ID. This cache exists purely to avoid making duplicate requests to RescueGroups.org and is not linked to you as an individual.
 
 ## Third-Party Services
 
-Tabby's backend queries the [RescueGroups.org](https://rescuegroups.org) public API to find adoptable cats. Your search location (ZIP code or coordinates) is sent to RescueGroups.org as part of that search — this is the only third party that ever receives your location, and only for the purpose of returning matching adoptable-cat listings. See [RescueGroups.org's own privacy policy](https://rescuegroups.org/privacy-policy/) for how they handle that request.
+Tabby's backend queries the [RescueGroups.org](https://rescuegroups.org) public API to find adoptable cats. Your search location (ZIP code or coordinates) is sent to RescueGroups.org as part of that search — this is the only third party that ever receives your location, and only for the purpose of returning matching adoptable-cat listings. It also receives the public listing IDs described above, with no location attached, to confirm those listings are still available. See [RescueGroups.org's own privacy policy](https://rescuegroups.org/privacy-policy/) for how they handle that request.
 
 Tabby also uses Chrome Web Store's built-in GA4 analytics, as described above — this collects only basic, aggregate usage metrics, not your location or any other data described in this policy. Tabby does not use any advertising or crash-reporting service. No data is sold, rented, or shared with any party other than RescueGroups.org and Google/Chrome Web Store as described in this policy.
 

README.md

@@ -6,7 +6,7 @@ Tabby is a Manifest V3 extension (Chrome and Edge) that replaces the new tab pag
 
 ## How Tabby works
 
-- New-tab UI with instant cached-card rendering and stale-while-revalidate refresh: the last-fetched batch renders immediately from `chrome.storage.local`, and a background refresh only fires once the cache is at least 5 minutes old **and** the user has seen at least 85% of the cached cards — so a batch the user hasn't finished browsing isn't discarded early.
+- New-tab UI with instant cached-card rendering and stale-while-revalidate refresh: the last-fetched batch renders immediately from `chrome.storage.local`, and a background refresh only fires once the cache is at least 5 minutes old **and** the user has seen at least 85% of the cached cards — so a batch the user hasn't finished browsing isn't discarded early. A batch that's sat unfinished can still go stale, though (listings get adopted or removed), so once its cards haven't been checked in 7 days, the next new tab first asks the backend (`POST /api/validate-cats`, one RescueGroups request per up-to-100 ids) which of them are still available and drops the rest: every still-available unseen card is kept, `page` never advances, and any failure just serves the cache as before (issue #79). If a photo fails to load anyway, Tabby skips ahead to another unseen cat instead of leaving a broken image on screen.
 - Browser-coordinate lookup with native postal-code fallback.
 - Server-side 25 -> 75 -> 150 -> 250 mile radius ladder: widens the radius, deduplicating by cat ID across steps, until at least 40 unique cats are accumulated or the 250-mile step is reached, whichever comes first — 40 is a floor, not a target. Whatever's accumulated is then capped at 100 cats (closest-first) before being sent to the extension.
 - RescueGroups `available/cats/haspic` query — only cats in "available" status with at least one photo are ever shown — plus nearest-first sorting, picture validation, organization join, and safe profile-url fallback.
@@ -47,7 +47,7 @@ Deploying the server is a separate step from packaging the extension; whatever h
 
 The in-memory cache is correct as-is for the intended deployment target: a single persistent Node process (for example Render, Railway, Fly.io, or Northflank). It would need to be replaced with a shared cache (for example KV/Redis) only if the server is ever scaled to multiple concurrent instances, or moved to a serverless/edge platform (Vercel functions, Cloudflare Workers) where in-process state isn't reliably shared or persistent between requests — those platforms would also require restructuring `server/index.js` away from its current `node:http` `createServer` model.
 
-`/api/nearby-cats` is also rate-limited per client IP (30 requests / 5 minutes, in-memory, same deployment assumption as the cache above) — a cache miss costs a real RescueGroups API call, so this bounds how much a script varying postal codes/coordinates can cost regardless of the response cache. The client IP is taken from `X-Forwarded-For` when present (Northflank and similar platforms terminate the real connection and forward, so `request.socket.remoteAddress` alone would otherwise be the platform's internal proxy address for every request), falling back to the raw socket address only when that header is absent, as in local dev. If a future host doesn't set `X-Forwarded-For` in front of this server, every request would be seen as one shared IP.
+`/api/nearby-cats` and `/api/validate-cats` (never cached: each call is one real RescueGroups request) are also rate-limited per client IP, sharing one budget (30 requests / 5 minutes, in-memory, same deployment assumption as the cache above) — a cache miss costs a real RescueGroups API call, so this bounds how much a script varying postal codes/coordinates can cost regardless of the response cache. The client IP is taken from `X-Forwarded-For` when present (Northflank and similar platforms terminate the real connection and forward, so `request.socket.remoteAddress` alone would otherwise be the platform's internal proxy address for every request), falling back to the raw socket address only when that header is absent, as in local dev. If a future host doesn't set `X-Forwarded-For` in front of this server, every request would be seen as one shared IP.
 
 - `ALERT_WEBHOOK_URL` — optional. When set, the server posts a Discord-compatible webhook message (a JSON body with a `content` field) whenever upstream RescueGroups failures spike: 5+ failures within a rolling 10-minute window, with a 30-minute cooldown between alerts so a sustained outage doesn't spam the channel. Left unset, alerting is a no-op — this exists because a real RescueGroups connectivity incident once went undetected for hours with errors only reaching server logs.
 
@@ -63,9 +63,9 @@ This bumps `manifest.json`/`package.json` to the given version and zips `manifes
 
 Three GitHub Actions workflows automate the release process end to end — see [RELEASE.md](RELEASE.md) for the step-by-step runbook:
 
-- **`ci.yml`** — runs `npm test` on every push and pull request to `main`/`dev`, pinned to Node 22 to match the Dockerfile. `main` has a ruleset (Settings → Rules → Rulesets) requiring this check to pass and requiring a pull request before merging — the workflow alone doesn't block anything, only the ruleset does.
+- **`ci.yml`** — runs `npm run lint` and `npm test` on every push and pull request to `main`/`dev`, pinned to Node 22 to match the Dockerfile. `main` has a ruleset (Settings → Rules → Rulesets) requiring this check to pass and requiring a pull request before merging — the workflow alone doesn't block anything, only the ruleset does.
 - **`tag-release.yml`** — on every push to `main`, tags the commit `v<version>` (read from `manifest.json`) if that tag doesn't already exist. Idempotent, so it's safe to fire on every push rather than needing to detect "was this actually a release."
-- **`deploy-verify.yml`** — on every push to `main` that touches `server/**` or `Dockerfile` (mirroring the `tabby` service's own Northflank build trigger), polls the live `/healthz` endpoint until its `sha` field (see below) matches the pushed commit, then smoke-tests `/api/nearby-cats`, `/api/photo-thumb`, and `/api/photo-share` against production. A **green run is the signal that it's safe to build and submit the release to CWS/EWS** — since Northflank deploys in minutes and store review takes hours, the server is always live and correct well before any user's browser updates to a new extension version, as long as server changes stay additive/backward-compatible with whatever extension version is still in the wild.
+- **`deploy-verify.yml`** — on every push to `main` that touches `server/**` or `Dockerfile` (mirroring the `tabby` service's own Northflank build trigger), polls the live `/healthz` endpoint until its `sha` field (see below) matches the pushed commit, then smoke-tests `/api/nearby-cats`, `/api/validate-cats`, `/api/photo-thumb`, and `/api/photo-share` against production. A **green run is the signal that it's safe to build and submit the release to CWS/EWS** — since Northflank deploys in minutes and store review takes hours, the server is always live and correct well before any user's browser updates to a new extension version, as long as server changes stay additive/backward-compatible with whatever extension version is still in the wild.
 
 `/healthz` reports `{ status: "ok", sha }`, where `sha` is Northflank's auto-injected `NF_DEPLOYMENT_SHA` runtime env var (the exact git commit of the running build) — `null` locally, where that variable is never set. This is what lets `deploy-verify.yml` confirm the *new* code is actually live, not just that some process answered the health check.
 
@@ -92,10 +92,25 @@ npm.cmd test
 
 Both run in CI as part of the same required `test` check; a lint failure blocks merge exactly like a test failure. Linting is `eslint.config.js`, no separate config file per directory — it enforces `no-eval`/`no-implied-eval`/`no-new-func`/`no-script-url` repo-wide (see AGENTS.md's "Security-First Coding") on top of `eslint:recommended`, deliberately without a formatter (no Prettier) or stylistic rules beyond that.
 
-Run the live RescueGroups integration test once against the real API before merging any change to search radius, pagination, or the RescueGroups query contract, to confirm the pagination contract still holds. It requires a real `RG_API_KEY` (loaded from `.env`, same as `start:server`) and is excluded from `npm test`/CI by design:
+Run the live tests against the real API before every release (see [RELEASE.md](RELEASE.md)) and before merging any change to search radius, pagination, or the RescueGroups query contract. `test:live` runs everything in `test-live/`: the pagination/radius contract and the by-id availability contract (`rescuegroups.live.js`), and the stale-cache revalidation flow end to end (`stale-cache.live.js`, which starts the real server in-process and drives the real `newtab.js` against the real API, so nothing needs to be running first). It requires a real `RG_API_KEY` (loaded from `.env`, same as `start:server`) and is excluded from `npm test`/CI by design:
 
 ```powershell
 npm.cmd run test:live
 ```
 
+`npm test` also includes `test/revalidation-integration.test.js`, which runs the real `newtab.js` (jsdom) against the real server with only RescueGroups faked, so the extension/server contract is covered in CI and not just each half against a mock of the other; `test-support/` holds the harness it shares with the live suite. `test/deploy-coverage.test.js` fails if a server route is added without a `deploy-verify.yml` smoke check and a mention here and in RELEASE.md, or if a file in `test-live/` isn't wired into `test:live`.
+
+### Manual QA: simulating an old cache
+
+The stale-cache check (issue #79) only fires for a cache that's 7+ days old, so seeing it in a real browser means aging one by hand. With the unpacked extension loaded and `npm run start:server` running on this branch, open a new tab, press F12, and in the console (type `allow pasting` first if Chrome asks):
+
+```js
+const { feedCache } = await chrome.storage.local.get("feedCache");
+const old = Date.now() - 8 * 24 * 60 * 60 * 1000;
+const fake = { ...feedCache.cards[0], id: "999999999999", name: "FAKE DEAD CAT" };
+await chrome.storage.local.set({ feedCache: { ...feedCache, cards: [fake, ...feedCache.cards], fetchedAt: old, validatedAt: old } });
+```
+
+Reload the tab. The Network tab should show one `POST /api/validate-cats`, "FAKE DEAD CAT" must never appear, and re-reading `feedCache` should show it gone with `page` unchanged, `fetchedAt` still 8 days old, and `validatedAt` about now. Also worth trying: stop the server first (a cat should still render with no notice, and `validationRetryAfter` should be set about an hour out), and set a few unseen cards' `imageUrl` to a bogus URL (a broken photo should skip ahead silently).
+
 The project design follows the architecture document in the parent workspace. The API key is intentionally absent from all source files.