fix: resolve advisory_packages join in postgres, skip 1.5TB purl scan (CM-1362)

[themarolt]

Summary

bootstrapOsspckgs has failed its last 5 weekly runs at ingestAdvisories — a BQ dry-run guard aborts because advisory_packages scans ~1.5 TB. Root cause: a purl_map CTE joins the entire PackageVersionsLatest table just to attach a purl used for one join key, which we already have locally in Postgres. This drops the BQ scan for that step to ~1.4 GB and stops the failure from stranding scorecard/ranking downstream.

Changes

• Drop the purl_map CTE from buildAdvisoryPackagesSql — no BQ-side purl lookup.
• Resolve advisory_packages.package_id in Postgres instead, joining on the same (ecosystem, namespace, name) identity packages already carries a unique index on (COALESCE(namespace,'') matches the index expression, so this stays an index lookup, not a scan).
• New shared helper queries/pgIdentity.ts (packageNameSplitSql) so the namespace/name split logic isn't duplicated between ingestPackages.ts and ingestAdvisories.ts.
• Drop the purl column from the advisory_packages staging DDL/columns; switch that DDL to DROP TABLE IF EXISTS + CREATE (matches ingestDependentCounts.ts) so a stale purl column doesn't silently survive CREATE ... IF NOT EXISTS.
• Lower BQ_DATASET_INGEST_ADVISORY_PACKAGES_MAX_BQ_GB 1500 → 50 (measured actual ~1.4 GB; keeps the ceiling a real regression gate).
• bqExportToGcs now throws a typed ApplicationFailure.nonRetryable(..., 'BQ_CEILING_EXCEEDED') on ceiling breach instead of a plain Error (a dry-run byte count is deterministic, so retrying is pure waste).
• ingestAdvisories unwraps that from ActivityFailure and rethrows a workflow-level ApplicationFailure of the same type; bootstrapOsspckgs now soft-fails only on that type around the advisories step (mirrors the existing dependent_counts/package_dependencies guards) — a ceiling breach no longer strands scorecard/ranking for the whole run.
• Bonus fix: resolveMissingPackageIds (osv.ts:316) built '@' || namespace || '/' || name for npm, but namespace already contains the @ — every scoped npm advisory package was silently unresolvable in this catch-up. One-line fix, same identity-reconstruction logic as the rest of this PR.

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Performance improvement
• Chore / dependency update
• Documentation

JIRA ticket

https://linuxfoundation.atlassian.net/browse/CM-1362

[cursor]

PR Summary

Medium Risk
Changes advisory-package ingest identity and bootstrap error handling; incorrect joins could leave package_id null, but existing data is untouched on soft-fail and resolveMissingPackageIds still catches gaps.

Overview
Fixes weekly bootstrapOsspckgs failures at ingestAdvisories by removing the BigQuery purl_map scan (~1.5 TB) from advisory_packages export and resolving package_id in Postgres on the same (ecosystem, namespace, name) identity as packages, via shared packageNameSplitSql in ingestPackages / ingestAdvisories.

Pipeline behavior: advisory_packages staging drops the purl column (DROP + CREATE DDL); BQ byte ceilings are tightened (e.g. advisory_packages 1500 → 50 GB). Dry-run ceiling breaches throw BQ_CEILING_EXCEEDED as non-retryable ApplicationFailure; bootstrapOsspckgs soft-fails only that case around advisories (so scorecard/ranking still run) and posts notifyBqCeilingSkip to Slack.

Bonus: resolveMissingPackageIds no longer double-prefixes @ for scoped npm package_name reconstruction.

^{Reviewed by Cursor Bugbot for commit cc2a3ea. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Reduces advisory-package BigQuery scanning by resolving package identities in PostgreSQL while allowing downstream jobs to continue after ceiling breaches.

Changes:

• Removes the costly BigQuery purl lookup and adds shared package-name parsing.
• Resolves package IDs through PostgreSQL’s indexed package identity.
• Adds typed, non-retryable ceiling failures and lowers the scan limit.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
services/libs/data-access-layer/src/packages/osv.ts	Fixes scoped npm identity reconstruction.
services/apps/packages_worker/src/deps-dev/workflows/ingestPackages.ts	Uses shared identity parsing.
services/apps/packages_worker/src/deps-dev/workflows/ingestAdvisories.ts	Resolves package IDs in PostgreSQL and handles ceiling failures.
services/apps/packages_worker/src/deps-dev/workflows/bootstrapOsspckgs.ts	Soft-fails advisory ceiling breaches.
services/apps/packages_worker/src/deps-dev/README.md	Documents revised scan ceilings.
services/apps/packages_worker/src/deps-dev/queries/pgIdentity.ts	Adds shared identity SQL generation.
services/apps/packages_worker/src/deps-dev/queries/advisoriesSql.ts	Removes the BigQuery purl scan.
services/apps/packages_worker/src/deps-dev/activities/bqExportToGcs.ts	Makes ceiling failures typed and non-retryable.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

services/libs/data-access-layer/src/packages/osv.ts:316

• The scoped-npm catch-up fix has no regression coverage. The existing resolveMissingPackageIds integration case only uses an npm package with namespace: null, so it would pass with the old double-@ expression. Add a scoped fixture (for example namespace @types, name node, package name @types/node) and assert that package_id resolves.

        WHEN p.ecosystem = 'npm' THEN p.namespace || '/' || p.name

services/apps/packages_worker/src/deps-dev/activities/notifyBqCeilingSkip.ts:20

• This alert runs after the advisory-header step has already merged rows, so “Existing data untouched” is inaccurate; only the advisory-package step was skipped. The message can mislead operators investigating a partially completed run.

        text: `Ingest skipped for this run so scorecard/ranking still complete. Existing data untouched. ${input.message}`,

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (1)

services/apps/packages_worker/src/deps-dev/activities/notifyBqCeilingSkip.ts:20

• The ceiling breach occurs after Step 1 has already merged advisory header rows, so “Existing data untouched” is inaccurate and may mislead operators. State specifically that advisory-package/range data was preserved and that advisory headers may already have been ingested.

        text: `Ingest skipped for this run so scorecard/ranking still complete. Existing data untouched. ${input.message}`,

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.