feat: add separate v2 health score sink to snowflake (IN-1212)

[gaspergrom]

Summary

Adds a new, standalone Tinybird sink pipe (health_score_v2_sink) that exports the full v2 health score picture — total score, the Health breakdown (3 categories), and the Impact breakdown (4 metrics + percentile bands) — to its own Kafka topic and, downstream, its own Snowflake table. Kept fully separate from the existing v1 health_score_sink pipe, which is untouched.

Per product guidance (Joana Maia): a new table is the right structure for a genuinely new data shape (lets v1 and v2 run in parallel until the health score migration is complete), and the sink should also export the sub-signal breakdown, not just the total score.

Pushed to production disabled — no EXPORT_SCHEDULE — per an explicit product request: Joana wants this inert until she's confirmed the rollout with external stakeholders (e.g. CNCF). The pipe exists in Tinybird and is queryable, but has no active cron and will not export anything on its own until an EXPORT_SCHEDULE line is added.

What's exported

• id, slug
• Total score: healthScoreV2, healthLabel, lifecycleLabel, impactScore, impactLabel
• Health breakdown (from project_insights_copy_ds directly, no join needed): maintainerHealthScoreV2 (0-40), securitySupplyChainScoreV2 (0-35), developmentActivityScoreV2 (0-25)
• Impact breakdown (joined from project_insights_impact_breakdown_ds on id): directDependents/directDependentsTopPct/directDependentsBand, transitiveDependents/transitiveDependentsTopPct/transitiveDependentsBand, downloads/downloadsTopPct/downloadsBand, centrality/centralityTopPct/centralityBand (the last three fields will be NULL for every row today — centralityScore is 0% populated in production, a known pre-existing data gap, not introduced here)
• date stamp

LEFT JOIN (not INNER JOIN) against the impact breakdown table, so a project with no linked packages still sinks its health fields instead of being dropped entirely.

• health_score_sink.pipe (v1) is untouched — zero diff against main.
• New Kafka topic: health_score_v2_sink (v1's topic, health_score_sink, is unchanged). Same EXPORT_CONNECTION_NAME (lfx-oracle-kafka-streaming) as v1.
• segmentId is not included — project_insights_copy_ds has no segmentId column (that field only exists on health_score_copy_ds, which v1 reads and this pipe does not).

Validation

• tb check clean.
• Confirmed no fan-out risk from the join: project_insights_impact_breakdown_ds has 13,548 total rows / 13,548 unique ids — exactly 1:1.
• Confirmed row count identical before/after adding the join: 11,626 rows for type='project' both with and without the LEFT JOIN, matching project_insights_copy_ds's own count exactly — no drops.
• Spot-checked Kubernetes against production: all 18 fields returned real, correct values (healthScoreV2=61, maintainerHealthScoreV2=27, directDependents=31058/Top 10%, centrality=null, etc.), matching every prior verification of this project's data in this ticket.

Deploy status

Deployed to production, disabled. Pipe exists and is queryable (tb pipe sink run health_score_v2_sink would work on-demand), but has no EXPORT_SCHEDULE, so it will not fire on its own. This is intentional per Joana's request — activating it (adding a schedule line) is a follow-up once external communication is confirmed.

Deploy history on this PR: an earlier version was briefly deployed then reverted (see prior commits); this version was validated read-only first, then deployed disabled.

Open items for reviewers

1. crowd-kube topic registration — already done and merged: crowd-kube #162 registered health_score_v2_sink in snowflake-sink.properties, and the Kafka Connect deployment was restarted to pick it up (confirmed live via the connector's own REST API).
2. Activation — when Joana confirms, add an EXPORT_SCHEDULE line (suggest mirroring the pattern used elsewhere in this ticket: run after this pipe's slowest dependency refreshes — project_insights_copy_ds at 03:00 UTC and project_insights_impact_breakdown_ds at 02:40 UTC, so 03:30 UTC or later is safe) and push again.

JIRA

IN-1212 (health score v1→v2 migration).

Test plan

• tb check clean
• Join verified 1:1, no fan-out (13,548/13,548 unique ids)
• Row count identical before/after join (11,626 = 11,626)
• Sample data verified correct for a real project (Kubernetes, all 18 fields)
• health_score_sink.pipe confirmed byte-identical to main
• Deployed to production, confirmed disabled (schedule: {} via live pipe API)
• crowd-kube topic registration merged and confirmed live on the running Kafka Connect pod

Checklist

• git commit --signoff -S
• Single file changed (health_score_v2_sink.pipe)
• Cross-repo follow-up completed (crowd-kube Update README.md #162, merged)

[cursor]

PR Summary

Low Risk
Adds a new read-only export pipe with no changes to v1 sinks or application logic; main operational risk is forgetting crowd-kube topic registration before relying on Snowflake data.

Overview
Adds a new Tinybird sink pipe health_score_v2_sink.pipe that exports v2 health and impact fields to Kafka topic health_score_v2_sink, separate from the existing v1 health_score_sink pipe (unchanged).

The sink selects project rows from project_insights_copy_ds (type = 'project') and LEFT JOINs project_insights_impact_breakdown_ds so projects without impact data still export health columns (healthScoreV2, labels, maintainer/security/development v2 scores) plus impact breakdown metrics and percentile bands when present. Export uses the same Kafka connection as v1 but no EXPORT_SCHEDULE in this file—intended on-demand via tb pipe sink run until product confirms recurring export.

Downstream Snowflake ingestion requires registering the new topic in crowd-kube snowflake-sink.properties (outside this PR).

^{Reviewed by Cursor Bugbot for commit 39ab1ab. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Extends the health-score Kafka/Snowflake export with v2 health and impact metrics while retaining all v1 fields.

Changes:

• Joins project-level insights by project ID.
• Exports five additional v2 health-score fields.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

services/libs/tinybird/pipes/health_score_sink.pipe:5

• This comment spans three lines, while the project convention in CLAUDE.md:72 limits warranted comments to two lines. Please condense it while retaining the scheduling rationale.

    -- EXPORT_SCHEDULE runs before both source tables refresh (health_score_copy_ds 00:50,
    -- project_insights_copy_ds 03:00), so this export always carries the prior day's snapshot
    -- of v1 and v2 fields alike. Pre-existing for v1; the v2 join inherits the same lag.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

services/libs/tinybird/pipes/health_score_v2_sink.pipe:20

• The 03:30 cron does not guarantee that the 03:00 project_insights_copy replacement has completed; these are independent scheduled jobs. If that COPY is delayed or takes over 30 minutes, this sink exports the previous snapshot while line 13 labels it with the current date, recreating the stale-data problem this schedule is intended to avoid. Please sequence the export from COPY completion, or use a monitored delay/freshness check that prevents exporting until the new snapshot is available.

EXPORT_SCHEDULE 30 3 * * *

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (3)

services/libs/tinybird/pipes/health_score_v2_sink.pipe:5

• The PR description and test plan say this sink runs daily at 03:30 UTC, but without EXPORT_SCHEDULE it will never export automatically after deployment. Since the comment says manual-only is intentional, update the PR description and deployment plan to reflect that activation behavior; otherwise restore the documented schedule.

    -- No EXPORT_SCHEDULE (disabled on purpose): pushed inert per product's request, only
    -- fireable on-demand via `tb pipe sink run` until external-stakeholder communication
    -- is confirmed. Add an EXPORT_SCHEDULE line to activate the recurring export.

services/libs/tinybird/pipes/health_score_v2_sink.pipe:39

• The description and validation still characterize this as a seven-field straight filter with no join, but the implementation now exports the health/impact breakdown and joins another datasource. Update the documented Snowflake contract and rerun cardinality/uniqueness validation against this query; the stated count equality “by construction” no longer validates the current implementation.

    FROM project_insights_copy_ds AS p
    LEFT JOIN project_insights_impact_breakdown_ds AS i ON p.id = i.id

services/libs/tinybird/pipes/health_score_v2_sink.pipe:13

• This block duplicates the selected columns and scheduling details already documented by the source pipes, making it likely to drift as those sources evolve. Retain only the non-obvious reasons for leaving the sink unscheduled and using a left join.

    -- Joins in the Health breakdown (maintainerHealthScoreV2/securitySupplyChainScoreV2/
    -- developmentActivityScoreV2, already columns on project_insights_copy_ds) and the
    -- Impact breakdown (directDependents/transitiveDependents/downloads/centrality +
    -- their percentile bands, from the separate project_insights_impact_breakdown_ds --
    -- materialized on its own 02:40 UTC schedule, see that datasource's pipe for why it's