In the bigquery-firestore-export kit (@firebase/bigquery-firestore-export, on the kits branch), a run that fails partway through writing its rows duplicates its whole output on every retry.
Cause
Rows are written with collection.add() in kits/bigquery-firestore-export/src/helper.ts, which mints a new document id per call, and the run document under .../runs/<runId> is only written once every row has been written. A run that dies partway leaves its rows in place, and the retry writes the whole result set again under fresh ids.
processMessages sets retry: true (kits/bigquery-firestore-export/src/index.ts), so the notification is redelivered until the subscription's retention expires, 24 hours by default. Nothing dedupes and nothing cleans up the partial attempt.
What happens
With a 3,000 row source on a live project, watched for 14 minutes: 3,400, then 6,400, then 9,848, then 12,900 documents under .../runs/<runId>/output, still climbing when the subscription was purged.
Fix
Write rows with deterministic ids derived from the run and the row index, so a retry overwrites rather than appends.
In the
bigquery-firestore-exportkit (@firebase/bigquery-firestore-export, on thekitsbranch), a run that fails partway through writing its rows duplicates its whole output on every retry.Cause
Rows are written with
collection.add()inkits/bigquery-firestore-export/src/helper.ts, which mints a new document id per call, and the run document under.../runs/<runId>is only written once every row has been written. A run that dies partway leaves its rows in place, and the retry writes the whole result set again under fresh ids.processMessagessetsretry: true(kits/bigquery-firestore-export/src/index.ts), so the notification is redelivered until the subscription's retention expires, 24 hours by default. Nothing dedupes and nothing cleans up the partial attempt.What happens
With a 3,000 row source on a live project, watched for 14 minutes: 3,400, then 6,400, then 9,848, then 12,900 documents under
.../runs/<runId>/output, still climbing when the subscription was purged.Fix
Write rows with deterministic ids derived from the run and the row index, so a retry overwrites rather than appends.