bigquery-firestore-export refuses to clear PARTITIONING_FIELD, but the Data Transfer API allows it.
Setting PARTITIONING_FIELD and later clearing it fails the update with:
Cannot remove partitioning_field from an existing transfer config. The BigQuery Data Transfer API does not support clearing this parameter once it has been set. To change partitioning, you must create a new transfer config with the desired partitioning settings.
The API does support it. Creating a new transfer config is a costly workaround, since run history is stored under the old config ID and does not come with you.
Reproduction
Run against a real project, us location:
- Create a scheduled query with
partitioning_field: created_at and destination template probe_{run_time|"%H%M%S"}. Two runs produced tables partitioned DAY (field: created_at).
- Clear it:
PATCH /v1/{name}?updateMask=params with "partitioning_field": "". Returns 200, and a follow-up GET reads it back as "".
- Trigger another run. It carries
partitioning_field: "", reaches SUCCEEDED, and reports no error.
- Compare the destination tables:
probe_152800 DAY (field: created_at) <- partitioning set
probe_152909 DAY (field: created_at) <- partitioning set
probe_153056 (none) <- after clearing
So the clear is accepted, persists, runs, and changes the output. It is not silently ignored and it does not fail.
The "you cannot un-partition an existing table" concern does not apply here: the destination template gives every run a new time-suffixed table, so nothing is mutated in place.
Where it comes from
The guard and its comment are in functions/src/dts.ts in the upstream extension:
// The BigQuery Data Transfer API does not support clearing this parameter
throw new Error(PARTITIONING_FIELD_REMOVAL_ERROR);
The kit inherited both in a34ba233. The kit and the extension differ only in what happens after the throw, so both are affected.
Suggested fix
Drop the guard and let an empty partitioning_field through the update mask, and remove PARTITIONING_FIELD_REMOVAL_ERROR along with the comment asserting the limitation. Worth confirming against the extension too, since the belief originated there.
Not verified: behaviour in locations other than us, and whether a destination table without a time-suffixed template behaves differently.
bigquery-firestore-exportrefuses to clearPARTITIONING_FIELD, but the Data Transfer API allows it.Setting
PARTITIONING_FIELDand later clearing it fails the update with:The API does support it. Creating a new transfer config is a costly workaround, since run history is stored under the old config ID and does not come with you.
Reproduction
Run against a real project,
uslocation:partitioning_field: created_atand destination templateprobe_{run_time|"%H%M%S"}. Two runs produced tables partitionedDAY (field: created_at).PATCH /v1/{name}?updateMask=paramswith"partitioning_field": "". Returns 200, and a follow-upGETreads it back as"".partitioning_field: "", reachesSUCCEEDED, and reports no error.So the clear is accepted, persists, runs, and changes the output. It is not silently ignored and it does not fail.
The "you cannot un-partition an existing table" concern does not apply here: the destination template gives every run a new time-suffixed table, so nothing is mutated in place.
Where it comes from
The guard and its comment are in
functions/src/dts.tsin the upstream extension:The kit inherited both in
a34ba233. The kit and the extension differ only in what happens after the throw, so both are affected.Suggested fix
Drop the guard and let an empty
partitioning_fieldthrough the update mask, and removePARTITIONING_FIELD_REMOVAL_ERRORalong with the comment asserting the limitation. Worth confirming against the extension too, since the belief originated there.Not verified: behaviour in locations other than
us, and whether a destination table without a time-suffixed template behaves differently.