Fix Doris overwrite metadata registration and routing

[jiangxt2]

Pull request

What changes were proposed in this pull request?

• Honor generic and Doris-specific overwrite requests using the active SparkSession configuration, and reject conflicting targets before registration or preparation.

• Bound task identities to 256 Unicode characters while preserving persisted legacy identities that already fit.

• Store complete selectors and predicates as TEXT, provide a data-preserving migration, refresh retry payloads, and reject failed metadata queries or missing state updates.

• Add redacted stage, SQLSTATE, length, and timing diagnostics; preserve primary failures when cleanup or failure recording also fails.

• Align CI policy expectations with 18 Doris cases and 35 cases across domain shards.

Why are the changes needed?

Eight comma-separated dates occupy 71 characters, exceeding the original 64-character metadata column. Combined task identities can exceed the 256-character key. Partial configuration, session settings read through SparkContext, and generic overwrite routed through JDBC can also leave writes without the requested metadata coordination.

Related issues

Fixes #6.

Does this PR introduce any user-facing change?

Yes. Generic overwrite now uses atomic connector replacement even for small or forced-JDBC outputs. The output path must match the configured Doris database/table; unqualified names resolve within that database. Completely absent metadata remains optional, but partial configuration and required metadata failures are errors. An explicitly configured empty password remains valid. JDBC append behavior is unchanged.

Existing stores must widen rdate and partition_filter before writing values beyond their legacy limits:

psql --set ON_ERROR_STOP=1 --file sqls/postgre/migrations/doris_task_metadata_text.sql

Use controlled PostgreSQL connection environment variables and a maintenance window for the table lock. The migration preserves task records, indexes, triggers, and foreign keys; it is not applied automatically. If metadata fails after publication, reconcile the target before retrying.

How was this patch tested?

Baseline: JDK 17.0.18, Scala 2.12.18, Spark 3.5.8, Maven 3.9.5, PostgreSQL 16.14, and native ARM64 Doris 3.0.6.2.

• Focused Doris unit coverage: 85 distinct cases passed, covering identity boundaries, legacy compatibility, routing, configuration, failure propagation, cleanup, and log redaction. Reproduction command: mvn -B -pl pista-connector -am test -DmembersOnlySuites=com.pista.spark.sql.doris.meta,com.pista.spark.sql.connector.doris,com.pista.spark.sql.connector.doris.batch.
• Real PostgreSQL/Doris evidence covers 18 distinct cases: task transitions, complete payloads, migration preservation, FE resolution, writes, transactions, whole-table/partition replacement, and conflicting-target protection. Domain entry point: scripts/run_domain_it.sh doris.
• Public overwrite boundary selection: mvn -B -pl pista-connector -am test -DmembersOnlySuites=com.pista.spark.sql.connector.doris -DtagsToInclude=com.pista.DorisOverwriteBoundaryIT -Dpista.it.exclude.tag=__pista_it_no_exclude__ — 3/3 passed, with no failed, skipped, canceled, or ignored cases.
• python -m pytest scripts/tests -q — 33 policy cases passed; ruff format --check scripts and ruff check scripts passed with Python 3.12, pytest 9.1.1, and Ruff 0.14.14.
• python3 -B scripts/check_public_content.py, python3 -B scripts/check_language.py, python3 -B scripts/check_markdown.py, python3 -B scripts/check_public_license.py, and git diff --check passed.

The verified target baseline is Doris 3.0.6.2 with PostgreSQL 16.14; this change does not establish additional recovery or cross-target consistency guarantees.