Pair the spark-4.2 profile with Delta 4.4 and run the Delta contrib tests on it #7455

Activity

1. dwsmith1983 commented on Nov 3, 2026

   

   dwsmith1983

   on Nov 3, 2026

   ContributorAuthor

   More actions

   take
2. github-actions

   assigned

   dwsmith1983

   on Nov 3, 2026
3. github-actions

   added

   requires-triage

   on Nov 3, 2026
4. ErikBPF commented on Nov 8, 2026

   

   ErikBPF

   on Nov 8, 2026

   Contributor

   More actions

   I reproduced the Spark 4.2 pairing with both Delta 4.4.0 and 4.4.1 on Comet 1.2.0 source (ab46419fcc99bcd34aa4d89f7b545d5d51694a88). Each full contrib run executes 279 tests: 278 passed, one failed, zero errors/skips/canceled/ignored/pending.

   The same failure appears in both lanes:

   DeltaScanContribSuite:
   discovery harness: every credential/encryption-shaped fs.s3a.* key Hadoop's
   Constants class declares is either compared by AllS3ConfigKeys or individually documented as exempt
   HashSet("fs.s3a.encryption.context") was not empty
   

   NativeScan (118), DML-decline (3) and local MinIO/S3 (3) pass; the 155-case admission suite contains the failure. Synthetic merge/clustering/hygiene workloads also pass 37 checks per lane with matching stock outputs, but that does not override the failed admission audit.

   After the matching native/JVM build, the root-reactor selection is:

   set -euo pipefail
   ./mvnw -Pspark-4.2,delta -Ddelta.artifact=delta-spark_4.2 \
     -Ddelta.spark.version=4.4.0 test -Dtest=none \
     -DwildcardSuites=org.apache.comet.contrib.delta
   # Repeat with -Ddelta.spark.version=4.4.1.

   Both artifact and version overrides matter. The source-matched packages in these runs reused the exact released AMD64 JNI; no native rebuild or current-main runtime is claimed. The actual failed-suite classpath has one Constants.class provider: hadoop-aws-3.5.0.jar (SHA256 66bd9eefb938b1a1657487324b63c13df2d8b17164a1dd24e0ef37b905bc5baa).

   The inspected current-main discovery test and #7026 head also lack classification for this key. Hadoop sends KMS context on write requests, but resolves/parses it during initialization. Fifteen pure in-memory Hadoop 3.5 reference checks passed; malformed context raises IllegalArgumentException through S3AUtils.buildEncryptionSecrets with unset, AES256, SSE-KMS and DSSE-KMS algorithms. For example:

   Configuration conf = new Configuration(false);
   conf.set("fs.s3a.encryption.context", "synthetic-malformed-pair");
   S3AUtils.buildEncryptionSecrets("synthetic-bucket", conf); // IllegalArgumentException

   Source: Hadoop encryption initialization, context resolution/parsing.

   This is pairing-gate evidence, not a reproduced Comet read-admission discrepancy or AWS KMS-policy failure. A blanket exemption cannot be justified solely because GET/HEAD omit the header. The narrow follow-up is to agree this property's classification with the S3 gate owner, coordinate with #7026, preserve existing SSE-C/CSE declines, and rerun the nonempty contrib suites. I am keeping that follow-up separate from the private-expression feature requests; no existing assertion was bypassed.
5. dwsmith1983 commented on Nov 9, 2026

   

   dwsmith1983

   on Nov 9, 2026

   ContributorAuthor

   More actions

   fs.s3a.encryption.context is new in Hadoop 3.5.0; 3.4.1 and 3.4.2 don't declare it, so only the spark-4.2 profile reaches it. The pairing PR will classify it.

   I plan to exempt it next to the CSE entries: the native Delta scan only reads, native code never reads the key, and Hadoop sends the context only on writes. For the malformed value you found, I'll add a test that it fails in S3A setup on the driver, before the native scan, so Comet fails where Spark does. If it doesn't, the key goes into AllS3ConfigKeys instead. The SSE-C and CSE declines stay, and I'll check it against #7026.