Repository navigation
Closed
Labels
area:ciCI/CD, GitHub Actions, build toolingCI/CD, GitHub Actions, build toolingenhancementNew feature or requestNew feature or requestrequires-triage
Description
What is the problem the feature request solves?
The Spark SQL suite sets the length of every merge-queue run: its build job (18 minutes with a warm cache) plus its slowest shard. Median shard durations across 129 queue runs between 2026-09-11 and 2026-09-29:
| Row | Median |
|---|---|
sql_hive-1 |
66.7 min |
sql_core-1 |
59.6 min |
sql_core-2 |
48.2 min |
sql_hive-3 |
44.9 min |
sql_core-3 |
28.1 min |
sql_hive-2 |
21.1 min |
catalyst |
14.0 min |
Two things make the long rows long:
org.apache.spark.sql.hive.client.HivePartitionFilteringSuitesrunsHivePartitionFilteringSuiteagainst every Hive client version and took 21 to 35 minutes ofsql_hive-1in the runs I sampled. It is a single class, so a name filter cannot split it further.org.apache.spark.sql.execution.datasources.*andorg.apache.spark.sql.connector.*account for 13 to 19 minutes ofsql_core-1, and the data source suites for another 11 minutes ofsql_core-2.
Describe the potential solution
Move them into two new rows with sbt's exclusion patterns (testOnly * -<glob>), keeping the tag filters so every test still runs exactly once:
sql_core-4:sql/testOnly org.apache.spark.sql.execution.datasources.* org.apache.spark.sql.connector.* -- -l org.apache.spark.tags.SlowSQLTest(their untagged and Extended tests), with both globs excluded fromsql_core-1andsql_core-2.sql_hive-4:hive/testOnly org.apache.spark.sql.hive.client.HivePartitionFilteringSuiteswithsql_hive-1's tag filters, and the class excluded fromsql_hive-1.
In the two runs I sampled, the slowest row would drop from 72 and 51 minutes to about 48 and 40. The cost is two more runners per Spark SQL run, each with about 9 minutes of setup.
Additional context
#6103 / #6113 shard the Iceberg extensions task, which is the next longest path (about 57 minutes).
Metadata
Metadata
Assignees
Labels
area:ciCI/CD, GitHub Actions, build toolingCI/CD, GitHub Actions, build toolingenhancementNew feature or requestNew feature or requestrequires-triage
Activity
ci: rebalance the Spark SQL test shards (apache#6393)