Skip to content

ci: rebalance the Spark SQL test shards #6388

Closed

Description

@andygrove

What is the problem the feature request solves?

The Spark SQL suite sets the length of every merge-queue run: its build job (18 minutes with a warm cache) plus its slowest shard. Median shard durations across 129 queue runs between 2026-09-11 and 2026-09-29:

Row Median
sql_hive-1 66.7 min
sql_core-1 59.6 min
sql_core-2 48.2 min
sql_hive-3 44.9 min
sql_core-3 28.1 min
sql_hive-2 21.1 min
catalyst 14.0 min

Two things make the long rows long:

  • org.apache.spark.sql.hive.client.HivePartitionFilteringSuites runs HivePartitionFilteringSuite against every Hive client version and took 21 to 35 minutes of sql_hive-1 in the runs I sampled. It is a single class, so a name filter cannot split it further.
  • org.apache.spark.sql.execution.datasources.* and org.apache.spark.sql.connector.* account for 13 to 19 minutes of sql_core-1, and the data source suites for another 11 minutes of sql_core-2.

Describe the potential solution

Move them into two new rows with sbt's exclusion patterns (testOnly * -<glob>), keeping the tag filters so every test still runs exactly once:

  • sql_core-4: sql/testOnly org.apache.spark.sql.execution.datasources.* org.apache.spark.sql.connector.* -- -l org.apache.spark.tags.SlowSQLTest (their untagged and Extended tests), with both globs excluded from sql_core-1 and sql_core-2.
  • sql_hive-4: hive/testOnly org.apache.spark.sql.hive.client.HivePartitionFilteringSuites with sql_hive-1's tag filters, and the class excluded from sql_hive-1.

In the two runs I sampled, the slowest row would drop from 72 and 51 minutes to about 48 and 40. The cost is two more runners per Spark SQL run, each with about 9 minutes of setup.

Additional context

#6103 / #6113 shard the Iceberg extensions task, which is the next longest path (about 57 minutes).

Activity

  1. self-assigned this
    on Sep 29, 2026
  2. added a commit that references this issue on Sep 29, 2026
    bd1d4d1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions