Spark Boot

Spark Boot is a JVM/Kotlin framework for defining, assembling, and running Spark pipelines with a clean separation between construction, composition, declarative configuration, and Spark execution.

It provides:

• a Kotlin DSL for developer-first pipeline authoring
• a SeaTunnel-style HOCON DSL for deployment/runtime configuration
• Dagger-based compile-time construction and factory registration
• a portable Flow / Node / Edge model
• a versioned JSON-friendly FlowDocument IR for tools and UI builders
• a ServiceLoader extension SPI so third-party libraries can contribute node types and job templates
• Spark 3.5 and Spark 4 runtime selection backed by org.openprojectx.spark.platform
• built-in Parquet, Kafka, JDBC source/sink, SQL filter/select/transform/action, Iceberg sink, and Hudi source/sink nodes

Compatibility target:

Spark Boot HOCON DSL supports SeaTunnel-style configuration shape compatibility,
not full Apache SeaTunnel runtime compatibility.

Modules

Module	Purpose
autoconfigure	Spring Boot-style config properties and connection/catalog registries.
core	Flow model, node definitions, factory contracts, and assembler.
runtime-spark	Shared Spark execution context, Spark node contracts, DAG validation, and runtime execution compiled against the Spark 3.5 Scala 2.13 baseline.
runtime-spark3	Spark 3.5 Scala 2.13 dependency carrier for applications that want the Spark 3 line.
runtime-spark4	Spark 4 dependency carrier for applications that want the Spark 4 line.
starter-spark3	Convenience starter for Spark 3.5 Scala 2.13 applications.
starter-spark4	Convenience starter for Spark 4 applications.
connectors	Built-in Spark nodes and config factories.
dagger	Dagger component, modules, and factory registry wiring.
dsl-kotlin	Kotlin DSL and fluent pipeline chaining.
dsl-hocon	SeaTunnel-style HOCON parser.
job-template	Parameterised job templates (Config -> FlowDefinition) and their descriptors.
cli	HOCON file runner for users who want to provide only config.
integration-tests	Local Spark integration tests.

Flow IR

FlowDefinition is the runtime graph model. UI tools should wrap it in FlowDocument, which adds a schema version and UI-only layout metadata:

{
  "schemaVersion": "spark-boot.flow/v1",
  "flow": {
    "name": "paid-orders",
    "nodes": [
      { "id": "orders", "type": "ParquetSource", "config": { "path": "s3a://warehouse/orders" } }
    ],
    "edges": []
  },
  "ui": {
    "nodes": {
      "orders": { "x": 80, "y": 160 }
    }
  }
}

Built-in node descriptors and structured validation diagnostics are exposed from the library so external graph builders do not need to hardcode node fields.

Extension SPI

Third-party libraries contribute node types and job templates through java.util.ServiceLoader, so a contributing library stays a plain JVM library with no dependency on the tool that consumes it:

class MyNodeDescriptorProvider : NodeDescriptorProvider {
    override val contributor = "my-lib"
    override fun descriptors() = listOf(/* NodeDescriptor(...) */)
}

Register it in META-INF/services/org.openprojectx.spark.boot.core.NodeDescriptorProvider, then assemble the palette with NodeCatalog.discover(). Built-in nodes use the same path — connectors registers BuiltinNodeDescriptorProvider — so they are not special-cased.

job-template adds the layer above the graph: a JobTemplate is a pure Config -> FlowDefinition compiler whose JobDescriptor describes the parameters it accepts, letting tools render forms, lint configs, and preview the resulting graph. Templates are discovered with TemplateCatalog.discover().

See docs/extensions.adoc.

Quick Start

Choose one Spark line in application builds. Spark 3 support intentionally uses Scala 2.13 artifacts, not Scala 2.12:

dependencies {
    implementation("org.openprojectx.spark.boot:starter-spark4:<version>")
}

or:

dependencies {
    implementation("org.openprojectx.spark.boot:starter-spark3:<version>")
}

Library modules that contain shared Spark-facing code compile against Spark 3.5 with Scala 2.13 as the lowest supported baseline. Spark 4 applications select Spark 4 dependencies through starter-spark4 / runtime-spark4.

Build and test:

env GRADLE_USER_HOME=/data/.gradle ./gradlew test --no-configuration-cache

Create and run a Kotlin DSL flow:

@SparkBoot
fun main(args: Array<String>) = runSparkBoot(args) {
    flow("paid-orders") {
        parquetSource("orders") {
            path = "data/orders"
        }
            .filterSql("paid-only") {
                condition = "status = 'PAID'"
            }
            .select("select-columns") {
                columns = listOf("id", "amount", "status")
            }
            .writeParquet("sink") {
                path = "output/paid-orders"
                mode = SaveMode.Overwrite
            }
    }
}

In the Kotlin DSL, names such as "orders", "paid-only", and "sink" are flow-local node ids. They are used to register nodes and wire DAG edges; they are not Dagger bean names, Spark table names, or paths. The configuration lambda customizes a newly-created node instance before execution. Dagger supplies node factories and runtime services, while SparkRuntime executes the completed flow later. Applications can contribute their own node factories through Dagger multibindings and create them in the DSL with node<MyNode>("orders", "MyNodeKind") { ... }. In that shape, "MyNodeKind" selects the Dagger-registered factory and "orders" remains the flow-local node id. DSL lambdas and built-in nodes run on the Spark driver and build Spark plans; they are not sent to executors. User-provided nodes can still enter executor-side Spark APIs such as Dataset.map, RDD map, foreachPartition, or UDF lambdas. Those closures must follow normal Spark serialization and executor classpath rules. See docs/user-guide.adoc for local-cluster testing guidance and examples of unsafe captures.

Spark Boot also supports starter-style environment configuration for shared infrastructure such as JDBC connections, S3, HMS, and Iceberg catalogs:

spark.boot {
  jdbc.connections.orders {
    url = "jdbc:mysql://localhost:3306/orders"
    user = "spark"
    password = "spark"
    driver = "com.mysql.cj.jdbc.Driver"
  }

  hms {
    uri = "thrift://localhost:9083"
    warehouse = "s3a://warehouse/iceberg"
    catalog = "hms"
  }

  catalogs.analytics_iceberg {
    type = "iceberg-hive"
    uri = "thrift://analytics-hms:9083"
    warehouse = "s3a://warehouse/analytics/iceberg"
  }

  catalogs.analytics_hive {
    type = "kyuubi-hive"
    uri = "thrift://analytics-hms:9083"
  }
}

This config can be partial. Stable values can live in src/main/resources/application.conf, while dynamic values such as Testcontainers ports can be supplied through system properties before the Dagger component is created.

Spring-style profiles are supported for classpath config. Spark Boot loads application.conf, overlays application-<profile>.conf, and leaves system properties as the final override layer. Use spark.boot.profiles.active, SPARK_BOOT_PROFILES_ACTIVE, or --profile ci with the CLI/Kotlin launcher. Profile-aware Dagger node factories can also be contributed so the same DSL kind resolves to different app-provided factories per profile.

Then flows can reference logical names instead of repeating connection details:

jdbcSource("orders") {
    connection = "orders"
    table = "jdbc_orders"
}.writeIceberg("sink") {
    catalog = "hms"
    table = "default.jdbc_orders"
}

Publish the root project to Maven local before running examples:

env GRADLE_USER_HOME=/data/.gradle ./gradlew publishToMavenLocal --no-configuration-cache

Run the standalone examples build:

env GRADLE_USER_HOME=/data/.gradle ./gradlew -p examples runAll --no-configuration-cache
env GRADLE_USER_HOME=/data/.gradle ./gradlew -p examples :kotlin-dsl:run --no-configuration-cache
env GRADLE_USER_HOME=/data/.gradle ./gradlew -p examples :spark-boot-app:run --no-configuration-cache
env GRADLE_USER_HOME=/data/.gradle ./gradlew -p examples :hocon:run --no-configuration-cache
env GRADLE_USER_HOME=/data/.gradle ./gradlew -p examples :jdbc-iceberg-hms:run --no-configuration-cache
env GRADLE_USER_HOME=/data/.gradle ./gradlew -p examples :kafka-hudi-hms:run --no-configuration-cache
env GRADLE_USER_HOME=/data/.gradle ./gradlew -p examples :multi-hms-catalogs:run --no-configuration-cache

The examples directory is an independent multi-module Gradle build. It is not included in the root build and consumes org.openprojectx.spark.boot:*:0.1.0-SNAPSHOT artifacts from Maven local. The Kotlin DSL examples create temporary Parquet input in code. The :spark-boot-app example shows the @SparkBoot application entry point and a user-provided Dagger node factory used from the DSL. The HOCON example is config-only: org.openprojectx.bigdata-test starts LocalStack S3 and prepares the Parquet input from TOML before the Spark Boot CLI runs paid-orders.conf. The :jdbc-iceberg-hms app starts LocalStack S3, Hive Metastore, and a MariaDB Testcontainers source, then writes JDBC data into HMS-backed Iceberg tables through named orders JDBC and hms catalog config; stable config is loaded from classpath application.conf and dynamic endpoints are supplied at runtime. The :kafka-hudi-hms app starts Testcontainers Kafka plus LocalStack S3 and Hive Metastore through org.openprojectx.bigdata-test, then writes Kafka JSON events into HMS-synced Hudi tables using both HOCON config and Kotlin DSL flows.

CLI

Applications can depend on the CLI module and provide only a HOCON config file:

java -cp "<app-and-dependencies>" org.openprojectx.spark.boot.cli.SparkBootCliKt paid-orders.conf

The CLI parses the config, assembles the flow with Dagger-backed built-in factories, and runs it with Spark.

HOCON Example

env {
  job.name = "paid-orders"
  job.mode = "BATCH"
}

source = [
  {
    plugin_name = "Parquet"
    path = "s3a://spark-boot-hocon-example/input/orders"
    plugin_output = "orders"
  }
]

transform = [
  {
    plugin_name = "Sql"
    plugin_input = "orders"
    plugin_output = "paid_orders"
    query = "select id, amount, status from orders where status = 'PAID'"
  }
]

sink = [
  {
    plugin_name = "Parquet"
    plugin_input = "paid_orders"
    path = "s3a://spark-boot-hocon-example/output/paid-orders"
    save_mode = "overwrite"
  }
]

See docs/user-guide.adoc for the detailed guide.