Repository navigation
CI/CD: Modularize the steps in the pipeline #2040
Description
User Story
As a developer I want to have the pipeline modularized, so that I can run certain jobs in parallel and make it easier to make changes.
Description
The main pipeline is currently structured such that everything runs after each other in one job. Using GitHub actions typically means creating a new job for each task like linting, running backend tests, etc.
The dependencies of jobs can then also be nicely graphed.
We currently already have a workflow that we can reuse in the pipeline: https://github.com/openremote/openremote/blob/master/.github/workflows/provision-host.yml
Acceptance Criteria
- 1. All significant steps must be converted to jobs.
- 2. Run jobs in parallel where possible.
Implementation Notes
All jobs could be written into one workflow file, however we could also split the jobs into their own workflow files using the reusable workflows feature. Reusable worflows can be added to an environment specific workflow (e.g. review), like so:
jobs:
format:
uses: ./.github/workflows/fmt.yamlJob dependencies can be setup like so: https://docs.github.com/en/enterprise-cloud@latest/actions/how-tos/write-workflows/choose-what-workflows-do/use-jobs
Activity
Ekhorn commentedon Aug 11, 2025
@pankalog feel free to add any suggestions as u suggested something similar a while back.
Ekhorn commentedon Oct 24, 2025
Ekhorn commentedon Oct 30, 2025
We should also think about how to share the pipeline with custom-projects.
pankalog commentedon Dec 16, 2025
@Ekhorn @denniskuijs I did some thinking for this. First, I think that every action that needs to be performed in the OpenRemote codebase should be done using Gradle. Gradle should be used a great base to allow us to automate a lot of actions that would need to be done using Python/Java/Bash. We should use Gradle in all steps of build-test-deploy-publish.
We can use #1998 to use the DevContainers generated, as a way to setup the pipeline environment. If we do that, then the pipeline will have the correct docker environments, Gradle, and other dependencies by default.
I would also like to have #2349 integrated in the pipeline, since we would then be able to check if a PR's test actually covers the code that it was written for. We can just have it there as an informational thing, or even then impose a certain percentage (70% code coverage per PR).
We can use Testcontainers in our Spock test infrastructure to reduce the dependency of a full Docker install. If we properly integrate Testcontainers, we essentially only require having a Docker socket available. My PoC worked quite nicely, but it needs some more work to make sure the runner wastes as little time as possible for taking the stack up/down.
At the end of the day, the pipeline should be broken up in multiple pieces, and they should be wired up to use artifacts efficiently from previous runs. Look at this workflow file for inspo. The pipeline should only invoke the Gradle command for performing real actions.
I would start by making a list of things that we need the pipeline to do. Once we have that, we can then map those requirements into tasks that gradle should do, with their inputs and outputs. After that, we can make a concrete implementation plan.
Let me know if you wanna jump on a call and talk about it!
Ekhorn commentedon Dec 16, 2025
Thanks @pankalog! I'm not sure if the DevContainers for setup would be more overhead than necessary, like configuring things e.g. cache (I guess just additional mountpoints) and spinning up containers is definitely not a zero cost abstraction. Not sure if we would want to experiment with this before or after separating the pipeline.
But all of the rest seems like a sensible plan.
Ekhorn commentedon Dec 22, 2025
7 remaining items
Ekhorn commentedon Jan 13, 2026
Ekhorn commentedon Jan 13, 2026
Ekhorn commentedon Jan 14, 2026
Ekhorn commentedon May 11, 2026
Updated list of issues to consider creating
Building
IaC
Additional jobs (tools)
Possible improvements
gradle -p deployment installDisttaskpackage.jsonversion bumps) to master branch.ebariaux commentedon Aug 26, 2026
v1 considered done when 4 first sub tasks are done