Separate the pipeline into environment workflows #2417

User Story

As a developer I want to separate the current ci_cd.yml file into environment workflow files, so that it's clearly defined what steps/jobs should be run per environment.

Description

The pipeline currently resides for the most part in the ci_cd.yml file. Making a change to this file must be done with great care to not affect custom projects or the pipeline on master. To make it clear what runs where we introduce the concept of an "environment workflow".

Please see #2040 (comment) for more details

Acceptance Criteria

• 1. Create the following environment workflow files
  • 1. Review (review.yml): Used for PRs
  • 2. Staging (staging.yml): Used for merge to master (and deploy to staging)
  • 3. Release (release.yml): Used to release and deploy to production
  • 4. Custom (custom.yml): Used for custom projects
  • 5. Custom Release (custom-release.yml): Certain custom projects make releases
• 2. All custom projects pipelines must continue to work like before and depend on the custom.yml and/or custom-release.yml workflow files.
• 3. Certain steps may not need to be added to all environment workflows.

Note: It's important that manual pipeline triggers remain functional. Deploying to test servers should be done from the review.yml workflow file. Until we rework the deployment process and introduce preview deployments in a separate issue.

Implementation Notes

@wborn mentioned there's also a concept called "Environments" in GitHub (see https://docs.github.com/en/actions/how-tos/deploy/configure-and-manage-deployments/manage-environments). However this more in relation to deployments and may be introduced separately.

I've used this feature before and have used something like the following to indicate to what "environment" to deploy within GitHub.

  deploy:
    runs-on: ubuntu-latest
    needs: publish
    environment:
      name: review-${{ github.event.number }}
      url: https://${{ github.event.number }}.openremote.app

This allowed me assign secrets to certain environments and apply protection rules. Though this isn't something that is strictly necessary.