Skip to content

CI/CD: Improve deployment security #1156

Description

@kartpop

Is your feature request related to a problem?
Our CI/CD pipelines have vulnerabilities that could lead to deployment incidents, similar to what occurred with the Glific team. Addressing these gaps is crucial to ensure deployment success and minimize failures.

Describe the solution you'd like

  • Ensure that Staging deploys wait for a successful CI run by updating deploy-staging.yml with workflow_run and adding branch protection on main.
  • Integrate a Discord webhook step to notify about staging and production deploy failures.
  • Implement health checks for deployments by enabling the ECS deployment circuit breaker and polling the rolloutState.
  • Add Docker image builds in CI for every PR to catch issues earlier.
  • Set up ECS staging rehearsals for every merge to main by scaling the staging ECS service and verifying it, with notifications on failures.
Original issue

Our CI CD pipelines can be hardened. There are gaps that need fixing, which will prevent from us from having similar deployment incidents like what Glific team shared recently.
More details here

1. Deploys wait for CI to be green

Staging deploy currently races CI instead of waiting for it. Change deploy-staging.yml to trigger only after a successful CI run (workflow_run), add branch protection on main with required checks, and make create-release.yml confirm the tagged commit passed CI before building.

2. Discord notifications on deploy failures

Add a Discord webhook step (if: failure()) to the staging and production deploy workflows. Today a red deploy on main is only visible if someone opens the Actions tab — and Sentry can't catch failures that happen before the app boots.

3. Verify deployments actually became healthy

  • Enable the ECS deployment circuit breaker on our services.
  • After update-service, poll the deployment's rolloutState until COMPLETED (pass) or FAILED / timeout (fail). A plain curl isn't enough — the old task can answer with 200 while the new one is failing.
  • Bake the git SHA into the image and return it from the health endpoint, so we can confirm the new code is the one responding.
  • On the EC2 staging box, wait for the container healthcheck after docker compose up -d instead of reporting success when containers start.

4. Build the Docker image in CI

on every PR CI never builds the Dockerfile or validates the compose files, so breakage there is only discovered during a deploy. Add docker build ./backend and docker compose config checks to CI.

5. ECS staging rehearsal on every merge to main

Staging runs on EC2 (to save costs), so the production ECS deploy path is never tested before a release. After the EC2 staging deploy completes (serial, not parallel — the EC2 path owns migrations), scale the staging ECS service from 0 to 1 with the new image, verify with the rolloutState poll, then scale back to 0 (if: always()). One-time prep: remove autoscaling on the staging service and enable its circuit breaker. A failed rehearsal pings Discord but does not roll back EC2 staging.

Activity

  1. changed the title [-]CI/CD: verify deployments and notify on failure[/-] [+]CI/CD: Improve deployment security[/+] on Aug 24, 2026
  2. self-assigned this
    on Aug 25, 2026
  3. moved this from To Do to In Progress in Kaapi-devon Aug 30, 2026
  4. moved this from In Progress to In Review in Kaapi-devon Aug 31, 2026
  5. Ayush8923 commented on Aug 31, 2026

    @Ayush8923
    Collaborator

    Need to check these 2 as well:

    • rolloutState polling — after the two aws ecs update-service calls, the workflow currently exits green immediately. ECS then swaps tasks in the background; if the new task crashes on boot, the Actions run still shows. The issue asks for a poll loop after update-service: describe the service's deployment until rolloutState is COMPLETED (pass) or FAILED/timeout (fail). This is the ECS equivalent of what the PR did do for the EC2 SSM command.
    • SHA in the health endpoint — bake the git SHA into the image and return it from /health, so you can confirm the new code is what's responding. This is why the issue says "a plain curl isn't enough — the old task can answer 200 while the new one is failing." Applies to prod primarily, but benefits staging too.
  6. moved this from In Review to In Progress in Kaapi-devon Aug 31, 2026
  7. Ayush8923 commented on Sep 2, 2026

    @Ayush8923
    Collaborator

    This is the Relevent PR of this issue: #1158

  8. moved this from In Progress to Closed in Kaapi-devon Sep 2, 2026
  9. moved this from Closed to In Progress in Kaapi-devon Sep 2, 2026
  10. moved this from In Progress to In Review in Kaapi-devon Sep 10, 2026
  11. github-actions commented on Sep 29, 2026

    @github-actions

    🎉 This issue has been resolved in version 1.7.0-main.10 🎉

    The release is available on GitHub release

    Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

  • Status
    Closed

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions