The Ideal CI/CD Deployment Pipeline

Before diving in, I recommend reading my article on the five types of software testing for reliable systems. I will use those testing concepts throughout this post.

First, let’s start with a brief overview of what a CI/CD pipeline is and what makes one better than another.

A CI/CD pipeline automates building, testing, and delivering code changes. With continuous deployment, changes that pass the required checks reach production without manual intervention. Continuous delivery keeps those changes ready to deploy, but may still involve a manual production release.

Historically, application deployments often involved someone following a detailed runbook. These deployments were time-consuming and error-prone. With rapid software delivery, especially with the recent growth of agent-driven coding, the rate of code changes is increasing.

Letting those changes accumulate makes releases harder to debug. When a release introduces a bug or higher error rates, finding the root cause across a large batch of changes can take hours, if not days. Smaller, automatically validated deployments make that process more manageable.

Now, onwards to what makes a good pipeline. The stage names below are conventions, and not every application needs every environment. Each stage should provide useful confidence before we move to the next one.

Throughout this article, a replica means one running copy of the application, whether that is on a virtual machine, in a container, or in a Kubernetes pod.

Build and initial tests

First, there needs to be a build process that makes sure the software still compiles, where applicable. Alongside that, we run static checks such as linting or Checkstyle, and unit tests.

Once these checks succeed, we can run the in-app integration tests. These exercise components together without requiring a deployment to a shared environment. They can run in a test environment with the necessary dependencies, permissions, and credentials.

These tests prevent us from deploying something that we can already determine is broken. This is also where we build the Docker image, if applicable. That same image should then move through the remaining stages so we are deploying what we tested.

Alpha or pre-devo

The first deployment goes to alpha, or pre-devo. For the initial deployment checks, a single application replica can be enough.

This stage verifies that the application starts correctly and that its configuration, credentials, and dependency connections work in the deployed environment. It catches setup problems early, although it cannot guarantee that another environment’s credentials are configured correctly.

After deployment, we run integration tests against the application’s endpoint or entry point. If those succeed, we can run load tests, if available.

Using a single replica gives us a baseline for how many transactions per second it can handle under the tested conditions. That result depends on resources, request patterns, and downstream dependencies, so it should not be treated as a complete estimate of production capacity.

We can also run fault injection tests here, using a tool such as Gremlin. With one replica, however, we cannot test how multiple replicas behave when one fails.

Why have this environment? Can’t we simply deploy to devo?

Devo is often shared by UI/UX designers, product managers, and other software engineers. Catching a broken deployment before it reaches that environment helps keep their work moving.

Beta or devo

Next comes beta, or devo: the shared development and testing environment in this pipeline.

This stage should have at least two/three application replicas so we can exercise deployments while the application continues serving requests. Two replicas alone do not guarantee 100% uptime. The deployment must also keep enough capacity available, wait for new replicas to become ready, and allow existing requests to finish/drain before stopping old replicas.

Devo should have its own dependencies, such as databases, separate from production.

After a successful deployment, we run the integration tests against the devo endpoint again. If continuous canary tests are available, the pipeline should also wait for successful results from those checks. Here, canary tests mean recurring synthetic requests that exercise important application workflows.

Some teams add a minimum bake time: a period during which the deployment stays in the environment while an aggregate alarm watches for problems. If the alarm triggers, the pipeline stops progressing and may automatically roll back.

This can catch issues that take time to appear, such as increasing error rates or a growing resource problem. A quiet environment provides limited evidence, so successful checks and sufficient activity matter alongside elapsed time.

Teams should still add tests for uncovered use cases instead of depending on someone happening to notice a problem. Bake time complements those tests by giving delayed failures a chance to surface.

Gamma or pre-prod

The next stage is gamma, or pre-prod. Its purpose is to test the application against dependencies, configuration, and data structures that closely represent production.

This does not necessarily mean connecting to the live production database, Elasticsearch cluster, or storage resources. Sharing those resources would allow pre-production code and tests to affect production. Usually, isolated resources with representative data and matching schemas are the better fit.

Why does this matter? We want to know whether the new code is compatible with the data structures it will encounter in production, including while old and new versions run together.

Compatibility also needs to work in the rollback direction: the previous version must still be able to handle data written by the new version. Passing tests against an existing schema alone does not establish that.

Integration tests, continuous canary tests, and bake time belong here too. If we find a compatibility problem, we stop before exposing customers to the new version.

OneBox or one-pod production

Next comes OneBox, or one-pod production: a small part of the production deployment running the new version.

We take the selected replica out of traffic while updating it, or replace it with a new replica. Once it is ready, we return it to service so it receives a limited share of real customer traffic.

This reduces the blast radius of problems that only appear under production conditions.

Metrics from this replica should be tagged or isolated so we can distinguish its behavior from the rest of production. Otherwise, errors from one replica may disappear into healthy fleet-wide averages.

We can run production-safe integration tests against its specific endpoint. At a minimum, canary checks must exercise the new version, and we should observe it through a bake period with an aggregate alarm tied to its metrics. Successful checks against only the old replicas tell us nothing about the deployment.

If a problem appears, stop the rollout and roll back promptly, provided rollback is safe. Reverting code does not automatically undo changes it has already made to data.

Production

Finally, we roll the change out to the remaining production replicas, keeping sufficient healthy capacity available throughout the deployment.

Canary tests and alarm monitoring continue during the rollout and the bake period afterward. If a problem appears, further deployment should stop and a safe rollback should begin promptly.

The purpose of this pipeline is to build confidence at each stage while keeping the impact of a failed change small. Every stage should help answer the same question: do we have enough evidence to safely move this change further?

Deployment windows for OneBox and production

Promotions to both OneBox and the rest of production should have a time-window blocker. Only allow deployments during the responsible team’s working hours, and never on Friday, or whichever day is the last working day of the local workweek.

The window should leave enough time to observe the deployment and respond before the team signs off. If a promotion reaches either stage outside that window, it waits until the next allowed period. Passing OneBox does not bypass the production deployment window.

This blocker applies to promotions, not automatic rollbacks. If a deployment causes problems, recovery should begin immediately, even outside working hours.

Put it all together

Each stage builds confidence before we expose more of the system to a change. We start with build checks and tests, verify the deployed application in increasingly realistic environments, then introduce real customer traffic through OneBox before rolling out to the rest of production.

The diagram below brings that sequence together, including the deployment windows before OneBox and production.

Not every application needs every stage, but each stage should earn its place by catching problems before they can have a wider impact. A good pipeline makes changes easier to ship, failures easier to contain, and recovery easier to trust.

I spent almost a decade at Amazon designing and implementing these kinds of tests. If you’re looking for someone to help build reliable software and a release process you can trust, I’m happy to help. Get in touch.

I wrote the ideas and the substance of this article. AI helped with proofreading, formatting and producing the diagram above. The technical details were accurate at publication.