CI/CD stands for Continuous Integration and Continuous Delivery, which sometimes extends to Continuous Deployment. It’s a set of practices aimed at ensuring code changes are frequently integrated, automatically tested, and taken through the steps required to be confident the software version can be deployed to production. In a CI/CD pipeline, code changes are automatically built, tested, and deployed, ensuring a flow from code commit to production deployment.
The CI process encourages frequent code integration and provides fast feedback on whether the changes result in a deployable software version. The first phase builds and tests the new software version and should let a developer know within 5 minutes if their changes have caused a problem. Longer-running tests are deferred to a later phase, but still precede any manual steps to ensure people don’t waste time on a bad software version.
The CD part of the pipeline automates the delivery of the software to different environments, ensuring the steps to deploy the new version are repeatable, reliable, and secure. This ensures the same process is used for all deployments and prevents outages caused by configuration errors.
What are the differences between Continuous Integration, Continuous Delivery, and Continuous Deployment?
Continuous Integration (CI) involves frequently integrating code changes into the main branch in a shared repository, enabling automated build and testing. This practice aims to identify integration issues early, reducing the complexity of merge conflicts and bugs. Each change triggers an automated build to create a new software version, with unit and integration tests that check that the code changes haven’t broken the system.
Continuous Delivery (CD) takes the new software version and moves through all the steps required to deploy it to production. This usually involves steps that increase the organization’s confidence in the new software version, some of which may be manual.
Continuous Deployment (CD) is a flavor of Continuous Delivery where the entire deployment pipeline is automated. Each change that passes the automated validation steps is deployed to production automatically.
CI/CD versus DevOps: What are the differences?
CI/CD is a subset of modern DevOps. While DevOps has a broad set of technical and cultural techniques and practices, CI/CD is focused on the flow of software changes and rapid feedback. The research has shown that CI/CD increases key outcomes such as team performance, software delivery performance, operational performance, and organizational performance.
DevOps aims to improve software delivery throughput and stability while aligning the goals of developers and IT operations teams to improve collaboration. CI/CD pipelines are one key enabler for achieving DevOps goals, but to fully implement DevOps, organizations must adopt additional practices and cultural changes beyond CI/CD.
What are CI/CD pipelines?
CI/CD pipelines are a sequence of steps that respond to changes to the code and bring confidence that the new software version is good to deploy. There are many benefits to be gained from automating the steps that follow the code commit, though many teams retain a reduced set of manual steps, which may be as small as scheduling the deployment.
A CI/CD pipeline orchestrates these tasks to ensure code changes are reliably and consistently moved from development to production. By automating these steps, teams can reduce the manual effort required for deployments, improve code quality, and accelerate the release process.
Benefits of CI/CD pipelines for Kubernetes
Kubernetes is a popular, open-source container orchestration system that has become the backbone of DevOps infrastructure for many organizations. Implementing CI/CD pipelines in a Kubernetes environment offers several advantages that enhance both the development process and operational efficiency:
- Deployment confidence: Kubernetes is an orchestrator for containerized applications with features for self-healing, service discovery, and the ability to scale on demand or in response to system conditions. CI/CD pipelines make sure only good versions of containerized applications make it to your production cluster.
- Reduce YAML sprawl and GitOps noise: CI/CD pipelines can use a single YAML manifest and substitute the values as the container is deployed to different environments or tenants. This reduces the number of YAML files under management and reduces the number of version control changes by eliminating noise from tasks such as key rotation.
- Better visibility: CI/CD pipelines make it easy to see what applications, services, and databases are deployed across multiple Kubernetes clusters and for other services, such as cloud database services or functions.
- Increased throughput and stability: CI/CD pipelines help improve throughput and stability for Kubernetes, just as they do for other technologies. Automated stages make the process more repeatable and reduces toil, which helps teams deliver more often and with fewer failures.
- Enhanced observability and monitoring: The Kubernetes ecosystem provides robust monitoring and logging capabilities, which can be integrated into CI/CD pipelines. Tools like Prometheus and Grafana can be used alongside CI/CD pipelines to create detailed dashboards and alerts, ensuring that the deployed applications are working as intended.
The key stages of a CI/CD pipeline
If you are planning to build a CI/CD pipeline, your first task should be to follow a change as it currently flows from development through to production. By capturing each step that must happen to progress the change, you’ll discover your organization’s existing pipeline.
The steps usually align with these CI/CD pipeline stages.
1. Commit stage
The commit stage is the initial step in a CI/CD pipeline, where developers push their code changes to a shared version control repository, such as Git. The developer may have to merge other developers’ changes into their working copy before they can commit their code, though frequent commits reduce the chance of code conflicts.
This stage typically includes pre-commit checks like linting, static code analysis, and security scans to catch syntax errors, enforce coding standards, and detect vulnerabilities early. These checks are crucial in ensuring that only high-quality secure code progresses further in the pipeline. Additionally, some pipelines might incorporate peer code reviews as part of this stage, which adds an extra layer of scrutiny before the code is integrated into the main branch.
2. Source stage
In the source stage, the pipeline retrieves the latest code from the version control system. This stage is vital for ensuring that the pipeline operates on the most up-to-date version of the codebase, reflecting all recent changes and merges.
The source stage isn’t only about fetching code; it also prepares the environment by downloading dependencies, setting up configurations, and ensuring that all necessary tools are in place for the build process. This preparation ensures that the subsequent stages can proceed smoothly without interruptions due to missing dependencies or misconfigurations.
3. Build stage
The build stage is where the source code is transformed into an executable form, such as a binary, a Docker image, or a deployable package. This stage is crucial as it compiles the code and links the necessary libraries, producing the artifacts that will be deployed to various environments.
The build process can vary significantly depending on the programming language, framework, and target environment. In addition to compiling, the build stage often includes tasks such as running automated unit tests, performing code coverage analysis, and packaging the application. A successful build results in a set of artifacts ready for deployment; if the build fails, the pipeline stops, signaling that issues in the code need to be resolved before continuing.
4. Artifact repository
When the build stage succeeds, it should lodge the resulting artifact in a repository. This artifact represents the software version, and your CI/CD pipeline should refer to this canonical version throughout, to ensure the artifact you validate is the same one you deploy to end users. You’ll also want to return to this artifact if you need to redeploy it later.
Artifact repositories provide the chain of custody for software versions, preventing problems that can occur if you build the same code with different dependencies or updated build settings. Preserving the output of the successful build is always easier than trying to recreate the precise context of the moment in time when it took place.
5. Acceptance test stage
In the acceptance test stage, the software version is deployed to an environment and automated smoke tests and acceptance tests are run. While the build stage validates the system works on a technical level, the acceptance test stage validates the system at a functional and non-functional level.
When a software version passes the acceptance test stage, it should result in high confidence the software behaves as users expect and meets the customer’s specifications. When a software version fails this stage, it prevents the software from progressing to stages where it would waste time (such as manual testing) or impact users.
6. User acceptance and capacity testing stage
Many organizations have additional testing requirements such as exploratory testing, user acceptance testing, or capacity testing. Where manual testing is required, it should be reserved for the user acceptance stage. This avoids wasting time on software versions that don’t meet the requirements of the acceptance test stage. Testers should be able to self-serve deployments to their test environments.
When issues are found during this stage, some thought should be given to how to detect them in an earlier automated stage. Where no issues are found over time, it may be possible to reduce or remove tasks in this stage.
7. Deploy stage
The deploy stage is the final and often the most critical step in the CI/CD pipeline. In this stage, the validated and tested artifacts are deployed to the production environment where end-users will interact with the application.
Deployment involves several intricate steps, such as provisioning infrastructure, setting up databases, configuring environment variables, and deploying application code. In modern DevOps practices, this stage is fully automated to ensure consistency and minimize human error.
For example, in a Kubernetes environment, the deploy stage might involve updating Kubernetes manifests, applying them to the cluster, and managing the deployment rollout process, such as handling rolling updates or rollbacks. The goal of the deploy stage is to ensure that the deployment is seamless, reliable, and can be performed repeatedly with minimal risk.
Steve Fenton is a Principal DevEx Researcher at Octopus Deploy and a 7-time Microsoft MVP with more than two decades of experience in software delivery.
CI/CD in the cloud
CI/CD in AWS
Amazon Web Services (AWS) offers a suite of tools for implementing CI/CD pipelines in the cloud. The key services include AWS CodePipeline, CodeBuild, CodeDeploy, and CodeCommit (no longer available to new customers). AWS’s security features include identity and access management (IAM) and virtual private cloud (VPC), while CloudWatch and CloudTrail provide insights into pipeline performance and security.
CI/CD in Azure
Azure provides a set of services for implementing CI/CD pipelines, including Azure Pipelines, Azure Repos, and Azure Artifacts, all of which are part of the Azure DevOps suite. Azure’s global network of data centers, along with its monitoring tools like Azure Monitor and Application Insights, ensures that CI/CD pipelines are resilient and performant.
CI/CD in Google Cloud
Google Cloud offers a range of tools for setting up CI/CD pipelines, including Google Cloud Build, Google Container Registry, and Google Kubernetes Engine (GKE). Google Cloud’s security features, such as Cloud IAM and VPC Service Controls, protect CI/CD pipelines from unauthorized access. Stackdriver provides monitoring and logging capabilities, with real-time insights into the health and performance of CI/CD processes.
Key features of CI tools
Let’s explore the key capabilities of the two primary components of a CI/CD pipeline—CI tools and CD tools, starting with CI.
Integration with version control systems
One of the fundamental features of Continuous Integration (CI) tools is their integration with version control systems (VCS) like Git, Subversion, and Mercurial. This integration allows CI tools to automatically trigger builds, tests, and other pipeline processes each time code is committed to the repository.
By continuously monitoring the repository for changes, CI tools ensure that the latest code is always tested, reducing the risk of integration issues and making the development process more efficient.
Build automation
Build automation is an essential feature of CI tools, transforming source code into executable artifacts, such as binaries, libraries, or container images. CI tools automate the entire build process, from compiling the code to packaging the artifacts, ensuring consistency and repeatability.
This automation reduces the potential for human error, speeds up the build process, and ensures that the build environment is consistent across different development stages, leading to more reliable software releases.
Automated testing
Automated testing ensures that code quality is maintained throughout the development cycle. CI tools automatically run a suite of tests, including unit tests, integration tests, and sometimes even functional tests, every time new code is pushed to the repository.
The immediate feedback from automated testing helps developers identify and fix issues early, before they can affect the stability of the project. Automated testing within CI tools is highly customizable, allowing teams to define specific test stages and conditions under which the pipeline should proceed or halt.
Artifact management
CI tools typically include or integrate with artifact management capabilities, allowing teams to store and manage build outputs in a centralized repository. These artifacts, which could include compiled code, Docker images, or packaged applications, are versioned and stored in a way that ensures they can be retrieved for future use, such as deployment to different environments or rollback scenarios.
Artifact management is crucial for maintaining traceability and ensuring that the exact versions of artifacts used in testing are available for production deployment.
Key features of CD tools
Technology independent
Continuous Deployment (CD) tools are designed to be technology-independent, allowing them to work with a wide range of platforms, languages, and infrastructure environments.
CD tools enable teams to deploy any type of application to any environment, including cloud, on-premises, virtual machines (VM), and Kubernetes. CD tools typically integrate with multiple orchestration tools, cloud providers, and deployment strategies, ensuring that they can accommodate diverse deployment scenarios.
Deployment automation
Deployment automation is a core feature of CD tools. It streamlines the process of moving code from testing environments to production. By automating the deployment process, CD tools minimize the risk of human error, reduce deployment time, and ensure consistency across environments.
This automation includes tasks such as configuring environments, managing infrastructure, and executing deployment steps. Automated deployments also support advanced strategies like blue/green and canary deployments, which improve application reliability and availability during deployments.
Environment progression
Environment progression in CD tools refers to the structured movement of code through various environments—such as development, testing, staging, and production—before it’s released to end-users. It enforces a strict deployment process and ensures the same artifacts and process are used throughout the development lifecycle.
CD tools manage this progression by enforcing policies, running additional tests, and capturing approvals at each stage. This controlled approach ensures that only thoroughly tested and validated code reaches production, reducing the likelihood of defects and ensuring a stable release process.
Variable management for environments, regions, locations, and instances
CD tools provide robust variable management capabilities, allowing teams to manage environment-specific configurations, such as environment variables, API keys, and deployment parameters. This feature is crucial for ensuring that applications behave correctly in different environments, regions, or physical locations.
CD tools also allow for customer-specific configuration, where unique parameters are required for different customer instances, ensuring that deployments are tailored to meet specific requirements.
Manual interventions and approval capture
While all tasks in a CD pipeline should be automated, having manual interventions and approval capture can provide an additional layer of control. Manual interventions allow the automated process to request human assistance, which can help overcome a transient problem during deployment, such as a timeout connecting to a deployment target.
Manual approval gates let stakeholders review and approve deployments, adding a safeguard against unintended changes reaching end-users. This can include integration with tools like ServiceNow or JIRA Service Management, enabling change management in an ITSM environment.
Role-based access control and audit trails
CD tools offer role-based access control (RBAC) to ensure that only authorized personnel can modify or trigger deployments. This feature is critical for maintaining security and compliance in the deployment process.
CD tools also maintain full audit trails, recording every action taken within the pipeline, including code changes, deployment activities, and manual interventions. These audit trails provide transparency and accountability, enabling teams to trace issues back to their source and ensuring that the deployment process complies with organizational policies and regulations.
Best practices for effective CI/CD
Commit frequently
Frequent commits are crucial for Continuous Integration as they reduce the chance of change collisions and make it easier to pinpoint where a problem was introduced. Smaller commits reduce the complexity of code reviews and provide more feedback opportunities from automated tests.
By committing code frequently, teams maintain a consistent development rhythm, minimizing the disruption caused by large, infrequent updates.
Reduce code review waits
A common problem in CI/CD pipelines is developers waiting for their changes to be reviewed. If code reviews take a long time, developers naturally start increasing the amount of work they do before they request a review. This makes the code reviews larger and delays them even further.
Where possible, use real-time code review techniques, such as pair or ensemble programming. If you can’t use a real-time process, monitor your review queues to make sure code reviews are prioritized above other work, especially further code changes.
Keep builds fast
Fast builds are crucial for maintaining a quick feedback loop in CI/CD pipelines. If your build takes more than 5 minutes, it will hinder development speed, delay feedback, and reduce productivity. Techniques like incremental builds, parallel processing, and caching can significantly reduce build times. You can also defer long-running tests to the acceptance stage.
Optimizing build processes ensures that code changes are validated swiftly, reducing the time developers spend waiting for builds. Quick builds enable faster iterations and shorter development cycles, contributing to the continuous delivery of software.
Run tests early and often
Running tests early and often in the CI/CD pipeline detects issues sooner. This reduces the number of bad versions reaching manual stages and wasting time and stops them reaching production where they’ll impact users. Early testing ensures that defects are identified and resolved promptly, reducing the cost and complexity of bug fixes.
Frequent testing fosters a culture of quality, ensuring that code changes are always validated against predefined standards. Early and continuous testing accelerates development cycles and enhances the reliability of deployed applications.
Use the same deployment process everywhere
Use the same process to deploy to every environment so the deployment process is tested as often as the software. Environment-specific variables can be replaced automatically during a deployment, which means the process is a reusable template.
If you automate deployments to pre-live environments but deploy to production manually, you increase the chance of a failed deployment. Where you depend on another team to perform the production deployment, you’ll need to work together to make it possible to use the same process everywhere.
Use feature flags for incomplete features
Feature flags decouple the deployment and release of new features. You can deploy a new software version with the feature switched off. You can then decide when to enable the feature for some or all users to gather feedback, knowing you can easily toggle it back off if there’s a problem.
You can deploy the in-progress features as often as necessary while they aren’t user-visible. That means you don’t need to use long-lived feature branches to prevent users from seeing the changes and can commit and deploy as often as necessary during the feature’s development.
Implement incremental deployments
Incremental deployments release updates in small, manageable increments rather than all at once. Techniques like blue/green deployments, rolling updates, and canary releases provide incremental deployment, reducing risk and keeping applications available during deployment.
By deploying changes incrementally, teams can closely monitor performance, quickly identify issues, and rollback changes if needed. Incremental deployments enhance reliability, ensuring that applications remain available and stable during updates.
Prefer best-in-class CD tools to all-in-one tools
When setting up a Continuous Deployment (CD) pipeline, it’s often more effective to use specialized, best-in-class tools rather than relying on all-in-one solutions. Modern CI tools can execute scripts as part of their pipelines, but they are missing features that let you scale your deployments or handle complex deployment scenarios. Often, they simply provide an execution environment for extensive and brittle homemade scripts.
Best-in-class CD tools, designed to excel in specific areas, such as deployment automation, environment management, and configuration management, offer greater flexibility, advanced features, and ultimately, more reliable CI/CD pipelines. By selecting the best tools for each aspect of the CD process, teams can tailor their pipeline to their specific needs, ensuring optimal performance and scalability.
Learn more in our detailed guide to CI/CD best practices
Deployment automation in your CI/CD pipeline with Octopus
Octopus is a world-leading CD tool that can handle complex deployments at scale. Octopus helps software teams streamline Continuous Delivery so they can deliver value faster. More than 4,000 organizations around the world use our solutions for Continuous Delivery, GitOps, and release orchestration to keep their software delivery running smoothly. Octopus manages software delivery across multi-cloud, Kubernetes, data centers, and hybrid environments, whether you’re working with containerized apps or legacy systems.
We support Platform Engineering teams in improving the Developer Experience (DevEx) while maintaining governance, risk, and compliance (GRC). We’re also committed to contributing to the developer community through open-source projects like Argo in the CNCF, and other efforts to advance software delivery and operations.
Help us continuously improve
Please let us know if you have any feedback about this page.
