Scalable CI/CD Pipeline Examples: Improve the Development Churn

CI/CD pipeline has gradually become a pillar of success in achieving sustainable delivery of bug-free code at high velocity. After all, customers love momentum! Month-long software release cycles belong to the bygone era for good. No wonder enterprises are giving their all to build a scalable CI/CD pipeline that benefits developers, end-users, and the business as a whole.

However, its efficiency depends heavily on automation. And that’s why it is essential to select the best CI/CD tools that enable meaningful and effective implementation of continuous integration and continuous delivery throughout the software delivery cycle.

Read the article to know the basics, essential stages, attributes, and examples of a properly configured CI/CD pipeline that ultimately leads to a faster and more agile development life cycle.

Let’s first understand how CI/CD improves the workflow by bridging the gaps between development and operation activities and integrating automation in various stages of software development.

CI/CD pipeline is a cornerstone of the DevOps approach, a series of orchestrated steps that drives software development by building codes, running tests, and deploying new software versions into all necessary environments.

The ultimate goal of the CI/CD pipeline is to minimize human errors and have a consistent software release by automating repetitive processes. By automating various phases of the CI/CD pipeline, development teams can work faster and improve software quality and other DevOps metrics. Here are some basic facts about the CI/CD pipeline:

• Software pipelines are mostly customized to suit the particular needs of the project and the business.
• Various types of tools for binary creation, code compilation, unit tests, code analysis, and security can be included in the pipeline.
• Containerized environments automate the packaging of the code into a container image that can be deployed across a hybrid cloud.
• Multiple pipelines are often required to take source code into production based on organizational and team structures.
• A CI/CD pipeline can be triggered by an event, like a code change (pull request), having a new artifact in an artifact repository, or some defined schedule to match a release cadence.

A CI/CD pipeline generally has several scripts that require execution in a specific order to achieve a common goal. Code – submitted at the start of the pipeline, passes through a series of stages inside the pipeline, and then gets published as production-ready code.

Generally, the CI/CD pipeline has four significant stages – source, build, test, and deploy. Each phase follows proper standards, well-detailed processes, and specific tools. Let’s explore these stages one by one.

Source

In this first stage, developers translate requirements into functional algorithms, features, and behaviors. Any change in a preconfigured flag in the code repository or the program triggers the CI/CD pipeline. Other common triggers are – user-initiated or automatically scheduled workflow, results of other pipelines, etc.

The tools employed for this stage depend mainly on the development languages and code-checking features, like vulnerability scanning, basic error detection, and adherence to established coding standards. Here are the commonly used tools –

• AWS CodeCommit
• Git (Distributed Version Control System)
• Subversion – SVN (Centralized Version Control System)
• Team Foundation Version Control – TFVC (Centralized)

Build

This stage of the CI/CD pipeline draws source code from the repository, links it to relevant libraries, dependencies, and modules, and builds an executable file. Besides building code, build automation also involves using tools to ensure code’s security and if it’s following the best CI/CD practices accordingly.

Meanwhile, tools used in the build stage also generate logs of the process, analyze the code for stylistic and programmatic errors, and notify developers about the code completion. Here are some standard tools used in the build stage:

• Jenkins
• Gradle
• Travis CI
• Azure Pipelines

Test

Now that the source code has been through static testing, the build enters the phase of dynamic testing that typically starts with the basic unit or functional testing. Then the build is subjected to various tests like regression testing, smoke testing, integration testing, compliance testing, end-to-end testing (E2E), etc., to validate its operation.

This phase lasts from seconds to hours as per the size and complexity of the project. Test specialists create highly-detailed test cases and criteria but rely on test tools. Here are some mainstream testing tools:

• Selenium (popular open-source Web UI automation testing suite)
• Appium (cross-platform)
• Jest (JavaScript testing framework)
• Playwright (cross-browser testing on complex applications)

Deploy

In a continuous delivery pipeline, the build is first sent to human stakeholders for approval, while the build automatically gets deployed after passing its test suite in a continuous deployment pipeline. In this phase, the approved code is packaged as an artifact and deployed to relevant environments, mainly to a staging environment first, then QA, and ultimately to the production environment.

This stage should be adapted to support the suitable deployment strategy, from blue-green deployments to canary deployments to in-place deployments. Deployment naturally involves additional precautions and live testing periods, including A/B tests, beta tests, blue/green tests, etc., to roll back unexpected errors and minimize the business impact. The deployment stage may involve infrastructure provisioning (Terraform), configuration (Puppet), and containerization (Docker, Kubernetes, etc.). Other popular tools include:

• Chef
• Ansible
• AWS Code Deploy
• AWS Elastic Beanstalk
• Azure Pipelines

The ultimate purpose of employing a CI/CD pipeline is to give the teams fast, reliable, accurate, and comprehensive feedback for their development cycle and let them deploy the best-quality software. Therefore, an outstanding CI/CD pipeline must have a few basic characteristics. Let’s see what they are:

Accuracy

The responsibility of modeling simple and complex workflows lies on CI/CD tools. This, in turn, removes manual errors in repetitive tasks and enables the CI/CD pipeline to run perfectly and have an accurate vision of the entire software delivery process.

Reliability

A reliable pipeline would run every time without throwing errors, eliminating developers’ frustration over increased workload on top of the compromised software quality. Self-hosted build agents offer higher flexibility but can often cause unreliable pipelines owing to maintenance issues. However, an experienced team of developers can successfully resolve the challenges and maintain reliability.

Speed

An optimized pipeline runs quickly and provides rapid feedback to the developers regarding the success or failure of the code, reducing the possibilities of distractions. Faster pipelines with lesser runtime also allow more deployments. However, the pipeline runs often result in a queued state. The solution is providing multiple agents enabling different pipelines to run in parallel. A serverless model or container orchestration dynamically scales the build agent capacity when demand is high.

There are multiple ways and, of course, the tools to create a pipeline. Some names like Jenkins, Docker, Kubernetes, and Git might instantly come to your mind when someone says as much as CI/CD. Fortunately, you can try different combinations of tools that serve your purpose.

Scaling microservices pipelines

More and more organizations today are adopting microservices. However, microservices pipelines are different from traditional ones. And with the growing number of microservices, managing separate CI/CD pipelines for each service might quickly get out of hand.

Challenges

• If an organization uses one CI/CD pipeline per service, they’d have to support hundreds or thousands of pipelines, each connected to a separate Git repository.
• A microservice might also have multiple pipelines for internal use, production, or testing environments, which leads to even more duplication.
• Scaling of microservices naturally results in the increased requirement of network and storage. Moreover, running validation and integration testing on thousands of CI/CD pipelines can be expensive.
• Consolidating codebases with shared libraries and plugins can be quite challenging. Shared libraries generally create version-specific conflicts.

Solutions

• Containerize a pipeline to let it run independently with different language versions.
• Use one malleable pipeline for every integration and deployment. Triggers can carry information like metadata or context, enabling a pipeline to operate accordingly.
• Adopt a canary release strategy that allows new software to get released and tested among a selected subgroup of users first.

Our first example is of an application that uses microservices, containers, and the container orchestration platform.

Containerized CI/CD pipeline with automation server

With various benefits and a vast list of use cases in the cloud-native world, Kubernetes has become a natural choice for many organizations to run their CI/CD pipelines. This container orchestration tool accelerates the coding process with an intuitive user interface.

You’d need various components to deploy a standard CI/CD pipeline to the Kubernetes cluster.

• A version control system like Git is generally the first place where continuous integration begins. Multiple developers push the code changes and updates into a code repository of a VCS.
• Containerizing the software with a container such as Docker helps achieve seamless encapsulation and integration.
• An appropriate CI/CD tool like Jenkins, Travis, etc., allows you to build the docker image, run a series of tests, and push the build image to the Kubernetes cluster.
• Kubernetes helps orchestrate and deploy the container for the software build verified by the CI/CD tool.

Although a Kubernetes CI/CD pipeline streamlines deployment, monitoring, and management for microservices, you’d need to adopt good CI/CD practices to ensure security.

Serverless CI/CD pipeline for front-end

Serverless computing abstracts the infrastructure, servers, and operating systems, enabling modern developers to focus on application development. And so, building CI/CD pipelines for serverless is a bit different from that of the traditional architecture. Also, deploying and managing such applications presents a unique set of challenges.

Serverless CI/CD pipeline for front-end

Above is an example of building a CI/CD pipeline with Azure services for the web front-end of a serverless application. As prerequisites for this application, you must have – a GitHub account, an Azure account, and an Azure DevOps organization.

• Keep your source code in an online version control system, which you can easily integrate with leading CI/CD services and is easier to maintain than the traditional on-premise VCS.
• The project files are kept in the GitHub repository.
• Azure pipelines let you automate the build and deploy processes. You can create various stages in the pipeline that’d run in a Windows or Linux container. And each stage will run based on the previous one’s result.
• Integrate modern build tools depending on your application’s programming language, platform, and additional functionality requirements. These tools may allow you to –
  • run ad-hoc commands,
  • order the execution of commands depending on dependencies,
  • parallelize different commands,
  • oversee file changes and run commands based on those changes,
  • configure and reconfigure
• Automate the build process to reduce the human errors
• The software contents are deployed to the Blob storage (for static website hosting), which acts as the Azure CDN’s origin server. Atomic deployment ensures that the users always get the content corresponding to the same version.
• A set of distributed servers – Azure CDN accelerates the content delivery over a vast geographical area.

With a complete set of CI/CD tools, Amazon Web Services offers scalable, reliable, and cost-effective cloud computing services. As CodePipeline is quite simple to use, it makes more sense to use it if you are already on the AWS cloud. Moreover, it can get along easily with AWS tools and the AWS ecosystem and provide added advantages, including Amazon’s security and IAM controls.

Here’s an example of how to build a complete CI/CD pipeline with AWS Developers Tools and services.

AWS for CI/CD pipeline

• Git repository is used in this example, but you can also use AWS CodeCommit repository or Amazon S3.
• With every code change, AWS CodePipeline automates the build, test, and deploy phase, depending on the defined release model.
• AWS CodeBuild – a fully managed CI service – compiles source code, runs tests, and packages the source code for deployment. CodeBuild scales continuously and can process multiple builds simultaneously, so the builds don’t have to wait in a queue.
• Amazon ECR is a managed container registry offering high-performance hosting, so you can reliably deploy artifacts and application images anywhere.
• AWS CloudFormation, as an infrastructure automation platform, deploys AWS resources in a recurrent, testable, and auditable manner.
• Amazon ECS, a fully managed container orchestration service, profoundly integrates with the rest of the AWS platform. So, you can easily deploy, manage, and scale containerized applications.

How did Simform help its client build a scalable CI/CD pipeline?

One of our clients wanted to build a white label software to help auction houses go digital. Their main priority was to have a scalable CI/CD pipeline that’s also secure and cost-effective. So, we

• Used a container-based pipeline, which made it easier to run different language versions independently.
• Made the pipelines unified to scale faster to deployment
• Ran tests on each push in every branch to fast-track code review
• Followed the ‘infrastructure as code’ and ‘pipeline as code’ approach

Here’s how we went about each of the stages, along with the tooling for the process.

As the project scales and more developers work on the software, there will be more releases, and the CI/CD pipeline can soon become a bottleneck. In this case, any degree of automation can bring a positive change to the entire software development process. Simform’s qualified team of engineers can efficiently manage your software development life cycle with the best-suited CI/CD tools and design or redesign the automation pipeline. So let’s introduce ongoing automation throughout the lifecycle of your dream project.