CI/CD: Necessary for modern software development, yet it carries a lot of risk

SCSW CI/CD over the past decade has become the cornerstone of modern software development.

The term – for continuous integration and continuous delivery (sometimes the "D" also can mean "deployment") – emerged in the late 2000s with the rise of DevOps, defining a way to more quickly create and update applications by leaning heavily on automation for everything from building to testing to deploying systems, pulling together contributions from myriad contributors into a pipeline, and speeding up release cycles.

Software is no longer built by a single developer on a single machine; instead, developers using disparate tools can contribute to the build within the pipeline without causing conflict. Organizations don't have to wait for software updates to be gathered together into a single large batch to be released at a set time and updates and improvements can be pushed out as soon as they are ready.

Builds are standardized, security shifts from shared to increasingly isolated resources and checks can be run on every change, and value is more quickly delivered, it's claimed. With CI/CD came a greater reliance on automation and infrastructure-as-code (IaC), more third parties being involved, and new frameworks and languages becoming rapidly adopted.

With speed comes risk

That said, the same speed that comes from a streamlined and automated shared CI/CD pipeline can also make it highly attractive to online miscreants.

"Today, CI/CD is where application code, build tools, third-party components, secrets, identities and even cloud resources come together," Adrian Diglio, principal program manager of secure software supply chain (S3C) at Microsoft, told The Register.

"CI/CD adoption grows at feature velocity speed and these interconnected pipelines outpace organizational maturity and their ability to keep them secure. This makes CI/CD a prime target for attackers."

CI/CD is where application code, build tools, third-party components, secrets, identities and even cloud resources come together

CI/CD expands the attack surface and intruders have become good at exploiting such systems to attack the software supply chain, as proven by the high-profile SolarWinds fiasco in 2020. In that case the Russia-linked Nobelium group compromised the IT software suite maker's build process and inserted malicious code into applications that subsequently went upstream to users.

Palo Alto Networks wrote in December 2022 that the number of supply chain attacks in the previous year jumped 51 percent. CI/CD pipelines are particularly vulnerable to such problems as misconfigurations (which can expose sensitive information and become entry points for malicious code) and permissive credentials (which can lead to lateral movement and CI poisoning).

Multiple threats to pipelines

Microsoft's Diglio added that the most prevalent initial access techniques are misconfiguration of software development lifecycle (SDLC) resources, malicious dependencies, and targeted developer attacks.

"In practice, this means attackers gain an initial foothold by manipulating CI/CD pipeline inputs, including code and configuration," he said.

By abusing broadly scoped tokens and other misconfigurations granting resource access, attackers can move deeper through their target's system

"Then attackers seek lateral movement. By abusing broadly scoped tokens and other misconfigurations granting resource access, often based on positional privilege, attackers can move deeper through their target's system and manipulate subsequent stages of software delivery."

From there, attackers can abuse production resources and compromise products distributed to third parties to spread attacks.

"CI/CD infrastructure compromises enable attackers to manipulate the software being built, making CI/CD infrastructure an attack surface for exploiting end users' trust," Diglio said.

CI/CD becomes an easier target

The pipelines are an easier target than more hardened and well-monitored production environments, according to John Steven, CTO at ThreatModeler. CI/CD pipelines tend to get less security attention and have little if any logging for what developers execute as part of the build, package, or deploy phases.

• SBOM is a 'massive galaxy of mess' for supply chain security
• Feeling VEXed by software supply chain security? You're not alone
• Datacenters in China, Singapore cracked by crims who then targeted tenants
• Open source software has its perks, but supply chain risks can't be ignored

Essentially, we're told, injecting malware or exploiting a vulnerability via an organization's CI/CD pipeline actions – or even into open source software or containers and images downloaded from external sources – is easier than successfully attacking a production environment without drawing notice. The invaders know this well.

"Attackers within an organization can add configuration to build phases that injects vulnerable or malicious dependencies," Steven told The Register.

"Build processes typically don't create – let alone retain – detailed logs of how code is constructed or transformed, so these injections would be 'invisible' compared to a nefarious configuration or source commit.

"Unless a later phase conducts detailed scanning of the produced binaries, those injections will remain undiscovered as they're orchestrated into production."

Unless a later phase conducts detailed scanning of the produced binaries, those injections will remain undiscovered as they're orchestrated into production

The Open Worldwide Application Security Project (OWASP) wrote about the recent surge in the number of incidents aimed at abusing the CI/CD ecosystem, with the frequency and magnitude of attacks also on the rise.

Defenses are in the early stages

Criminals are rapidly adapting techniques to target CI/CD, while many defenders are in the early stages of figuring out how to detect, understand and manage the risks.

"Seeking the right balance between optimal security and engineering velocity, security teams are in search for the most effective security controls that will allow engineering to remain agile without compromising on security," OWASP wrote.

Diglio said the large number of components that come with modern software delivery, and the increasing complexity of CI/CD, complicate software supply chain security considerations.

"Organizations must lead with a defense-in-depth approach spanning source integrity, build integrity, release integrity, dependencies, and access controls," he said.

The Microsoft executive outlined a number of steps enterprises can take to harden CI/CD pipelines, including performing an assessment using the Secure Supply Chain Consumption Framework (S2C2F), a tool developed and used by the software behemoth since 2019 to secure its own development processes.

In November 2022 Microsoft contributed the S2C2F to the OpenSSF (Open Source Security Foundation). The framework is designed to address real-world supply chain threats that are specific to open source software. An assessment using it will help organizations understand how to improve the security of open source consumption practices, Diglio said.

Security steps to take

Enterprises also need to address CI/CD misconfigurations, limit access to the CI/CD infrastructure and related services, and extend detection to the CI/CD infrastructure. They also need to harden IaC against tampering.

DevOps teams should "start addressing third-party dependency risk today by taking inventory and understanding dependencies, reducing them where practicable, and monitoring them," Diglio said, pointing to testing and debugging tools like Dependabot in Microsoft-owned GitHub. "Work incrementally on pinning, proxying and rebuilding those dependencies in-house as your organizational maturity and confidence increases."

In addition, developers need to be included in their companies' security programs. This includes enabling multi-factor authentication (MFA) and conditional access, and reviewing existing permissions across the CI/CD infrastructure as part of the principle of least privilege in a zero-trust strategy.

"Educate your developers about security risks and how security threats can put their enterprise at risk," Diglio said.

CI/CD pipelines aren't going anywhere. There are integral to the larger DevOps push and adoption of agile development. However, all that makes them attractive and, for now, vulnerable areas for attacks. Miscreants understand this and are putting a focus on the software supply chain. Organizations now have to take the steps to harden the process. ®