Key Takeaways from the State of DevSecOps Report by Datadog

In an era of relentless cyber threats and high-profile data breaches, a new philosophy has taken center stage in the software development world: DevSecOps. The practice of integrating security into every phase of the development lifecycle isn’t just a buzzword; it has become a necessity nowadays. With the ultimate goal of shipping secure code rapidly and scalably, you require a deep understanding of the current threat landscape and the adoption of modern security practices.

Datadog’s latest State of DevSecOps report provides a comprehensive, data-driven look into how organizations are faring on this journey. By analyzing tens of thousands of applications, container images, and cloud environments, the report uncovers crucial trends and provides improvement ideas, making it a goldmine for DevOps and platform engineers and more professionals.

I have distilled the comprehensive report into the most critical takeaways and sorted to tackle the most pressing issues first. This is what you and your teams need to know to bolster your security posture, drive operational excellence, and focus on what truly matters – in the age of data and AI.

The Ticking Time Bomb: Securing CI/CD with Short-Lived Credentials

In cloud-native environments, the CI/CD pipeline is the medium for fast deployments, but it’s also a prime target for attackers. These pipelines often require privileged access to deploy applications and infrastructure, making their credentials the “keys to the kingdom.” The report reveals a dangerous trend: the widespread use of long-lived credentials that creates a massive attack surface.

The Alarming Statistics

63% of organizations using GitHub Actions with AWS have used long-lived IAM user credentials.
A staggering 42% of organizations exclusively used these static, long-lived credentials.
Only 37% have adopted the secure authentication method using short-lived credentials via OpenID Connect (OIDC) for AWS IAM (Identity and Access Management).

The report states that leaked long-lived credentials are one of the most common root causes of major data breaches. In the case of the Codecov breach , attackers gained access to their CI/CD environments through a single compromised credential, which impacted thousands of downstream customers. When a static key is hardcoded or leaks through logs, build artifacts, or a compromised dependency, it provides an attacker with a persistent, privileged entry point into your cloud environment.

The Path Forward: Embrace Keyless and Short-Lived

The solution is clear and often easier to implement. Modern authentication mechanisms like OpenID Connect (OIDC) allow CI/CD platforms like GitHub Actions or GitLab to establish a trust relationship with your cloud provider (e.g., AWS or GCP). Instead of storing a static secret, the pipeline requests a temporary, short-lived token for each specific job. Benefits of this approach include:

Dramatically Reduced Risk: If a token were to leak, its lifespan is measured in minutes or hours, not months or years, severely limiting the window of opportunity for an attacker.
Simplified Secret Management: No more rotating static keys or worrying about where they are stored. The process is automated and dynamic, reducing attack surface.
Improved Operational Efficiency: As the report notes, good operational practices often lead to better security outcomes. For example, a one-time configuration is required to set up OIDC that enhances security and simplifies pipeline management too – in the long run.

Drowning in Noise: Why Most Alerts are a Distraction and What to Look for

Security teams are facing a deluge of alerts due to automated security scanners and bots constantly probing for vulnerabilities in publicly-facing applications and networks. This creates a overwhelming amount of noise that can easily obscure real, actionable threats.

The report’s findings confirm this reality in stark terms: only 0.0065% of millions of malicious requests from automated security scanners actually resulted in a successful exploit. This means that only 1 of every 15,000 alerts captured by a log analysis tool or WAF might correspond to a genuine, successful attack. The result: security teams relying on these noisy signals is a recipe for alert fatigue and burnout, leaving digital defenders chasing ghosts while real threats slip through the cracks.

The Real Culprit: Your Application’s Dependencies (Especially in Java-based Applications)

The report points a firm finger at third-party and open-source dependencies – the libraries and frameworks that form the foundation of modern applications. This is especially true for Java when compared to other popular development ecosystems like Python and Javascript.

90% of all Java services have at least one critical or high-severity vulnerability originating from a third-party library. This is nearly double the average of 47% for apps in other languages.
55% of Java services are affected by vulnerabilities listed in CISA’s Known Exploited Vulnerabilities list, which is actively used by attackers. This compares to just 7% for other languages.
63% of these high-stakes vulnerabilities in Java come from indirect dependencies. These are the dependencies of your dependencies, often pulled into your application without your direct knowledge, making them incredibly difficult to track using in-house methods.

Vulnerabilities like Log4Shell, Spring4Shell, and the recent RCE in ActiveMQ are prime examples of how a single flaw in a popular, deeply nested library can expose thousands of organizations overnight. The key takeaway is twofold: First, you must have a mechanism to filter out the noise from low-impact, unsuccessful scans. Second, you must shift your focus inward to the software supply chain. Knowing your dependencies, both direct and indirect, is paramount in the modern era.

The Art of Prioritization: Not All Vulnerabilities Are Created Equal

The sheer volume of identified vulnerabilities is overwhelming. In 2023 alone, over 5,000 new high or critical CVEs were published. The report finds that any average application is vulnerable to 19 such issues. With development backlogs already full, how can teams possibly address them all?

Academic research has shown that only about 5% of all vulnerabilities are ever actively exploited by attackers. The challenge lies in identifying those critical 5%. Relying solely on a vulnerability’s base CVSS score is insufficient since it lacks the real-world context of your specific environment.

The report advocates for an “adjusted score” that enriches CVE data with critical runtime context. To determine which vulnerabilities truly warrant immediate attention, ask these three questions:

Is the vulnerable service publicly exposed to the Internet? An internal-only service with an RCE flaw is far less dangerous and urgent than an Internet-facing one.
Is the service running in a production environment? A vulnerability in a temporary dev or test environment carries a lower immediate risk than one in production.
Is there publicly available exploit code? The existence of a PoC exploit or attack instructions for the vulnerability dramatically increases the likelihood of an attack.

When this prioritization framework was applied, the results were transformative:

63% of organizations initially having “critical” vulnerabilities per CVE scores got to 0.
Another 30% saw their number of critical vulnerabilities down by 50% or more.

This is a game-changer. It transforms vulnerability management from an endless game of whack-a-mole into a focused, risk-driven strategy. By integrating application context, security teams can provide developers with a short, high-impact list of what to fix now, building trust and making security an achievable goal rather than an always-growing backlog and a source of frustration.

The Automation Gap: Growing IaC Adoption vs. Persistent “ClickOps”

Infrastructure as Code (IaC) has been a cornerstone of the DevOps movement for years. Tools like Terraform, CloudFormation, and Pulumi allow teams to define, version, and deploy infrastructure through code, bringing immense benefits for traceability, reproducibility, and also security.

The adoption rates are impressive, reflecting a mature understanding of these benefits:

Over 71% of organizations on AWS use at least one popular IaC technology.
55% of organizations on Google Cloud Platform have adopted IaC.
Terraform stands out as the most popular cross-cloud choice.

IaC is a massive win-win strategy for both development and security teams. It enforces peer review for infrastructure changes, enables automated scanning for misconfigurations before deployment, and lays the groundwork for limiting direct human access to production environments.

The “ClickOps” Contradiction

However, the data reveals a stark contradiction. Despite high IaC adoption, the practice of “ClickOps” – making manual changes directly through the cloud provider’s web console – remains rampant. The Datadog’s report found that 38% of organizations on AWS had performed manual ClickOps in all of their AWS accounts, including production, within a recent 14-day window.

This is a significant security gap. Manual changes are untracked, bypass peer review, and are prone to human error. They lead to configuration drift, where the reality of your production environment no longer matches the state defined in your codebase. This not only creates security vulnerabilities but also makes it nearly impossible to quickly recover from an incident. Teams looking to mature in DevSecOps practices need to close this automation gap by enforcing IaC-only deployment pipelines.

Less Is More: The Undeniable Security Benefit of Lightweight Containers

In software world, a smaller attack surface is always better. The report provides compelling evidence that this principle holds true for container images as well. The choice of a base image for your containers has a direct and significant impact on your application’s vulnerability profile; for example:

Container images smaller than 100 MB have an average of just 4.4 high or critical vulnerabilities.
Images between 250 MB and 500 MB have 42.2 vulnerabilities on average.
Images larger than 500 MB have nearly 80 vulnerabilities.

The reason is simple: larger images, often based on classic Linux distributions like Ubuntu, are packed with system libraries, package managers, and shell utilities that are rarely needed by the application itself. Each of these components is a potential source of vulnerabilities.

Solution: Go Lightweight or Distroless

The State of DevSecOps report lists a simple solution for engineering teams: invest the time to slim down your container images. The security and operational returns are well worth the effort.

Reduced Attack Surface: Lightweight images (like those based on Alpine Linux) or “distroless” images (which contain only the application and its minimal runtime dependencies) drastically reduce the number of third-party packages an attacker can exploit.
Hardened Environments: As Jay Beale, CEO of InGuardians, notes in the report, distroless images make an attacker’s job much harder. Without tools like curl, wget, or even a basic shell, many common post-exploitation techniques become almost impossible.
Operational Benefits: Smaller images lead offer benefits beyond security such as reduced storage costs, faster network transfer times, and quicker deployment cycles.

The State of DevSecOps report by Datadog paints a clear picture of engineering community making progress but still facing significant challenges. The path to a mature security posture is not about buying more tools or hiring more people; it’s about adopting smarter, more focused practices.

Organizations must up their security posture game in this digital era. By embracing these data-driven insights, you and your teams can move beyond a reactive security stance and build a proactive culture where security is a shared responsibility between development and security teams.

References

State of DevSecOps [ Datadog (original) (archived) ]
Bash Uploader Security Update [ Codecov by Sentry (original) (archived) ]

Table of Contents