Blog
DevOps Security

DevSecOps best practices: how to secure your pipeline

Christian Gonzalez
Author
Christian Gonzalez
DevOps Engineer

Key Points

DevOps has supercharged software development by meshing the gears of coding and operations to crank up efficiency and speed. But this has often been at the expense of security because of the cost, impact on deployment, or simply neglect - developers aren’t security experts and often kick it down the road until deployment.  

But integrating security right from the start in an agile DevOps approach can actually speed up development so you're not just making things faster; you're building a secure app as you go.

Welcome to DevOps security - or DevSecOps - the art and science of weaving security into the development fabric at every stage, ensuring your code is as tough as it is brilliant.

Let's dive into the nitty-gritty of folding security into the DevOps process without missing a beat - we're talking DevSecOps best practices and top tools to help fortify your DevOps from start to finish.

What is DevOps security?

DevOps security, or DevSecOps, focuses on integrating security practices into the DevOps process and throughout the software development lifecycle (SDLC). This promotes cross-team collaboration among development, operations, and security teams.

The role of DevOps security in modern software development is multifaceted. It aims to address security proactively, promote a culture of shared responsibility, and facilitate continuous monitoring and automation to identify and rectify vulnerabilities as you go, rather than during the final testing and deployment phase. In short, DevOps security is the secret sauce that makes software development rock solid, ensuring secure, up-to-code software apps.

Top 10 DevOps security challenges

Sounds simple, but integrating security into your DevOps doesn’t come without its challenges. Here are some of the most common issues you can face:

1. Rapid deployment and speed

In the DevOps world, continuous integration and continuous deployment (CI/CD) pipelines are designed for speed. But this can sometimes lead to deployments that bypass critical security checks. You need to integrate automated security testing tools into these pipelines to catch vulnerabilities like code injections or buffer overflows before they reach production.

2. Collaboration barriers

While tools like Jenkins, Git, and Docker have facilitated collaboration, there's still a cultural gap between devs, security experts, and operations. Misunderstandings or misconfigurations, especially in infrastructure-as-code (IaC) setups, can lead to security loopholes.

3. Containerization concerns

Containers, often orchestrated by systems like Kubernetes, can have vulnerabilities in their base images or configurations. Ensuring your containers are isolated, using trusted base images, and regularly scanning for vulnerabilities is essential.

4. Process integration

Introducing tools like SonarQube or OWASP Dependency-Check can help to identify vulnerabilities in your codebase and third-party libraries. But integrating these tools into an established DevOps process requires a shift in culture and mindset.

5. Weak access controls

Role-based access control (RBAC) in tools like Kubernetes or cloud platforms requires meticulous configuration. Misconfigured access controls can expose your sensitive operations or data to unauthorized users, leading to potential breaches.

6. Risks with DevOps tools

Open-source tools, while powerful, can be targets for supply-chain attacks. Ensuring your tools are sourced from trusted repositories, verifying checksums, and keeping them updated is crucial.

7. API and web application security

Beyond standard vulnerabilities, APIs can be exposed to issues like insecure direct object references (IDOR) or insecure endpoint configurations. Tools like OWASP ZAP can help you identify and mitigate such risks.

8. Cloud security in DevOps

Multi-cloud strategies can introduce complexity in managing security postures. Using infrastructure-as-code tools like Terraform or Pulimi require careful handling to avoid exposing secrets or misconfiguring resources.

9. Configuration drift

Over time, as systems evolve and configurations are modified, there's a risk of configuration drift where production environments differ from their intended state. This can introduce unintended vulnerabilities. Tools like Terraform can help you maintain and audit desired configurations.

10. Secrets management

Hardcoding secrets like API keys or database credentials in code or configuration files is a significant risk. By employing solutions like HashiCorp Vault or AWS Secrets Manager you can manage and inject secrets into applications and infrastructure securely.

Addressing these challenges requires a multifaceted strategy encompassing robust security practices, continuous monitoring, and the adoption of advanced security tools to enhance the resilience of your DevOps pipeline.

If you're looking for more tools recommendations, we have an article dedicated to our top DevSecOps tools.

Find your weaknesses,
before the hackers do

Try Intruder for free

DevOps security best practices

Step it up with DevSecOps

Think of DevSecOps as your development process getting a security upgrade. It's not just about switching up how you do things; it's about stitching security into your project instead of addressing issues at the end. No more waiting for the “big reveal” of security tests when you're about to roll out. Instead, you're keeping an eye on the security of your code every step of the way.

With this always-on security, the chances of bugs slipping through are reduced; and if they do find their way in, you can fix them on the spot. DevSecOps is a safety net - it catches any falls so security hiccups don't turn into full-blown headaches after you've gone live. It's about making security part of your team's DNA - so it's there, doing its thing, without slowing anyone down.

Shift left, secure early

Picture your software development lifecycle as a big timeline, with the start of the project on the left and the finish line on the right. "Shifting left" brings security into the picture way earlier in the game. It's like deciding to check the sturdiness of a boat when you're still sketching the blueprints, not when you're about to sail.

Adopting a shift-left strategy means "test early, test often". Instead of pushing off security checks until you deploy, you're embedding them into every coding session from day one. This way, potential security slip-ups get caught early. This is the bread and butter of DevSecOps best practices: making sure security is baked in from the start. When you adopt a shift-left mindset, you're basically putting on your security goggles from the get-go, paving the way for software that's awesome but rock solid.

Scanning and penetration testing

Let's talk now about how to keep your software safe. Imagine your app as a castle. As you build and expand, you've got to look for weak spots where attackers could break in. That's where scanning and penetration testing comes into play, and in the world of DevSecOps, they're non-negotiable.

Think of vulnerability scanning as your sentries that patrol the walls and halls, automatically checking for open windows and unlocked doors - your first guard against possible entry. Penetration testing, though, is like staging a simulated assault on your castle to uncover any hidden cracks in your defenses, showing you what it takes to keep the attackers at bay. Together, they're the dynamic duo that helps you stay a step ahead of attackers, making sure your app is safe and sound.

Automation is the answer

When it comes to DevSecOps best practices, automation is key. It's all about ensuring security works like clockwork, ticking along so your team can focus on building software. Imagine having a robot buddy that not only looks for bugs in your code but also keeps things moving smoothly through the assembly line, from coding to launch and beyond.

This is what happens when you introduce automation to your CI/CD pipeline – a conveyor belt that carries your code from "just an idea" to "it's live". Like setting up dominoes in a line; once you hit the first (your code being ready), the rest fall into place without you having to lift a finger until your code is out there. This way, your code isn't just built well; it's built securely and fast.

Start with vulnerability management

Getting started with DevSecOps is about folding in a smart vulnerability management program. Think of it as a software's health check: finding vulnerabilities, figuring out which ones are the real troublemakers, and then removing, patching or mitigating them before they can cause any problems. It's not just about fixing but keeping tabs on them - through continuous monitoring and regular penetration testing.

But how often should you be scanning? For programmers, keeping scans going non-stop during the whole coding party - especially before you send out any shiny new updates - is critical. Regular check-ins with your code paired with an eagle-eyed watch for new threats, mean you're always a step ahead. And the best part? Automation can make this whole routine a breeze.  

Level up your DevOps security with Intruder

Intruder makes it easy to develop and deploy secure applications by integrating with your CI/CD pipeline, automating the discovery of any weaknesses or misconfigurations. You can automatically synchronize your cloud IPs and hostnames to stay on top of your infrastructure and make cloud security a breeze, streamline and optimize your workflow with our API by adding Intruder to your CI/CD pipeline. Plus get notifications and push vulnerabilities to Slack or Microsoft Teams, send issue information to JIRA for remediation, or extend to 2,000+ other apps with Zapier. Ready to get started with your 14-day trial? Try it for free.

Get our free

Ultimate Guide to Vulnerability Scanning

Learn everything you need to get started with vulnerability scanning and how to get the most out of your chosen product with our free PDF guide.

Sign up for your free 14-day trial

7 days free trial
Many DevOps advantages actually create security challenges. Find out how to help secure your dev pipeline with Intruder.
back to BLOG

DevSecOps best practices: how to secure your pipeline

Christian Gonzalez

DevOps has supercharged software development by meshing the gears of coding and operations to crank up efficiency and speed. But this has often been at the expense of security because of the cost, impact on deployment, or simply neglect - developers aren’t security experts and often kick it down the road until deployment.  

But integrating security right from the start in an agile DevOps approach can actually speed up development so you're not just making things faster; you're building a secure app as you go.

Welcome to DevOps security - or DevSecOps - the art and science of weaving security into the development fabric at every stage, ensuring your code is as tough as it is brilliant.

Let's dive into the nitty-gritty of folding security into the DevOps process without missing a beat - we're talking DevSecOps best practices and top tools to help fortify your DevOps from start to finish.

What is DevOps security?

DevOps security, or DevSecOps, focuses on integrating security practices into the DevOps process and throughout the software development lifecycle (SDLC). This promotes cross-team collaboration among development, operations, and security teams.

The role of DevOps security in modern software development is multifaceted. It aims to address security proactively, promote a culture of shared responsibility, and facilitate continuous monitoring and automation to identify and rectify vulnerabilities as you go, rather than during the final testing and deployment phase. In short, DevOps security is the secret sauce that makes software development rock solid, ensuring secure, up-to-code software apps.

Top 10 DevOps security challenges

Sounds simple, but integrating security into your DevOps doesn’t come without its challenges. Here are some of the most common issues you can face:

1. Rapid deployment and speed

In the DevOps world, continuous integration and continuous deployment (CI/CD) pipelines are designed for speed. But this can sometimes lead to deployments that bypass critical security checks. You need to integrate automated security testing tools into these pipelines to catch vulnerabilities like code injections or buffer overflows before they reach production.

2. Collaboration barriers

While tools like Jenkins, Git, and Docker have facilitated collaboration, there's still a cultural gap between devs, security experts, and operations. Misunderstandings or misconfigurations, especially in infrastructure-as-code (IaC) setups, can lead to security loopholes.

3. Containerization concerns

Containers, often orchestrated by systems like Kubernetes, can have vulnerabilities in their base images or configurations. Ensuring your containers are isolated, using trusted base images, and regularly scanning for vulnerabilities is essential.

4. Process integration

Introducing tools like SonarQube or OWASP Dependency-Check can help to identify vulnerabilities in your codebase and third-party libraries. But integrating these tools into an established DevOps process requires a shift in culture and mindset.

5. Weak access controls

Role-based access control (RBAC) in tools like Kubernetes or cloud platforms requires meticulous configuration. Misconfigured access controls can expose your sensitive operations or data to unauthorized users, leading to potential breaches.

6. Risks with DevOps tools

Open-source tools, while powerful, can be targets for supply-chain attacks. Ensuring your tools are sourced from trusted repositories, verifying checksums, and keeping them updated is crucial.

7. API and web application security

Beyond standard vulnerabilities, APIs can be exposed to issues like insecure direct object references (IDOR) or insecure endpoint configurations. Tools like OWASP ZAP can help you identify and mitigate such risks.

8. Cloud security in DevOps

Multi-cloud strategies can introduce complexity in managing security postures. Using infrastructure-as-code tools like Terraform or Pulimi require careful handling to avoid exposing secrets or misconfiguring resources.

9. Configuration drift

Over time, as systems evolve and configurations are modified, there's a risk of configuration drift where production environments differ from their intended state. This can introduce unintended vulnerabilities. Tools like Terraform can help you maintain and audit desired configurations.

10. Secrets management

Hardcoding secrets like API keys or database credentials in code or configuration files is a significant risk. By employing solutions like HashiCorp Vault or AWS Secrets Manager you can manage and inject secrets into applications and infrastructure securely.

Addressing these challenges requires a multifaceted strategy encompassing robust security practices, continuous monitoring, and the adoption of advanced security tools to enhance the resilience of your DevOps pipeline.

If you're looking for more tools recommendations, we have an article dedicated to our top DevSecOps tools.

Find your weaknesses,
before the hackers do

Try Intruder for free

DevOps security best practices

Step it up with DevSecOps

Think of DevSecOps as your development process getting a security upgrade. It's not just about switching up how you do things; it's about stitching security into your project instead of addressing issues at the end. No more waiting for the “big reveal” of security tests when you're about to roll out. Instead, you're keeping an eye on the security of your code every step of the way.

With this always-on security, the chances of bugs slipping through are reduced; and if they do find their way in, you can fix them on the spot. DevSecOps is a safety net - it catches any falls so security hiccups don't turn into full-blown headaches after you've gone live. It's about making security part of your team's DNA - so it's there, doing its thing, without slowing anyone down.

Shift left, secure early

Picture your software development lifecycle as a big timeline, with the start of the project on the left and the finish line on the right. "Shifting left" brings security into the picture way earlier in the game. It's like deciding to check the sturdiness of a boat when you're still sketching the blueprints, not when you're about to sail.

Adopting a shift-left strategy means "test early, test often". Instead of pushing off security checks until you deploy, you're embedding them into every coding session from day one. This way, potential security slip-ups get caught early. This is the bread and butter of DevSecOps best practices: making sure security is baked in from the start. When you adopt a shift-left mindset, you're basically putting on your security goggles from the get-go, paving the way for software that's awesome but rock solid.

Scanning and penetration testing

Let's talk now about how to keep your software safe. Imagine your app as a castle. As you build and expand, you've got to look for weak spots where attackers could break in. That's where scanning and penetration testing comes into play, and in the world of DevSecOps, they're non-negotiable.

Think of vulnerability scanning as your sentries that patrol the walls and halls, automatically checking for open windows and unlocked doors - your first guard against possible entry. Penetration testing, though, is like staging a simulated assault on your castle to uncover any hidden cracks in your defenses, showing you what it takes to keep the attackers at bay. Together, they're the dynamic duo that helps you stay a step ahead of attackers, making sure your app is safe and sound.

Automation is the answer

When it comes to DevSecOps best practices, automation is key. It's all about ensuring security works like clockwork, ticking along so your team can focus on building software. Imagine having a robot buddy that not only looks for bugs in your code but also keeps things moving smoothly through the assembly line, from coding to launch and beyond.

This is what happens when you introduce automation to your CI/CD pipeline – a conveyor belt that carries your code from "just an idea" to "it's live". Like setting up dominoes in a line; once you hit the first (your code being ready), the rest fall into place without you having to lift a finger until your code is out there. This way, your code isn't just built well; it's built securely and fast.

Start with vulnerability management

Getting started with DevSecOps is about folding in a smart vulnerability management program. Think of it as a software's health check: finding vulnerabilities, figuring out which ones are the real troublemakers, and then removing, patching or mitigating them before they can cause any problems. It's not just about fixing but keeping tabs on them - through continuous monitoring and regular penetration testing.

But how often should you be scanning? For programmers, keeping scans going non-stop during the whole coding party - especially before you send out any shiny new updates - is critical. Regular check-ins with your code paired with an eagle-eyed watch for new threats, mean you're always a step ahead. And the best part? Automation can make this whole routine a breeze.  

Level up your DevOps security with Intruder

Intruder makes it easy to develop and deploy secure applications by integrating with your CI/CD pipeline, automating the discovery of any weaknesses or misconfigurations. You can automatically synchronize your cloud IPs and hostnames to stay on top of your infrastructure and make cloud security a breeze, streamline and optimize your workflow with our API by adding Intruder to your CI/CD pipeline. Plus get notifications and push vulnerabilities to Slack or Microsoft Teams, send issue information to JIRA for remediation, or extend to 2,000+ other apps with Zapier. Ready to get started with your 14-day trial? Try it for free.

Release Date
Level of Ideal
Comments
Before CVE details are published
🥳
Limited public information is available about the vulnerability.

Red teamers, security researchers, detection engineers, threat actors have to actively research type of vulnerability, location in vulnerable software and build an associated exploit.

Tenable release checks for 47.43% of the CVEs they cover in this window, and Greenbone release 32.96%.
Day of CVE publish
😊
Vulnerability information is publicly accessible.

Red teamers, security researchers, detection engineers and threat actors now have access to some of the information they were previously having to hunt themselves, speeding up potential exploit creation.

Tenable release checks for 17.12% of the CVEs they cover in this window, and Greenbone release 17.69%.
First week since CVE publish
😐
Vulnerability information has been publicly available for up to 1 week.

The likelihood that exploitation in the wild is going to be happening is steadily increasing.

Tenable release checks for 10.9% of the CVEs they cover in this window, and Greenbone release 20.69%.
Between 1 week and 1 month since CVE publish
🥺
Vulnerability information has been publicly available for up to 1 month, and some very clever people have had time to craft an exploit.

We’re starting to lose some of the benefit of rapid, automated vulnerability detection.

Tenable release checks for 9.58% of the CVEs they cover in this window, and Greenbone release 12.43%.
After 1 month since CVE publish
😨
Information has been publicly available for more than 31 days.

Any detection released a month after the details are publicly available is decreasing in value for me.

Tenable release checks for 14.97% of the CVEs they cover over a month after the CVE details have been published, and Greenbone release 16.23%.

With this information in mind, I wanted to check what is the delay for both Tenable and Greenbone to release a detection for their scanners. The following section will focus on vulnerabilities which:

  • Have CVSSv2 rating of 10
  • Are exploitable over the network
  • Require no user interaction

These are the ones where an attacker can point their exploit code at your vulnerable system and gain unauthorised access.

We’ve seen previously that Tenable have remote checks for 643 critical vulnerabilities, and OpenVAS have remote checks for 450 critical vulnerabilities. Tenable release remote checks for critical vulnerabilities within 1 month of the details being made public 58.4% of the time, but Greenbone release their checks within 1 month 76.8% of the time. So, even though OpenVAS has fewer checks for those critical vulnerabilities, you are more likely to get them within 1 month of the details being made public. Let’s break that down further.

In Figure 10 we can see the absolute number of remote checks released on a given day after a CVE for a critical vulnerability has been published. What you can immediately see is that both Tenable and OpenVAS release the majority of their checks on or before the CVE details are made public; Tenable have released checks for 247 CVEs, and OpenVAS have released checks for 144 CVEs. Then since 2010 Tenable have remote released checks for 147 critical CVEs and OpenVAS 79 critical CVEs on the same day as the vulnerability details were published. The number of vulnerabilities then drops off across the first week and drops further after 1 week, as we would hope for in an efficient time-to-release scenario.

Figure 10: Absolute numbers of critical CVEs with a remote check release date from the date a CVE is published

While raw numbers are good, Tenable have a larger number of checks available so it could be unfair to go on raw numbers alone. It’s potentially more important to understand the likelihood that OpenVAS or Tenable will release a check of a vulnerability on any given day after a CVE for a critical vulnerability is released. In Figure 11 we can see that Tenable release 61% their checks on or before the date that a CVE is published, and OpenVAS release a shade under 50% of their checks on or before the day that a CVE is published.

Figure 11: Percentage chance of delay for critical vulnerabilities

So, since 2010 Tenable has more frequently released their checks before or on the same day as the CVE details have been published for critical vulnerabilities. While Tenable is leading at this point, Greenbone’s community feed still gets a considerable percentage of their checks out on or before day 0.

I thought I’d go another step further and try and see if I could identify any trend in each organisations release delay, are they getting better year-on-year or are their releases getting later? In Figure 12 I’ve taken the mean delay for critical vulnerabilities per year and plotted them. The mean as a metric is particularly influenced by outliers in a data set, so I expected some wackiness and limited the mean to only checks released 180 days prior to a CVE being published and 31 days after a CVE being published. These seem to me like reasonable limits, as anything greater than 6 months prior to CVE details being released is potentially a quirk of the check details and anything after a 1-month delay is less important for us.

What can we take away from Figure 12?

  • We can see that between 2011 and 2014 Greenbone’s release delay was better than that of Tenable, by between 5 and 10 days.
  • In 2015 things reverse and for 3 years Tenable is considerably ahead of Greenbone by a matter of weeks.
  • But, then in 2019 things get much closer and Greenbone seem to be releasing on average about a day earlier than Tenable.
  • For both the trendline over an 11-year period is very close, with Tenable marginally beating Greenbone.
  • We have yet to have any data for 2021 for OpenVAS checks for critical show-stopper CVEs.
Figure 12: Release delay year-on-year (lower is better)

With the larger number of checks, and still being able to release a greater percentage of their remote checks for critical vulnerabilities Tenable could win this category. However, the delay time from 2019 and 2020 going to OpenVAS, and the trend lines being so close, I am going to declare this one a tie. It’s a tie.

The takeaway from this is that both vendors are getting their checks out the majority of the time either before the CVE details are published or on the day the details are published. This is overwhelmingly positive for both scanning solutions. Over time both also appear to be releasing remote checks for critical vulnerabilities more quickly.

Written by

Christian Gonzalez

Recommended articles

Ready to get started with your 14-day trial?
try for free