Credit: Dreamstime
The emergence of the DevOps culture over the past several years has fundamentally changed software development, allowing companies to push code faster and to automatically scale the infrastructure needed to support new features and innovations.
The increased push toward DevSecOps, which bakes security into the development and operations pipelines, is now changing the state of application security, but gaps still remain according to data from new industry reports.
Security testing ownership
A new report by the Enterprise Strategy Group (ESG), which surveyed 378 application developers and application security professionals in North America, found that many organisations continue to push code with known vulnerabilities into production despite viewing their own application security programs as solid.
Releasing vulnerable code is never good but doing so knowingly is better than doing it without knowing, since the decision usually involves some risk assessment, a plan to fix, and maybe temporary mitigations. Half of respondents said their organisations do this regularly and a third said they do it occasionally.
The most often cited reasons were meeting a critical deadline, the vulnerabilities being low risk or the issues being discovered too late in the release cycle (45 per cent).
The findings highlight why integrating security testing as early in the development process as possible is important, but also that releasing vulnerable code is not necessarily a sign of not having a good security program because this can happen for different reasons and no single type of security testing will catch all bugs.
However, the report also found that many organisations are still in the process of expanding their application security programs, with only a third saying their programs cover more than three quarters of their codebase and a third saying their programs cover less than half.
Who takes responsibility for the decision of pushing vulnerable code into production can vary from organisation to organisation, the survey found. In 28 per cent of organisations the decision is taken by the development manager together with a security analyst, in 24 per cent by the development manager alone and in 21 per cent by a security analyst.
This could actually be a sign of application security programs maturing, because DevSecOps is about moving security testing as early as possible in the development pipeline, whereas in the past security testing fell solely in the sphere of security teams who used to perform it after the product was complete.
In organisations where the development team does the security testing as a result of integrations into their processes and also consumes the results, it's normal for the development manager to make decisions regarding which vulnerabilities are acceptable, either in collaboration with the security team or even inside their own organisation if they have a security champion -- a developer with application security knowledge and training -- on their team.
Such decisions, however, should still be taken based on policies put in place by the CISO organisation, which is ultimately responsible for managing the entire company's information security risk and can, for example, decide which applications are more exposed to attacks or contain more sensitive information that hackers could target. Those applications might have stricter rules in place when it comes to patching.
If the risk is not evaluated correctly, shipping code with known vulnerabilities can have serious consequences. Sixty percent of respondents admitted that their production applications were exploited through vulnerabilities listed in the OWASP top 10 over the past 12 months.
The OWASP Top-10 contains the most critical security risks to web applications and include problems like SQL injection, broken authentication, sensitive data exposure, broken access controls, security misconfigurations, the use of third-party components with known vulnerabilities and more. These are issues that should not generally be allowed to exist in production code.
According to ESG's report, companies use a variety of application security testing tools: API security vulnerability (ASV) scanning (56 per cent), infrastructure-as-code security tools to protect against misconfigurations (40 per cent), static application security testing (SAST) tools (40 per cent) and software composition analysis (SCA) testing tools (38 per cent).
This is in addition to interactive application security testing (IAST) tools (38 per cent), dynamic application security testing (DAST) tools (36 per cent) and plugins for integrated development environments (IDEs) that assist with security issue identification and resolution (29 per cent), alongside scanning tools for images used in containers, repositories and microservices (29 per cent), fuzzing tools (16 per cent) and container runtime configuration security tools (15 per cent).
However, among the top challenges in using these tools, respondents listed developers lacking the knowledge to mitigate the identified issues (29 per cent), developers not using tools the company invested in effectively (24 per cent), security testing tools adding friction and slowing down development cycles (26 per cent) and lack of integration between application security tools from different vendors (26 per cent).
While almost 80 per cent of organisations report that their security analysts are directly engaged with their developers by working directly to review features and code, by working with developers to do threat modelling or by participating in daily development scrum meetings, developers themselves don't seem to get a lot of security training.
This is why in only 19 per cent of organisations the application security testing task is formally owned by individual developers and in 26 per cent by development managers. A third of organisations still have this task assigned to dedicated security analysts and in another 29 per cent it's jointly owned by the development and security teams.
In a third of organisations less than half of developers are required to take formal security training and only 15 per cent such training is required for all developers. Less than half of organisations require developers to engage in formal security training more than once a year, 16 per cent expecting developers to self-educate and 20 per cent only offering training when a developer joins the team.
Furthermore, even when training is provided or required, the effectiveness of such training is not properly tracked in most organisations. Only 40 per cent of organisations track security issue introduction and continuous improvement metrics for development teams or individual developers.
Veracode, one of the application security vendors who sponsored the ESG research, recently launched the Veracode Security Labs Community Edition, an in-browser platform where developers can get free access to dozens of application security courses and containerised apps that they can exploit and patch for practice.
Open source components targeted in supply chain attacks
Any mature application security program should also cover any open source components and frameworks because these make up a large percentage of modern application code bases and carry risks of inherited vulnerabilities and supply chain attacks.
Almost half of respondents in ESG's survey said that open source components make up over 50 per cent of their code base and eight per cent said they account for two thirds of their code. Despite that, only 48 per cent of organisations have invested in controls to deal with open source vulnerabilities.
In its 2020 State of the Software Supply Chain report, open source governance company Sonatype noted a 430 per cent year-over-year growth in attacks targeting open source software projects.
These attacks are no longer passive where attackers exploit vulnerabilities after they've been publicly disclosed, but ones where attackers try to compromise and inject malware into upstream open source projects whose code is then pulled by developers into their own applications.
In May, the GitHub security team issued a warning about a malware campaign dubbed Octopus Scanner that was backdooring NetBeans IDE projects. Malicious or compromised components have also been regularly distributed on package repositories like npm or PyPi.
The complex web of dependencies makes dealing with this issue difficult. In 2019, researchers from Darmstadt University analysed the npm ecosystem, which is the primary source for JavaScript components. They found that any typical package loaded an average of 79 other third-party packages from 39 different maintainers. The top five packages on npm had a reach of between 134,774 and 166,086 other packages.
"When malicious code is deliberately and secretly injected upstream into open source projects, it is highly likely that no one knows the malware is there, except for the person that planted it," Sonatype said in its report. "This approach allows adversaries to surreptitiously set traps upstream, and then carry out attacks downstream once the vulnerability has moved through the supply chain and into the wild."
According to the company, between February 2015 and June 2019, 216 such "next-generation" supply chain attacks were reported, but from July 2019 to May 2020 an additional 929 attacks were documented, so this has become a very popular attack vector.
In terms of traditional attacks where hackers exploit known vulnerabilities in components, companies seem unprepared to respond quickly enough. In the case of the Apache Struts2 vulnerability that ultimately led to the Equifax breach in 2017, attackers started exploiting the vulnerability within 72 hours after it became known.
More recently, a vulnerability reported in SaltStack was also exploited within three days after being announced, catching many companies unprepared.
A Sonatype survey of 679 software development professionals revealed that only 17 per cent of organisations learn about open source vulnerabilities within a day of public disclosure. A third learn within the first week and almost half after a week's time. Furthermore, around half of organisations required more than a week to respond to a vulnerability after learning about it and half of those took more than a month.
Both the availability and consumption of open source components is increasing with every passing year. The JavaScript community introduced over 500,000 new component releases over the past year pushing the npm directory to 1.3 million packages.
Until May developers downloaded packages 86 billion times from npm, Sonatype projecting that by the end of the year the figure will reach one trillion downloads. It's concerning that the University of Darmstadt research published last year revealed that nearly 40 per cent of all npm packages contain or depend code with known vulnerabilities and that 66 per cent vulnerabilities in npm packages remain unpatched.
In the Java ecosystem, developers downloaded 226 billion open source software components from the Maven Central Repository in 2019, which was a 55 per cent increase compared to 2018. Given the statistics seen in 2020, Sonatype estimates that Java components downloads will reach 376 billion this year.
The company, which maintains the Central Repository and has deep insights into the data, reports that one in ten downloads was for a component with a known vulnerability.
A further analysis of 1,700 enterprise applications revealed that on average they contained 135 third-party software components, of which 90 per cent were open source. Eleven per cent of those open source components had at least one vulnerability, but applications had on average 38 known vulnerabilities inherited from such components.
It was also not uncommon to see applications assembled from 2,000 to 4,000 open source components, highlighting the major role the open source ecosystem plays in modern software development.
Similar component consumption trends were observed in the .NET ecosystem and the microservice ecosystem, with DockerHub receiving 2.2 container images over the past year and being on track to seeing 96 billion image pull requests by developers this year. Publicly reported supply chain attacks have involved malicious container images hosted on DockerHub and the possibility of having images with misconfigurations or vulnerabilities is also high.
Infrastructure-as-code needs remediation-as-code
The DevOps movement has fundamentally changed software development and made possible the new microservice architecture where traditional monolith applications are broken down into individually maintained services that run in their own containers.
Applications no longer contain just the code necessary for their features, but also the configuration files that dictate and automate their deployment on cloud platforms, along with the resources they need. Under DevSecOps, development teams are not only responsible for writing secure code, but also deploying secure infrastructure.
In a new report, cloud security firm Accurics, which operates a platform that can detect vulnerable configurations in infrastructure-as-code templates and cloud deployments, 41 per cent of organisations had hardcoded keys with privileges in their configurations that were used to provision computing resources, 89 per cent deployments had resources provisioned and running with overly permissive identity and access management (IAM) policies and nearly all of them had misconfigured routing rules.
According to the company, the September 2019 breach at CenturyLink that exposed 2.8 million customer records; the August 2019 breach at Imperva that exposed a database snapshot containing emails, hashed, and salted passwords; and the July 2019 breach at Capital One that affected 100 million US residents were the result of attacks that exploited at least one of those issues.
"Policy as code should be implemented to ensure that obvious best practices are employed such as encrypting databases, rotating access keys, and implementing multi-factor authentication," Accurics said.
"However, automated threat modelling is also necessary to determine if changes such as privilege increase and route changes create breach paths in a cloud deployment. As a result, organisations must augment policy as code with security as code when infrastructure is defined during development (infrastructure as code)."
According to Om Moolchandani, CTO of Accurics, a new practice called “remediation as code” is emerging where security tools don't just check for vulnerabilities in cloud configuration templates or in the running deployments themselves, but also generate the code necessary to automatically fix the issue and propose it to the developers.
This could improve organisations' time to remediate, which is a major issue, because until now this process has largely been manual, leading to many issues being ignored.
Over the past several years, many application and infrastructure security vendors have been working on reengineering their products so they integrate well with development tools, since developers have different expectations and work differently than security teams. Many open source tools are also available to test cloud deployments.
Penetration testing firm Bishop Fox released a tool called Smogcloud during the Black Hat USA conference that can help security engineers and cloud administrators find their exposed AWS cloud assets, including issues like internet-facing FQDNs and IPs, misconfigurations or vulnerabilities, assets that are no longer in use, services not currently monitored and other shadow IT problems. Accurics also has a collection of open source tools and a free version of its cloud deployment testing platform.
