The Forrester Wave: Static Application Security Testing 2021

Why Read This Report

In our 28-criterion evaluation of static application security testing (SAST) providers, we identified the 12 most significant ones — CAST, Checkmarx, GitHub, GitLab, HCL Software, Micro Focus, Parasoft, Perforce Software, SonarSource, Synopsys, Veracode, and WhiteHat Security — and researched, analyzed, and scored them. This report shows how each provider measures up and helps security and risk professionals select the right one for their needs.

Key Takeaways

Veracode, Synopsys, Checkmarx, And Micro Focus Lead The Pack

Forrester’s research uncovered a market in which Veracode, Synopsys, Checkmarx, and Micro Focus are Leaders; HCL Software and CAST are Strong Performers; GitHub, Parasoft, GitLab, Perforce Software, and SonarSource are Contenders; and WhiteHat Security is a Challenger.
Developer Enablement, New Architecture Support, And Accuracy Are Key Differentiators

As development speeds continue to increase and teams embrace new development methodologies, SAST solutions that build security into the software development lifecycle (SDLC), regardless of how and where the application is built, will lead the pack. Vendors that offer deep integration with the CI/CD pipeline; quickly expand to protect new architectures like containers, APIs, and infrastructure-as-code (IaC); and continuously improve on performance and accuracy, position themselves to delight both security and developer stakeholders.

Table of Contents

• NEXT GENERATION SAST SOLUTIONS FOCUS ON DEVELOPER ENABLEMENT
• EVALUATION SUMMARY
• VENDOR OFFERINGS
• VENDOR PROFILES
• EVALUATION OVERVIEW

NEXT GENERATION SAST SOLUTIONS FOCUS ON
DEVELOPER ENABLEMENT

Static application security testing (SAST) tools were initially built for security pros and neglected the needs of developers. As a result, developers were frustrated by false positives, lack of application context, and being forced out of their day-to-day workflows — and that frustration and friction hindered adoption. Happily, SAST vendors have shifted their thinking to include the developer as a key stakeholder in the application security process. Web application attacks were the top cause of external breaches in 2020, and SAST remains a critical tool to address vulnerabilities in proprietary code — as long as it continues to align with developer workflows and helps security pros prioritize and address application security weaknesses early in the SDLC.( See note 1)
As a result of these trends, SAST customers should look for providers that:

• Embrace the developer persona. SAST solutions must build into the developer experience, allowing developers to work efficiently in the tools that they already know. Look for SAST solutions that overlay the CI/CD pipeline through out-of-the-box-integrations with popular IDEs, build tools, and code repositories. In addition, seek solutions that provide actionable remediation guidance, with code samples and interactive training reachable through the developer’s toolset.
• Go beyond the traditional definition of code. Firms don’t only build applications with traditional languages like C++ and Java or newer languages like Swift and Kotlin. APIs have become a common application building block, citizen developers have emerged to build apps using low-code platforms, and developers are using infrastructure-as-code (IaC) to define cloud configurations. As your firm’s definition of code expands, and as developers come from outside the traditional development organization, look for SAST tools that will scan for vulnerabilities in these new types of “code.”
• Deliver accurate results quickly. Even as SAST has advanced with new features, the basic requirements of low false positives and short scan times remain. A number of customers still list accuracy and performance as challenges. Go beyond asking about false positive rates and performance metrics — ask what tuning is required out of the box and how to optimize scanning performance given your applications’ structure and architecture.

EVALUATION SUMMARY

The Forrester Wave™ evaluation highlights Leaders, Strong Performers, Contenders, and Challengers. It’s an assessment of the top vendors in the market and does not represent the entire vendor landscape. You’ll find more information about this market in our “Now Tech: Static Application Security Testing, Q3 2020” and “The Forrester Tech Tide™: Application Security, Q4 2020.”

We intend this evaluation to be a starting point only and encourage clients to view product evaluations and adapt criteria weightings using the Excel-based vendor comparison tool (See Figure 1 and See Figure 2). Click the link at the beginning of this report on Forrester.com to download the tool.

Figure 1: Forrester Wave™: Static Application Security Testing, Q1 2021

Figure 2: Forrester Wave™: Static Application Security Testing Scorecard, Q1 2021

VENDOR OFFERINGS

Forrester included 12 vendors in this assessment: CAST, Checkmarx, GitHub, GitLab, HCL Software, Micro Focus, Parasoft, Perforce Software, SonarSource, Synopsys, Veracode, and WhiteHat Security (See Figure 3).

Figure 3: Evaluated Vendors And Product Information

VENDOR PROFILES

Our analysis uncovered the following strengths and weaknesses of individual vendors.

Leaders

• Veracode has invested in the developer experience. Veracode’s offering is fully software as a service (SaaS) and includes three different levels of scans: IDE Scan (for the individual developer), Pipeline Scan (scanning as code is added into the pipeline), and Policy Scan (the predeployment check). Veracode recently launched Security Labs Community Edition, a free set of secure development training modules — a paid enterprise version is available as well, though neither version directly integrates into its SAST tool today. Veracode has recently updated its pricing model to simplify the purchasing experience. Prioritization is a strength, with Veracode’s graphical representation of code flaws according to risk and ease of fix unmatched in the market. Veracode has pushed to become more developer friendly in recent years — an important initiative, considering that a number of customer references for other vendors called out Veracode’s prior deficiencies in developer focus. In considering developer experience, Veracode’s own customer references rated the remediation guidance highly but had some challenges with developer workflow and the API. References complimented Veracode’s premium support — “the relationship really stands out” — and consolidated platform for all business units. For firms looking for an enterprise-grade SAST tool, Veracode remains a top choice.
• Synopsys offers accuracy and deep IDE integration, but pricing raises questions. Synopsys offers a range of prerelease scanning tools, with Coverity as its SAST solution. Coverity is available either on-prem or via SaaS through the Synopsys Polaris platform. The included CodeSight IDE plug-in offers real-time scanning in the developer’s IDE, built-in remediation guidance, and integrated reporting of both SAST and software composition analysis (SCA) findings (for those customers using both solutions from Synopsys). Customers who also purchase Synopsys eLearning — an add-on that is not part of the Coverity package — can get context-appropriate links to training as part of the remediation guidance. Synopsys has also introduced Intelligent Orchestration, a policy-as-code module that leverages the customer-defined risk profile to determine what sort of testing is necessary at what time. References were particularly complimentary of Coverity’s low false positive rate, flexible reporting, and customer support. One reference commented that if a Coverity scan flagged an issue, “the general consensus on the developer team is that it’s accurate and [we] need to look at it.” A big downside was cost — even as they were happy with the product, references questioned whether the pricing was in line with the value they received. Also, while happy with performance for smaller applications and microservices, references were less satisfied with scanning times for very large, monolithic applications. Synopsys is a good fit for firms looking for a strong SAST solution that is also part of an overall AST platform.
• Checkmarx’s CxSAST extends into new dev approaches, but reporting has gaps. CxSAST is available either on-prem or as a managed service through Checkmarx’s AppSec Accelerator offering. Checkmarx integrates its training offering, Codebashing, into CxSAST’s remediation guidance — this is complimentary for all CxSAST users. CxSAST has invested in support for new development approaches, including scans for OWASP Top 10 API issues and support for Salesforce’s VisualForce and Lightning frameworks. Checkmarx’s out-of-the-box rules are a strength, and users can augment them with custom rules with the CxQL language. Checkmarx has numerous integrations with source control and build tools, and customers can enable webhooks to enforce code scanning at key stages of the development process. Risk scoring and workflow modification were weak points, with limited customizability. Customer references were eager to provide feedback — they rated CxSAST particularly well in scanning performance and language coverage and highlighted the strength of the vendor relationship. References did complain about reporting limitations — particularly key metrics missing from reports — and one commented, “The user interface leaves something to be desired.” Customers embracing modern development methodologies will benefit from Checkmarx’s API support and deep integrations with CI/CD tooling.
• Micro Focus leverages third-party integrations to enrich user experience. Micro Focus Fortify is available as an on-prem solution or through a SaaS model with Fortify on Demand (FoD). The FoD solution can be fully automated to produce scan results or augmented with a manual Micro Focus security expert review. Micro Focus offers a suite of application security tools, including dynamic application security testing (DAST) — Micro Focus WebInspect — that can be combined with Fortify for unified reporting and analytics. In addition, Micro Focus supports parser plug-ins for several third-party security tools, allowing those findings to appear in the dashboard alongside Fortify results. Micro Focus also builds in training platform SecureCodeWarrior for all Fortify customers, providing a contextual and gamified learning experience. Fortify offers strong language support and IDE integrations that identify flaws in real time. Some key features come from third-party integrations, such as the integration with SecureCodeWarrior to support developer remediation guidance, and the parser plug-ins that feed third-party tools’ scanning data into Fortify. Areas for improvement include the risk score, which is limited to a non-customizable five-star rating system, and workflow modification. Micro Focus reference customers complimented Fortify’s strong APIs and ease of use, though there were requests for better documentation and native integration with ServiceNow. Customers looking to provide additional context to their SAST results through first- and third-party integrations should consider Fortify.

Strong Performers

• HCL Software has broad language support but weak remediation guidance. HCL purchased the AppScan portfolio from IBM in 2019 and has worked to merge the AppScan Source and AppScan on Cloud (ASoC) solutions into a more unified offering with a single engine. HCL customers have the option of on-prem, cloud, or hybrid AppScan deployments. HCL also released CodeSweep, a community edition of AppScan, in 2020. HCL offers a wide range of language support, including modern languages like Groovy, Go, Kotlin, and Swift. Under its “Bring Your Own Language” framework, customers can write rules and remediation advice for niche languages not already supported. While HCL provides language and API specific articles to assist with remediation, references rated HCL’s remediation guidance poorly, describing it as “insufficient” and “not very helpful to an inexperienced developer.” Across the board, HCL’s references were among the least satisfied with key features, including performance, false negatives, and reporting. References did praise the business relationship and felt that HCL would work with them to resolve feature issues. Customers seeking a flexible deployment model and broad language support should consider HCL AppScan.
• CAST offers architecture level analysis but lacks SDLC integrations. CAST Application Intelligence Platform (AIP) and CAST Imaging go beyond analyzing the source code and delve into the data structures, code components, and interdependences. A unique feature of CAST is the architecture blueprint, an interactive view into the application details based on a CAST Imaging scan. CAST customers can deploy CAST AIP and CAST Imaging on-prem or as a managed service through CAST or a partner. Among the vendors we evaluated, CAST AIP has one of the broadest out-of-the-box rulesets available, covering industry standards, coding standards, and mobile applications. A major downside is the dearth of out-of-the-box integrations with development and CI/CD tools — AIP has no integrations with IDEs and only integrates with Jira out-of-the-box for ticketing. Customer references were particularly effusive about CAST’s service and support teams, noting, “The CAST engineering team is really helpful in coming up with all sorts of support.” They also noted that the blueprint helped preserve institutional knowledge, even as developers and architects turned over on the team. References would like to see more innovation from the R&D team going forward. Organizations looking for a scanning solution that analyzes both code and architecture at a system level should consider CAST AIP and CAST Imaging.

Contenders

• GitHub is strong on vision but lagging in basics like reporting and language support. GitHub obtained a SAST solution when it acquired Semmle and its CodeQL analysis engine in 2019, and the company has spent the last year building CodeQL into the GitHub experience. CodeQL is part of GitHub’s Advanced Security offering, which is free for all GitHub open source users and available to commercial users on a per committer pricing model. GitHub integrates with several of the other competitors included in this evaluation, so GitHub users can augment their CodeQL usage with other SAST tools. GitHub’s community focused approach also exhibits itself through users contributing custom CodeQL queries, enabling additional rules and language support beyond what GitHub provides out of the box. The community element also comes to the fore when GitHub discusses its vision — as a leading open source repository, it views a built-in SAST offering as critical to making the software supply chain safer. That vision has the potential to be game changing, but today’s reality, which GitHub acknowledges, is it is new to the SAST market and must catch up on features. One major weakness is reporting, which is limited to a single, non-customizable PDF detailing the results of each scan. Reference customers agreed that the reporting needs work, but they appreciated the ease of use that comes from having a SAST tool fully integrated into GitHub. CodeQL’s integrations outside of GitHub are negligible, however. Customers that are all-in on GitHub for development will benefit from the baked-in SAST that CodeQL provides.
• Parasoft excels in custom reporting, but its remediation guidance needs attention. Parasoft comes from the code quality space and positions security as a quality issue. The Parasoft SAST subscription starts with static analysis for C/C++, C#, and Java — it then adds on several other packs with security and compliance features, extended language support, process intelligence, and traceability. The extended language support is a set of open source code analysis tools that feed results into Parasoft, but Parasoft does not maintain them. Customers can deploy Parasoft on-prem or in the cloud. While many of the vendors in this Wave struggled with reporting, Parasoft’s out-of-the-box and custom reporting capabilities stand out. Customers can create their own dashboards based on Parasoft-provided templates or start with an empty dashboard and add Parasoft-provided and custom widgets. The customizable risk scoring is also a strength. Reference customers rated Parasoft highly for accuracy and appreciated the new dashboards. The remediation guidance received mixed reviews, with one reference complaining that the code samples didn’t make a great deal of sense or were even incorrect on occasion. Parasoft is a good choice for customers looking for a solution that can address both web and embedded applications and that can report at the developer, security, and executive levels.
• GitLab meets developer needs but doesn’t compete on key SAST features … yet. Over the last few years, GitLab has invested heavily to integrate security into the GitLab experience. GitLab provides the individual developer with scans on the feature branch and security pros with scans on the default branch and on applications deployed to production. SAST scanning is available to all GitLab customers — Core/Free customers can run SAST scans from GitLab, and Gold/Ultimate customers have access to security dashboards and integrated workflow. Pricing is per seat, with no limit on the number of scans or number of applications scanned. Customer references see value from the CI/CD integration and the developer focus, but when it comes to other features, opinions were mixed. Scanning performance and the lack of available tuning was a sore spot. One reference described GitLab as a low-cost, operationally efficient, and highly usable solution that is not best-of-breed but is good enough. The question is whether this will still be the case in a few years. GitLab’s near-term product roadmap includes “catch-up” features like additional language support, reporting improvements, and custom rulesets, but proposed features a couple of years out are similar to what other vendors are planning. GitLab continues to aggressively invest in security, and its strength is in offering a SAST solution that works seamlessly with its developer experience. Customers fully using GitLab for development will want to investigate whether the SAST solution will meet their needs.
• Perforce Software’s Klocwork is best for functional safety and compliance. Perforce acquired Rogue Wave in 2019, bringing the Klocwork SAST solution into its portfolio of automated testing, code management, application management, and agile management solutions. Klocwork focuses on compliance and functional safety and caters to customers in industries such as automotive, medical, aerospace, and manufacturing that build mission-critical applications. Klocwork focuses on depth rather than breadth, covering only four programming languages — C, C++, C#, and Java. Keeping up with evolving security standards like PCI DSS, CERT, and MISRA is a prominent part of the product roadmap. Klocwork integrates with a wide array of IDEs and flags issues in real time as the developer introduces them. However, its out-of-the-box integrations with other tools were more limited — even integrating with one of Perforce’s own agile management tools, Helix ALM, requires the customer to write a python script. Forrester received limited feedback from Perforce reference customers, who also declined to go into much detail about their experiences with Klocwork. References were generally satisfied with key features like accuracy, performance, and out-of-the-box rules, but they were ambivalent about Perforce’s ability to put out new features and bugfixes in a timely manner. Organizations that build mission-critical applications for compliance-heavy industries will value Perforce’s laser focus on standards and functional safety.
• SonarSource’s SonarQube is a solution that targets developers. SonarSource comes from the code quality space and has added SAST to the SonarQube product that also looks at quality, reliability, and technical debt. The company extended its security investment in May of 2020 by acquiring German SAST vendor RIPS Technologies. SonarSource estimates that over 200,000 companies use the open source community edition of SonarQube — the commercial Developer and Enterprise editions add features such as dataflow analysis, OWASP security reports, and portfolio management. In its messaging and strategy, SonarSource targets developers as its core audience and stresses the importance of developers getting actionable data and real time feedback. That developer-first messaging runs up against some of SonarQube’s feature gaps. Today, SonarQube supports only Java, C#, PHP, and Python for SAST, and it provides no native integration with ticketing tools like Jira. A strong point is the custom rules, which can be created either via a query language or a UI. References viewed SonarQube as a great developer tool but were challenged to manage a large number of projects at the enterprise level — for example, reporting was viewed as helpful for developers in finding and understanding flaws but not as useful for getting an enterprisewide view. One reference commented that SonarQube “lacks the flexibility that you would typically want at an enterprise.” Organizations looking for a development tool that also does some security scanning will want to consider SonarQube — the large open source adoption means that many developers will already be familiar with it.

Challengers

• WhiteHat Security offers a semi-managed SAST solution. WhiteHat’s Sentinel Source is part of the WhiteHat Sentinel SaaS platform that also includes DAST, MAST, and SCA. Key to the customer experience is the threat research team, which creates and maintains rules and assigns proprietary scores to different flaws. Users can also send the threat research team questions about secure coding practices and vulnerabilities directly through the platform, the IDE, or the build tools and get responses to support their specific remediation efforts. Note that customers cannot create rules themselves and must make requests to the threat research team. WhiteHat could only provide limited customer feedback — those references praised Sentinel’s accuracy but noted that they depend on WhiteHat’s human interaction to eliminate false positives. Customers are also looking for support for newer languages, such as Kotlin and Swift. WhiteHat’s postsale support gets high marks, an important consideration given customers’ reliance on the threat research team. WhiteHat is a good option for customers looking for a semi-managed solution and willing to give up some control and agility in exchange for having a third party do a lot for them.

EVALUATION OVERVIEW

We evaluated vendors against 28 criteria, which we grouped into three high-level categories:

• Current offering. Each vendor’s position on the vertical axis of the Forrester Wave graphic indicates the strength of its current offering. Key criteria for these solutions include accuracy, remediation guidance and education, breadth of coverage, reporting, rule management, and integration with the SDLC.
• Strategy. Placement on the horizontal axis indicates the strength of the vendors’ strategies. We evaluated product vision, execution roadmap, market approach, planned enhancements, and performance.
• Market presence. Represented by the size of the markers on the graphic, our market presence scores reflect each vendor’s installed base and revenue.

Vendor Inclusion Criteria

Forrester included 12 vendors in the assessment: CAST, Checkmarx, GitHub, GitLab, HCL Software, Micro Focus, Parasoft, Perforce Software, SonarSource, Synopsys, Veracode, and WhiteHat Security. Each of these vendors has:

• A comprehensive, enterprise-class SAST tool. All vendors in this evaluation offer a range of SAST capabilities suitable for developers and security pros. We required participating vendors to have most of the following capabilities out of the box: source code scanning with broad language support, quality gates, reporting, and integrations with developer tools such as IDEs and build tools.
• At least $10M in SAST revenue and interest from or relevance to Forrester clients. Vendors in this evaluation earned $10 million or more in global revenue directly from SAST capabilities, and Forrester clients often discuss the participating vendors and products during inquiries and interviews. Alternatively, the participating vendor may, in Forrester’s judgement, have warranted inclusion because of technical capabilities or market presence.

End Notes

1. Base: 480 global security decision-makers with network, data center, app security, or security ops responsibilities who experienced an external attack when their company was breached. Source: Forrester Analytics Business Technographics® Security Survey, 2020.