MartinFederal Logo

The Challenge of Realizing Pentest Value

November 12th, 2020 • Category: Blog

By Dr. Wesley McGrew, Senior Cyber Fellow

Since recently joining MartinFederal, I have had the opportunity to meet with many people, both internally and externally,  that want to understand offensive approaches to cybersecurity—penetration testing, red teaming, vulnerability analysis, and similar services. Most of their prior exposure to this field is in the form of contracting vulnerability scans and penetration tests that are required of them by regulation and policy. I was not surprised to find that these experiences were not as positive or productive as they could have been. While a properly conducted penetration test with experienced operators can result in some of the best value for the money in identifying real-world vulnerabilities that would otherwise lead to expensive breaches, a lack of rigor and hands-on care in client interaction, scope, and communicating results, penetration tests can be disruptive and unproductive.

The relationship between penetration testers and individuals at the client organization often becomes adversarial, and when this happens, the test is not as efficient or helpful. A penetration test where there is animosity between the testers will result in the latter contesting the results, rather than taking useful action on them. It is the responsibility of the penetration testing firm to engage with language and communication that does not set up the IT staff to feel like they are the “fall guy.” The IT staff can put themselves on unnaturally-high guard for the limited duration of a test, and influence the scope of the test in a way that makes them look better, but does not represent what a real attacker could do to the organization. While it is counter to much of the culture of offensive security practitioners, penetration testers must not communicate in terms of “us versus them” and “gotchas.” Reports should explain vulnerabilities and impacts without using language that implies a judgment on the IT staff’s skill. A test should be framed as a mechanism for the staff to get the mandate and resources they need to make the network better, by leveraging talents and skill sets that they cannot afford to have in-house.

Scoping is an important concern for both the client and the effort provided by the tester. Too many clients limit the scope of an engagement to only involve external attack surface or limited to systems that are directly related to compliance. This ignores the nature and timeline of modern attacks. While a penetration test may take place over a set period of weeks following the kickoff meeting, a real attacker has the luxury of time. That time allows the attacker to look for opportunities that a penetration tester may not have in their more-limited window. They will be able to play a longer “social engineering” game to convince an insider to run malicious software, or they may be able to hack a trusted third-party site or partner. A previously unknown vulnerability may be disclosed publicly or discovered by the attacker that gives them a foothold on the organization. The fluid nature of cybersecurity means that preventing all possibility of the initial compromise of individual systems is unrealistic. The goal of the organization should be to prevent an attack’s potential impact by limiting movement around a network. A penetration test can only help achieve this goal if the scope of the test is comprehensive of all external and internal hosts.

The value of engaging in a penetration test suffers when care is not put into presenting the results. If you talk to testers that do the technical work of penetration testing, whether it’s fully automated or involves some amount of manual testing, they will tell you that their least favorite part of the engagement is reporting. This preference should not be surprising when you realize that their training and education focuses on the technical aspects of their job—the “fun part”—with comparatively little time spent on deliverables.

I have seen a lot of penetration test reports. To help stakeholders (owners, executives, IT managers, and more) get an idea of what kind of offensive testing they’ve had, I sit down with them and review their reports from previous penetration tests with a wide variety of vendors. What I see most often is the absolute minimum effort: a report generated automatically by a vulnerability scanner. If you have been required to engage in vulnerability scanning or contracted a penetration test that left you feeling underwhelmed, you have seen the same output. They have the following negative traits in common:

  • The overwhelming length often inches in printed thickness. Every single potential issue identified in every single host is given room on the page to vie for the limited attention that you can afford to give the report.
  • A lack of useful prioritization. Your resources for addressing issues are finite, and you cannot solve them all at the same time. Many reports will include severity for findings. Without significant analysis on the part of an analyst—not a program—this score is not useful for allocating resources to fix issues. A true prioritization will incorporate post-exploitation impact, ease of remediation, and other risk-based analysis that is specific to your organization’s environment.
  • False positives. These are findings that are the result of errors made by vulnerability scanning software. These errors will always exist, because it is often impossible to accurately test for a vulnerability without going through the full process of exploiting it, with all the context of the target environment. Many vulnerabilities have no known path to exploitation. While those vulnerabilities should be addressed, they should not be given the space and positioning in a report that verified, high-impact findings are given.

Penetration tests have been commoditized. There are enough firms that have purchased licenses to automated vulnerability scanning software (that require no training to operate) that advertise penetration testing services that it can be difficult to identify which providers actually use skilled operators in a meaningful, human-driven way. Limitations in scope, both by the client and in technical effort by the provider, can lead to significant gaps in coverage by a penetration test. Adversarial relationships and poor communication of findings can lead to resources being held back or allocated in the wrong areas. Without a clear and prioritized path to remediation, the high value that a penetration test can potentially provide cannot be realized. When contracting a provider for offensive security testing, have a conversation about the points made in this article to understand their approach and whether it will provide value for your organization.