Four reasons why Breach and Attack Simulation (BAS) cannot validate security

Breach and attack simulation (BAS) tools are designed to determine whether security controls can detect and respond to attacks. BAS tests security efficacy by simulating attack scenarios and providing an analysis of how security controls perform. Gartner defines BAS technology as "tools that allow enterprises to continuously and consistently simulate the full attack cycle (including insider threats, lateral movement and data exfiltration) against enterprise infrastructure, using software agents, virtual machines and other means."

BAS has proven to be useful for security teams who need a snapshot of how specific security controls are performing. However, as the threat landscape and attack surfaces expand, and remote working becomes more prevalent, BAS tools are insufficient to truly validate security performance. Gartner’s definition of BAS implies that these tools are an ideal testing option, but they lack critical features needed to continuously validate security performance across an organisation's entire security infrastructure — which companies now need to continuously assess their risk profile and operational capabilities.

Breach and attack simulation tools differ from security validation in several key areas. First, they are simulations rather than real attacks, which are often not treated as real threats by security systems. In addition, BAS lacks threat intelligence inputs, preventing security teams from pinpointing the threats most relevant to their testing. Here are four key areas where security validation provides capabilities that BAS lacks, enabling CISOs and their teams to gain a comprehensive understanding of the security levels of people, processes, and technology.

1. Real attacks matter

Simulations conducted by BAS are incomplete and unrealistic, and security controls often do not recognise them as threats. Security controls may "see" the simulation but fail to trigger alerts as they would for a real attack, resulting in a false sense of security. On the contrary, security validation executes simulated attacks across the entire kill chain and provides a comprehensive, accurate overview. CISOs should explore validation solutions that simulate real attacks for the following reasons:

1) Executed under a safe and controlled state, real attacks provide evidence of how security controls perform during an actual attack, giving CISOs control over the results.

2) Attack simulations provide a pre-attack and post-attack picture across the entire kill chain to comprehensively identify all vulnerabilities.

3) Simulated or reverse-engineered binaries will cause machine learning and AI to learn incorrect behaviours, rendering their learning ineffective.

2. Testing across the security infrastructure provides an accurate picture

BAS solutions can be partially integrated, but do not support the full suite of systems, meaning they only provide a partial picture of what is happening in a company's IT environment. For example, many BAS providers only support testing of endpoint security products and cannot determine if other threat vectors serve as entry points. This is like assessing someone's health solely by studying what they eat for lunch.

To get a complete view of security control effectiveness, CISOs should look for security validation solutions that support testing across the entire attack lifecycle and the entire security infrastructure, including network, email applications, cloud, and endpoints. In this way, they gain a holistic understanding of efficacy and vulnerabilities — much like observing a person's overall diet, exercise routine, and other habits provides a comprehensive view of their general health.

3. Comprehensive attack libraries to identify relevant threats

Testing solutions are based on the contents of the vendor's attack library, which lacks sufficient reliability. BAS solutions do not provide sufficiently timely and reliable intelligence on the latest tactics and techniques being used by attackers, so security teams cannot determine which threat actors are the most important to test against. Furthermore, BAS tools lack support for attack frameworks like MITRE ATT&CK and the Kill Chain, which also limits testing capabilities. Overall, the lack of threat intelligence can produce inconsistent testing results and lead to an inaccurate understanding of an organisation’s true security posture and risk profile.

Instead, CISOs should consider security validation, which provides up-to-date threat intelligence. Based on a comprehensive library of attack behaviours, tactics, techniques, and procedures, security teams can prioritise testing the most relevant attack threats based on attack data specific to an organisation or industry.

4. Monitoring environmental drift to strengthen defences

IT environments are constantly changing as systems are added or reconfigured, new devices join the network, or patches and upgrades are automatically introduced. These changes can affect how security controls behave, but without continuous monitoring for IT environmental drift, security threats can occur without the security team's knowledge. BAS tools do not perform this type of continuous monitoring and alerting, so teams cannot remediate as needed. This can ultimately degrade the health of the security infrastructure.

When evaluating security validation, security teams should ensure three key capabilities related to the IT environment:

1) Automatic and continuous monitoring of environmental drift;

2) The ability to alert relevant personnel when changes are detected;

3) Providing recommendations on how to remediate the situation.

Minimise risk and enhance security operations capabilities

True security validation requires the aforementioned capabilities to guarantee the accuracy and completeness of testing results. Without them, companies remain unaware of their security posture, exposing themselves to attack threats. Unlike BAS, security validation is automated, continuous, comprehensive, and holistic, and is based on timely, actionable threat intelligence — all of which provides CISOs and business leadership with quantitative data on security effectiveness across technology, processes, and people. These can effectively minimise a company's risk profile and help maintain security operations capabilities.