Live Stream Recap | Seminar on Continuous Security Effectiveness Validation

In August 2023, Anzai New Media, as a leading third-party professional cybersecurity media service provider in China, released a major report — "2023 China Continuous Security Effectiveness Validation User Survey Report". From the perspective of users and incorporating mainstream industry viewpoints, the report provides a comprehensive analysis, outline, summary, and outlook on security validation.

Using the release of the report as an opportunity, Anzai New Media specially organized a themed live broadcast, inviting experts such as Wang Jingpu, CTO/Co-founder of Cyritex, Zhao Rui, Chief Security Officer of a multinational enterprise, and Bruce ZHANG, founder of Shiyuan Business Consulting, to discuss hot topics and share insights. The live broadcast was hosted by Zhang威, a partner of Anzai, creator of the report, and host of this session.

(From top to bottom, left to right: Zhang Wei, Wang Jingpu, Bruce ZHANG, Zhao Rui)

Topic Sharing: "Security Validation Technical Sharing"

Wang Jingpu

Cyritex CTO/Co-founder

01

What is Security Validation?

Gartner defines it as: Security validation is the convergence of technologies, processes, and tools used to validate how potential attackers exploit identified threat exposures, and the actual response of security defenses and processes. Blue team and red team tools are converging towards highly customized and flexible intrusion to more effectively test an enterprise's defense capabilities, including the effectiveness and configuration of security controls and monitoring tools. The resulting validation results make cross-team decision-making easier and help organizational decision-makers allocate relevant resources.

And Cyritex defines it as: Security validation utilizes attack scenarios covering the entire cyber kill chain of real hacker attacks to automatically and continuously validate the effectiveness of security defenses. This includes validation of "security operations - basic operations", validation of "security product defense capabilities", validation of "security operations - correlation analysis", and validation of "security operations - incident response". It helps customers assess the current state of defense effectiveness, optimize and supplement based on mitigation suggestions from validation results, and ultimately achieve the improvement or maintenance of overall defense capabilities.

Wang Jingpu introduced that scanning tools emerged in the 1990s, such as Nessus, which are still used today. By 2000, patch management similar to vulnerability scanning began to appear. 2013 was a watershed; the famous "PRISM" incident made more security practitioners aware of APT attacks, leading to manual POC validation and manual penetration testing validation. In 2015, automated penetration validation emerged, as well as service-oriented validation methods like red-blue confrontation/offensive and defensive drills. From 2016, the inaugural year of the "Hu Wang" (Protecting the Network) campaign arrived. Therefore, from manual validation all the way to Hu Wang, everyone has actually been subtly exposed to validation methods.

In 2017, Gartner defined the BAS concept. In its initial definition of the domain, Gartner used Breach and Attack Simulation. In 2021, Gartner listed BAS technology as one of the top eight security and risk management technology trends of the year. In the security operations maturity curve of the same year, BAS technology reached its peak of popularity, and it was also referred to as security validation in China around the same time.

In 2022, Gartner's prediction for the widespread adoption of BAS technology changed from 5-10 years to 2-5 years, showing that BAS technology is being rapidly accepted by the market. In 2023, Gartner evolved the BAS technology field into the concept of "security validation".

By 2020, the concept of security validation had already appeared abroad and expanded upon the BAS concept, with terms like security validation, security control validation, continuous security validation, and cyber security testing and validation. These developed corresponding products based on the needs of different client users.

02

What is the difference between security validation and traditional BAS?

Security validation is an expansion and development of the BAS concept; we can understand it as BAS 2.0. Theoretically, security validation solutions, in addition to having the functions of BAS, should also possess capabilities in the following aspects:

1. Destructive Attack Simulation

Realize real simulation of destructive attacks while ensuring no damage to business operations, while achieving full kill-chain attack simulation.

2. Built-in Attack Scenarios

Built-in large number of pre-defined attack scenarios (e.g., industry-specific attack scenarios, APT group scenarios, etc.) rather than single vulnerability exploitation scenarios.

3. Continuous Security Validation

Continuous and automated attack simulation and closed-loop validation to monitor security defense drift in the IT environment. The significance of continuity is extremely important; only with continuity can we know the actual defense level of an enterprise over 365 days.

4. Real-time Threat Intelligence Linkage

Real-time linkage with third-party threat intelligence automatically converts attacks into experiments, thereby enriching attack methods and corroborating the realism of attacks.

5. Overall Validation and Assessment

Aims at validating the overall security defense system and security operations capability, which can later evolve into risk assessments for enterprise operations.

There are many technical challenges in security validation, the most difficult and core of which is the attack library. The quantity, richness, and coverage of the attack library determine the core capability of the product. First, in terms of the cyber kill chain, it must cover all stages, rather than just marking points in the initial access phase; it must cover the entire chain. Second, it must cover various techniques and tactics, as summarized by the world-famous ATT&CK framework.

The third point is to cover a wide variety of attack scenarios, such as phishing, cryptomining, ransomware, or APT groups. The fourth point is to cover various threat actors, including the attack methods of relevant groups, and the volume each group possesses, which determines whether the enterprise's simulation has enough output across different aspects. If the quantity, richness, and coverage of the library are insufficient, it is difficult to achieve comprehensive attack simulation and hard to evaluate the current state of the client's entire defense system.

Other technical difficulties include updates; only timely updates can keep up with the generation and use of hacker attack methods. Another is making attack simulations harmless. Since current security validation is used in real production environments, it must ensure the simulation of the entire kill chain while guaranteeing absolutely no impact on the overall environment. Furthermore, there is the simplification of integration linkage and the practical implementation of mitigation suggestions.

In terms of deployment key points, first is the understanding of the environment; the vendor needs to see if their product is suitable. Second is the deployment method, which can be diversified. The third point is consideration; for clients, it should include convenience, safety, labor costs, and scalability. The fourth point is operations; enterprises must formulate executable security validation operation plans so that they can integrate with current security operations to reflect the value of security validation.

With security validation, it is possible to cover the entire complete kill chain and simulate different attack methods at different attack stages. When the security validation platform is deployed in a real environment, as long as it is located in a zone, it is fine as long as it can communicate with the deployed nodes throughout the environment. For users, they only need to orchestrate attack scenarios on the platform, using either the platform's built-in scenarios or custom orchestrations.

The platform + nodes will simulate the entire process of a hacker attack. In different stages and different technical and tactical parts, there can be attacks of different volumes and different methods, ultimately validating whether the enterprise's current various defense means are effective.

Client Doubts

After the presentation, Zhao Rui expressed that, compared to past BAS, security validation shows a significant improvement in measurement and feedback. In the past in DevOps or DevSecOps, for the development, testing, and operations phases, clients would look at relevant indicators, such as the number of vulnerabilities in open-source components, vulnerability discovery and remediation rates before going live, during operations, during development/testing, and during production operations, or in red-blue confrontations, as well as false positive and success rates of some code security scans. In addition, it also included vulnerability discovery, false positive, and remediation rates for black-box scanning, and the number of vulnerabilities, discovery rate, and false positive rate scanned by white-box tools, etc. Although enterprises have large platforms for overall management, after Cyritex's security validation came out, it integrated many of these functions.

Zhao Rui pointed out that many people think doing security is just about stacking products, but the more products are stacked, security capabilities do not necessarily improve; the key is whether the client uses the products well. Some clients are more diligent and will go through processes with vendors to better integrate security products into the enterprise's security management lifecycle, officially launching them only when false positive rates are low. However, some clients see a good product elsewhere and immediately put it online, resulting in many false positives, which actually reduces security capabilities and the efficiency of security work.

To this end, Zhao Rui raised a question: in many lifecycles of the client, can Cyritex validation replace existing tools, such as automated tools for code detection, penetration testing, and offensive and defensive drills?

Wang Jingpu replied: First of all, our goal is not to replace the client's original methods. Rather, it can be understood that security validation uses another approach, testing through the dimension of attack simulation. It focuses more on simulating attacks from a hacker's perspective, so vulnerabilities are only part of it. It focuses more on various attack methods and is a comprehensive defense system measurement platform based on various hacker techniques and tactics. Therefore, many tests have nothing to do with vulnerabilities and simulate the existing means of hackers.

Next, Zhao Rui raised a second question: Currently, it is not uncommon for vendors to use existing, public, and exploitable vulnerabilities to conduct POC tests. However, testing in combination with business scenarios, as well as the industry, system, and architecture where different enterprises are located, is rare. Therefore, he hoped Wang Jingpu could further introduce the content where attack simulation scenarios can be set and recorded.

Wang Jingpu stated: The focus of security validation revolves around defense effectiveness, which measures the defense effectiveness of various defense means at the periphery of applications and systems against attacks. Therefore, peripheral protection and vulnerability detection inside applications and systems are two different dimensions, and both should exist.

One exhausting point for clients is that applications and systems will always have vulnerabilities, and they can never be fully patched. Therefore, most of the time, the client's applications and systems rely on peripheral defense lines to see if these defense means can prevent various attack attempts against applications and systems.

Security validation products are currently undergoing development iterations, so in the future, they will also cover "making attack attempts against a certain target system." However, relatively speaking, they cannot truly replace manual penetration testing yet, though the integration of AI capabilities into products in the future cannot be ruled out. Currently, for target systems, security validation can complete part of the work, at least telling the client which vulnerabilities exist and which can be exploited.

Unlike the domestic market, for overseas markets, Bruce ZHANG stated that from the perspective of investment and industry research, client enterprises have paid close attention to security validation in recent years, and many new companies of this kind have emerged globally. Next, Bruce ZHANG also raised his own question, wanting to know how custom attack methods in security validation are operated.

Wang Jingpu replied: Customization in security validation has two levels. The first is based on the existing tool library of security validation products, where client enterprises can define and design attack scenarios themselves. That is, whether it is an attack incident attribution report obtained from external resources or an internal incident attribution report, as long as the library contains the attack methods mentioned in these reports, enterprises can select and define them into attack scenarios, and then run attack simulations.

The second level of customization is more challenging for clients. First, the client must be able to obtain the attack method, and second, analyze the details of the attack method, so that through the customization function provided by security validation products, the attack method can be defined into an attack experiment that can run on the platform. Of course, clients can also hand over tasks like collection and analysis to the vendor to let them define it.

Bruce ZHANG remarked that some foreign companies have indeed integrated security validation into the ecosystem in a truly automated way. Unlike other tools that only complete projects or actions requiring clients to iterate the tool based on reports, security validation can link with other tools in actual work to complete updates. What made Bruce ZHANG curious is whether the future development trend of security validation will lean more towards "being a platform," just like game consoles, where the game cartridge company and the game console company are two different entities. Is it possible that in the future, those making security validation products and those customizing attack methods will be on two different tracks?

Wang Jingpu replied: This is entirely possible. As long as clients do not have time to collect and analyze various attack methods, they will buy services from relevant companies, and these service providers can run them on security validation products as long as they define the attack methods through a standard platform. Of course, at this stage, it is best for the custom module and the security validation platform to be provided by one company, as compatibility issues may be involved.

Since the API interaction ecosystem among various security products in China is not yet mature, direct operation and interaction between platforms are very difficult. Of course, there are also platforms that can act as intermediate links, connecting security validation platforms with other security products to automate intermediate processes and iterate based on the validation results and the validation methods used.

Report Sharing

"2023 China Continuous Security Effectiveness Validation Product Enterprise User Survey Report"

Zhang Wei

Anzai Partner, Report Publisher

From the perspective of security development, security in the past was like protective gear, which business departments often discarded. Later, security became like a bodyguard; although its status in business improved, business always felt security affected efficiency. With development, current security is more like white blood cells, becoming a part of the business, no longer a burden, and security is completely implanted into the entire ecosystem chain. Therefore, security validation has become a very important node for the future.

The report points out that in the field of security validation, continuous automated security validation has become a new trend to replace traditional manual security validation technologies. Through highly automated and highly secure attack simulations, automated security validation products allow users to validate, assess, and continuously optimize the effectiveness of enterprise security protection in a practical way, continuously ensuring the comprehensive and continuous security status of user networks under dynamic threat postures. In the future, it will play an important role in security operations, red-blue confrontation, security decision-making, and security supervision.

Currently, in the Chinese market, security validation can indeed replace part of the role of penetration testing and red-blue confrontation. In client enterprises, penetration testing is carried out by internal teams, while also using external third parties to further validate their internal protection results. In addition, the emergency response format of enterprises still mainly relies on manual and system-driven methods, meaning the level of automation is not strong, and a potential market and demand for security validation exist.

The survey shows that corporate users' awareness of automated security validation products is rapidly improving, with those self-rating 5 points or above exceeding 57%, and those with a high level of familiarity (8-10 points) accounting for 15.9%.

Corporate users have a high acceptance of automated security validation products, with 96.8% of users accepting them, and only 3.2% rejecting them. Reasons for rejection mainly include "the organization does not allow the use of SaaS products," "worry about leaking internal information," "not a necessary product," "worry about the instability of automated security validation products," and "too expensive."

Although large overseas enterprises have high requirements for implementation and continuity, currently in China, small and medium-sized enterprises, including many large enterprises, do not have as high requirements for product ecosystems and system iterations as those overseas, so some users remain skeptical about security validation.

The report points out that China's cybersecurity market is growing steadily and is expected to exceed 250 billion RMB by the end of 2023, with a compound annual growth rate of 19.3% from 2021 to 2023. Among them, the automated security validation product market is growing rapidly, with a market size of about 430 million RMB in 2019, which is expected to reach 1.01 billion RMB by 2023, representing a compound annual growth rate of 23.8%.

The survey shows that among more than a thousand users, 20.93% of corporate users have already deployed automated security validation products; 33.06% have not purchased but plan to purchase; 31.94% are considering; and only 14.07% of users feedback that automated security validation is not necessary for their enterprises.

Among enterprises that have participated in national-level offensive and defensive drills, 28% of corporate users have deployed, 37% plan to purchase, 26% are considering, and 9% deem it unnecessary. Among enterprises that have not participated in national-level offensive and defensive drills, 14% of corporate users have deployed automated security validation products, 30% plan to purchase, 37% are considering, and 19% deem it unnecessary. Obviously, corporate users who have participated in national-level drills are more willing to accept and deploy automated security validation products.

The survey shows that automated security validation products are currently concentrated in industries such as the Internet, finance, government, smart manufacturing, and smart connected vehicles/Internet of Vehicles. Among these enterprises, categorized by size, large enterprises with more than 1,000 people account for 41%, medium-sized enterprises with 300 to 1,000 people account for 35%, and small and micro enterprises with fewer than 300 people account for 24%. Categorized by enterprise type, government/state-owned enterprises account for 39%, foreign-invested enterprises account for 20%, private enterprises account for 29%, and individual businesses account for 12%.

The main roles of corporate users deploying automated security validation products currently are conducting security compliance assessments, offensive and defensive drills/SOC drills, corporate security defense posture assessments, and replacing manual penetration testing. The size of the corporate security team is correlated with the deployment of automated security validation products, with teams of 5 to 20 people showing the highest willingness to organize and deploy automated security validation products.

The report analyzes vendors that have already deployed products, including 6 domestic and 18 enterprises. The analysis mainly focuses on four dimensions: "richness of simulated attack scenarios", "ease of deployment", "usability", and "completeness and comprehensiveness of output reports".

The survey shows that the main problems current users face when using automated security validation products include limited simulated attack scenarios, insufficient automation capabilities, inconsistent validation, incomplete reporting functions, and a lack of standardized testing methods.

The survey shows that corporate users who already have plans to deploy automated security validation products are mainly distributed in the Internet, finance, government, smart manufacturing, and other industries, and are also more common in medium and large enterprises.

The survey shows that corporate users planning to purchase and deploy automated security validation products expect to deploy within half a year (24%), within 1 year (56%), and within 2 years (20%), with the majority deploying within 1 year. Corporate users still considering deploying automated security validation products are mainly distributed in the Internet, finance, smart manufacturing, government, healthcare, and other industries.

In the future, the development trend of security validation must be platform-based, with its core competitiveness being the attack library, and it will also be reflected in intelligence, full-scenario application, growth, and operation.

Roundtable Discussion

After the report sharing, experts from all sides began the roundtable discussion.

Zhao Rui: The client's demand for security is, first of all, legal compliance, and secondly for the enterprise itself, which is to make more money while ensuring the safety and stability of the system. Looking at the report released by Anzai, attack methods and techniques have been constantly changing in recent years, and correspondingly, the ways to protect the stable and normal operation of client business production are also constantly evolving. The differences in using new products and new technologies will be huge depending on the size of the enterprise and the number of security team members.

Therefore, target users of new products are still concentrated in large, strictly regulated enterprises, while small and medium-sized enterprises actually face the same threats on the Internet, just on a smaller scale, so they receive less attention. With the promotion and application of new products and technologies, subsequent security products had better develop towards a trend that allows small and medium-sized enterprises to use them easily, so that our market environment will be better.

Wang Jingpu: Currently, Cyritex has released the SaaS version of security validation. We hope that through promotional policies and marketing efforts of the SaaS version, more SMEs can understand, experience, and use security validation products. The SaaS version of security validation needs to deploy nodes internally. Since SMEs do not have many internal security zones and have relatively fewer points of concern, the number of nodes and scenarios will not be too many, and the pressure on the team will not be too great. Bruce ZHANG: Security has so many scenarios, whether it is ransomware, anti-fraud, or at the business level. Ultimately, it still needs to return to the concept of scores. For example, an enterprise with 20 problems is better than one with 30 problems. This requires a weighted scoring system. If security validation can be linked with this system, its value will be more prominent. For example, in the scoring system, if an enterprise does not do something well, its score will become 59, and if improved, it will become 81, or the score can suggest what the enterprise should fix first and validate first, which would be more perfect.

Regarding Bruce ZHANG's suggestion, Zhang Wei stated that many enterprises are reluctant to score their own security posture, so he asked how the client views this matter.

Zhao Rui: Different enterprises have different focuses. In many client enterprises, what is actually assessed at the end is not security, but development, testing, and operations. For example, for classified protection grading, every enterprise will start relatively low, and then improve year by year up to 80 or 90 points, after which it becomes difficult to improve further. Subsequently, as standards are updated and regulators issue new requirements, the score will drop again. At this time, the security department can explain to management and other business department leaders that because of the new regulatory requirements, our score in a certain part has dropped. But overall, security will not directly admit where it has not done well.

Zhang Wei asked: What preparations do users need to make to deploy security validation?

Wang Jingpu: For enterprises with many internal security zones and strict execution of original security policies, there will be more preparation work, which is also more detailed. For example, enterprises need to prepare resource applications, node planning, and the opening of network policies. On the other hand, after security validation is deployed, dedicated personnel are required to control it; it cannot just run completely automatically. If enterprises do not conduct corresponding analysis and subsequent optimization and improvement, it is equivalent to violating the original intention of using security validation.

Currently, the products available on the market are not that complex, mainly focusing on analyzing after results come out to see how to optimize and improve. This requires dedicated personnel responsible for the platform and leaders of different security dimensions to discuss and analyze missed detections together. Therefore, relatively speaking, it emphasizes internal cooperation rather than having a dedicated person stare at it overall.

Zhang Wei asked: If an enterprise wants to deploy security validation, which performance aspects or indicators will the client focus on more?

Zhao Rui: For security validation, we first measure how the benefits it brings us are, for example, whether it can bring beneficial supplements besides code detection, vulnerability scanning, and penetration testing. Secondly, since all products and services require corresponding employees to manage them, in terms of resource allocation, personnel input and the cost-benefit ratio are also the focus of our attention.

Mainly, it is reflected in four aspects: "Does such a problem actually exist?" "Does regulation have corresponding requirements?" "What is the software cost?" "What are the corresponding impacts brought by manpower?" If security validation can give the security team more say in the enterprise and better serve corporate business operations, everyone will be willing to purchase and use it.

Zhang Wei asked: Assuming there are two people in the red team finding vulnerabilities, doing attack and defense, and penetration testing, can deploying security validation replace part of the original workload?

Wang Jingpu: For example, if an internal penetration tester wants to complete manual validation, he needs about half a month. First, he needs to collect vulnerability exploits or a certain attack method, which takes about three to five days. After getting the attack method, he spends one to two days studying it. Next, he needs to build an environment internally to validate whether the researched content meets expectations, and finally, actually implement and validate the target. The whole process takes two weeks.

But with security validation products, if the attack method or vulnerability already exists in the attack library, it can be run directly. Because the attack nodes and attacked nodes of security application products are ready-made, tasks that originally took ten days can now be completed in half a day. At the same time, relevant personnel can use the saved time for deeper research, such as discovering some business logic vulnerabilities, which automation cannot do. In summary, if there are many people, staffing can be adjusted; if there are few people, security validation can perform basic work, giving technical personnel time to perform higher-level tasks.

Zhang Wei asked: What is the value of the security validation track?

Bruce ZHANG: Security tools have been growing explosively in the past two years. Under this trend, many products that can directly bring value to customers will rank higher. Secondly, from the perspective of the overall economic environment this year, security products closer to the business side are more likely to gain the client's attention. Therefore, overall, the security validation track is quite good.

On the other hand, how does security validation compare to cyber ranges? One is controlling the scale in a real environment, and the other is real fighting in a fake environment. Which one will the client choose first, as the value points brought by the two may be different. Personally, I feel customers may prefer to polish their security tools in real environments.

Zhao Rui: Cyber ranges and security validation, different companies make different choices. Some enterprises worry that touching production will impact their business, so they prefer cyber ranges. But some enterprises like to test in real environments because the impact of attacks on real business is hard to discover in cyber ranges. Therefore, products have their respective target customers. In terms of actual effects, security validation done in real environments can better demonstrate security capabilities and provide convincing proof through actual effects.

Zhang Wei asked: What are the factors limiting the development of security validation?

Bruce ZHANG: It is standardization. Although the definition of a specific attack method is definitely standardized, combining them seems difficult to achieve unified standardization. In short, cartridges and game consoles cannot match, and they have some degree of adhesion in design. At the same time, foreign countries have not developed very well in the matching of "cartridges and game consoles" either.

Wang Jingpu: Therefore, security validation must be linked with threat intelligence. This can be security validation vendors directly cooperating with third-party threat intelligence vendors, or clients with custom attack method platforms finding domestic suppliers with real value. Whether "cartridges" are provided by security validation vendors themselves or by third-party suppliers, the purpose is to enrich the attack library.

As for which scenarios the validation effect is better in, this is directly related to the client's expectations. Some enterprises want to validate basic problems in basic operations, some want to validate the capabilities of security products themselves, and some want to validate automated response processes. On the other hand, products need to have flexibility and cannot just be strong in one aspect. Of course, products that are large and comprehensive but lack highlights are not good either. All in all, the stronger the comprehensive strength, the better it can meet customer expectations.

Zhang Wei asked: From the client's perspective, what problems do you hope security validation can help you solve?

Zhao Rui: First, I hope it can reduce the waste of resources caused by false positives of tools for security personnel. Second, I hope that after using security validation, efficiency can be improved. At the same time, I also hope that security validation can be like low-code or zero-code platforms, providing relevant configurations for business security when business raises corresponding demands, so that security and business can cooperate better.

Zhang Wei asked: What is the future prospect of the security validation market?

Bruce ZHANG: If a product can connect the source with the end user, business personnel can directly write demands, and then let large models automatically implement them, generating playbooks for simulated validation. Everyone's role in the security circle is different, and their expertise is also different. People who are extremely good at attack and defense do not need to study how to build game consoles; they can just focus on playbooks. The existing white hats are a ready-made community, which means constant vitality for the platform.

When it comes to front-end business, if some new technologies can be used, maybe every company will no longer be troubled by building the entire lifeline, but can make the entire security validation into an ecosystem chain, with everyone doing one of the links well, and the final product delivered to customers being assembled, standardized, and of high quality. If security validation can become such a model in the future, the market will be extremely large. The relevant report has been uploaded to "Zhuziyun Knowledge Planet", scan the code to get it.