Critical Vulnerability | Citrix NetScaler ADC Session Hijack and Gateway Vulnerability Analysis (CVE-2023-4966)

▌LockBit Ransomware Gang Actively Exploiting Citrix Bleed Vulnerability in High-Profile Attacks

Recently, following the tracking of attacks against several companies including a certain bank, DP World, Allen & Overy, and Boeing, all of them were found to be affected by the Citrix Bleed vulnerability. The latest news also confirms that LockBit's attack on the bank was achieved by exploiting the Citrix Bleed vulnerability (CVE-2023-4966).

The LockBit ransomware exploits CVE-2023-4966 to breach the systems of large organisations, steal data, and encrypt files. This vulnerability is a highly critical, remotely exploitable information disclosure vulnerability.

On 10 October 2023, Citrix released a security bulletin regarding a sensitive information disclosure vulnerability (CVE-2023-4966) affecting NetScaler ADC and NetScaler Gateway appliances, claiming the vulnerability had been patched. However, thousands of internet-exposed endpoints are still running vulnerable appliances.

Citrix Server Distribution Map

SycSec has observed zero-day exploits starting from late August 2023, as well as n-day exploits following the Citrix release. Based on the successful exploitation of CVE-2023-4966 in these attack incidents, this vulnerability allows legitimate user sessions on NetScaler ADC and Gateway appliances to be taken over. Session takeover bypasses passwords and multi-factor authentication.

The following sections will discuss the indicators that can be used to identify exploitation activity and highlight some of the post-exploitation techniques observed.

▌Vulnerable Endpoints

Prior to 10 October, tracking revealed that attackers were taking over users' NetScaler sessions through unknown means. Before Citrix's release and the development of the PoC, we believed the session takeover was the result of a zero-day exploit of an unknown vulnerability. We have identified the vulnerable endpoints along with a PoC to validate the vulnerability.

URLs Vulnerable to CVE-2023-4966

By crafting an HTTP GET request with an HTTP Host header exceeding a certain length, a vulnerable appliance will return the contents of system memory. The memory contents can include valid NetScaler AAA session cookies. This cookie is issued after authentication, which can include multi-factor authentication checks. An attacker with access to a valid cookie can establish an authenticated session with the NetScaler appliance without knowing the username, password, or having access to the multi-factor authentication token or device.

▌Analysis

The challenge in analysing vulnerable appliances to exploit CVE-2023-4966 is that the web server running on the appliance does not log requests (or errors) to the vulnerable endpoints. We do not know what configuration changes can be applied to force logging of requests to these endpoints. To identify attempted exploit requests, organisations will need to rely on Web Application Firewalls (WAF) or other network appliances to log HTTP/S requests targeting NetScaler ADC or Gateway appliances.

Therefore, the following methods can be used to identify potential exploitation of CVE-2023-4966 and subsequent session hijacking. These include the following techniques:

• Analysing requests sent from WAF to vulnerable HTTP/S endpoints

• Identifying suspicious login patterns based on NetScaler logs

• Identifying suspicious Virtual Delivery Agent Windows registry keys

• Memory core dump file analysis

Note: Identifying historical exploitation is challenging due to the limited forensic evidence left behind by exploiting this vulnerability. Patterns of suspicious activity associated with session hijacking can vary from environment to environment, and the techniques described below may not be applicable or feasible in all situations.

▶ Suspicious Login Patterns on NetScaler

Following successful exploitation of CVE-2023-4966, two types of logs have been identified that can be used to identify historical evidence of session hijacking. While these logs can be used to find evidence of suspicious login patterns, determining whether a session was hijacked by an attacker requires context and further analysis. For example, during analysis, analysts need to consider whether the appliance logs the true client IP address (as opposed to a Network Address Translation [NAT] address) and what normal user behaviour looks like (whether users frequently change IP addresses).

Ns.log Analysis

Citrix NetScalers log system message information in the ns.log file, stored in /var/log/ns.log. Log messages recorded in nslog can include connection statistics for SSLVPN and ICA proxy sessions.

Note: The default log rotation configuration on NetScaler allows for 25 files of 100 KB for each log type (e.g., ns.log), for a total of 2.5 MB logged. In most cases, this log size does not provide enough space to cover the entire investigation period. Additionally, log forwarding via syslog can be configured to forward logs off the NetScaler appliance.

1. Sessions Accessed from Different Source IP Addresses

Suspicious SSLVPN sessions can be identified by comparing the IP addresses Client_ip and the source fields recorded in TCPCONNSTAT events. TCPCONNSTAT events record TCP connection-related information for connections belonging to SSLVPN sessions. Please refer to the example log event in the figure below (field values have been replaced with placeholders).

Example of a TCPCONNSTAT message with mismatched client and source IP addresses

Based on our analysis, the Client_ip field is based on the client IP address that originally created the SSLVPN session, and the source field is based on the client IP address of the connection being recorded. For hijacked sessions, we observed TCPCONNSTAT events where there was a mismatch between the Client_ip and source fields, and upon analysis, the source IP address was suspicious. IP address changes during a session can also occur due to legitimate circumstances such as network changes (e.g., enabling/disabling Wi-Fi, switching to VPN, etc.). Investigating the ASN and geolocation of the Client_ip and source values can provide insight into whether this behaviour is potentially malicious or a false positive. NetScaler appliances may also log similar connection statistics in other log messages.

2. Multiple Sessions from the Same IP Address

Another detection opportunity identified through analysis is a source IP address accessing multiple user accounts within a short period of time, recorded in the ns.log file or logs forwarded via syslog. While attackers may choose to access a single account from a single source IP address, we observed attackers accessing multiple accounts from the same source IP address within a span of a few hours.

Note that multiple user sessions originating from the same IP address can also occur legitimately, depending on the network used by the organisation (e.g., employees from a remote location accessing Citrix sessions using the same Internet Point of Presence, NAT IP addresses, etc.).

Windows Registry Keys

When a NetScaler appliance successfully proxies a user session, the target Citrix Virtual Delivery Agent (VDA) system logs the source host, source host local IP, and associated UserSid in the Windows registry of the Citrix endpoint.

By correlating these registry values with the timestamps and user accounts recorded in ns.log that may relate to the hijacked session, the attacker's original host name (i.e., DESKTOP-1ABC2DE3) and local host IP (i.e., 10.0.2.15) can be determined. Once identified, the attacker's host name and local IP address can be used to hunt for other attacker sessions within the Citrix VDA environment in the registry and Windows Event Logs (specifically Event IDs 4624, 21, 23, 24, and 25). This data can also be used to increase confidence in suspicious sessions identified in log files, for example, if the recorded client host name is a default Windows host name (i.e., DESKTOP-########).

▶ Memory Analysis of NetScaler Memory Core Dump Files

In some cases, evidence of exploitation can be found in the memory space of the appliance's NSPPE process. Exploiting CVE-2023-4966 does not crash the NSPPE process and generate a memory core dump file. To analyse NetScaler memory core dump files, they must be collected. It is important not to reboot the appliance before generating the core dump for analysis.

To collect these files, SycSec recommends following the instructions published by Citrix:

• Verify that there is more than 5 GB of free disk space on the NetScaler appliance.

• Create an NSPPE core dump file on the NetScaler.

• Collect the NSPPE core dump file from the NetScaler. Copy the core dump file from /var/core/<N>, where N is the folder with the highest number. The dump file will start with "NSPPE-".

In the core files of appliances exploiting CVE-2023-4966, there are human-readable strings exceeding 120,000 characters, padded with more than 99% "0" characters. This string is part of the response object that exposes memory areas. If the FQDN of the appliance in the response string is a long sequence of characters instead of a legitimate FQDN, searching for strings contained in legitimate responses to vulnerable endpoints might also indicate exploitation. The figure below shows a line of code used to print all readable strings in a core file along with their lengths and additional processing.

Commands used to analyse strings in the NSPPE core dump

The figure below shows the server's expected response to an OAuth configuration endpoint. The placeholder values will be replaced on the server side with the value of the HTTP Host header.

Typical server response to a configuration endpoint

The figure below shows a truncated sample of evidence of an exploitation attempt obtained from a core dump file. In this case, the attacker sent a request with the HTTP Host header set to 24,799 "0" characters.

Example of Server Response After an Attack Request

Important Note: The above evidence does not guarantee successful exploitation. Evidence of exploitation attempts can be found in core dump files generated on newer NetScaler versions that are not vulnerable to CVE-2023-4966. However, this data alone cannot distinguish between failed and successful exploitation attempts.

▌Post-Exploitation Activity

Following successful exploitation of CVE-2023-4966, we observed a variety of post-exploitation tactics, techniques, and procedures (TTPs). Once attackers are able to successfully hijack a session, they perform actions including host and network reconnaissance of the victim's environment, credential harvesting, and lateral movement via RDP. Among these, the use of non-native binaries (such as net.exe) was identified. Additionally, we observed the use of SoftPerfect Network Scanner (netscan.exe) to perform internal network enumeration.

Example of AD reconnaissance commands executed by the attacker

In some instances, attackers used 7-zip to create encrypted, split archives to compress reconnaissance results. The attackers then used the built-in certutil utility to Base64 encode the segments.

Example commands used by attackers to compress and encode output files

In one case, certutil was used to decode multiple files associated with credential theft. We also observed attackers using e.exe to load d.dll into the lsass process memory space. When run, this utility creates a memory dump file located at %temp%\1.png and prints "success" to the console upon completion. Attackers can process the memory dump file offline to extract credentials. Meanwhile, sh3.exe is a utility suspected of running Mimikatz LSADUMP commands.

Example commands to decode the credential harvester

In another example, an attacker used certutil to decode a file we identified as a newly tracked backdoor that uses Slack for command and control. Known in the industry as FREEFIRE, it is a lightweight backdoor written for .NET. FREEFIRE communicates with hardcoded channels to retrieve commands and upload responses. It supports loading arbitrary .NET assemblies encoded as Base64 sent to it via chat comments. We observed the attacker deploying FREEFIRE using the following certutil command.

certutil command used to decode FREEFIRE

We also observed the deployment of various Remote Monitoring and Management (RMM) tools following successful exploitation of CVE-2023-4966. Deployments of Atera, AnyDesk, and SplashTop have been observed to establish and maintain a foothold after exploiting CVE-2023-4966.

▌Continuous Tracking

SycSec is currently tracking the attacks and threat groups exploiting this vulnerability. We have found that these threat groups exhibit low similarity in the post-exploitation phase, such as using the same reconnaissance commands and utilities available on Windows. Common tools include:

• csvde.exe

• certutil.exe

• local.exe

• nbtscan.exe

Some threat groups use Mimikatz to dump process memory.

▌General Detection Rules (YARA Rules)

We have also identified some general detection rules in the open-source community for reference.

▌MITRE ATT&CK Technique Mapping

Initial Access

T1133: External Remote Services

T1190: Exploit Public-Facing Application

Execution

T1059.001: PowerShell

T1059.003: Windows Command Shell

T1059.006: Python

T1569.002: Service Execution

Defense Evasion

T1027: Obfuscated Files or Information

T1027.002: Software Packing

T1055: Process Injection

T1070: Indicator Removal

T1070.004: File Deletion

T1070.006: Timestomp

T1112: Modify Registry

T1140: Deobfuscate/Decode Files or Information

T1620: Reflective Code Loading

T1036.008: Masquerade File Type

T1202: Indirect Command Execution

T1218.011: Rundll32

Credential Access

T1003.001: LSASS Memory

T1003.003: NTDS

T1555.003: Credentials from Web Browsers

T1555.004: Windows Credential Manager

Collection

T1115: Clipboard Data

T1213: Data from Information Repositories

T1560: Archive Collected Data

T1560.001: Archive via Utility

Discovery

T1007: System Service Discovery

T1012: Query Registry

T1016: System Network Configuration Discovery

T1016.001: Internet Connection Discovery

T1033: System Owner/User Discovery

T1069.001: Local Groups

T1069.002: Domain Groups

T1082: System Information Discovery

T1083: File and Directory Discovery

T1087: Account Discovery

T1087.001: Local Accounts

T1087.002: Domain Accounts

T1135: Network Share Discovery

T1482: Domain Trust Discovery

T1614.001: System Language Discovery

Persistence

T1543: Create or Modify System Process

T1543.003: Windows Service

Command and Control

T1071.001: Web Protocols

T1095: Non-Application Layer Protocol

T1071.004: DNS

T1102: Web Service

Impact

T1489: Service Stop

Lateral Movement

T1021.001: Remote Desktop Protocol

T1563: Remote Service Session Hijacking

▌Citrix Vendor Security Bulletin and Mitigation Recommendations

https://support.citrix.com/article/CTX579459/netscaler-adc-and-netscaler-gateway-security-bulletin-for-cve20234966-and-cve20234967

To ensure that the measures you take achieve the desired effect, you can search for the keywords "LockBit" and "Citrix" in the SycSec Security Metric Validation Platform to obtain relevant attack simulation validation actions and related scenarios. This allows you to verify whether your security defence system can effectively counter ransomware attack methods, while ensuring harmlessness during validation through an industry-exclusive approach.