Memory forensics against Citrix ADC

The Nixu Digital Forensics and Incident Response team has received a bunch of cases related to the wide-spread exploitation of the Citrix CVE-2019-19781 vulnerability after the proof-of-concept exploit code was published. Our team started looking into possibilities to perform memory forensics on the specific version of FreeBSD that the virtual appliance uses. Here is a brief description of how we managed to do Netscaler forensics with Volatility.

Background information

• The Citrix NetScaler ADC is based on FreeBSD version 8.4 released in 2005. It's End-of-Life was in 2015.
• Recently, a remote code execution vulnerability CVE-2019-19781 was found in Citrix Netscaler.
• The go-to tool to perform memory forensics is Volatility
• Luckily a few months back in the annual Volatility plugin contest Antoine Brodin implemented FreeBSD support for Volatility

Old customized kernel is a challenge

We needed to create a profile for the specific installation/version we wanted to analyze before we could even try if the FreeBSD plugins would work. Initial thoughts about potential obstacles were:

1. Part of the profile creation requires compiling a kernel module, which in turn requires kernel sources (`/usr/src/sys`) and we were skeptic whether kernel sources exist in Citrix products
2. Citrix has customized the kernel, so creating the profile in a vanilla FreeBSD 8.4 will probably not work
3. The Volatility plugins were written against "modern FreeBSD systems" and much water has flown under the bridge since the retirement of FreeBSD 8.4, so the plugins might not work at all

Getting past compilation errors

The initial step was to create the profile to see if anything worked. As we didn't have a live Citrix at that point, we decided to get the kernel module compiled and debug information extracted from vanilla FreeBSD 8.4, hoping that it would be sufficient. The module wouldn't compile, as FreeBSD 8.4's `/usr/include/sys/filedesc.h` didn't have the `fdescenttbl` and`filedescent` structures at all. We commented those out in the hopes that some of the plugins would still work. It worked and we were able to produce `module.dwarf`. Then we extracted the symbols from the Citrix ADC kernel (`/ns-xx.y-xx.yy.gz`) with `nm` and packaged that data into a ZIP file which would form the profile.

Our first target was the version 12.0-57.24. If you are investigating a different version, you need to copy the kernel, gunzip it and extract the symbols with `nm` as seen in the Makefile. Then ZIP the two files to create the profile and place it into `volatility/plugins/overlays/freebsd/`. Check that Volatility properly detects the profile by running

vol.py --info | grep FreeBSD

The initial test with `freebsd_version` plugin worked with the profile we created:

$ vol.py -f Snapshot1.vmsn --profile FreeBSD-8_4-NETSCALER-12_0-NS64-amd64 freebsd_version

Volatility Foundation Volatility Framework 2.6.1

Version FreeBSD 8.4-NETSCALER-12.0 #0: Fri Apr 13 12:40:52 PDT 2018

Rewriting code based on old versions

Some of the plugins did not work because of the following changed kernels structures:

• `struct pmap`: had no member `pm_ucr3` or `pm_cr3`
• `struct filedesc`: had no member `fd_files`
• `struct cdev`: `si_name` member is a character pointer instead of a character array (this didn't prevent the plugin from working, but the data was incorrect)

We bravely decided to fix these issues by checking out the historical version of the FreeBSD 8.4.0 base from the subversion repository of the project and seeing how these structures looked like back then. With a couple of modifications, we were able to run all the available FreeBSD plugins successfully.

Unfortunately, the available plugins did not include one for listing active network connections. In some of the cases, that was essential information to dig out. So, with the help of previous work by Elyse Bond, we quickly implemented a new plugin `freebsd_tcpconns` for listing active TCP-connections.

Successful results

With some tweaking, we were able to add support of FreeBSD 8.4 to Volatility memory forensics framework. This allowed us to view network connections, processes, and process arguments at the time of taking memory snapshots of compromised Netscaler instances. This snapshot in time gave us some information on whether individual compromised hosts were used for further attacks into customers' networks or not.

The core kernel structures in FreeBSD haven't changed that much in the last 15 years, so memory forensics was possible, and the timing of the Volatility plugin contest and the FreeBSD support was more than perfect. Thus we want to thank Antoine Brodin for the FreeBSD Volatility plugins that were base for our work.

There are still some problems to be solved as we had difficulties with some NetScaler memory images, and we are currently not entirely sure what is causing the issue.

Find our modified version of Volatility here and profile for Netscaler version 12.0-57.24 here.

Authors: Janne Kauhanen, Miika Turkia, Patrik Nisén

Would you like to stay up to date with the new cybersecurity trends? Subscribe to the Nixu newsletter.