Back to Insights

Vulnerabilities in Citrix/Netscaler appliances exploited actively

January 18, 2020 at 15:19

Note: This blog relates to on-going incidents. Facts, recommendations and impact may change as more information becomes available.

UPDATE 24.1.2020: 

incidents

Vulnerabilities found in Citrix Gateway, Application Delivery Controller and Citrix SD-WAN WANOP appliances are being exploited in the wild and Nixu has now seen multiple cases in the Nordics where attackers are exploiting the flaw. These devices were previously known under the NetScaler brand. The exploit allows unauthenticated attackers to carry out code execution on the device – essentially giving anyone access to the device and potentially the network behind it. The vulnerability has the CVE number CVE-2019-19781. (https://nvd.nist.gov/vuln/detail/CVE-2019-19781)

The vulnerability is based on path-traversal attack on servers that are exposed to the internet. Attackers have been proven to be able to run commands and download malicious software on customer servers. A public proof-of-concept exploit is available, greatly lowering the threshold for criminals to exploit this vulnerability.

The vulnerability is critical and is currently being exploited with automated scripts. The mitigations provided by Citrix may have also been compromised, and can not be relied on to completely protect a vulnerable server. Citrix has promised a patch for the server software, but it has not yet been released. You can follow the release schedule for your version of the server from Citrix's support page: https://support.citrix.com/article/CTX267027

Citrix has released a write-up of the incident with instructions and a list of affected devices: https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

Any organization with these appliances should perform the recommended mitigation actions immediately to prevent the devices from being compromised. As these devices are often deployed to the borders of networks, they can act as a way to attack other services and devices behind them.

These types of incidents happen regularly in today’s connected world. Any organization should be prepared to react quickly and decisively. To ensure adequate reaction times and capabilities, the following activities are recommended:

Keep up to date asset inventory of all devices and software being used

  • When vulnerabilities are released, knowing whether your organization is affected and which assets need patches or mitigations is critical

Follow vulnerability notices from vendors or utilize a service that does it for you

  • For larger enterprises, manually following the notices is often infeasible as the number of assets and vendors grows
  • Ensure that a process exists for quickly applying patches or mitigations outside of normal patching windows when the situation calls for it
  • Test the processes regularly to ensure that they can be utilized during a real crisis

Regularly patch all assets, whether software or hardware appliances

  • Ensuring that all assets have up to date patches is generally the best defense against attacks. In most cases, patches are already available or become available within hours after attack becomes known

Ensure that you have in-house capabilities or a partner to support with incident response

  • If mitigations aren’t done in time or the organization is hit very early on, simply patching assets without addressing the on-going incident that may have escalated beyond the affected assets is no longer effective

Attack mitigations

To further complicate things, several of the NetScaler ADC servers are running business-critical functions and can not be turned off to wait for a patch to be published. If an instance of NetScaler ADC is *not* business critical, Nixu suggests that volatile data should be collected from the machine following the instructions on Evidence Preservation. After collecting evidence and storing it in a secure location, the appliance should be paused until a patch is released and applied. It is also important to take a snapshot of the running machine before pausing it to avoid losing evidence in case of an error. It is not advisable to power down the appliance as parts of the file system are held in memory during runtime.

The following mitigations can be applied if turning off the server is not a possibility:

  • Apply recommended mitigations for different device types as listed on Citrix KB article: https://support.citrix.com/article/CTX267679
  • Limiting access to the server using all available methods, for example whitelisting IP addresses, VPN or GeoIP blocking.
  • Limiting access from the NetScaler ADC server to other servers.
  • Revoking all certificates stored on the server. The attacker user level ‘nobody’ has read rights to private encryption keys.
  • Revoking all passwords stored on the server. The passwords are encrypted with a hard-coded key that is retrievable by the attacker.
  • Restoring the NetScaler ADC Server from a backup to a state earlier than 10.1.2020 when the vulnerability was published. Restoring the server should be done only after other mitigations are in place, otherwise the server can be reinfected.
  • Reinstalling the NetScaler ADC Server.

Before restoring or reinstalling the server, it is advisable to store a snapshot of the current state for further forensic analysis. As a part of the operating system is running on a ramdisk, it is important to store a snapshot that preserves memory contents.

Indicators of compromise

Appearance of the following addresses may be an indicator of a compromised system:

      138[.]68.14.63
      95[.]179.163.186
      185[.]178.45.221
      159[.]69.37.196
      hxxps://pastebin.com/raw/d3SY1erQ
      hxxps://pastebin.com/raw/2zds3h2T
      hxxps://pastebin.com/raw/8xNac8At
      hxxps://pastebin.com/raw/UrJnnijX

Evidence Preservation

Before running any commands, preserve evidence by saving the following files:
      /var/cron/tabs
      /netscaler
      /tmp

Save the process list to a file and save that too:
      ps -axu | grep -i nobody > /var/tmp/processlist.txt

You can easily combine the files to an archive with the following command:
      tar -czf /var/evidence_from_ramdisk.tar.gz /var/cron/tabs /netscaler /tmp /var/tmp/processlist.txt

Look for processes running as nobody.
      ps -axu | grep -i nobody

Check /netscaler/portal/templates directory and look for suspicious XML files.
      ls -lah /netscaler/portal/templates
      cat suspicious.xml

Remember to download the preserved evidence files before analyzing the system further.

Incident Triage

Check bash and sh logs to see what commands have been executed on the system. Note that some of the logs have been rotated and archived.
      grep -i nobody /var/log/bash*
      grep -i nobody /var/log/sh*

Check cronjobs
      crontab -l -u nobody
      cat /var/cron/tabs/nobody

Need help?

Nixu provides services around cyber defense, incident response and vulnerability management. To understand more how we can help with preventing, mitigating and responding to incidents such as this one, see our service listing for cyber defense. https://www.nixu.com/services/cyber-defense

If you suspect that your organization has been affected and you have an active cybersecurity incident on-going, support is available to organizations that are not yet clients. To get started quickly, visit: https://www.nixu.com/service/security-incident

Further information about the vulnerability: