In the wake of CVE-2019-19781 vulnerability…and the lessons learned


Nearly before Christmas 2019, Citrix publicly announced the CVE-2019-19781 vulnerability in Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP appliances that could lead to arbitrary code execution; affecting the majority of the NetScaler products that are under active maintenance.

This development has led to disarrays among Information Technology professionals given that in most IT departments annual leaves were already in effect.

Therefore, Incident Response procedures have been a bit harder to follow. It doesn’t seem so promising to have to deal with this while your leave has just started, right?

The days that followed, the vendor urged customers to apply the provided mitigation steps to stop exploitation attempts until the release of a patch that addresses the vulnerability.

Meanwhile, malicious actors have been lurking around the corner planning their hits, in an attempt to seize the opportunity.

The facts

Α lot of things have happened the period between vulnerability announcement and the official patch release.

From one side, we had those who were in charge of defending the networks and from the other side; those who had bad intentions towards the vulnerable appliances.

In the meantime, the researching community started setting up honeypots to verify the threat level.

It was only a matter of time until the malicious actors started attacking the honeypots successfully; thus, it was then official that the enemy was at the gates!

In the meantime, while Citrix was working on issuing the patches, they partnered with a group of cybersecurity experts.

This partnership led to the release the Indicator of Compromise (IoC) scanner tool which was able to tell whether a vulnerable system has been hacked or not. The script accomplishes that by checking for any modifications on several system files.

Shortly after, Citrix gradually rolled out official patches for the affected appliances, starting from the 19th of January 2020 for 11.1 and 12.0 versions and ending on the 24th of January 2020 with the patch release for 10.5 version.

According to the community feedback, a significant number of susceptible appliances have been attacked; with many ending compromised.

Since it doesn’t make a lot of sense to patch a system that is compromised already, from that point, the only way to move forward was to rebuild those systems from scratch.

Lessons learned

As with every similar incident (remember Ransomware havoc back in 2017), the IT industry becomes a bit wiser both in being proactive as well as reactive in terms of Incident management.

Concerning the first, in-place defences get reevaluated while in relation to the second, Incident Response procedures get improved by adding elements that were previously missing (e.g. distinct assigned roles to IT personnel).

Below follows a piece of advice on how to align with best practices for deployments across every organization, regardless of its size:

  • Segment your network

In today’s hybrid world, companies spread their assets across different locations while, at the same time, the working landscape has also changed rapidly; with trends like telecommuting being on the rise.

This reality adds complex challenges to the IT departments since they now must pay attention not only to their datacenter(s) but also to every location where company data reside and from where users can access corporate resources.

Well, that might be known already but still, how to deal with this?

The answer is, segment your network! Isolate your workloads by creating secure zones and protect each separately; thus, adding security layers to your infrastructure.

Let’s dig a bit more on this using a real-world example:

Presumably, you just got your costly next-gen firewall to protect the company’s network.

At the same time, you ensured to configure it according to best practices. So, you are now protected, right?

Well, indeed you are protected against most of the threats but what happens in case of the vendor, on a later time identifies a significant security flaw, rolls out an urgent patch to address it but you fail to apply in time?

You got it; a breach is around the corner!

Now here comes the real value of this architecture: Your unpatched device may be hacked, compromised or whatsoever but the resources that the perpetrator could access would be significantly less than otherwise.

As a result, the chances that the break-in could lead to a potential compromise of most of the infrastructure parts or even worse of the entire infrastructure, are significantly limited.

  • Monitor your infrastructure

So, you have your services published securely and; you are confident nothing is going to happen. Or at least nearly nothing, besides cyberattacks are on the rise, remember?

However, in case something happens; chances are you would want to be notified as earlier as possible so you can work your remediation actions.

That is why you need to have the right tools in place to detect any abnormal activity throughout your infrastructure. Products based on Machine Learning and Behaviour analysis can be a true ally towards that direction; hence, consider to play around with some of them to get better overall perception and see how well they could fit in your infrastructure.

  • Have an Incident Response procedure in place

Sooner or later, most of the organizations deal with some kind of an incident. Both the strategy and the procedures an organization has set concerning Incident Response; play a vital role both for dealing with the alarming situation as well as for the implications the incident could have on its reputation publicly.

The two key features when dealing with such incidents; would first be to discover the attack as soon as possible and secondly to effectively contain the damage.

The two will make integrity restoration for the entire infrastructure more comfortable and practical.

For the plan to succeed; it is crucial to assign specific roles and duties to the personnel, hold relevant training sessions regularly and last but not least; to ensure to document all the procedures for everyone to be aware of their responsibilities.

And as you might have guessed already, the last one on the list is …

  • Backup, backup, backup!

You got it right; I am referring to this old school practice for which everyone knows a thing or two and often goes unnoticed among IT professionals.

In reality; backups are a few of the last (for most Small Businesses most likely the last) cards an affected organization has to restore its infrastructure partially or even completely.

However, this invaluable option comes with strings attached for the personnel that is in charge of backups since they will have to ensure they are following the principles of a proper backup strategy.

Such a strategy; typically includes periodic checks of the integrity and validity of backups.

Although we will discuss more on this in a future post, it is necessary to mention that as a rule of thumb, you can only feel confident for the ability to restore from backups if you have no less than two copies of the backup sets; if not more.

Correct, that is both onsite as well as at an offsite location!

Since you made it that far; I would like to thank you for your attention.

We will meet again in another blog post but until then; take care!

Leave a Reply