It started out as a normal Thursday for Tony Mendoza, senior IT director at Spectra Logic, a data storage company based in Boulder, Colorado. And then the ransomware attack began.
“We got some notifications of some system failings and it quickly turned into a lot of unrelated systems failing, which is really abnormal,” says Mendoza. He realised that the company was under attack – and that its files were being encrypted.
“When it hit, we ran to our server room and data centre and started pulling plugs out so it couldn’t propagate itself – which brought our entire infrastructure down,” he says.
In total, three-quarters of the production environment was compromised with ransomware. The hackers left a ransom note demanding a payment of $3.6 million in bitcoin in exchange for the decryption key.
“Figuring out what it was was fairly simple, because they tell you who they are, and they tell you where to send the money. It was NetWalker because it said so in the ransomware letter,” explains Mendoza.
Another problem: the attack came in May 2020, when many employees had just started to work remotely because of the COVID-19 outbreak, so there was no way of easily communicating what was going on outside the building.
Despite that, the IT team had to assess the damage that had been done and what the options were for getting data back – if it was going to be possible at all. There was some hope – the company had backups, which were separate from the rest of the network and safe from the incident.
“We’re still under attack, we’re still trying to stop the bleeding, we still don’t know what the extent of the damage was – but we knew we had data to work with,” says Mendoza.
Every organization that falls victim to a ransomware attack ultimately has to face one major question – do they they give in to the ransom demand in order to retrieve their data?
Cybersecurity companies and law enforcement agencies around the world argue against giving into extortion surrounding ransomware attacks, because not only does it hand over hundreds of thousands or even millions of dollars in bitcoin to criminals, it proves that the attacks work, which encourages ransomware attackers to continue with campaigns.
However, some victims feel as if they’ve got no choice and they’ll pay the ransom, perceiving it to be the quickest and easiest way to get their data returned and the network back up and running – although that isn’t without issues. There are instances where attackers have either taken the money and ran, or taken the ransom then just returned with a second attack.
Spectra Logic had cyber insurance, which could potentially have covered the cost of paying the ransom. That might have been the simpler short-term decision for restoring the network, but it was quickly decided that with the backups still available, Spectra Logic wouldn’t give in to the ransom demand.
So instead of communicating with the cyber criminals at all, Mendoza contacted the FBI.
“I went from being in a panic to being reassured by them that they’d seen it before, we’re not alone in this and they’re going to put tools in place to start protecting us. That was the biggest thing, getting protected,” he explained.
The FBI also assigned a specialist team to help Spectra Logic deal with the immediate fallout from the attack over the course of the days that followed.
Attempting to restore the network turned out to be a 24/7 job for the small team over the course of the following week. For much of that time, people were sleeping at the office in order to have the most time possible to focus on restoring the network.
“From the Thursday morning, we spent 24 hours everyday for the next five days working on this – we slept in shifts. Three of us would work through the night while two people slept for a few hours,” said Mendoza.
“There was no leaving and coming back, it was go sleep on the couch in case we need you. It was five days of all hands on deck.”
As well as this, he was having to provide the board with updates on the ongoing situation. They wanted answers about when the network was going to be restored and when business was going to be back to normal.
“I’m dealing with leadership in the company and I don’t want to lie to them and say I know when it’ll be up – I had to tell them I don’t know what’s going on or when systems will be up,” he says.
It took days of working around the clock but eventually the IT department, with the aid of cybersecurity specialists, was able to restore some functionality to the network a week after the ransomware attack, without paying out to the attackers.
“Our cybersecurity team provided us with the expertise and tools, monitoring and logging to get the threat out of our system. Monday morning they give us a green light; it’s done, they’ve stopped it and removed it,” Mendoza remembers.
“The FBI told us we’re going the hard way, but the right way – and it ended up being the easy way when we came back and said we were back up eight days later; it was shocking for them,” he added.
But it didn’t mean everything was immediately back to normal – it took weeks more to bring back systems that weren’t critical to the business, and during that whole time careful attention was required just to make sure the attackers hadn’t somehow managed to spread the ransomware again, which meant constantly monitoring all activity on the network for another month.
A lot of ransomware attacks never become public knowledge, and examples of companies that go into detail about what happened are still few and far between.
But Mendoza says it’s important to be transparent about dealing with a ransomware attack, because it’s important to show that it is possible to recover from an attack without lining the pockets of cyber criminals.
“What we realised was we protected our data and there’s a way to thwart ransomware. We couldn’t find public information when we were looking for it, so we wanted to make it a common thing, that it’s okay to talk about being impacted by ransomware,” he said.
So what is the key lesson Mendoza would say that other organisations need to take away from Spectra Logic’s experience? It’s backup your systems – and do so offline – so, if the worst happens and the organisation falls, you still have backups offline.
“You’ve got to limit your attack blast radius. Backup your data in multiple locations on multiple mediums and the key is to air-gap it. Whether it’s physical air-gap or virtual air-gap, you’ve got to put a wall between an attack and your data,” he said.
And how did the company end up falling victim to a ransomware attack in the first place? Analysis of the incident revealed a phishing email sent to an employee working from home was how hackers gained their initial access to the network.
In the aftermath of the ransomware attack, Spectra Logic has worked to improve its cybersecurity culture, both on-site and for remote workers in an effort to learn from the incident. The company is now actively looking for potential cybersecurity threats that might have been missed before.
“Initially after the attack, when the wounds were fresh, we talked about security. Six months later, we’re still concerned about security and we’re more aware of phishing attacks. We were kind of complacent before,” he says: now staff will notify him if a phishing email isn’t picked up by the malware system. “There’s more awareness now.”
At Raptor IT Consultants, our goal is establishing a foundation for your business network that empowers its users to work efficiently, while leveraging technologies that save time and money, and offering scalable IT solutions that work with any business model. #raptoritnetwork