In his fourth blog in our series on cybersecurity, Steve Matheson, Product Manager for BridgeHead’s RAPid™ Data Protection solutions, discusses a recent meeting with a customer where they undertook post-event analysis following a cyber event with some interesting insights, conclusions, lessons learned, and follow up activities. He also highlights two new cyber threats that have gained traction and offers some advice on how hospitals can prepare by staying on top of their recovery architecture.
Recovery after a successful cyberattack
I was asked by a colleague to sit in on a cyber recovery wrap-up meeting with the IT team at one of our partner hospitals. The facility’s onsite digital systems environments had been breached at numerous points in the infrastructure by a successful cyberattack. An important premise of this meeting was to evaluate our collective teams’ efforts in facilitating the recovery from the significant ransomware event – one that had an hourly negative impact to every aspect of the hospital’s operations. The second premise was to review the details of the specific breaches and discuss how, in the future, these points of vulnerability would be either reduced or eliminated.
At the end of a very lengthy discussion, I concluded that both teams did everything right as it related to the planning for, and the process of recovery from, a significant cyberattack-created outage. The general feeling was that, while the plan didn’t account for every possible data loss contingency, it had worked as intended. With such close collaboration during the cyber event, the two teams had achieved a full recovery with limited digital system downtime. You could tell everyone was proud that they had designed a recovery architecture that worked.
However, we collectively identified areas for future improvement. This led to us carrying out extensive scenario planning for the recovery of each significant digital system. We redesigned the recovery architecture – providing more frequent and safer restore points. And, finally, we implemented a few schedule changes in the creation of those restore points. All-in-all, a very productive meeting that led to immediate activities to enhance the customer’s data protection and recovery.
Is a cyberattack a finite event?
Obviously, the stress of the cyberattack, and the days of working ‘around the clock’ during the outage; all contributed to a state of fatigue among both teams. The customer was just ‘glad it was over’ and delighted that hospital operations had resumed. I sensed that, with the cyberattack defeated, there was a view that it was a ‘finite event’. Thus, once each affected system was restored back to some earlier point-in-time, and end users were able to login in and transition from the manual workflows to their everyday digital workflows, that was a wrap!
What we hadn’t acknowledged was the elephant in the room. The next successful cyberattack will likely come in through different attack vectors and affect different attack surfaces. If we were to have the same success in recovery as we had in this case, we needed to establish an ongoing process of reviewing current cyber threats and evolving the recovery architecture to provide recovery points for those attacks.
As cyberattacks evolve, so too must recovery architectures
The sad news is that successful cyberattacks are not ‘one and done’. Unfortunately, for the foreseeable future, these attacks will be the precipitator of the majority of recovery events hospitals will have to contend with. The processes and mechanics for their recovery must evolve as the methods of cyberattack evolve – with rising levels of sophistication. Many IT folks believe that, once they have a successful recovery architecture in place, that’s it – they have it covered. This is a dangerous assertion. It’s absolutely the case that parts of the plan can be re-used, but I would advise caution here. Don’t fall into the trap of assuming your plan will continue to be effective without continual monitoring and adaptation to the ever-changing cyberattack landscape.
New cyberattack targets: firmware
One specific cyberattack surface method that has been gaining attention is the focus on firmware within digital systems. In case your reading list hasn’t recently included detailed articles of how computer hardware interfaces to operating systems, let me give you a quick synopsis of the firmware issue and why I think it changes your recovery architecture. (If you want more detailed information, there is an informative article on the absence of firmware security titled, “The Missing Middle – Addressing the Absence of Firmware Security” written by Michael Sugden at FDD.org.
In the article, Michael defines firmware as “the code embedded into a device’s hardware” which he states “functions as a bridge between the device’s software and hardware”. Essentially, firmware directs the hardware how to operate. And, of course, firmware is ubiquitous – it’s on your laptop, your server, your storage arrays and, not to put too fine a point on it, but also many of your medical monitoring devices.
The security issue is straightforward – it’s hard to detect malware has been planted within firmware and, harder still, to remove it. Consequently, as malware is difficult to remove from the hardware layer, it wouldn’t matter if you do a backup every hour or a storage snapshot each minute – it will do you no good if you don’t have clean hardware to restore them to. This requires a recovery architecture that compensates for hardware loss.
New attack targets: ‘living-off-the-land’ binaries (LOLBins)
A second widely documented cyberattack vector is facilitated by utilizing ‘living-off-the-land’ binaries (LOLBins), which are legitimate commands and pre-installed executables of the operating system commonly used by organizations such as Microsoft. However, cybercriminals are increasingly using LOLBins to perform malicious activities. A common LOLBin is Cerutil – a command-line utility used to display certification authority (CA) configuration information, configure Certificate Services, and back up and restore CA components, is being used to manipulate certificate services on Windows machines. The method details are too voluminous for me to delve into here but, if you ‘Google it’, you will have a week’s reading ahead of you!
So, if it’s well documented, and there are products out there that can discover LOLBin manipulation, why does it require a change in the recovery architecture? Well, you know those digital systems that you have kept in the corner rack in the server room? Yes, the systems that you claim are only used for legacy data access in order to meet your regulatory and legal compliance obligations. If you are not going to retire the data and decommission those systems, you must include them in your recovery architecture. Because these systems run on much older versions of hardware (often with out-of-date firmware) as well as using legacy (often unsupported) versions of Microsoft Windows, they are more vulnerable to attack. If you are going to keep them running in your environment, you must have a recovery architecture that can restore them should you suffer a cyber event.
There are no one-size fits all recovery architectures
Recovery, in the age of cybercrime, is tough. And it’s only going to get tougher! The methods of attack will continue to evolve and your recovery architecture must evolve in parallel. Old attack methods may not work for newer software and hardware, but they are still viable for your legacy systems. With every new, clever feature release of operating system software and infrastructure hardware, new vulnerabilities for cyberattack are discovered. If you think about your recovery architecture as something dynamic – one that you can continually tweak in order to adjust to new threats – you’ll significantly improve your resilience to cybercrime. And, of course, if you’d like to explore how we can help you navigate this proverbial minefield, feel free to reach out to us. Given our large install base, we’ve seen much of ‘what works’ and ‘what doesn’t’ and would be happy to share our insights.
Steve Matheson is BridgeHead Software’s Product Manager for its RAPid™ Data Protection solutions.
Steve has previously held leadership roles in high profile organizations focused on data management and data protection, with global experience covering both hardware and software. These include Vice President of Channel Sales for CommVault, Vice President of Sales at Cambridge Computer Systems, and Senior Director of Channel Sales at EMC.
To learn how your healthcare organization can enhance its recovery architecture to insulate you from current and future cyberattacks…
