Combined offerings from HCIS and data protection experts keep hospital IT safe
April 2011 marks the 10th anniversary of the business relationship between MEDITECH and BridgeHead Software. BridgeHead specializes in healthcare data and storage management solutions which complement the MEDITECH Health Care Information System (HCIS) with a scalable, future-proof platform to overcome rising data volumes and increasing storage costs while delivering peace of mind around the access, availability and protection of critical electronic patient data.
A Data Protection Solution for MEDITECH’s Home Team
In Spring 2001, MEDITECH’s Phil Polimeno was faced with a problem. As Director of Corporate Technology at MEDITECH, Polimeno is responsible for maintaining the two data centers that house all of the company’s servers, the majority of which contain copies of every MEDITECH customer’s software. These copies are vital, not only for customer support, but also in assisting the software development process by ensuring new versions of the latest releases are fully tested and compatible with a customer’s configuration. Clearly, backing up and protecting these systems is a priority for Polimeno and his team. But, at that time, MEDITECH’s backup provider was failing to protect the huge number of files resident on these systems. Consequently, MEDITECH was unable to meet either its recovery point objective (RPO) or total recovery time objective (RTO) targets.
“Backup drives a lot of decisions. It’s the tail wagging the dog,” said Polimeno. “We had a problem with our incumbent vendor’s file-by-file backup. Just to build the backup index took hours. Backing up Windows files was tough, and the backup software was just choking.”
Polimeno, along with Senior Systems Analyst, Paul MacDonald, and IT Manager and Network Administrator, Tom Parisi, set out to solve the problem.
“With our Client/Server architecture, the file that our then backup vendor created to point to the backup was bigger than the files themselves,” explained Parisi. “Although it was only a 100 gigabyte backup, there were 10 million files. Once the backup product started walking the tree on that 10-million-file structure, it took three days to complete.”
MacDonald, a backup and SAN expert, figured that there had to be other companies with similar problems backing up one million flat files at a time. He identified BridgeHead Software as one such backup vendor that might be able to help.
BridgeHead Software to the Rescue
BridgeHead Software’s Chief Technology Officer, Gareth Griffiths, (along with software specialist Brendan O’Leary) was a key member of the BridgeHead team that set out to solve MEDITECH’s backup challenges 10 years ago. “After we attempted to back up and restore MEDITECH’s files, we appreciated the severity of the problem. It took 24 hours to do a daily backup, and restore took twice that time. We ran lots of tests, modeled the process, but we just couldn’t open and close files fast enough to backup effectively.”
Instead the BridgeHead team recommended a different approach. “We suggested a whole disk backup, as opposed to file by file,” says Griffiths. “Sequentially reading a disk from beginning to end is very fast, and restore is the same speed. The problem is that you can’t do that while data is changing. We needed to devise a way to get a read_only copy of the disk. This was the genesis of what was to become the BridgeHead ISB/IDR [Integrated Serverless Backup/Integrated Disaster Recovery offering for MEDITECH customers].”
MEDITECH transitioned from its previous backup provider to BridgeHead Software over several months in 2001.
“The BridgeHead team was extremely responsive,” said Tom Parisi. “They showed up at our offices so, unusually, we got to know names and faces. Who were we going to call at Microsoft or our previous backup vendor? The MEDITECH / BridgeHead partnership is not a traditional customer-vendor relationship. There was no need for an official training program. Instead, Brendan O’Leary and Gareth Griffiths practically lived at MEDITECH headquarters for four months.”
Parisi continued, “It was a collaborative effort to back up Windows 2000 with both BridgeHead and MEDITECH trying to reach the same goal. For us to see another software company change their code to accommodate our needs – well that’s what we do too, so it was very refreshing to work with BridgeHead.”
“Looking back, the thing I remember most,” said Parisi, “is we were sitting with the BridgeHead tech folks, and they kicked off a file-based backup, and four hours later the progress bar still hadn’t completed counting up the number of objects. So the BridgeHead image backup changed that process. It was all gravy from there. BridgeHead works as advertised, and BridgeHead people do what they say they will do.”
“These days, any backup problems are our own fault. BridgeHead works,” said MacDonald.
A Data Protection Solution for MEDITECH Customers
Just as MEDITECH’s internal IT team had trouble backing up its systems, MEDITECH customers were experiencing a similar problem. The Enterprise Medical Record (EMR) application, with its millions of files and folders, was the most problematic for customers in terms of backup and, more importantly, restore.
“When customers would call up to complain that their backup didn’t work,” said Polimeno, “they’d ask, ‘what do you use?’ and we’d tell them to try BridgeHead Software.”
In fact, while BridgeHead was solving MEDITECH’s internal backup problem, Phil Polimeno introduced them to MEDITECH’s Director of System Technology Development, Mike Belkner, who had been charged with resolving the backup problem for MEDITECH’s Client/Server platform customers.
“When our Client/Server product line was released to customers in 1996, we chose not to provide native backup tools as we had done with MAGIC. Rather, we told customers they could use any product capable of backing up Windows files and folders – since that’s what Client/Server was running on. However, five years later, some customers were starting to experience challenges, especially with their Enterprise Medical Record (EMR) application,” said Belkner. “EMR used a different database structure than the rest of the Client/Server applications, which resulted in the creation of potentially millions of Windows files and folders that needed to be backed up daily. These customers were facing the same problem Phil was having internally – traditional Windows backup products were taking far too long to back up EMR. Given BridgeHead Software’s success in solving Phil’s problem, I felt confident they could work with us to help our customers.”
Belkner continues, “This was also at a time when our storage partners were introducing Storage Area Network (SAN) technology to our customers. We initially certified the EMC Symmetrix, which had some very interesting capabilities around creating Business Continuance Volumes (BCVs), which I knew we could leverage to affect image-based backups of MEDITECH databases. Given that BridgeHead had taken a similar approach in solving Phil’s problem on Direct-Attached Storage (DAS) systems, it didn’t seem like a big leap that they could work with the SAN tools to do the same thing.”
The Integrated Serverless Backup (ISB), offered through MEDITECH and BridgeHead Software, takes advantage of the “snapshot” capabilities (sometimes called clones, snap copies, or flash copies, depending on the technology) of various SANs to affect a static, image-based backup of the MEDITECH database. A dedicated backup server, running BridgeHead’s HT Backup software controls the backup schedule and communicates with MEDITECH to ensure that a quiesced, static image of the database is backed up to the tape library. One benefit of this solution is that the MEDITECH host does not spend CPU cycles backing up or resynching data, thus making multiple daily backups feasible.
Backups exercise a system like nothing else; the amount of I/O is huge. MEDITECH supports a 24×7 industry, so backups don’t take place on production servers. With BridgeHead ISB, there is no I/O or CPU load on the hospital’s server because the backup is offloaded to the BridgeHead server.
“I remember being in a room with a major MEDITECH customer explaining our backup solution. I hadn’t realized that they had been struggling for several days to restore an EMR backup from their incumbent supplier,” says Griffiths. “During a break, I took a call from one of our existing clients who had just completed a successful restore of their EMR system in 90 minutes. I did wonder why the room went rather silent when I explained the call.”
Next came BridgeHead Integrated Disaster Recovery (IDR), which was developed as a way to give better disk-based recoverability with more frequent restore points. It complements BridgeHead ISB by allowing hospitals to make multiple point-in-time copies and keep them close at hand on an array for a short period of time. Where ISB reduces backup and recovery times for traditional tape backup, IDR is an effective tool for reducing both the recovery point objective (RPO) and total recovery time objective (RTO) in many types of disaster recovery and operational recovery scenarios.
Since collaborating with MEDITECH to create BridgeHead ISB a decade ago, BridgeHead continues to work closely with the MEDITECH development team. BridgeHead CTO, Gareth Griffiths, has made in-person visits from his UK development office to MEDITECH’s MA development offices every six weeks since that initial project.
“Whether it’s the next storage array or a specific problem to be solved, BridgeHead Software and MEDITECH work very closely together on an engineer-to-engineer level,” said Griffiths. “Both teams respect the skills of the other side. We are there for the long term. Backup is not a short term business.”
Chris Anschuetz, Vice President System Technology, MEDITECH, added “We have found BridgeHead able to respond with alacrity to our customers’ changing needs. Despite being headquartered in the UK, they are regular visitors to our facilities in Massachusetts. They participate actively with us in development efforts, at a conceptual design level as well as at the more pragmatic level of step-by-step tuning for performance and accuracy. They have made a complete commitment to MEDITECH and to our industry, and it shows in their products and in their flexibility.”
BridgeHead ISB/IDR is available for all MEDITECH platforms (MAGIC, Client/Server and 6.0) and supports all MEDITECH certified storage devices, which currently include dozens of storage arrays from IBM, HP and EMC. As MEDITECH certifies a new storage technology, BridgeHead develops in collaboration with them ensuring ISB/IDR’s continued availability across storage platforms.
And while ISB/IDR may be the premier data protection solution for a MEDITECH environment, BridgeHead is not content to rest on its laurels. It continues to find ways to enhance and complement the existing solution to the benefit of its customers. Take for example the MEDITECH Scanning and Archiving (SCA) application, which generates terabytes of essentially static data in a single year – many times that of a typical MEDITECH application. BridgeHead Software was able to optimize the storage by introducing its FileStore EHR product into the SCA environment. By removing static SCA content from the backup stream and securing it in a multi-media, multi-copy, multi-location archive, BridgeHead is able to ensure that a customer’s RPO and RTO are not compromised by a single application.
“MEDITECH is glad to acknowledge and celebrate our 10 years’ association with BridgeHead Software as providers of backup and archival storage products to our customers,” said Chris Anschuetz. “A relationship lasting 10 years is not something that happens by accident. Both sides have worked hard for this, but the real winner here is our customers. This has been a good partnership for all concerned, and we look forward to many more years.”
