1

[ EXECUTIVE SUMMARY ]

INTRODUCTION Managing enterprise chromatography data systems is even more challenging in today’s unprecedented times. According to McKinsey & Company, Chief information-security officers must balance two priorities to respond to the COVID-19 pandemic:

■ Protecting against new cyberthreats and

■ Maintaining business continuity1

Empower™ CDS is a key part of many labs’ operations, and disruption to generating chromatography data is a real concern in 2020 and beyond. In a recent Waters webinar on disaster recovery and business continuity, the audience indicated that the top 2 concerns when it comes to extended disruptions in the lab were: 1. Down server/computer, 2. Cyberattack (figure 1). In this executive summary and companion webinar, we discuss disaster recovery and business continuity topics that address these top 2 concerns.

THE MODERN LABORATORY

Disaster recovery and business continuity have been very active topics of late due to the changing nature of the types of business interruptions companies are experiencing.

In the past when hard drive failures, power failures and other infrastructure problems arose, they failed in the local data center where one could plan for and control many of the risks. With modern design, where companies build regional and global environments using scaling and redundant hardware, many of those smaller scale interruptions have been eliminated or at least severely reduced.

A side effect of this change is that while business is less likely to be interrupted by minor hardware failures, when there are interruptions, the events are larger and last for a longer period.

Natural Disasters and malicious attacks have the potential to impact an entire site and all its infrastructure. The threats are very different than they were just a few years ago as seen in figure 2.

Disaster Recovery and Business Continuity for Durable Data

Shirley WongAmericas Informatics Field Marketing ManagerWaters Corporation

Figure 1. Poll results from Empower Your Lab Webinar Part 3 shows Down server/computer and Cyber attacks as top 2 distruptions organizations are most concerned about.

Figure 2. Slides presented during webinar on Cyberattacks and climate disasters on the rise.

Tim BaraPrincipal Informatics Consultant, Americas Professional ServicesWaters Corporation

Larry MugaveroPrinciple Informatics Architect, Americas Professional ServicesWaters Corporation

https://event.on24.com/wcc/r/2540983/D91E722E2567D511C6203577DCA52F40https://event.on24.com/wcc/r/2540983/D91E722E2567D511C6203577DCA52F40

[ EXECUTIVE SUMMARY ]

2

WHAT’S IN A BACKUP When we talk about data backup there can be some confusion because there are a couple of terms that get used interchangeably. One definition of the term backup is what we consider a long-term archive of the data (How does FDA use the term “backup” in § 211.68(b))2. In Empower, this is when you backup a project and remove it from the active data environment. These get stored in secure long-term storage and are a true copy that can be restored to active use if needed. These are important and often cited in FDA warning letters when they go missing3,4.

For now, we are focusing on system backups. These are a point in time copy of full sections of an environment. For example, a backup of the raw data share or application drives. These backups are used to recover the system from an outage. While they are vital to restoring a full system after a crash or corruption, they are not designed to provide access to specific data for the longterm. Further discussion on this topic of backup vs archive and why you need both can be found in this white paper5.

WHAT’S IN A DISASTER RECOVERY PLAN?The keystone of disaster recovery plan starts with good high-quality backups of all the primary components of the Empower CDS. That would include the Empower database and all associated raw data files. Keeping three weeks of the weekly full and daily incremental backups of those two components is critical to maintaining the integrity of the system in the event of a corruption or an outright failure in key components. The near-line storage device should only be engaged with the active corporate network as needed and the backup set or the backup device should be encrypted to protect the integrity of the backup.

As a best practice, the backup and restore process of the fully functional system should be documented and tested before moving into production and retested at least annually. Testing is the only way to ensure all the moving parts are working together properly, and retesting ensures that no changes occurred over time as personnel change and new backup software is introduced into the equation.

Working hand in hand with a good disaster recovery solution is a good high availability (HA) solution. Here are some examples of deployment architectures that create a highly available environment. In figure 3 in the shaded blue area is the primary data center that contains the working production environment. The secondary or standby data center is shaded in pink, which hosts the near real-time replica of the primary production environment. Between the two we have options that help create and manage the replication between the two data centers. Options such as these can help reduce the likelihood of a single point of failure in the infrastructure. These HA options can be deployed with an on-premise installation and all provide redundancy of Empower architecture components across data centers and regions. They allow the primary site to failover to the secondary site quickly to maintain the integrity of the Empower application and its use by the lab sites connected to it.

There are also cloud specific services that will manage and maintain the infrastructure replication when deployed with a cloud vendor. Deploying Empower into the cloud can take advantage of inherent infrastructure redundancies offered by the cloud vendors. The secondary site could be maintained at an at-risk location, should the site be isolated from corporate communications due to a disaster scenario.

Figure 3. High Availability (HA) Deployment Configuration Options

https://www.waters.com/waters/library.htm?cid=513068&lid=134872580

[ EXECUTIVE SUMMARY ]

3

WHAT IS IN A BUSINESS CONTINUITY PLAN?While the disaster recovery plan is a technology document that leverages some processes, the business continuity plan is more of a process document that leverages some technologies.

Some key questions to address:

■ What temporary procedures will you need if you are using a system that is local to the lab?

■ How will a secondary reviewer access that data?

■ If you must shift to an outside lab, how will you provide them with testing information they will require?

■ Once the emergency is complete how is data analysis finalized and how will you secure the data from that alternate system?

When we look at these three incident recovery tools: system backup, disaster recovery plans and business continuity plans, you can see in figure 4 how they interrelate and build on each other. The backup and restore files and technologies are critical for getting the system back in a working state with minimal impact to data. The disaster recovery plan will use those backups and other resources to help recover from the interruption. Finally, the business continuity plan details how you can sustain your business and collect, process and report reliable and accurate testing data during the interruption of your systems.

NEW SOLUTIONSDespite the best efforts to create a sound disaster recovery plan, a complete production site isolation due to natural disasters or malware attacks, or both, could render the lab idle.

In this case, we now can control and collect data from key instrumentation with the LAC/E™ without having to be connected back to the data center. That NEW capability is called the Waters Business Continuity LAC/E with SecureSync™ or BC LAC/E. The BC LAC/E creates a functional unit, at the laboratory bench level, providing for operational granularity. This allows for a controlled and audited switchover from regular LAC/E mode to BC Mode for as long as required until the site can re-establish connection back to the corporate network

after the disaster.

The BC LAC/E with SecureSync™ replicates specified information needed to operate the BC LAC/E for extended periods of time without a network connection. Waters SecureSync™ takes care of the administrative overhead needed to replicate appropriate information from the Enterprise to the BC LAC/E. Among many other things, it manages project template replication with up-to-date methods allowing you to run autonomously with your validated methods while in BC mode.

EXPERIMENTAL (H1)

LC conditions (H3 Sentence case) Experimental Text: data

Column: data

Column temp.: data

Sample temp.: data

Injection volume: data

Flow rate: data

Mobile phase A: data

Mobile phase B: data

Gradient: data

MS conditions (H3 Sentence case)LC system: data

Column: data

Column temp.: data

Sample temp.: data

Injection volume: data

Flow rate: data

Mobile phase A: data

Mobile phase B: data

Gradient:

Time %A %B Curve0.0 0.0 0.0 0.00.0 0.0 0.0 0.00.0 0.0 0.0 0.00.0 0.0 0.0 0.0

Figure 4. These pieces go together as tools that help businesses recover from both natural and man-made disasterss

BC LAC/E Overview

[BC LAC/E switched from

standard LACE/E]

■ Same LAC/E is used in both Enterprise mode and Business Continuity mode

■ SecureSync™ streamlines your readiness by copying relevant information from the Empower Enterprise database server to the BC LAC/E

Oracle Database

ServerBC LAC/E with SecureSync™

Enterprise Production System

Waters Corporation 34 Maple Street Milford, MA 01757 U.S.A. T: 1 508 478 2000 F: 1 508 872 1990 www.waters.com

[ EXECUTIVE SUMMARY ]

Waters, The Science of What’s Possible, SecureSync, and Empower are trademarks of Waters Corporation. All other trademarks are the property of their respective owners.

©2020 Waters Corporation. Produced in the U.S.A. December 2020 LM-PDF

PLANNING THE BEST SOLUTION FOR YOUR SITUATIONThere is no one size fits all answer, only the best option for your situation.

Small enterprise labs, such as a single building lab with only a few users, may be successful with a minimal approach. A sound backup strategy including backups that are stored off-site, instruments and Business Continuity LAC/Es, paired with a basic disaster recovery and business continuity plan may be all that is needed.

Larger multi-site implementations are where options can become staggering. If the company has the in-house expertise and technology, they can plan for very robust and highly available systems. Or by implementing a cloud-based solution they can rely on the vendor to provide that “Always On” capability. Yet, it’s still a big issue if a site cannot access the network. They will need access to the local Businss Continuity LAC/e to provide testing recourses while communication is being re-established. The disaster recovery and business continuity plans are often global policies that require each site to define how they will implement the global strategy depending on their business needs.

HOW CAN WATERS HELP?Waters can work with your team, providing guidance on technologies and process, and help identify pitfalls. If you already have a strategy designed but have not had to test it in a real disaster, it can be helpful to bring Waters in to review the plan with an outsider perspective to identify and mitigate any gaps.

Lastly, during an actual event, having your vendor available while recovering your system may be invaluable. If the circumstances fall outside of the situations you have planned for, Waters may be able to help determine a new strategy to get you back up and functional.

WHERE CAN I FIND MORE INFORMATION?■ Disaster Recovery and Business Continuity for Durable Data Webinar.

■ www.waters.com/empoweryourbusiness

■ Please send any questions to Shirley_Wong@waters.com

References1. Cybersecurity’s dual mission during the coronavirus crisis: https://www.mckinsey.com/business-functions/risk/our-INSIGHTs/

cybersecuritys-dual-mission-during-the-coronavirus-crisis?cid=other-eml-alt-mip-mck&hdpid=37ca2ac9-1fd2-48e6-8096-0b133f41946e&hctky=12292244&hlkid=fbd8a00ab58d43209aab906926d53fc0#

2. Data Integrity and Compliance With Drug CGMP Questions and Answers Guidance for Industry: https://www.fda.gov/media/119267/download

3. U.S. Food and Drug Administration. [Online] Accessed September 2020: https://www.fda.gov/files/drugs/published/Compliance-Record-483--Dr.-Reddy’s-Laboratories-Ltd.--March-9--2018.pdf

4. U.S. Food and Drug Administration. Warning Letter. [Online] May 27, 2015: http://www.fda.gov/ICECI/EnforcementActions/WarningLetters/2015/ucm448433.htm

5. Backup vs Archive: https://www.waters.com/waters/library.htm?cid=513068&lid=134872580

6. Cybersecurity Ventures Ransomware Damage Report: https://cybersecurityventures.com/global-ransomware-damage-costs-predicted-to-reach-20-billion-usd-by-2021/

http://www.waters.comhttps://event.on24.com/wcc/r/2540983/D91E722E2567D511C6203577DCA52F40https://www.mckinsey.com/business-functions/risk/our-insights/cybersecuritys-dual-mission-during-the-coronavirus-crisis?cid=other-eml-alt-mip-mck&hdpid=37ca2ac9-1fd2-48e6-8096-0b133f41946e&hctky=12292244&hlkid=fbd8a00ab58d43209aab906926d53fc0# https://www.mckinsey.com/business-functions/risk/our-insights/cybersecuritys-dual-mission-during-the-coronavirus-crisis?cid=other-eml-alt-mip-mck&hdpid=37ca2ac9-1fd2-48e6-8096-0b133f41946e&hctky=12292244&hlkid=fbd8a00ab58d43209aab906926d53fc0# https://www.mckinsey.com/business-functions/risk/our-insights/cybersecuritys-dual-mission-during-the-coronavirus-crisis?cid=other-eml-alt-mip-mck&hdpid=37ca2ac9-1fd2-48e6-8096-0b133f41946e&hctky=12292244&hlkid=fbd8a00ab58d43209aab906926d53fc0# https://www.fda.gov/media/119267/downloadhttps://www.fda.gov/files/drugs/published/Compliance-Record-483--Dr.-Reddy's-Laboratories-Ltd.--March-9--2018.pdfhttps://www.fda.gov/files/drugs/published/Compliance-Record-483--Dr.-Reddy's-Laboratories-Ltd.--March-9--2018.pdfhttp://www.fda.gov/ICECI/EnforcementActions/WarningLetters/2015/ucm448433.htmhttp://www.fda.gov/ICECI/EnforcementActions/WarningLetters/2015/ucm448433.htmhttps://www.waters.com/waters/library.htm?cid=513068&lid=134872580https://cybersecurityventures.com/global-ransomware-damage-costs-predicted-to-reach-20-billion-usd-by-2021/https://cybersecurityventures.com/global-ransomware-damage-costs-predicted-to-reach-20-billion-usd-by-2021/

Figure1-Poll: Poll Large: ClosePollButton: Figure 3 Original: Figure3 Large: Closebox: Figure4: BCP Large: CloseBox for BCP: