2009-10 xAT Tutorial_v5.ppt

8/11/2019 2009-10 xAT Tutorial_v5.ppt

1/33

Protecting Your Investment:

Creating an Effective FAT/SAT/BAT

Procedure

October 21st, 2009

Presented in conjunction with the

Laboratory Robotics Interest Group

New England

Open Source Software Initiative


2/33

Agenda

Introduction

Types of Laboratory Automation Systems

Creating a Project Plan

Factory Acceptance Testing

Site Acceptance Testing

Biological / Chemical Acceptance Testing

Lessons Learned

Summary


3/33

Some Definitions

FAT- Factory Acceptance Testing

SAT- Site Acceptance Testing

BAT- Biological Acceptance Testing

CAT- Chemical Acceptance Testing


4/33

Introduction

Purchasing Laboratory Automation is an investment

What are you buying (workstation or integrated system?)

Adding a new capability or enabling technology?

Not always about efficiency

Properly plan for success

Define your project goals and criteria for success before

engaging any vendor

Design a Test Plan which uses the actual workflow(s) Test the new capability successfully adds value


5/33

Getting Started

Map your process

Build a Project Charter Why automate?

Added value?, Criteria for success?

What can kill the project?

Setup a Project Team and Determine Stakeholders Include an Automation Lead/ Project Manager, Scientific Lead, &

Purchasing Representative

A Facility Lead and Software Lead may be necessary dependingon the complexity of the system

Add Reagent Incubate Add Reagent Read


6/33

Getting Started (II)

Capture User Requirements Specification (URS)

Use requirements to define an objective Request for

Proposal (RFP) process

Select a vendor which..

Is willing to work with you to properly test the system

Is willing to tie milestone payments to project phases

URS RFP Vendor

Selected

Contract


7/33

Workstations

Workstations are standalone pieces of equipment

integrated with a small # of peripheral devices

Typically stacker based and designed for a specific

workflow


8/33

Integrated Systems

Integrated Systems vary in complexity and usuallyinclude 1 or more robotic arms and a number of

peripheral devices

Typically support a number of different workflows and

are controlled with a scheduler package


9/33

Integrated Systems Continued

Integrated systems can be designed to serve a

number of processes

Historically integrated systems have been builtto support:

High Throughput Screening (HTS)

Ultra High Throughput Screening (uHTS)

Compound Storage and Replication

Combinatorial Chemistry

Cell Culture


10/33

Project Plan

In general, the project plan is similar in all cases

and should include:

project goals

the process map(s) Schedule and milestones

Resource plan

Payment terms

Reference to test plan documents (ATPs)


11/33

Project Plan Continued

Consider breaking project into separate milestones with paymentpercentages tied to the completion of these milestones

Ensure that there is a written and clear understanding withvendor for what acceptance testing will be required

Recommend leaving a percentage of total project payment as afinal milestone payment

Identify a clear procedure for change management

Consider applying payment penalties for

Missing key deadlines

Not satisfying system / design requirements


12/33

Example Project Plan Timeline

PO

Accepted

Define

System

Specifications

Build and Test Ship and Install

Kickoff

Meeting

Agreement on

Acceptance

Testing

Define Functional

Design Spec

Define

Assay/Consumables/

Factory

Acceptance

Test

Site Acceptance

Test

Bio/Chem

Acceptance

Test


13/33

Acceptance Test Plan (ATP)

Draft a formal document (ATP) for testing procedures foreach test phase (FAT / SAT / BAT)

ATP should include language which Outlines requirements for testing procedures

Outlines Testing Scripts

Defines criteria for success / failure

Defines when / what / if interventions are acceptable

Roles & responsibilities and more


14/33

Example Assay

Labware

Source Plate/Assay Plate

Stated Protocol Dispense 10l BufferA

Dispense 10l BufferB

Incubate 60 min at Room Temperature

Wash

Read


15/33

Additional Assay Information

Unstated Protocol

Read Barcodes from Source and Assay Plate

Shake for 1min after 2nddispense

Seal plate with clear heat seal before reading Use Barcode for name of resulting reader file


16/33

Acceptance Test Plan (ATP) (II)

Any contract language regarding payment structure,penalties etc. should be in the purchase agreement andnot in the ATP

The ATP can reference milestone payments

Between the Project Plan and/or Purchase Agreement,all parties should have a clear understanding of financial

responsibilities

Request that the FAT and SAT be performed on a wetsystem


17/33

Acceptance Test Plan (ATP) (III)

The ATP document should be finalized and agreed upon

well in advance of the actual FAT

allows the vendor to run through the FAT testing (pre-FAT)

Vendor is responsible for unit testing of Custom and

Standard devices prior to FAT

The Actual FAT should not occur until the vendor has

confirmed that they have gone through and tested the

FAT process (pre-FAT )


18/33

FAT: Workstations and Integrated Systems

FAT will test the purchased equipment using your

materials and protocols at the vendors site

FAT should use all of the equipment on the system Should not be limited to 1 giant all encompassing test

should entail the primary task(s) for which the system was

purchased

FAT should not execute an experiment or protocol whichwill not be used in actual lab


19/33

Workstation FAT

Not all vendors will agree to complete a FAT for a workstationpurchase

Insist upon executing a FAT for any system with customization

If the vendor is unwilling to include a successful FAT as criteria forpayment, offer to pay for application labor to execute the FAT

FAT should include: module level testing

error recovery testing

testing of new interfaces and any data handling that the system willgenerate

safety requirement testing

QC testing

Not all vendors will agree to QC testing, however most will providean exception in the event it does not work in the SAT or BAT


20/33

Integrated System FAT

All modules should be tested If any of the modules will be run in standalone mode confirm that

they will operate in that mode

Module level and system level error recovery should be tested

QC and Safety testing are required

1 sample should be run from start to end so the entireprocess can be monitored, before running multiplesamples

If appropriate, test throughput matches requirements


21/33

Integrated System FAT (II)

Run the most complex process using the maximumexpected run size (regardless of run duration)

Information System testing should be done as part of

FAT

Always check the error recovery option

Review System documentation Confirm the most current layout is provide

Manuals should be provide for standard and custom equipment

Cable layout a bonus, in particular when labeled.


22/33

Navigating through FAT defects

System requirements will define when interventions areacceptable. For example: Day time run semi-automated systemInterventions Acceptable

Fully automated system which runs overnightInterventions NotAcceptable

When FAT runs require interventions refer to the ATP

acceptable to proceed without starting over?

The # of successful continuous days of running shouldbe defined in advance and required for the system topass the FAT


23/33

Navigating through FAT defects (II)

Any exceptions / defects should be documented and

witnessed by both the vendor and customer

Only accept defects to be fixed later when you are

confident they will be resolved easily

It is typically riskier, more expensive and time consuming

to fix problems on the customers site compared to the

vendors site


24/33

Examples of Exception

A non-recoverable error

Communication Errors between hardware and

controlling software


25/33

SAT

SAT should replicate the FAT testing procedures

SAT confirms that the system was not impacted by transport andinstalled successfully in the customer lab

Both customer and vendor representatives should be present for all

SAT testing

Repeat testing for any exceptions that occurred during the FAT confirms defects have been corrected to the customers satisfaction

Document and witness any preexisting or new exceptions that occurduring the SAT

Only accept SAT when all critical system exceptions have beenrectified and re-tested


26/33

To BAT or not to BAT?

A Biological Acceptance Test or Chemical Acceptance Test is a finaltesting phase which

Uses actual experiment materials

Confirms the actual experiment / process can be executed successfullyusing the system

Vendors rarely agree to execute BAT tied to a payment milestone

Numerous variables outside of their control

External factors can impact experiment performance

Include as requirement within RFP

BAT should be applied when

Purchasing a large / complex integrated system

Experiment or process can be repeated manually with good precision


27/33

BAT

Set expectations for BAT before signing the purchase

agreement / contract

A successful BAT will show that the automation does not

negatively impact experiment / process results

Precision and accuracy of data should be equal to or better than

running the experiment manually

Apply financial recourse for un-successful BAT Final milestone


28/33

Lessons Learned

Avoid scheduling major milestone at the end of a

quarter or end of the year.

Time constraints may influence the acceptance of the

FAT/SAT/BAT/CAT testing Financial constraints from users companies

Negotiate service contract quote and associated

services (parts and labor) before contract signoff


29/33

Summary

Understand the process you wish to automate and the

instrumentation you decided to purchase to complete the

task

A FAT, SAT and BAT should be performed on any

integrated system. For workstation a SAT and BAT

should be performed

Work with your team to protect your investment. It may

not be your money but your name is associated with the

purchase


30/33

Acknowledgments

Ted Manley

William Goode

John Davies

Brian Duffy

Matt OBrien

Josh Martin

Agilent/Velocity 11


31/33

Backup Slides


32/33

Resources


33/33

Example Project Plan

2009-10 xAT Tutorial_v5.ppt

Documents

Transcript of 2009-10 xAT Tutorial_v5.ppt