NCS Grid ServiceNCS Grid Service

Ken Meacham, IT InnovationKen Meacham, IT Innovation

Crystal Grid Workshop,Crystal Grid Workshop,13-17 Sept 200413-17 Sept 2004

OverviewOverview

• DesignDesign– objectivesobjectives– workflowworkflow– architecturearchitecture– implementationimplementation

• DemoDemo• ConclusionsConclusions

ObjectivesObjectives• NCS Grid Service objectivesNCS Grid Service objectives

– to allow remote users to interact with their experimentsto allow remote users to interact with their experiments– to exploit this collaboration to improve NCS efficiencyto exploit this collaboration to improve NCS efficiency– to be compatible with NCS operational processesto be compatible with NCS operational processes

• Vision at start of 2003Vision at start of 2003– monitoring of sample status / progressmonitoring of sample status / progress– read-access to experimental software via VNCread-access to experimental software via VNC– audio/video link to a human experimenteraudio/video link to a human experimenter

• Vision and detailed objectives changed during 2003Vision and detailed objectives changed during 2003– no direct interaction with a human experimenterno direct interaction with a human experimenter– write-access to experimental parameterswrite-access to experimental parameters– requires restricted steering interface: no use for VNC / videorequires restricted steering interface: no use for VNC / video– automation of x-ray diffractometer S/W (Nonius)automation of x-ray diffractometer S/W (Nonius)– automated structure refinementautomated structure refinement

ObjectivesObjectives

• Problems encountered during 2004Problems encountered during 2004– automation of Nonius S/W not mature enoughautomation of Nonius S/W not mature enough– automatic structure refinement not reliableautomatic structure refinement not reliable– cannot define all possible S/W parameters in steering toolcannot define all possible S/W parameters in steering tool

• Consequent changes during 2004Consequent changes during 2004– experiment steering interface put “on ice”experiment steering interface put “on ice”

– work continues on automation aspectswork continues on automation aspects

– enhancements to Status Serviceenhancements to Status Service– addition of Admin User facilities (tracking other samples)addition of Admin User facilities (tracking other samples)

– changes to security model (in PHP code)changes to security model (in PHP code)– improved selection of users / projects / samplesimproved selection of users / projects / samples

– downloading of experimental results (uploaded by Admin)downloading of experimental results (uploaded by Admin)

Sample ProcessingSample Processing

Scheduled

Add to queue

Scheduled:sample prep

underway

Update schedule

Scheduled:sample prepared

Running:expt setup

started

Running:unit cell:

collecting scanx (of y)

Running:expt setupcompleted

Running:unit cell:

collected scanx (of y)

Running:unit cell:

calculatingunit cell

Running:unit cell:

generating imagex (of y)

Running:unit cell: image

x (of y) available

Running:unit cell:

calculationcomplete

Running:unit cell:

unit cell dataavailable

Start sample preparation

Finish sample preparation

Setup experiment(mount sample,create directory)

Collect scan

Scans complete.Start unit cell calculation

Finish unit cellcalculation

Start unit celldetermination

Start imagegeneration

Finish imagegeneration.

Publish image

Publish unit celldata

Start next scan

Discard unit cell

Running:crystal

discarded

Running:sample prep

underway

Running:sample prepared

Start sample preparation

Finish sample preparation

Discard crystal

Start newexperiment

Startdata collection

Initiated by Lab Technician

Executed automatically

Key

New samplereceived

Sample datastored

(pendingschedule)

Unpack sample,process and store data

Receive new sample

Start

Sample ProcessingSample Processing

Running:data collection:

started

Running:data collection:collecting scan

x (of y)

Running:data collection:collected scan

x (of y)

Expt finished:data collection:

completed

Start scan

Collect scan

Scans complete

Next scan

Post-processing:started

Start post-processing

Post-processing:completed(HKL file

generated)

Post-processing:completed(HKL fileavailable)

Process images,create HKL file

Publish HKL

Calculatestructure

Structurecalculation failed

Structurecalculationcompleted

Structureavailable

Calculation succeeded

Store / publish structure

Calculationfailed

Recalculatestructure

Calculatingstructure

Startdata collection

Initiated by Lab Technician

Executed automatically

Key

User Authorisation StatesUser Authorisation States

(Setup running)can request statuscan monitor status

can connect to conferencecan download images

(Prep new crystal)can request statuscan conference?

can monitor?

(Prev. expt finishes)Start new expt (launch control GUI),

notify user

Authenticatedusers

Revoke users

Discard crystal

Discard unit cell

New crystal ready

(New)can send sample

(Pending schedule)can request status

(Scheduled)can request statuscan monitor status

(Ready to start)can request statuscan monitor status

Unpack, enter data,send confirmation

Schedule for tomorrow,notify user

Sample prepared,notify user

(Data collection)can request statuscan monitor status

(Raw data available)can request statuscan download HKL

(Structure available)can request statuscan download HKL

can download structure

(Finished)

Experiment finished + postprocess completed(HKL available)

Structure determined(structure data available)

Publish data?Archive data?

Timeout?

Setup completed

Abort

Client ImplementationClient Implementation

• Web-based Client (portal)Web-based Client (portal)– access via browseraccess via browser

– no additional client S/W requiredno additional client S/W required– restricted (well defined) write accessrestricted (well defined) write access– steering of X-ray experimentsteering of X-ray experiment

– secure connection via HTTPSsecure connection via HTTPS– certificate-based authenticationcertificate-based authentication– no client side firewall problemsno client side firewall problems

Server ImplementationServer Implementation

• Sample Tracking DatabaseSample Tracking Database– samples, users, projectssamples, users, projects– replacement for old NCS sample databasereplacement for old NCS sample database– experiment information experiment information

– schedulingscheduling– statusstatus

– Java / web (JDBC) interfaceJava / web (JDBC) interface– NCS use onlyNCS use only

Server ImplementationServer Implementation

• Sample Status ServiceSample Status Service– client interface to sample tracking databaseclient interface to sample tracking database– secure access to personal sample data onlysecure access to personal sample data only

– (Admin User may access other user’s samples)(Admin User may access other user’s samples)

– implemented in PHP (dynamic HTML)implemented in PHP (dynamic HTML)

• Control ServiceControl Service– collaborative access to X-Ray diffractometercollaborative access to X-Ray diffractometer

– immediate access to raw data (images, unit cell data)immediate access to raw data (images, unit cell data)– steering of experimentsteering of experiment

– access rights linked to whose sample is runningaccess rights linked to whose sample is running– implemented in PHPimplemented in PHP

NCS Grid Service ArchitectureNCS Grid Service Architecture

NCS Grid Service DemoNCS Grid Service Demo

ConclusionsConclusions

• Core NCS grid services deployedCore NCS grid services deployed– sample tracking databasesample tracking database– status / control servicesstatus / control services

• What next?What next?– operational roll out of existing NCS Grid Serviceoperational roll out of existing NCS Grid Service

– testing of systems / procedurestesting of systems / procedures(internal staff then selected external NCS clients)(internal staff then selected external NCS clients)

– improvements / fixes to current servicesimprovements / fixes to current services– background developments by NCS into automation, etcbackground developments by NCS into automation, etc

– integration with other Comb-e-Chem servicesintegration with other Comb-e-Chem services– e.g. results, data, computation servicese.g. results, data, computation services

– collaboration with other crystallography services?collaboration with other crystallography services?