National Alliance for Medical Image Computing http://na-mic.org

Grid Computing with BatchMake

Julien JomierKitware Inc.

National Alliance for Medical Image Computing http://na-mic.org

Motivation

Algorithm Design & Implementation

Data Processing

Reports & Analysis

National Alliance for Medical Image Computing http://na-mic.org

Outline

• Introduction to BatchMake

• Grid Computing with BatchMake

• Reporting with the BatchBoards

• BatchMake with Slicer

National Alliance for Medical Image Computing http://na-mic.org

Introduction to BatchMake

• Batch script generator• Open-Source BSD license • Common Language (CMake-like)• Generation for:

- Shell script - Condor - KWGrid

• Run locally or on grid• Cross platform command line and GUI tool• C++ with ITK and FTLK

National Alliance for Medical Image Computing http://na-mic.org

BatchMake Workflow

CommandLine Executable

ApplicationWrapper

BatchMake

Script

Local Execution

Condor Script

KWGrid Script

Shell Script

National Alliance for Medical Image Computing http://na-mic.org

Application Wrapper

• Describe command line arguments- Manually (GUI)- Automatically using itkMetaCommand

• Internal XML format

• .bmm files (BatchMake Modules)

• Compatible with Slicer execution modules

National Alliance for Medical Image Computing http://na-mic.org

MetaCommand

• C++ class for command line parsing

• Output recognized by- BatchMake (-vxml)- Slicer (--xml)

• command.SetOption("compileScript","c",false,"Compile a script");

command.AddOptionField("compileScript","filename", MetaCommand::STRING,true);

• http://www.itk.org/Wiki/MetaIO

• Insight/Utilities/MetaIO

National Alliance for Medical Image Computing http://na-mic.org

Example Script

1 - SetApp(imSeg @ImageSegmentation)2 - SetAppOption(imSeg.threshold 10)3 - 4 - Set(workdir '/BatchMake/Demo/Ellipse_Images')5 - ListFileInDir(files ${workdir} '*.mha')6 -7 - Foreach(file ${files})8 - Set(filename '${workdir}/${file}')9 - Set(outfile '${workdir}/out_${file}')10- SetAppOption(imSeg.inputFilename ${filename})11- SetAppOption(imSeg.outputFilename ${outfile})12- Run(output ${imSeg})13- EndForeach(file)

National Alliance for Medical Image Computing http://na-mic.org

Condor

• Specialized workload management system for compute-intensive jobs

• Created by UW-Madison• Free for use• Provides job queuing mechanism,

scheduling policy, priority scheme, resource monitoring, and resource management.

• Unix and Windows

National Alliance for Medical Image Computing http://na-mic.org

Condor Example Script

Universe = vanillaExecutable = EMSegmenterArguments = ‘brain.mha segmentation.mha --brain’Output = emsegmentation.out Error = emsegmentation.error Log = emsegmentation.logRequirements = (OpSys =="LINUX" && Arch=="INTEL")Should_transfer_files = YES When_to_transfer_output = ON_EXIT_OR_EVICTTransfer_input_files = brain.mhaQueue

National Alliance for Medical Image Computing http://na-mic.org

Condor Workflow (Vanilla)

Submit Machine

Central Manager

Execute Machines

Condor Pool

Submit job

Issue job

Idle Workstation

Busy Workstation

Available Execute Machine

Push out job to machine

Central Manager

Central ManagerUnavailable Execute Machine

Dispatch to Central manager

= Available

Dispatch to Central manager

= Unavailable

National Alliance for Medical Image Computing http://na-mic.org

BatchMake Condor Generation

National Alliance for Medical Image Computing http://na-mic.org

BatchMake Condor Generation

• Application/Module description - Inputs/Outputs - Data to transfer to nodes

• Loop unrolling

• Scope definition

• Directed Acyclic Graph (DAG)

• Ensure order of execution of jobs

National Alliance for Medical Image Computing http://na-mic.org

Monitoring Condor Jobs

• Flock Monitor (Condor)• Condor Watcher (FLTK)• Condor Web Watcher (Ajax/PHP)

National Alliance for Medical Image Computing http://na-mic.org

Reporting with BatchMake

CVS/SVN

CVS/SVN

Results posted on web(i.e., the batchboard)

CTest/DART compilessource code

Developers review results

Developers check-in

code

MIDASXNat

MIDASXNat

BatchMake validateson datasets

+

National Alliance for Medical Image Computing http://na-mic.org

Reporting with BatchMake

National Alliance for Medical Image Computing http://na-mic.org

BatchMake Server

• PHP Scripts with MySQL database

• Simple C++ HTTP post

• Statistical Analysis (Factor analysis: Taguchi).

• Dashboard creation capability

• User management

National Alliance for Medical Image Computing http://na-mic.org

BatchBoard Script

DashboardHost(http://www.insight-journal.org/batchmake)

DashboardUser(‘Julien Jomier')

DashboardKey('FKH3BPM1y11VXsMf')

CreateExperiment(exp ‘Liver Project‘ ‘Segmentation Experiment' '')

CreateMethod(Meth exp 'Registration' 'Register a T1 with noise to a translated T1')

AddMethodInput(noiseImageName Meth 'Fixed Image')

Set(fixedSlice '${fixedImage}Slice.png')

ExtractSlice('${fixedImage}' '${fixedSlice}' 2 1)

AddMethodInput(fixedSlice Meth 'Fixed Slice with noise' png)

DashboardSend(Meth)

National Alliance for Medical Image Computing http://na-mic.org

BatchMake and Slicer

• BatchMake distributed with Slicer• Some modules can be run on a

collection of data- EMSegment- GaussianBlur

• Generic way to run moduleson a collection

• Support for XNat

National Alliance for Medical Image Computing http://na-mic.org

GridWizard

• Open-Source• Written in Java• BIRN and Na-Mic • Support for Sun Grid Engine and

Condor• Shell-like scripts

National Alliance for Medical Image Computing http://na-mic.org

GridWizard Example

bin/gwiz-run prog -t 1:3

prog -t 1 prog -t 2prog -t 3

bin/gwiz-run prog -i sftp://host.com/opt/mri/subjects/*.img

prog -i sftp://host.com/opt/mri/subjects/s1.img prog -i sftp://host.com/opt/mri/subjects/s2.img

prog -i sftp://host.com/opt/mri/subjects/s3.img

National Alliance for Medical Image Computing http://na-mic.org

References

• www.batchmake.org

• www.insight-journal.org/batchmake

• http://forums.gridsphere.org/index.shtml

National Alliance for Medical Image Computing http://na-mic.org

Condor Job Submission Workflow

Collector

Submit Machine

Scheduler

Starter

Negociator

Central Manager Execute Machine

MasterClassAd

MasterMatch

ClassAd

Job

1

23

4

5