S U M M I T - Amazon Web Services... · Task2/Slide1 Task Dispatcher Spark Driver Spark Worker...

S U MM I TB E R L I N

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT

Building an Image Analysis auto-scaling hybrid HPC to research cancerAmador PahimQuality Assurance EngineerDefiniens AG

S e s s i o n I D


Our vision is to improve patient lives by matching patients to the

best therapies based on the most comprehensive digital profiling


Our proprietary technology finds structures, patterns and textures

in the tumor tissue image to better understand the disease biology


Project Steps

Project

Initiation

Receiving

Inspection

Region

annotations

Image

Analysis

Data

processingQC

Report and

Delivery


Project

Initiation

Receiving

Inspection

Region

annotations

Image

Analysis

Data

processingQC

Report and

Delivery

Definiens

Proprietary

Software

Project Steps


Internal Grid System

Tissue

Blur

Nucleus

Detectio

n

Tumor

Stroma

Annotation

s

Cell

Segmentatio

n

Level

Aggregatio

n


800 cores


Tissue

Blur

Nucleus

Detectio

n

Tumor

Stroma

Annotation

s

Cell

Segmentatio

n

Level

Aggregatio

n


Tissue

Blur

Nucleus

Detectio

n

Tumor

Stroma

Annotation

s

Cell

Segmentatio

n

Level

Aggregatio

n

800 cores

35TB of RAM



Project

Initiation

Receiving

Inspection

Region

annotations

Image

Analysis

Data

processingQC

Report and

Delivery

Definiens

Proprietary

Software

Project Steps


• Multiple data sets

• Different types of tasks

• Hybrid cloud support

• Auto scaling

• Job flow control

• Easy deployment

Requirements


Task Dispatcher

Executor

API Web UI

Executor Executor

Executor

Task Scheduler

Architecture


Executor

Executor

Executor

Executor

Task Dispatcher

API Web UI

Task Scheduler

Deployment


User Input

Tasks:

• Task1

• Type: python

• App: convert_format.py

• Task2:

• Type: spark

• App: heatmaps_calculation.py

• Upstream tasks: Task1,

Input Data

• Slide1

• Slide2

Task1/Slide1 Task1/Slide2

Task2/Slide1 Task2/Slide2

Resulting Tasks Workflow

First Level of Parallelismper input data


Task Dispatcher

Second Level of Parallelismmultiprocessing

Task1/Slide1


Task Dispatcher


Task1/Slide1

Python

Executor

- Python Executor

provisioning


Task Dispatcher


Task1/Slide1- Python Executor

provisioning

- Task parameters

Python

Executor


Task Dispatcher



provisioning

- Task parameters

- Parallel execution of App

run() methods

Python

Executor


Task Dispatcher



provisioning

- Task parameters


run() methods

- Results report

Python

Executor


Task Dispatcher



provisioning

- Task parameters


run() methods

- Results report

- Executor teardown


Task2/Slide1

Task Dispatcher

Second Level of Parallelismdistributed processing


Task2/Slide1

Task Dispatcher

Spark

Driver

- Spark Driver provisioning



Task2/Slide1

Task Dispatcher

Spark

Driver


- Task parameters



Task2/Slide1

Task Dispatcher

Spark

Driver

Spark

Worker

Spark

Worker

Spark

Worker


- Task parameters

- Spark Workers

provisioning



Task2/Slide1

Task Dispatcher

Spark

Driver

Spark

Worker

Spark

Worker

Spark

Worker


- Task parameters

- Spark Workers

provisioning

- Processing orchestration



Task2/Slide1

Task Dispatcher

Spark

Driver

Spark

Worker

Spark

WorkerSpark

Worker


- Task parameters

- Spark Workers

provisioning


- Results report



Task2/Slide1

Task Dispatcher


- Task parameters

- Spark Workers

provisioning


- Results report

- Cluster teardown



• DSS - Data Streaming Service

• Serves tiles from multiple file formats

• Standard data access service for internal applications

• Can be executed as a container

• Supports multiple storage backends (S3 included)

Data Access


DSS

Executor

DSS

Data Access


from jpc import Job

from jpc import InputData

from jpc import PythonTask

task1 = PythonTask(name='heatmaps',

app='Heatmaps.py',

app_args=['-p', '-r'],

repository_url='[email protected]:projects/12312.git')

input_data = InputData([['dss://dss.definiens.com/projects/12312/slide1'],

['dss://dss.definiens.com/projects/12312/slide2']])

job = Job(name='heatmaps_generation',

tasks=[task1],

input_data=input_data)

job_status = job.submit()

User Interface – First Version


User Interface – First Version


• Data Access Framework

• Final User Interface: Portal Integration

• More executors to come:

• Amazon SageMaker

• Amazon EMR

• Amazon Lambda experiment

• Projects billing

Next Steps


Thank you!


Amador [email protected]

S U M M I T - Amazon Web Services... · Task2/Slide1 Task Dispatcher Spark Driver Spark Worker...

Documents

Transcript of S U M M I T - Amazon Web Services... · Task2/Slide1 Task Dispatcher Spark Driver Spark Worker...