Active Folder : Integrating All Activities of Simulation on File System

Suntae Hwang ( sthwang@kookmin.ac.kr )Daeyoung Heo ( dyheo@kookmin.ac.kr )

School of Computer ScienceKookmin University

Apr 10 2014

2

Active Folder: Integrating All Activities of Simulation on File System

⊙Active Folder – good for case comparative study●Tasks○Described as regular folders and files

●Products○Input or output of simulation ○Can be handled like regular file by using legacy software○Contains provenance information (meta data, task info,

etc)○Can be reproduced by the task which is extracted from the

provenance information●Apps(Computing Resource)○Computing server(Local, Grid, Cloud, what ever, …) is reg-

istered as regular folders and files○To submit a Job(task), just Drag&Drop the task folder to the

folder which represents computing server

ActiveFolder

3

Active Folder: Integrating All Activities of Simulation on File System

Storage(Cloud, Local, ..)

Producing

SharingAnalizing

• Products• Tasks• Apps

User’s own environment

Computing(Cloud, Grid, Local, …)

Or Searching(DB, IR, …)

4

Formal Definition of Spaces in Active Folder

Definition 1. Set P for Product

Definition 2. Set T for taskDefinition 3. Set A for producing/reproducingDefinition 4. Set M for monitoring.

5

Active Folder: Product and Reproducing

DATAHistory

TaskProduct

Fall3D

Visualize & Analysis

Procedure 𝒍𝒆𝒕 𝒕=𝒕𝒂𝒔𝒌 (𝒑)𝒂𝒑𝒑(𝒕 ,𝑲 ,𝑷𝒌)

Differentparame-ters

Task can be ex-tracted from prod-ucts and used for producing similar results with differ-ent parameters

Product consists of data, history info and task(provenance)

6

Active Folder: Products Searching and Comparison

⊙Similar or interested products can be searched by their provenance information and compared by var-ious legacy tools on conventional file system

7

Case Study : Volcano Eruption Simulation using Active Folder

8

Short history of Active Folder

⊙Active Folder on DropBox+EC2●Cost & Performance Problem with very large files

⊙Active Folder on NAS⊙NAS(Network Attached Storage)● Large Volume Storage● Network File System ( NFS,

SMB/CIFS, AFP … )● Most vendors support Cloud

solution like DropBox

at PRAGMA 25

at PRAGMA 24

http://www.youtube.com/watch?v=vXTh3tCHEPE

9

Spin

1000

Temperature273K

Planning

File & Folder

Everything is

Applications

ExecutionDrag & Drop

SSH

CHARMM

AMBER

NAMD

GAUSSIAN

ActiveFolder . . . ...

CHARMM

SimulationScript

Activity

CHARMMT-SPACE

ssh.server.com

Server

A-SPACE

Active Folder Features 1:Computing by manipulating files or folders, which makes only files visible by hiding correlation of computing method, parameters, result files and computing re-sources

10

Spin

1000

Temperature273K

Activ

eFold

er . .

. ...

CHARMM

Simula

tion

Scrip

t

How to create this product?

Products

P-SPACE

Product

Product

Data Provenance

Relationship

PROVIDED / USED

History BoardCreated by CHARMM Activity

From ssh.server.com

At 2014.m.d

StageOut : 2014.m.d 12:30

StageOut : 2014.m.d 13:02

Active Folder Features 2:Keep the way of computing meth-ods and parameters with result files silently

11

Active

Fold

er . .

. ...

CHARMM

Simula

tion

Scrip

t

Temperature300K

Temperature293K

Searching

Data Provenance

Parameters

Parameters

Para

meter

s

Para

meter

s

Parameters

Parameters

Parameters

Parameters

Active Folder Features 3:Searching result files and other core-lated ones

12

Activ

eFold

er . .

. ...

CHARMM

Simula

tion

Scrip

t

Reproduction

Using data provenance

Extracting

Activity

CHARMMT-SPACE

Parameters

Parameters

Parameters

Temperature293K

ApplyingNew Methods/Parameters

Parameters

Parameters

Parameters

Parameters

Parameters

Parameters

Execution

Active Folder Features 4:Computing by manipulating files or folders, Re-generating either the same or similar ones by existing re-sult files

13

ssh.server.com

Server

A-SPACE

CHARMMSSH

Products

Charmm.dcd file created

StageOut : 2014.m.d 12:30

Charmm.dcd file modified

StageOut : 2014.m.d 1:13

stdout.afsx.log file created

StageOut : 2014.m.d 1:14

stdout.afsx.log file modified

StageOut : 2014.m.d 1:15

Tim

elin

e

Data Synchronization

Between server and your active folder

• Automatic Stage-Out to ActiveFolder• Incremental update.

Active Folder Features 5:Returns computing results by progressive synchronization without connecting computing server

14

File Navigator

Products

P-SPACE

Activity

CHARMMT-SPACE

ssh.server.com

Server

A-SPACE

SAMBA

FTP

NFS

Data Delivery

Support Legacy File ServiceYour Desktop, Laptop or Mobile

Microsoft

Window

s

MacO

SX

Linux

Smartphone

CloudStorage

ActiveFolder

Apps Activities Products

Open this files by your favorite softwares

Active Folder Features 6:Support various file service such as NFS, Samba(CIFS) and FTP

15

Active Folder: Overall View

Products

P-SPACE

Activity

CHARMMT-SPACE

ssh.server.com

Servers

A-SPACE

Searching and Analysis

ActiveFolder

16

Active Folder: Software Architecture

17

ActiveFolder

ComputingEnvironment(Large scale

service)

ActiveFolder

Satellite Service

* Private Space - File Management - Private Archiving* For Power-User

Satellite Service

ActiveFolder

Satellite Service

Usage of Active Folder

⊙Good for managing & sharing data in a small group of Lab

⊙For Power-users us-ing legacy SW in their own computing environment (with-out GUI sometimes)

18

Active Folder Site

⊙Active Folder is released

●http://www.activefolder.org

⊙Downloads ●Binary file for general installation(Synology(NAS) tested)●RPM for Redhat 6/CentOS 6

⊙Coming soon●Product searching interface●Synology packaging for NAS●Integrated using NAS and DropBox together

Attention! .org only, not .com

19

Thank you!

Questions?