CSinParallel Workshop OnRamp: An Interactive Learning ...€¦ · • May 2015 • SIGCSE Grant...

CSinParallel Workshop� �

OnRamp:�An Interactive Learning Portal for� Parallel Computing Environments�

Samantha Foley [email protected]

http://cs.uwlax.edu/~ssfoley

1

Josh Hursey [email protected]

http://cs.uwlax.edu/~jjhursey/

Project supported by SIGCSE Special Projects Grant (May 2015), Blue Waters Student Internship program (2015), and the UW-L Computer Science Department

https://github.com/ssfoley/onramp

OnRamp to Parallel Computing Project�

There exists a significant barrier to entry for learning how to become productive in a Parallel Computing Environment (PCE)

due to often unfamiliar and complex system software, programming interfaces, and tools.

2

Parallel Programming Patterns, Tradeoffs, Scaling

UNIX Shell (Scripting)

Compilation Environment (Makefile, mpicc, nvcc)Runtime Environment (mpirun, srun, aprun)

Parallel Architectures

Batch Submission Systems (PBS, SLURM, LSF)

Remote System Access & File Transfer

Parallel Programming Paradigms (MPI, OpenMP)

CS1/CS2 – (Java & Eclipse IDE)

Trad

ition

al In

stru

ctio

n Pa

th



The OnRamp Project, provides a web-based portal that coaches users through interactive tutorials that teach them about the

software ecosystem and parallel computing while allowing them to launch & explore parallel applications from day one.

3








CS1/CS2 – (Java & Eclipse IDE)Level 1

Level 1.5Level 2

Level 3



• High-Level Goals•  Encourage students to explore parallel and distributed computing

concepts without the overhead of PCE system software peculiarities. •  Help students transition to using the native PCE, eventually. •  Bring together existing educational hardware & curriculum modules

efforts into a flexible, portable architecture.

• Basic Architecture

4 https://github.com/ssfoley/onramp

OnRamp Server

LittleFeOnRamp PCE

VM ClusterOnRamp PCE

XSEDEOnRamp PCE

ClusterOnRamp PCE

Portal


• Web Portal (client facing side of the OnRamp Server)•  Users do not need accounts on each PCE (use a shared UNIX account). •  Users can be grouped into one or more Workspaces. •  Workspaces are assigned PCE & Module combinations.

•  Users view Module instructions, launch jobs, view results, transfer files. •  Automatically generated & validated custom forms for each Module.

•  Administrative panel to manage PCEs, one-click deploy Modules, manage Users, manage Workspaces, monitor usage, …

5

OnRamp Server

LittleFeOnRamp PCE


XSEDEOnRamp PCE

ClusterOnRamp PCE

Portal

Later release will allow users to edit code



•  OnRamp Server•  Broker between the users of the OnRamp Portal and the PCEs. •  Enforce policy, authenticate users, and manage security. •  Cache files between Users and PCEs.

•  OnRamp PCE•  Custom drivers for each type of software environment (SLURM, PBS, LSF). •  Manage modules, and user files associated with jobs. •  Launch and monitor jobs on the system.

6

OnRamp Server

LittleFeOnRamp PCE


XSEDEOnRamp PCE

ClusterOnRamp PCE

Portal

Developing ‘turn-‐key’ scripts to setup virtual

clusters.



•  Curriculum Modules•  Complete freedom to structure the module as you like. •  Write a few python scripts to ‘hook’ into the OnRamp architecture. •  Configuration files allow you to specify custom, module-specific parameters and

validation requirements for users running the code. •  Custom documentation/instructions •  Can be tailored to the PCE environment

7

OnRamp Server

LittleFeOnRamp PCE


XSEDEOnRamp PCE

ClusterOnRamp PCE

Curriculum modules

A

Z

B

Z

A

B

R

Z

Portal


OnRamp to Parallel Computing Project�Curriculum Module Life cycle�

• Admin deploys Module on PCE using Portal • Module can be deployed from git, svn, tar.gz, file system • OnRamp PCE unpacks Module and runs onramp_deploy.py•  Script can compile the code, generate documentation, …

• OnRamp PCE sends back Module generated docs and configuration files.

• Once deployed: • Admin can ‘test’ the module by trying to run it. • Admin can assign it to the users in one or more Workspaces.

8

LittleFeOnRamp PCE

OnRamp Server

Portal

A

A



• Users configure & run Modules •  View Module and PCE documentation and instructions • Custom parameters defined in onramp_uioptions.spec• On job submission: (user defined parameters are available) • OnRamp PCE runs onramp_preprocess.py• OnRamp PCE sets up batch script and submits to queue •  Batch script will run onramp_run.py• On job completion •  Batch script will run onramp_postprocess.py

9

LittleFeOnRamp PCE

OnRamp Server

Portal A



• After job completion, users can: •  View result output (stdout/stderr) • Compare results to other runs • Access files from the run and module •  Resubmit the job after adjusting some parameters •  (Future) Visualize, graph results •  (Future) Share results and/or collaborate

10

LittleFeOnRamp PCE

OnRamp Server

Portal A


OnRamp to Parallel Computing Project�Roadmap�

11 https://github.com/ssfoley/onramp

• May 2015•  SIGCSE Grant started (project development started)

• Sept. 2015 (alpha release) •  PCEs: SLURM, LittleFe (PBS) •  Server: Workspaces/Users, job launch/monitoring, basic admin. capabilities •  Modules: MPI Hello World & Ring, MPI/OpenMP Area Under the Curve

(Shodor)

• Jan. 2016 (beta release) •  PCEs: Rocks Cluster, Virtual Machine (scripted) •  Server: Module customized submissions, full admin. Capabilities •  Modules: HPL, Monte Carlo (CSinParallel), Parameter search exemplar

• March 2016 (v1.0 release) •  PCEs: Cloud provisioned PCEs (scripted) •  Server: Demonstrate Coaching up to Level 2


The OnRamp Project, provides a web-based portal that coaches users through interactive tutorials that teach them about the

software ecosystem and parallel computing while allowing them to launch & explore parallel applications from day one.

12








CS1/CS2 – (Java & Eclipse IDE)Level 1

Level 1.5Level 2

Level 3


CSinParallel Workshop OnRamp: An Interactive Learning ...€¦ · • May 2015 • SIGCSE Grant...

Documents

Transcript of CSinParallel Workshop OnRamp: An Interactive Learning ...€¦ · • May 2015 • SIGCSE Grant...