INTEL High Performance Data Division 1

HPC Technologies for Big Data

Brent Gorda GM High Performance Data Division March 2013

INTEL High Performance Data Division 2

Introduction

One year ago:

- Community

- Roadmap

- Current Development

- Looking Forward

Whamcloud’s goal was to preserve Lustre for the community.

INTEL High Performance Data Division 3

Along the way – Intel Agreed

July 13, 2012

The move to Intel completes the journey started by Eric Barton, Robert Read and myself in 2010

Lustre is on stable ground again

INTEL High Performance Data Division 4

Intel and Lustre

•  Intel continues the Whamcloud business model: •  Multi-vendor Lustre development

•  Open source, community friendly, single source tree

•  Staff retention at 98% since acquisition (lost 1)

•  Intel supports all Whamcloud business (incl. Fastforward)

•  A member of EOFS (Whamcloud’s position)

•  Recently joined OpenSFS as a promoter ($500K)

“Intel  bought  Whamcloud  to  be  Whamcloud”              –  Boyd  Davis  

INTEL High Performance Data Division 5

Dir, Datacenter Computing Paresh Pattani

Deb Christensen, Administrative Assistant

Datacenter Software Division (DSD) Boyd Davis VP, GM DSD

HR Lea Ann Hansen

Finance Fawwad Qureshi

Legal TBD

GM., Enterprise Software Strategy Pauline Nist

GM, Intel Hybrid Cloud Bridget Karlin

CTO, DSD & GM, Big Data Software & Services Girish Juneja

Dir., Datacenter Management Jeff Klaus

GM, Attached Platform Storage Software

Steve Dalton

Dir., Manageability Software Ben Vrvilo

Curtis McKee, Technical Assistant

GM., High Performance Data Brent Gorda

Dir, Solutions and Technologies Lynn Comp

CTO, HPC Ecosystem Mark Seager

INTEL High Performance Data Division 6

Evolving Lustre Community

An increasing number of organizations contribute

http://git.whamcloud.com/?p=fs/lustre-release.git;a=summary

0   10000   20000   30000   40000   50000   60000   70000   80000   90000   100000  

[2.1]  

2.1  

2.2  

2.3  

Bull  

CEA  

Cray  

DDN  

EMC  

Fujitsu  

LLNL  

NICS  

ORNL  

TACC  

Ultrascale  

UVT  

Whamcloud/Intel  

Xyratex  

Sun/Oracle  

INTEL High Performance Data Division 7

INTEL High Performance Data Division 8

Department of Energy - Fast Forward Challenge

•  FastForward RFP provided US Government funding for Exascale research and development

•  Sponsored by 7 leading US national labs

•  Aims to solve the currently intractable problems of Exascale to meet the 2020 goal of an Exascale machine

•  RFP elements were Processor, Memory and Storage

•  Whamcloud won the Storage (filesystem) component –  HDF Group – HDF5 modifications and extensions –  EMC – Burst Buffer manager and I/O Dispatcher –  Cray - Test

•  Contract renegotiated on Intel acquisition of Whamcloud –  Intel - Arbitrary Connected Graph Computation –  DDN - Versioning OSD

INTEL High Performance Data Division 9

Jitter Bulk synchronous programming

– Simplifies application development – Susceptible to jitter – Makes strong scaling harder

Asynchronous programming –  Loosen coupling between

processes – No waiting at barriers – Closes “gaps” provided jitter

balances out over time

INTEL High Performance Data Division 10

Transactions

Consistency and Integrity •  Guarantee required on any and all failures

–  Foundational component of system resilience •  Required at all levels of the I/O stack

–  Metadata at one level is data to the level below

No blocking protocols •  Non-blocking on each OSD •  Non-blocking across OSDs

I/O Epochs demark globally consistent snapshots •  Guarantee all updates in one epoch are atomic •  Recovery == roll back to last globally persistent epoch

–  Roll forward using client replay logs for transparent fault handling •  Cull old epochs when next epoch persistent on all OSDs

Time

Upd

ates

I/O Epochs

INTEL High Performance Data Division 11

Exascale I/O Architecture

Compute    Nodes  

I/O  Nodes  

Burst  buffer  NVRAM  @  200  TB/s  

Disk  Metadata  NVRAM  

Storage    Servers  

Site  Storage    Network  @20  TB/s  

Exascale Machine Shared Storage

Exascale  Fabric  

INTEL High Performance Data Division 12

Project Goals

Make storage a tool of the Scientist •  Manage the explosive growth

and complexity of application data and metadata at Exascale

•  Support complex / flexible analysis to enable scientists to engage with their datasets

•  Provide storage performance and capacity required for Exascale science

Overcome today’s filesystem scaling limits •  Share nothing •  Move compute to data or data to compute as appropriate

Provide unprecedented fault tolerance •  Design ground-up to handle failure as the norm rather than the exception •  Guarantee data and application metadata consistency and integrity

INTEL High Performance Data Division 13

Exascale Filesystem

Integrated I/O Stack •  Epoch transaction model •  Non-blocking scalable object I/O

HDF5 •  High level application object I/O model •  I/O forwarding

I/O Dispatcher •  Burst Buffer management •  Impedance match application I/O performance to

storage system capabilities

DAOS •  Conventional namespace for administration, security & accounting •  DAOS container files for transactional, scalable, object I/O

/projects

/Legacy /HPC /BigData

Simulation data OODB metadata

data data data data data data data data data data data data data data data

OODB metadata OODB metadata OODB metadata OODB metadata

Posix striped file

a b c a b c a b c a

MapReduce data

data data data

data data data

data data data

Blocksequence

INTEL High Performance Data Division 14

Hadoop on Lustre

14

70%  

•  Hadoop on Lustre: extend Hadoop analytics to HPC environments o  Exploit performance and scalability of shared storage o  Scale storage and compute nodes separately

•  Initial work demonstrates Lustre performance advantage •  In plan to enable/optimize Hadoop analytics stack on Lustre in 2013 •  Early results indicate existing HPC sites do well changing HDFS->Lustre

•  No need to put disk back in the compute nodes •  Scale compute and I/O separately to balance work

InfiniBand  Interconnect  

Hadoop  Cluster/Compute  Nodes  

Lustre  Storage  

INTEL High Performance Data Division 15

Intel @ intersection of forces behind big data

Enabling exascale computing on massive data sets

Helping  enterprises  build  open  interoperable  clouds  

Commi_ed  and  contribu`ng    code,  process,  and  capital      

HPC   Cloud   Open  Source  

Intel®  TrueScale  Infiniband  

*  Other  names  and  brands  may  be  claimed  as  the  property  of  others.  

INTEL High Performance Data Division

Thank You brent.gorda@intel.com

16

INTEL High Performance Data Division 18

Today: A Vibrant Open Solution Ecosystem

Advocacy   Storage   Compute   Integra=on   Development  

INTEL High Performance Data Division 19

Another view of the code effort

LLNL  23  

ORNL  5   TACC  2  

Whamcloud  250  

Xyratex  5  Bull  1   CEA  3   Cray  5  

DDN  3  LLNL  10   NICS  1  

ORNL  15  

TACC  3  

Whamcloud  258  

Xyratex  19  

Bull  4   CEA  3   Cray  4   DDN  1  

EMC  38  

Fujitsu  1  

Intel  435  

LLNL  19  ORNL  17  

TACC  5  

Ultrascale  2  

UVT  1   Xyratex  28