Innovative Applications and Technology Pivots –A Perfect Storm in Computing

Wen-mei Hwu

Professor and Sanders-AMD Chair, ECE, NCSA

University of Illinois at Urbana-Champaignwith

Jinjun Xiong (IBM, C3SR Co-Director), Abdul Dakkak and Carl Pearson

Agenda

• Revolutionary paradigm shift in applications

• Technology pivot to heterogeneous computing

• Cognitive computing systems research

A major paradigm shift

In the 20th Century, we were able to understand, design, and manufacture what we can measure• Physical instruments and computing systems allowed us to see farther, capture

more, communicate better, …

A major paradigm shift

In the 20th Century, we were able to understand, design, and manufacture what we can measure• Physical instruments and computing systems allowed us to see farther, capture

more, communicate better, understand natural processes, control artificial processes…

In the 21st Century, we are able to understand, design, and create what we can compute• Computational models are allowing us to see even farther, going back and

forth in time, learn better, test hypothesis that cannot be verified any other way, …

Examples of Paradigm Shift20th Century

Small mask patterns

Electronic microscope and Crystallography with computational image processing

Anatomic imaging with computational image processing

Optical telescopes

Teleconference

GPS

21st Century

Optical proximity correction

Computational microscope with initial conditions from Crystallography

Metabolic imaging sees disease before visible anatomic change

Gravitational wave telescopes

Tele-emersion – augmented reality

Self-driving cars

What is powering the paradigm shift?

• Large clusters (scale out) allow solving realistic problems• 1.5 Peta bytes of DRAM in Illinois Blue Waters• E.g., 0.5 Å (0.05 nm) grid spacing is needed for accurate molecular dynamics

• interesting biological systems have dimensions of mm or larger• Thousands of nodes are required to hold and update the grid points.

• Fast nodes (scale up) allow solution at realistic time scales• Simulation time steps at femtosecond (10-15 second) level needed for accuracy

• Biological processes take milliseconds or longer• Current molecular dynamics simulations progress at about one day for each 100

microseconds of the simulated process.• Interesting computational experiments take weeks (used to be months)

What types of applications are demanding computing power today? • First-principle-based models

• Problems that we know how to solve accurately but choose not to because it would be “too expensive”

• High-valued applications with approximations that cause inaccuracies and lost opportunities

• Medicate imaging, earthquake modeling, weather modeling, astrophysics modeling, precision digital manufacturing, combustion modeling, ….

• Applications that we have failed to program• Problems that we just don’t know how to solve• High-valued applications with no effective computational methods• Computer vision, natural language dialogs, stock trading, fraud detection, …

We know what we want but don’t know how to build it.

Deep Learning Object DetectionDNN + Data + HPC

Traditional Computer VisionExperts + Time

Deep Learning Achieves “Superhuman” Results

0%10%20%30%40%50%60%70%80%90%

100%

2009 2010 2011 2012 2013 2014 2015 2016

Traditional CV

Deep Learning

ImageNet

Slide courtesy of Steve Oberlin, NVIDIA

2M training images

Some different modalities of Real-world Data

Image Vision features Detection

This seems to be a combinational logic design problem.

?

Combinations Logic Specification –Truth Table

10

Inputoutputa b c

0 0 0 0

0 0 1 1

0 1 0 1

0 1 1 0

1 0 0 1

1 0 1 0

1 1 0 0

1 1 1 1

a’ a b’ b c’ c

What if we did not know the truth table?

• Look at enough observation data to construct the rules

• 000 → 0

• 011 → 0

• 100 → 1

• 110 → 0

• If we have enough observational data to cover all input patterns, we can construct the truth table and derive the logic!

11

LeNet-5, a convolutional neural network for hand-written digit recognition.

12

This is a 1024*8 bit input, which will have a truth table of 2 8196 entries

Convolutional Layer

Weights W

Input Features

X OutputFeatures

Y

1M training data is approximately 0%

The adoption of full cognitive business applications has exploded since …

13

14

Back in 2011

The cognitive application is built and optimized for the underlying infrastructure manually

3 sec response time!

Illinois-IBM C3SR faculties & students(Launched 9/20/2016)

Suma Bhat Julia HockenmaierMinh Do Deming Chen Wen-mei Hwu Nam Sung Kim Dan Roth Lav VarshneyRakesh Nagi

…

A C3SR App: CELA for Personalized Education

List of

available

materials

Database of

existing science

projects

Database of

STEM required

concepts Mapping of

concepts &

projects

Creative

Science

Project

Advisor

Web/text

sources for

science projects

STEM

curriculum,

textbooks etc

Materials

at hand

Image

Recognition

Hand Inputs

Camera Suggested

science project

experience

Dialog

system for

Q&A

Learner’s

background

Model

Learner’s

past test

results Questions

(answers) to

guide

experience

Observing

experienceVideo

Comprehension

Videos

Deep Learning-based

Extract concept graphs from next generation science standard (http://www.nextgenscience.org/)

• Five blocks of information:• Performance Expectations

• Science and Engineering Practices

• Disciplinary Core Ideas

• Crosscutting Concepts

• Connections

Paradigm shift for cognitive application development• Traditional programming approaches failed to deliver cognitive

applications for decades

• With the wide adoption of machine learning (deep learning), the core of application development has shifted to model training (including model customization)

• Experimentation with a large amount of data is on the critical path of application development

• The nature of functional verification, performance tuning, and debugging is fundamentally different

Cognitive Application Builder (CAB)

• CAB: A language, compiler, and runtime for easy development of cognitive applications

• System-aware to exploit accelerators and efficient communication• Introspection for debugging and performance evaluation• Workflow optimization and orchestration for system-level performance• Decentralized application architecture for scalability, composability, testing,

and development

A system-level challengeWorkflow description

Innovative AI techniquesHigh-performance, scalable,

robust applications

CELA as a driving use case for CAB

• CAB will simplify component connection, workflow description, and iterative development

Correlate

User

Video Processing

Learner’s

background

Model

Science Project

Sequence

Inference Engine

Dialogue

Q&A

System

STEM

Concepts

and Projects

1 2

3

4

CELA Time Warp for Efficiency

• CAB automatically transforms workflows for high-performance execution

Correlate

User

Video Processing

Learner’s

background

Model

Science Project

Sequence

Inference Engine

Dialogue

Q&A

System

STEM

Concepts

and Projects

1 2

1

Previous Frame

1

C3SR Experimental Heterogeneous Infrastructure

2 x P8 Minsky with NVLink GPUs DGX-1

www.ptopenlab.comSuperVessel

Watson developer cloud

4 x P8 Tuleta (S824L)

FPGAAFU

PSL

FPGA CAPI over PCIe

ConTutto over DMI

Courtesy: Jinjun Xiong, IBM

Workload acceleration research at C3SR based on CAB/TANGRAM Software Synthesis • Focus on impactful cognitive workloads for acceleration

• Matrix factorization on GPU• Long-term Recurrent Convolutional Network acceleration• ResNet inference acceleration• Neuron Machine Translation acceleration• DNN inference acceleration• Graph analytic acceleration

• In discussion with other CHN centers to collect performance critical cognitive workloads

• Plan to deliver a set of cognitive benchmarks optimized for OpenPOWER

Matrix factorization: one of key workloads

Predict missing ratings

Group similar users/items

Match query and document In machine learning and HPC applications

MatrixFactorizationLink prediction

Vertices clustering

Latent semantic model

Word embedding as input to DNN

Recommender systems

Complex network

Web search

Natural language processing

Tensor decomposition

Model compression

Embedding layer

Deep learning

Ratings (R)

n items

mu

sers

* * **

*

*

*

*

≈

x

Use

rs

items

T

xT

u

vX

f

f

R

cuMF acceleration

• cuMF formulation: factorize matrix R into

• while minimizing the empirical lost

• Connect cuMF to Spark MLlib via JNI

• cuMF_ALS @4 Maxwell ($2.5/hour)≈ 10x speedup over SparkALS @50 nodes≈ 1% of SparkALS’s cost ($0.53/hour/node)

• Open source @ http://github.com/cuMF/

• Demoed at SC’16 and GTC’16 on Minsky

• Presented to Jen-Hsun Huang on Feb 1, 2017

• cuMF_ALS w/ FP16 on Maxwell and Pascal• LIBMF: 1 CPU w/ 40 threads• NOMAD

• 32 nodes for Netflix and Yahoo• 2-10x as fast

Conclusion and Outlook

• Applications have very large appetite for more computing power• Both larger scale clusters and faster devices

• Heterogeneity has become the norm for all hardware systems• HPC community are currently seeing about 2-3x application speedup

• Recent positive spiral between deep learning and GPU computing

• Cognitive Computing Systems Research• Game changing applications (CELA)

• Next generation heterogeneous system – democratizing compute and bandwidth (100x)

• High productivity development with software synthesis (CAB)