Deep dive into deeplearn.js
-
Upload
kai-sasaki -
Category
Software
-
view
1.051 -
download
0
Transcript of Deep dive into deeplearn.js
![Page 1: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/1.jpg)
Deep Dive into deeplearn.jsKai Sasaki
Treasure Data Inc
![Page 2: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/2.jpg)
About me
Kai Sasaki - 佐々木 海 (@Lewuathe)
Software Engineer at Treasure Data Apache Hivemall Committer Hadoop, Spark, deeplearnjs Contributor
![Page 3: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/3.jpg)
What is deeplearn.js?
deeplearn.js is a deep learning library running on your browser accelerated by WebGL. Mainly developed by Google PAIR project.
https://research.googleblog.com/2017/08/harness-power-of-machine-learning-in.html
![Page 5: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/5.jpg)
Similar Products
WebDNN - https://mil-tokyo.github.io/webdnn/ Keras.js - https://github.com/transcranial/keras-js
![Page 6: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/6.jpg)
Technology
- Written in TypeScript - Available in TypeScript and ES6 project - Accelerated by WebGL - Tested on Chrome and Firefox - Can run both training and inference
https://www.npmjs.com/~types
![Page 7: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/7.jpg)
Architecture
Graph: Representing computation data flow much like graph in TensorFlow etc Op: A graph node that represents a computation on tensor NDArray: Real data copied from/to CPU and GPU backed by Typed Array
![Page 8: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/8.jpg)
Computation Graph
![Page 9: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/9.jpg)
Graph
Node
Node
Node Tensor
Tensor
Tensor
![Page 10: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/10.jpg)
Graph
Node
Node
Node
Node
Node Tensor
Tensor
Tensor
![Page 11: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/11.jpg)
Graph
Node
Node
Node
Node
Node
Node
Tensor
Tensor
Tensor
Tensor
Tensor
add
multiply
conv2d
reluetc
![Page 12: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/12.jpg)
Operation
Op
Op
Op
Op
Op
Op
NDArray
NDArray
NDArray
NDArray
NDArray
![Page 13: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/13.jpg)
Kernel Implementation
![Page 14: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/14.jpg)
math
Web Browser
WebGL
![Page 15: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/15.jpg)
math
NDArrayMathCPUNDArrayMathGPU
Web Browser
WebGL
![Page 16: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/16.jpg)
math
math
NDArrayMathCPUNDArrayMathGPU
Web Browser
WebGL
kernelNDArray
![Page 17: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/17.jpg)
math
math
NDArrayMathCPUNDArrayMathGPU
Web Browser
WebGL
kernelNDArray
Conv2d softmax etc…
Graph
matmul
![Page 18: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/18.jpg)
Kernel Exampleimport {Array1D, NDArrayMathGPU, Scalar} from 'deeplearn';
const math = new NDArrayMathGPU();const a = Array1D.new([1, 2, 3]);const b = Scalar.new(2);
const result = math.add(a, b);
// Option 1: With async/await.// Caveat: in non-Chrome browsers you need to put this in an async function.console.log(await result.data()); // Float32Array([3, 4, 5])
// Option 2: With a Promise.result.data().then(data => console.log(data));
// Option 3: Synchronous download of data.// This is simpler, but blocks the UI until the GPU is done.console.log(result.dataSync());
![Page 19: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/19.jpg)
TensorFlow like Exampleconst graph = new Graph();// Make a new input in the graph, called 'x', with shape [] (a Scalar).const x: Tensor = graph.placeholder('x', []);// Make new variables in the graph, 'a', 'b', 'c' with shape [] and random// initial values.const a: Tensor = graph.variable('a', Scalar.new(Math.random()));const b: Tensor = graph.variable('b', Scalar.new(Math.random()));const c: Tensor = graph.variable('c', Scalar.new(Math.random()));
//…
// At this point the graph is set up, but has not yet been evaluated.// **deeplearn.js** needs a Session object to evaluate a graph.const math = new NDArrayMathGPU();const session = new Session(graph, math);
![Page 20: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/20.jpg)
TensorFlow Integration
![Page 21: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/21.jpg)
Port TensorFlow model
deeplearn.js can import TensorFlow model from checkpoint. Need to port the weights from checkpoint files.
https://deeplearnjs.org/demos/mnist/mnist.html
![Page 22: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/22.jpg)
Porting weights
$ python scripts/dump_checkpoints/dump_checkpoint_vars.py \ —model_type=tensorflow \ —output_dir=demos/mnist/ \ —checkpoint_file=/tmp/tensorflow/mnist/logs/fully_connected_feed/model.ckpt-1999
# Will save a lot of files # (one file per variable and manifest.json)
![Page 23: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/23.jpg)
Porting weights
import {CheckpointLoader} from 'deeplearn';
const varLoader = new CheckpointLoader('.');varLoader.getAllVariables().then(async vars => { const math = new NDArrayMathGPU(); const hidden1W = vars[‘hidden1/weights’] as Array2D; const hidden1b = vars[‘hidden1/biases’] as Array1D; //…
}
![Page 24: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/24.jpg)
GPU Acceleration
![Page 25: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/25.jpg)
WebGL
WebGL (Web Graphics Library) is a JavaScript API for rendering interactive 3D and 2D graphics within any compatible web browser without the use of plug-ins.
- Mozilla Developer Network
![Page 26: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/26.jpg)
WebGL
- Provides the standard API - Implemented by major web browsers such as Chrome, Firefox, Safari and Opera - Hardware Agnostic
![Page 27: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/27.jpg)
WebGL Graphic Pipeline
https://www.tutorialspoint.com/webgl/webgl_graphics_pipeline.htm
![Page 28: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/28.jpg)
WebGL Graphic Pipeline
http://math.hws.edu/graphicsbook/c6/s4.html
![Page 29: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/29.jpg)
WebGL
1. deeplearnjs compiles GLSL shader 2. Send input NDArray as texture into GPU 3. Run a fragment shader 4. Fetch result NDArray from GPU
![Page 30: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/30.jpg)
1. Compile Shader
float unaryOperation(float x) {return (x < 0.0) ? 0.0 : x;
}
void main() {float x = getAAtCoords(); // Sample a value from a texturefloat y = unaryOperation(x);
setOutput(y);}
![Page 31: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/31.jpg)
2. Upload NDArray to GPU
gl.bindTexture(gl.TEXTURE_2D, texture);gl.texSubImage2D( gl.TEXTURE_2D, 0, 0, 0, width, height, textureFormat, getTextureType(gl), data);
// Texture format is RGBA// Texture type is gl.UNSIGNED_BYTE or gl.FLOAT
gl.bindTexture(gl.TEXTURE_2D, null);
![Page 32: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/32.jpg)
3. Run shader program
gl.bindBuffer(gl.ARRAY_BUFFER, buffer);
// Map buffer data to texture coordinationgl.vertexAttribPointer( loc, arrayEntriesPerItem, gl.FLOAT, false, itemStrideInBytes, itemOffsetInBytes);
// Draw somethinggl.drawElements(gl.TRIANGLES, 6, gl.UNSIGNED_SHORT, 0);
![Page 33: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/33.jpg)
4. Download the result
getBufferSubDataAsyncExtension.getBufferSubDataAsync( gl2.PIXEL_PACK_BUFFER, 0, downloadTarget);
// Download as Typed Array -> NDArray// Typed Array is backed by raw buffer memory (ArrayBuffer) // accessed through views (e.g. Float32Array)
https://www.khronos.org/registry/webgl/extensions/WEBGL_get_buffer_sub_data_async/
![Page 34: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/34.jpg)
Future Works
![Page 35: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/35.jpg)
Automatic TensorFlow to deeplearn.js
Direct importing from TensorFlow GraphDef format makes model importing easier.
https://deeplearnjs.org/docs/roadmap.html
![Page 36: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/36.jpg)
Eager Mode
Eager execution mode enables us to run computation without Graph definition.
Similar to TensorFlow eager mode.
![Page 37: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/37.jpg)
Decoupling NDArray from storage mechanism
NDArray is now tightly coupled with storage layer (GPU texture)
Tracking NDArray to avoid memory leak often becomes hard work.
![Page 38: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/38.jpg)
Great Demos
One of the reason we use deeplearn.js is that it enables us to create interesting application by using deep learning easily.
deeplearn.js is now enhancing demos to show the potential of deep learning on Web Browser.
![Page 39: Deep dive into deeplearn.js](https://reader033.fdocuments.net/reader033/viewer/2022050614/5a64787c7f8b9a27568b4615/html5/thumbnails/39.jpg)
Patches Welcome!