Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf ·...
Transcript of Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf ·...
![Page 1: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/1.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Introduction to TensorFlow Internals
Guangcong Liu
Software Architect@ZTE
2017-11-16
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 1 / 44
![Page 2: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/2.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Contents
1 Architecture Overview
2 Programming Model
3 Execution Model
4 Model Training
5 Bibliography
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 2 / 44
![Page 3: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/3.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Architecture Overview1 System Architecture2 Design Principles3 Graph Transformation
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 3 / 44
![Page 4: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/4.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
System Architecture
System Architecture
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 4 / 44
![Page 5: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/5.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Design Principles
Design Principles
Deferred Execution:The construction and execution of graph areseparated, and the graph execution is delayed.
Primitive OP:OP is the basic computation unit.
Abstract Accelerator:Support CPU, GPU, and ASIC.
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 5 / 44
![Page 6: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/6.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Graph Transformation
Graph Construction
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 6 / 44
![Page 7: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/7.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Graph Transformation
Graph Exection
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 7 / 44
![Page 8: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/8.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Graph Transformation
Split Graph
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 8 / 44
![Page 9: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/9.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Graph Transformation
Register Graph
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 9 / 44
![Page 10: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/10.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Graph Transformation
Run Graph
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 10 / 44
![Page 11: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/11.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Programming Model1 Dataflow Graph2 Variable3 Session4 Graph Construction & Exection
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 11 / 44
![Page 12: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/12.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Dataflow Graph
Graph = Set{OP}+ Set{Tensor}
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 12 / 44
![Page 13: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/13.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Dataflow Graph
OP: Abstract Computation
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 13 / 44
![Page 14: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/14.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Dataflow Graph
Tensor: Dataflow
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 14 / 44
![Page 15: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/15.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Variable
Initialization Model
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 15 / 44
![Page 16: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/16.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Variable
Initialization Dependency
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 16 / 44
![Page 17: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/17.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Session
Life Cycle: Python
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 17 / 44
![Page 18: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/18.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Session
Life Cycle: C++
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 18 / 44
![Page 19: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/19.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Graph Construction & Exection
Graph Construction & Serialization
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 19 / 44
![Page 20: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/20.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Graph Construction & Exection
Example: OP Constructor
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 20 / 44
![Page 21: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/21.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Graph Construction & Exection
Example: Create OP
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 21 / 44
![Page 22: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/22.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Graph Construction & Exection
Example: Graph Construction
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 22 / 44
![Page 23: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/23.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Execution Model1 Execution Model2 Distributed Example
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 23 / 44
![Page 24: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/24.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Execution Model
Local Runtime
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 24 / 44
![Page 25: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/25.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Execution Model
Distributed Runtime
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 25 / 44
![Page 26: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/26.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Client Session
Create ClientSession
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 26 / 44
![Page 27: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/27.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Client Session
Polymorphism Creation
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 27 / 44
![Page 28: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/28.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Client Session
Create MasterSession
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 28 / 44
![Page 29: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/29.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Client Session
MasterSession Model
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 29 / 44
![Page 30: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/30.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Run Step
Split Graph by Task)
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 30 / 44
![Page 31: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/31.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Run Step
Split Graph by Device
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 31 / 44
![Page 32: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/32.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Example
Split Graph
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 32 / 44
![Page 33: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/33.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Example
Receive Tensor
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 33 / 44
![Page 34: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/34.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Training Model1 Compute Gradients2 Apply Gradients3 Training Workflow
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 34 / 44
![Page 35: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/35.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Compute Gradients
Optimizer: Compute Gradients
�class Optimizer(object):
def minimize(self, loss, var_list=None, global_step=None):
grads_and_vars = self.compute_gradients(
loss, var_list=var_list)
return self.apply_gradients(
grads_and_vars,
global_step=global_step)
def compute_gradients(loss, var_list):
grads = gradients(loss, var_list, grad)
return list(zip(grads, var_list))
def gradients(loss, var_list, grads=1):
ops_and_grads = {}
for op in reversed_graph(loss).topological_sort():
grad = op.grad_fn(grad)
ops_and_grads[op] = grad
return [ops_and_grads.get(var) for var in var_list] � �Guangcong Liu (ZTE) TensorFlow Internals 2017.11 35 / 44
![Page 36: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/36.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Compute Gradients
Gradient Function
�@ops.RegisterGradient("op_name")
def grad_func(op, grad):
"""construct gradient subgraph for an op type.
Returns:
A list of gradients, one per each input of op.
"""
return cons_grad_subgraph(op, grad) � �(y1, y2, ..., ym) = f (x1, x2, ..., xn)
(∂L/∂x1, ∂L/∂x2, ..., ∂L/∂xn) = g(x1, x2, ..., xn; ∂L/∂y1, ∂L/∂y2, ..., ∂L/∂yn
)
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 36 / 44
![Page 37: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/37.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Compute Gradients
Example:Square
�@ops.RegisterGradient("Square")
def SquareGrad(op, grad):
x = op.inputs[0]
with ops.control_dependencies([grad.op]):
return grad * (2.0 * x) � �
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 37 / 44
![Page 38: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/38.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Apply Gradients
Apply Gradients
�def apply_gradients(grads_and_vars, learning_rate):
for (grad, var) in grads_and_vars:
apply_gradient_descent(learning_rate, grad, var) � �
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 38 / 44
![Page 39: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/39.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Training Workflow
Critical Path: RunStep
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 39 / 44
![Page 40: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/40.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Training Workflow
Distributed Initialization
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 40 / 44
![Page 41: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/41.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Bibliography
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 41 / 44
![Page 42: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/42.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Bibliography
Papers
TensorFlow: Large-Scale Machine Learning on HeterogeneousDistributed Systems, Google Inc.
TensorFlow: A System for Large-Scale Machine Learning, GoogleInc.
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 42 / 44
![Page 43: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/43.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Acknowledgments
Q&A
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 43 / 44
![Page 44: Introduction to TensorFlow Internalsbos.itdks.com/08ec88d5acc74cd4a95064d1efc7397e.pdf · LifeCycle:Python GuangcongLiu (ZTE) TensorFlowInternals 2017.11 17/44..... ArchitectureOverview](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec47bf64978dd3a4e4de1f7/html5/thumbnails/44.jpg)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Architecture Overview Programming Model Execution Model Model Training Bibliography
Acknowledgments
Thanks
Guangcong Liu (ZTE) TensorFlow Internals 2017.11 44 / 44