Università degli Studi di Pavia Deep Learning and TensorFlow · Deep Learning and TensorFlow...

[1]Deep Learning and TensorFlow – Episode 4

Deep Learningand TensorFlowEpisode 4TensorFlow Basics

Part 1

Università degli Studi di Pavia

Summary

Main References

What is TensorFlow?

Why TensorFlow? (alternatives?)

Why TensorFlow?

Example application: classify skin cancer

Example application: Neural Style Transfer

The structure of a TensorFlow program

𝑦 = (𝑎 ∗ 𝑏)𝑚𝑢𝑙

+ (𝑎 + 𝑏)𝑎𝑑𝑑

𝑎𝑑𝑑

Graphs: an intuitive view

Graphs: an intuitive way (with matrices)

1 23 4

5 67 8

1 23 4

5 67 8

1 23 4

5 1221 32

6 810 12

11 2031 44

What is Tensorflow?

Sessions: computing flow graphs

The first TensorFlow program

import tensorflow as tf

y = tf.multiply(a, b) + tf.add(a, b)

print(y)

>> Tensor("Add:0", shape=(), dtype=int32)

print(y)

>> Tensor("Add:0", shape=(), dtype=int32)

Sessions

import tensorflow as tfy = tf.add(3, 7)sess = tf.Session()print(sess.run(y))sess.close()

Sessions

import tensorflow as tfy = tf.add(3, 7)sess = tf.Session()print(sess.run(y))sess.close()

with tf.Session() as sess:

print(sess.run(y))

A graph in TensorFlow

print(sess.run(y))

A graph in TensorFlow (with matrices)

a = tf.constant([1,2,3,4], shape=(2,2))

b = tf.constant([5,6,7,8], shape=(2,2))

print(sess.run(y))

>> [[11 20]

[31 44]]

print(sess.run(y))

>> [[11 20]

[31 44]]

tf.constant

Why graphs?

Visualization with TensorBoard

a = tf.constant(3, name='a') # equivalent to a = 3

b = tf.constant(7, name='b')

y = tf.multiply(a, b, name='mul_op') + tf.add(a, b, name='add_op')

print(sess.run(y))

tf.add

print(sess.run(y))

tf.add

y = tf.add(tf.multiply(a, b, name='mul_op'), tf.add(a, b, name='add_op'))

writer = tf.summary.FileWriter('./graphs', tf.get_default_graph())

print(sess.run(y))

writer.close() # close the writer when you’re done using it

writer = tf.summary.FileWriter('./graphs', tf.get_default_graph())

print(sess.run(y))

writer.close() # close the writer when you’re done using it

with tf.Session() as sess:writer = tf.summary.FileWriter('./graphs', sess.graph)print(sess.run(y))

Run it!

$ python3 [yourprogram].py$ tensorboard --logdir="./graphs" --port 6006

Run it!

$ python3 [yourprogram].py$ tensorboard --logdir="./graphs" --port 6006

http://localhost:6006/

The same graph in Tensorboard

a = tf.constant(3, name='a')b = tf.constant(7, name='b')y = tf.multiply(a, b, name='mul_op') +

tf.add(a, b, name='add_op')

The same graph in Tensorboard

x = 2y = 3

add_op = tf.add(x, y)mul_op = tf.multiply(x, y)

useless = tf.multiply(x, add_op,name='useless')

pow_op = tf.pow(add_op, mul_op)

with tf.Session() as sess:z = sess.run(pow_op)

Subgraphs

x = 2y = 3

Subgraphs

x = 2y = 3

Subgraphs

pow_oppow_op uselessuseless

Subgraphs

x = 2y = 3

with tf.Session() as sess:z, not_useless =

sess.run([useless, pow_op])

Subgraphs

x = 2y = 3

with tf.Session() as sess:z, not_useless =

sess.run([useless, pow_op])

sess.run

TensorFlow and GPUs

Distributed computation

# Creates a graph.

with tf.device('/gpu:2'):

a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='a')

b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='b')

y = tf.multiply(a, b)

# Creates a session with log_device_placement set to True.

sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

# Runs the op.

print(sess.run(y))

Protocol buffers

print(tf.get_default_graph().as_graph_def())

Protocol buffers

node {name: "mul_op/a"op: "Const"attr {key: "dtype"value { type: DT_INT32 }

}attr {key: "value"value { tensor

{ dtype: DT_INT32tensor_shape {}int_val: 7 }

Protocol buffers

node {name: "mul_op"op: "Mul"input: "mul_op/a"input: "mull_op/b"attr {key: "T"value {type: DT_INT32

Protocol buffers

node {name: "add"op: "Add"input: "mul_op"input: "add_op"attr {key: "T"value {type: DT_INT32

Tensors, constants and ops

What is a tensor?

Tensors as objects tf.Tensor

float32 int32 string

r = tf.rank(cat_image) # r is a tensors = tf.shape(cat_image) # s is a tensor# after evaluation r will be 3# s will be (28, 28, 3)

Tensors Containers

Tensorflow Operations

Graph representations of tensors and containers

Constants tf.constant(value,

dtype=None,shape=None,name='Const',verify_shape=False)

# constant of 1d tensor (vector)a = tf.constant([2, 2], name="vector")

# constant of 2x2 tensor (matrix)b = tf.constant([[0, 1], [2, 3]], name="matrix")

x = tf.multiply(a, b, name='mul_op')

>> x = [[0 2][4 6]]

# constant of 1d tensor (vector)a = tf.constant([2, 2], name="vector")

# constant of 2x2 tensor (matrix)b = tf.constant([[0, 1], [2, 3]], name="matrix")

x = tf.multiply(a, b, name='mul_op')

>> x = [[0 2][4 6]]

Constants filled with specific valuestf.zeros(shape, dtype=tf.float32, name=None)

tf.zeros_like(input_tensor, dtype=None, name=None, optimize=True)

tf.ones(shape, dtype=tf.float32, name=None)

tf.ones_like (input_tensor, dtype=None, name=None, optimize=True)

tf.fill(shape, value, name=None)

tf.zeros([2, 3], tf.int32)

tf.ones([2, 3], tf.int32)

0 0 00 0 0

1 1 11 1 1

# input_tensor is [[0, 1], [2, 3], [4, 5]]

tf.zeros_like(input_tensor)

tf.ones_like(input_tensor)

0 0 00 0 0

1 11 11 1

1 1 11 1 1

0 00 00 0

# input_tensor is [[0, 1], [2, 3], [4, 5]]

tf.zeros_like(input_tensor)

tf.ones_like(input_tensor)

tf.fill([2, 3], 8)

0 0 00 0 0

1 11 11 1

8 88 88 8

1 1 11 1 1

0 00 00 0

Constants as sequences

tf.lin_space(start, stop, num, name=None)

tf.lin_space(10.0, 13.0, 4) ==> [10. 11. 12. 13.]

tf.range(start, limit=None, delta=1, dtype=None, name='range')

tf.range(3, 18, 3) ==> [3 6 9 12 15]

tf.range(5) ==> [0 1 2 3 4]

Constants as sequences

tf.lin_space(start, stop, num, name=None)

tf.lin_space(10.0, 13.0, 4) ==> [10. 11. 12. 13.]

tf.range(start, limit=None, delta=1, dtype=None, name='range')

tf.range(3, 18, 3) ==> [3 6 9 12 15]

tf.range(5) ==> [0 1 2 3 4]

tf.range

for _ in tf.range(4):# TypeError

Randomly generated Constants

tf.random_normal

tf.truncated_normal

tf.random_uniform

tf.random_shuffle

tf.random_crop

tf.multinomial

tf.random_gamma

Randomly generated Constants

tf.random_normal

tf.truncated_normal

tf.random_uniform

tf.random_shuffle

tf.random_crop

tf.multinomial

tf.random_gamma

tf.set_random_seed(seed)

Data types

t_0 = 19 # scalars are treated like 0-d tensorstf.zeros_like(t_0) # ==> constant 0-d tensor with value 0tf.ones_like(t_0) # ==> constant 0-d tensor with value 1

t_1 = ["apple", "peach", "grape"] # 1-d arrays are treated like 1-d tensorstf.zeros_like(t_1) # ==> ['' '' '']tf.ones_like(t_1) # ==> TypeError

t_2 = [[True, False, False],[False, False, True],[False, True, False]] # 2-d arrays are treated like 2-d tensors

tf.zeros_like(t_2) # ==> 3x3 tensor, all elements are Falsetf.ones_like(t_2) # ==> 3x3 tensor, all elements are True

Data types

Variables

Shortcomings of constants

attr {key: "value"value {

tensor {dtype: DT_FLOAT

tensor_shape {dim { size: 2 }

}tensor_content:

"\000\000\200?\000\000\000@"}

my_const = tf.constant([1.0, 2.0], name="my_const")with tf.Session() as sess:

print(sess.graph.as_graph_def())

attr {key: "value"value {

tensor {dtype: DT_FLOAT

tensor_shape {dim { size: 2 }

}tensor_content:

"\000\000\200?\000\000\000@"}

my_const = tf.constant([1.0, 2.0], name="my_const")with tf.Session() as sess:

print(sess.graph.as_graph_def())

Variables

tf.Variable

Variables

tf.Variable

# create variables with tf.Variable

s = tf.Variable(2, name="scalar") m = tf.Variable([[0, 1], [2, 3]], name="matrix") W = tf.Variable(tf.zeros([784,10]))

Variables

tf.Variable

# create variables with tf.get_variable

s = tf.get_variable("scalar", initializer=tf.constant(2)) m = tf.get_variable("matrix", initializer=tf.constant([[0, 1], [2, 3]]))W = tf.get_variable("big_matrix", shape=(784, 10),

initializer=tf.zeros_initializer())

Variables

tf.Variable

# create variables with tf.get_variable

s = tf.get_variable("scalar", initializer=tf.constant(2)) m = tf.get_variable("matrix", initializer=tf.constant([[0, 1], [2, 3]]))W = tf.get_variable("big_matrix", shape=(784, 10),

initializer=tf.zeros_initializer())

get_variable

Variables operator tf.Variable

x = tf.Variable(…)

x.initializer # init opx.value # read op

# and more..

W = tf.get_variable("big_matrix", shape=(784, 10),initializer=tf.zeros_initializer())

print(sess.run(W))

# and more..

W = tf.get_variable("big_matrix", shape=(784, 10),initializer=tf.zeros_initializer())

print(sess.run(W))

>> FailedPreconditionError: Attempting to use uninitialized value Variable

Variables initialization

with tf.Session() as sess:sess.run(tf.global_variables_initializer())

with tf.Session() as sess:sess.run(tf.variables_initializer([a, b]))

W = tf.Variable(tf.zeros([784,10]))

with tf.Session() as sess:sess.run(W.initializer)

# W is a random 700 x 100 variable object

W = tf.Variable(tf.truncated_normal([700, 10]))

sess.run(W.initializer)

Variables evaluation

print(W)

>> Tensor("Variable/read:0", shape=(700, 10), dtype=float32)

print(W)

print(W.eval())

print(sess.run(W))print(W.eval())

>> [[-0.76781619 -0.67020458 1.15333688 ..., -0.98434633 -1.25692499-0.90904623]

..., [ 0.19076447 -0.62968266 -1.97970271 ..., -1.48389161 0.681706431.46369624]]

print(sess.run(W))print(W.eval())

Variable Assignments

W = tf.Variable(10)

W.assign(100)

print(W.eval())

W = tf.Variable(10)

W.assign(100)

print(W.eval())

# >> 10

W = tf.Variable(10)

W.assign(100)

print(W.eval())

# >> 10

W.assign(100)

W = tf.Variable(10)

W.assign(100)

print(W.eval())

# >> 10

W = tf.Variable(10)

assign_op = W.assign(100)

sess.run(assign_op)

print(W.eval())

W.assign(100)

W = tf.Variable(10)

W.assign(100)

print(W.eval())

# >> 10

W = tf.Variable(10)

assign_op = W.assign(100)

sess.run(assign_op)

print(W.eval())

# >> 100

W.assign(100)

# create a variable whose original value is 2

my_var = tf.Variable(2, name="my_var")

# assign a*2 to a and call that op a_times_two

my_var_times_two = my_var.assign(2*my_var)

2*my_var my_varmy_var_times_two

sess.run(my_var.initializer)

sess.run(my_var_times_two) # >> the value of my_var now is 4

Variables in multiple sessions

W = tf.Variable(10)

sess1 = tf.Session()

sess1.run(W.initializer)

sess1.close()

sess2.close()

W = tf.Variable(10)

print(sess1.run(W.assign_add(10)))

print(sess2.run(W.assign_sub(2)))

sess1.close()

sess2.close()

W = tf.Variable(10)

sess1.close()

sess2.close()

# >> 20# >> 8

W = tf.Variable(10)

sess1.close()

sess2.close()

# >> 20# >> 8

W = tf.Variable(10)

sess1.close()

sess2.close()

# >> 20# >> 8

# >> 120# >> 108

Placeholders

Placeholder and feed dicts tf.placeholder(dtype, shape=None, name=None)

# create a placeholder for a vector of 3 elements, type tf.float32

a = tf.placeholder(tf.float32, shape=[3])

b = tf.constant([5, 5, 5], tf.float32)

# use the placeholder as you would a constant or a variable

c = a + b # short for tf.add(a, b)

with tf.Session() as sess:print(sess.run(c))

>> InvalidArgumentError: a doesn’t have an>> actual value

with tf.Session() as sess:print(sess.run(c))

with tf.Session() as sess:print(sess.run(c, feed_dict={a: [1, 2, 3]}))

>> [6, 7, 8]

Feed dicts

print(sess.run(some_op, feed_dict={a: [1, 2, 3]}))

Feed dicts

a = tf.placeholder(tf.float32, shape=None)

Feed dicts

for a_value in list_of_values_for_a:

print(sess.run(some_op, {a: a_value}))

Feed dicts

a = tf.add(2, 5)

b = tf.multiply(a, 3)

# compute the value of b given a is 15

sess.run(b, feed_dict={a: 15})

Feed dicts

a = tf.add(2, 5)

Feed dicts

a = tf.add(2, 5)

tf.Graph.is_feedable(tensor

A common mistake

Lazy loading

x = tf.Variable(10, name='x')

y = tf.Variable(20, name='y')

z = tf.add(x, y)

writer = tf.summary.FileWriter('./graphs/normal_loading', tf.get_default_graph())

sess.run(tf.global_variables_initializer())

for _ in range(10):

sess.run(z)

writer.close()

Lazy loading

z = tf.add(x, y)

for _ in range(10):

sess.run(z)

writer.close()

Lazy loading

for _ in range(10):

sess.run(tf.add(x, y))

writer.close()

Lazy loading

for _ in range(10):

sess.run(tf.add(x, y))

writer.close()

Lazy loading

node {

op: "Add"

input: "x/read"

input: "y/read"

attr {

key: "T"

value {

type: DT_INT32

Lazy loading

node {

op: "Add"

input: "x/read"

input: "y/read"

attr {

key: "T"

value {

type: DT_INT32

node {

op: "Add"

node {

op: "Add"

Lazy loading

node {

op: "Add"

input: "x/read"

input: "y/read"

attr {

key: "T"

value {

type: DT_INT32

node {

op: "Add"

node {

op: "Add"

Lazy loading

@property

Next step…build the first machine learning model!

The first machine learning model in TF

Now's the time for part 2!

Università degli Studi di Pavia Deep Learning and TensorFlow · Deep Learning and TensorFlow...

Documents

Transcript of Università degli Studi di Pavia Deep Learning and TensorFlow · Deep Learning and TensorFlow...

TensorFlow Deep Learning Projects - iedu.us€¦ · TensorFlow Deep Learning Projects 10 real-world projects on computer vision, machine translation, chatbots, and reinforcement learning

TensorFlow Serving, Deep Learning on Mobile, and Deeplearning4j on the JVM - Enterprise Deep Learning.pdf

Learning in TensorFlow Deep ReinforcementLearning in TensorFlow Danijar Hafner · Stanford CS 20SI · 2017-03-10. Gu16. Barron16. Hafner16. Repeat until end of episode: Most methods

Deep Learning and TensorFlow - PaviaDeep Learning: a theoretical introduction –Episode 2 [1] Deep Learning and TensorFlow Episode 2 The Quest for Deeper Networks Università degli

Deep Learning: R with Keras and TensorFlow

Deep Learning and TensorFlow - Pavia€¦ · Deep Learning and TensorFlow–Episode 1 [1] ... Artificial Neural Networks Università degli Studi di Pavia. Deep Learning and TensorFlow–Episode

Deep Learning, Keras, and TensorFlow

Early Stabilizing Feature Importance for TensorFlow Deep ... 2017-v2-final.pdf · Early Stabilizing Feature Importance for TensorFlow Deep Neural Networks IJCNN 2017, May 18, 2017

NVIDIA DEEP LEARNING INSTITUTE · >>Deep Learning Workflows with TensorFlow, MXNet, and NVIDIA-Docker >>Image Segmentation with TensorFlow >>Image Classification with Microsoft Cognitive

Introduction to tensorflow · Introduction to tensorflow. Why do you need a deep learning framework? Speed: ... - Deep-learning/Machine learning specific - Simpler to do common tasks.

Deep Water - Bringing Tensorflow, Caffe, Mxnet to H2O

Deep-Learning-TensorFlow Documentation - Read the · PDF fileDeep-Learning-TensorFlow Documentation, Release stable This repository is a collection of various Deep Learning algorithms

- Deep learning hardware - PyTorch and TensorFlow

RSTensorFlow: GPU Enabled TensorFlow for Deep … GPU Enabled TensorFlow for Deep Learning on Commodity Android Devices ... lar Snapdragon Qualcomm SoC.

Deep Learning Lab11: TensorFlow 101€¦ · TensorFlow 4 Read and Preprocess Data tf.keras Premade Estimators TensorFlow Hub Distribution Strategy GPU CPU TPU SavedModel TensorFlow

TENSORFLOW - ajou.ac.krcvml.ajou.ac.kr/wiki/images/5/55/Ch4_1_TensorFlow.pdf · 2017-01-12 · What is TensorFlow? •TensorFlow is a deep learning library open-sourced by Google.

Introdução ao Deep Learning com o TensorFlow

Deep-Learning-TensorFlow Documentation - Read the · PDF fileDeep-Learning-TensorFlow Documentation, Release latest Thisprojectis a collection of various Deep Learning algorithms implemented

Large Scale Deep Learning with TensorFlow

Deep Learning with Keras and TensorFlow