Università degli Studi di Pavia Deep Learning and TensorFlow · Deep Learning and TensorFlow...

[1]Deep Learning and TensorFlow – Episode 4

Deep Learningand TensorFlowEpisode 4TensorFlow Basics

Part 1

Università degli Studi di Pavia

http://vision.unipv.it/AI/AIRG.html


Summary


Main References

https://www.tensorflow.org/programmers_guide/


What is TensorFlow?


Why TensorFlow? (alternatives?)


Why TensorFlow?


Example application: classify skin cancer


Example application: Neural Style Transfer


The structure of a TensorFlow program

𝑦 = (𝑎 ∗ 𝑏)𝑚𝑢𝑙

+ (𝑎 + 𝑏)𝑎𝑑𝑑

𝑎𝑑𝑑


Graphs: an intuitive view


Graphs: an intuitive way (with matrices)



1 23 4

5 67 8



1 23 4

5 67 8

5 67 8

1 23 4



5 1221 32

6 810 12



11 2031 44


What is Tensorflow?


Sessions: computing flow graphs


The first TensorFlow program

import tensorflow as tf

a = 3

b = 7

y = tf.multiply(a, b) + tf.add(a, b)




a = 3

b = 7





a = 3

b = 7


print(y)




a = 3

b = 7


print(y)

>> Tensor("Add:0", shape=(), dtype=int32)


Sessions


Sessions

sess

y


Sessions

sess

y

import tensorflow as tfy = tf.add(3, 7)sess = tf.Session()print(sess.run(y))sess.close()


Sessions

sess

y

import tensorflow as tfy = tf.add(3, 7)sess = tf.Session()print(sess.run(y))sess.close()

with




a = 3

b = 7





a = 3

b = 7


with tf.Session() as sess:

print(sess.run(y))




a = 3

b = 7



print(sess.run(y))

>> 31


A graph in TensorFlow


a = 3

b = 7





a = 3

b = 7



print(sess.run(y))




a = 3

b = 7



print(sess.run(y))

>> 31


A graph in TensorFlow (with matrices)


a = tf.constant([1,2,3,4], shape=(2,2))

b = tf.constant([5,6,7,8], shape=(2,2))









print(sess.run(y))








print(sess.run(y))

>> [[11 20]

[31 44]]








print(sess.run(y))

>> [[11 20]

[31 44]]

tf.constant

shape


Why graphs?


Visualization with TensorBoard


a = tf.constant(3, name='a') # equivalent to a = 3

b = tf.constant(7, name='b')

y = tf.multiply(a, b, name='mul_op') + tf.add(a, b, name='add_op')


print(sess.run(y))








print(sess.run(y))








print(sess.run(y))

tf.add







print(sess.run(y))

tf.add

y = tf.add(tf.multiply(a, b, name='mul_op'), tf.add(a, b, name='add_op'))






writer = tf.summary.FileWriter('./graphs', tf.get_default_graph())


print(sess.run(y))

writer.close() # close the writer when you’re done using it







with tf.Session() as sess:writer = tf.summary.FileWriter('./graphs', sess.graph)print(sess.run(y))



Run it!

$ python3 [yourprogram].py$ tensorboard --logdir="./graphs" --port 6006


Run it!

$ python3 [yourprogram].py$ tensorboard --logdir="./graphs" --port 6006

http://localhost:6006/


The same graph in Tensorboard

a = tf.constant(3, name='a')b = tf.constant(7, name='b')y = tf.multiply(a, b, name='mul_op') +

tf.add(a, b, name='add_op')


The same graph in Tensorboard

name




x = 2y = 3

add_op = tf.add(x, y)mul_op = tf.multiply(x, y)

useless = tf.multiply(x, add_op,name='useless')

pow_op = tf.pow(add_op, mul_op)

with tf.Session() as sess:z = sess.run(pow_op)

Subgraphs


x = 2y = 3





Subgraphs


x = 2y = 3





Subgraphs

pow_oppow_op uselessuseless


Subgraphs

x = 2y = 3




with tf.Session() as sess:z, not_useless =

sess.run([useless, pow_op])


Subgraphs

x = 2y = 3




with tf.Session() as sess:z, not_useless =

sess.run([useless, pow_op])

sess.run


TensorFlow and GPUs


Distributed computation


Distributed computation

# Creates a graph.

with tf.device('/gpu:2'):

a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='a')

b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='b')

y = tf.multiply(a, b)

# Creates a session with log_device_placement set to True.

sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

# Runs the op.

print(sess.run(y))


Protocol buffers

print(tf.get_default_graph().as_graph_def())


Protocol buffers


node {name: "mul_op/a"op: "Const"attr {key: "dtype"value { type: DT_INT32 }

}attr {key: "value"value { tensor

{ dtype: DT_INT32tensor_shape {}int_val: 7 }

}}

}


Protocol buffers


node {name: "mul_op"op: "Mul"input: "mul_op/a"input: "mull_op/b"attr {key: "T"value {type: DT_INT32

}}

}


Protocol buffers


node {name: "add"op: "Add"input: "mul_op"input: "add_op"attr {key: "T"value {type: DT_INT32

}}

}


Tensors, constants and ops


What is a tensor?


Tensors as objects tf.Tensor

float32 int32 string




r = tf.rank(cat_image) # r is a tensors = tf.shape(cat_image) # s is a tensor# after evaluation r will be 3# s will be (28, 28, 3)




r = tf.rank(cat_image) # r is a tensors = tf.shape(cat_image) # s is a tensor# after evaluation r will be 3# s will be (28, 28, 3)

r s


Tensors Containers


Tensorflow Operations


Graph representations of tensors and containers




Constants tf.constant(value,

dtype=None,shape=None,name='Const',verify_shape=False)




# constant of 1d tensor (vector)a = tf.constant([2, 2], name="vector")

# constant of 2x2 tensor (matrix)b = tf.constant([[0, 1], [2, 3]], name="matrix")

x = tf.multiply(a, b, name='mul_op')

>> x = [[0 2][4 6]]


Constants filled with specific valuestf.zeros(shape, dtype=tf.float32, name=None)

tf.zeros_like(input_tensor, dtype=None, name=None, optimize=True)

tf.ones(shape, dtype=tf.float32, name=None)

tf.ones_like (input_tensor, dtype=None, name=None, optimize=True)

tf.fill(shape, value, name=None)







tf.zeros([2, 3], tf.int32)

tf.ones([2, 3], tf.int32)

0 0 00 0 0

1 1 11 1 1









# input_tensor is [[0, 1], [2, 3], [4, 5]]

tf.zeros_like(input_tensor)

tf.ones_like(input_tensor)

0 0 00 0 0

1 11 11 1

1 1 11 1 1

0 00 00 0









# input_tensor is [[0, 1], [2, 3], [4, 5]]

tf.zeros_like(input_tensor)

tf.ones_like(input_tensor)

tf.fill([2, 3], 8)

0 0 00 0 0

1 11 11 1

8 88 88 8

1 1 11 1 1

0 00 00 0


Constants as sequences

tf.lin_space(start, stop, num, name=None)

tf.lin_space(10.0, 13.0, 4) ==> [10. 11. 12. 13.]

tf.range(start, limit=None, delta=1, dtype=None, name='range')

tf.range(3, 18, 3) ==> [3 6 9 12 15]

tf.range(5) ==> [0 1 2 3 4]


Constants as sequences

tf.lin_space(start, stop, num, name=None)

tf.lin_space(10.0, 13.0, 4) ==> [10. 11. 12. 13.]

tf.range(start, limit=None, delta=1, dtype=None, name='range')

tf.range(3, 18, 3) ==> [3 6 9 12 15]

tf.range(5) ==> [0 1 2 3 4]

tf.range

for _ in tf.range(4):# TypeError


Randomly generated Constants

tf.random_normal

tf.truncated_normal

tf.random_uniform

tf.random_shuffle

tf.random_crop

tf.multinomial

tf.random_gamma


Randomly generated Constants

tf.random_normal

tf.truncated_normal

tf.random_uniform

tf.random_shuffle

tf.random_crop

tf.multinomial

tf.random_gamma

tf.set_random_seed(seed)


Data types


Data types

t_0 = 19 # scalars are treated like 0-d tensorstf.zeros_like(t_0) # ==> constant 0-d tensor with value 0tf.ones_like(t_0) # ==> constant 0-d tensor with value 1

t_1 = ["apple", "peach", "grape"] # 1-d arrays are treated like 1-d tensorstf.zeros_like(t_1) # ==> ['' '' '']tf.ones_like(t_1) # ==> TypeError

t_2 = [[True, False, False],[False, False, True],[False, True, False]] # 2-d arrays are treated like 2-d tensors

tf.zeros_like(t_2) # ==> 3x3 tensor, all elements are Falsetf.ones_like(t_2) # ==> 3x3 tensor, all elements are True


Data types


Variables


Shortcomings of constants



attr {key: "value"value {

tensor {dtype: DT_FLOAT

tensor_shape {dim { size: 2 }

}tensor_content:

"\000\000\200?\000\000\000@"}

}}

my_const = tf.constant([1.0, 2.0], name="my_const")with tf.Session() as sess:

print(sess.graph.as_graph_def())


Variables

tf.Variable


Variables

tf.Variable

# create variables with tf.Variable

s = tf.Variable(2, name="scalar") m = tf.Variable([[0, 1], [2, 3]], name="matrix") W = tf.Variable(tf.zeros([784,10]))


Variables

tf.Variable



# create variables with tf.get_variable

s = tf.get_variable("scalar", initializer=tf.constant(2)) m = tf.get_variable("matrix", initializer=tf.constant([[0, 1], [2, 3]]))W = tf.get_variable("big_matrix", shape=(784, 10),

initializer=tf.zeros_initializer())


Variables

tf.Variable



# create variables with tf.get_variable

s = tf.get_variable("scalar", initializer=tf.constant(2)) m = tf.get_variable("matrix", initializer=tf.constant([[0, 1], [2, 3]]))W = tf.get_variable("big_matrix", shape=(784, 10),

initializer=tf.zeros_initializer())

get_variable


Variables operator tf.Variable

x = tf.Variable(…)

x.initializer # init opx.value # read op

# and more..





# and more..

W = tf.get_variable("big_matrix", shape=(784, 10),initializer=tf.zeros_initializer())


print(sess.run(W))





# and more..

W = tf.get_variable("big_matrix", shape=(784, 10),initializer=tf.zeros_initializer())


print(sess.run(W))

>> FailedPreconditionError: Attempting to use uninitialized value Variable


Variables initialization

with tf.Session() as sess:sess.run(tf.global_variables_initializer())




with tf.Session() as sess:sess.run(tf.variables_initializer([a, b]))




with tf.Session() as sess:sess.run(tf.variables_initializer([a, b]))

W = tf.Variable(tf.zeros([784,10]))

with tf.Session() as sess:sess.run(W.initializer)


# W is a random 700 x 100 variable object

W = tf.Variable(tf.truncated_normal([700, 10]))


sess.run(W.initializer)

Variables evaluation

print(W)






>> Tensor("Variable/read:0", shape=(700, 10), dtype=float32)


print(W)







print(W.eval())







print(sess.run(W))print(W.eval())






>> [[-0.76781619 -0.67020458 1.15333688 ..., -0.98434633 -1.25692499-0.90904623]

..., [ 0.19076447 -0.62968266 -1.97970271 ..., -1.48389161 0.681706431.46369624]]


print(sess.run(W))print(W.eval())


Variable Assignments

W = tf.Variable(10)

W.assign(100)



print(W.eval())



W = tf.Variable(10)

W.assign(100)



print(W.eval())

# >> 10



W = tf.Variable(10)

W.assign(100)



print(W.eval())

# >> 10

W.assign(100)



W = tf.Variable(10)

W.assign(100)



print(W.eval())

# >> 10

W = tf.Variable(10)

assign_op = W.assign(100)



sess.run(assign_op)

print(W.eval())

W.assign(100)



W = tf.Variable(10)

W.assign(100)



print(W.eval())

# >> 10

W = tf.Variable(10)

assign_op = W.assign(100)



sess.run(assign_op)

print(W.eval())

# >> 100

W.assign(100)



# create a variable whose original value is 2

my_var = tf.Variable(2, name="my_var")

# assign a*2 to a and call that op a_times_two

my_var_times_two = my_var.assign(2*my_var)







2*my_var my_varmy_var_times_two








sess.run(my_var.initializer)

sess.run(my_var_times_two) # >> the value of my_var now is 4



Variables in multiple sessions

W = tf.Variable(10)

sess1 = tf.Session()


sess1.run(W.initializer)


sess1.close()

sess2.close()



W = tf.Variable(10)





print(sess1.run(W.assign_add(10)))

print(sess2.run(W.assign_sub(2)))

sess1.close()

sess2.close()



W = tf.Variable(10)







sess1.close()

sess2.close()

# >> 20# >> 8



W = tf.Variable(10)









sess1.close()

sess2.close()

# >> 20# >> 8



W = tf.Variable(10)









sess1.close()

sess2.close()

# >> 20# >> 8

# >> 120# >> 108


Placeholders


Placeholder and feed dicts tf.placeholder(dtype, shape=None, name=None)



# create a placeholder for a vector of 3 elements, type tf.float32

a = tf.placeholder(tf.float32, shape=[3])

b = tf.constant([5, 5, 5], tf.float32)

# use the placeholder as you would a constant or a variable

c = a + b # short for tf.add(a, b)








with tf.Session() as sess:print(sess.run(c))








>> InvalidArgumentError: a doesn’t have an>> actual value

with tf.Session() as sess:print(sess.run(c))








with tf.Session() as sess:print(sess.run(c, feed_dict={a: [1, 2, 3]}))








with tf.Session() as sess:print(sess.run(c, feed_dict={a: [1, 2, 3]}))

>> [6, 7, 8]


Feed dicts


Feed dicts

print(sess.run(some_op, feed_dict={a: [1, 2, 3]}))


Feed dicts



Feed dicts


None


Feed dicts


None

a = tf.placeholder(tf.float32, shape=None)


Feed dicts


None



Feed dicts


None



for a_value in list_of_values_for_a:

print(sess.run(some_op, {a: a_value}))


Feed dicts


Feed dicts

a = tf.add(2, 5)

b = tf.multiply(a, 3)


# compute the value of b given a is 15

sess.run(b, feed_dict={a: 15})


Feed dicts

a = tf.add(2, 5)






Feed dicts

a = tf.add(2, 5)





tf.Graph.is_feedable(tensor


A common mistake


Lazy loading

x = tf.Variable(10, name='x')

y = tf.Variable(20, name='y')

z = tf.add(x, y)

writer = tf.summary.FileWriter('./graphs/normal_loading', tf.get_default_graph())


sess.run(tf.global_variables_initializer())

for _ in range(10):

sess.run(z)

writer.close()


Lazy loading



z = tf.add(x, y)




for _ in range(10):

sess.run(z)

writer.close()


Lazy loading






for _ in range(10):

sess.run(tf.add(x, y))

writer.close()


Lazy loading


Lazy loading

node {

name: "Add"

op: "Add"

input: "x/read"

input: "y/read"

attr {

key: "T"

value {

type: DT_INT32

}

}

}


Lazy loading

node {

name: "Add"

op: "Add"

input: "x/read"

input: "y/read"

attr {

key: "T"

value {

type: DT_INT32

}

}

}

node {

name: "Add_1"

op: "Add"

...

}

...

node {

name: "Add_10"

op: "Add"

...

}


Lazy loading


Lazy loading

@property


Next step…build the first machine learning model!


The first machine learning model in TF


Now's the time for part 2!

Università degli Studi di Pavia Deep Learning and TensorFlow · Deep Learning and TensorFlow...

Documents

Transcript of Università degli Studi di Pavia Deep Learning and TensorFlow · Deep Learning and TensorFlow...