Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with...

27
Lecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym Sung Kim <[email protected]>

Transcript of Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with...

Page 1: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Lecture 7: DQN

Reinforcement Learning with TensorFlow&OpenAI GymSung Kim <[email protected]>

Page 2: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Q-function Approximation: Q-Nets

(1) state, s

(2) quality (reward)for all actions(eg, [0.5, 0.1, 0.0, 0.8] LEFT: 0.5, RIGHT 0.1 UP: 0.0, DOWN: 0.8)

111 11111

1

1

Page 3: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Q-Nets are unstable

Page 4: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Convergence

Tutorial: Deep Reinforcement Learning, David Silver, Google DeepMind

min

✓

TX

t=0

[

ˆQ(st, at|✓)� (rt + �max

a0ˆQ(st+1, a

0|✓))]2

Page 5: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Reinforcement + Neural Net

http://stackoverflow.com/questions/10722064/training-a-neural-network-with-reinforcement-learning

Page 6: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

NATURE.COM/NATURE26 February 2015 £10

Vol. 518, No. 7540

EPIDEMIOLOGY

SHARE DATA IN OUTBREAKS

Forge open access to sequences and more

PAGE 477

COSMOLOGY

A GIANT IN THE EARLY UNIVERSE

A supermassive black hole at a redshift of 6.3

PAGES 490 & 512

QUANTUM PHYSICS

TELEPORTATION FOR TWO

Transferring two properties of a single photon

PAGES 491 & 516

INNOVATIONS INThe microbiome

Self-taught AI software attains human-level

performance in video games PAGES 486 & 529

T H E I N T E R N AT I O N A L W E E K LY J O U R N A L O F S C I E N C E

Tutorial: Deep Reinforcement Learning, David Silver, Google DeepMind

Page 7: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Two big issues

Tutorial: Deep Reinforcement Learning, David Silver, Google DeepMind

Page 8: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

1. Correlations between samples

Playing Atari with Deep Reinforcement Learning - University of Toronto by V Mnih et al.

Page 9: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

1. Correlations between samples

Playing Atari with Deep Reinforcement Learning - University of Toronto by V Mnih et al.

Page 10: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

1. Correlations between samples

Page 11: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Prerequisite: http://hunkim.github.io/ml/ orhttps://www.inflearn.com/course/기본적인-머신러닝-딥러닝-강좌/

Page 12: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

1. Correlations between samples

Page 13: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

2. Non-stationary targets

min

✓

TX

t=0

[

ˆQ(st, at|✓)� (rt + �max

a0ˆQ(st+1, a

0|✓))]2

Page 14: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

2. Non-stationary targets

min

✓

TX

t=0

[

ˆQ(st, at|✓)� (rt + �max

a0ˆQ(st+1, a

0|✓))]2

Y = Q(st, at|✓) Y = rt + �max

a0ˆQ✓(st+1, a

0|✓)

Page 15: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

DQN’s three solutions

1. Go deep

2. Capture and replay • Correlations between samples

3. Separate networks: create a target network • Non-stationary targets

Tutorial: Deep Reinforcement Learning, David Silver, Google DeepMind

Page 16: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Human-level control through deep reinforcement learning, Nature http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html

Solution 1: go deep

Page 17: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Solution 1: go deep

ICML 2016 Tutorial: Deep Reinforcement Learning, David Silver, Google DeepMind

Page 18: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Problem 2: correlations between samples

Page 19: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Solution 2: experience replayDeep Q-Networks (DQN): Experience Replay

To remove correlations, build data-set from agent’s own experience

s1, a1, r2, s2s2, a2, r3, s3 ! s, a, r , s 0

s3, a3, r4, s4...

s

t

, at

, rt+1, st+1 ! s

t

, at

, rt+1, st+1

Sample experiences from data-set and apply update

l =

✓r + � max

a

0Q(s 0, a0,w�) � Q(s, a,w)

◆2

To deal with non-stationarity, target parameters w� are held fixed

Capturerandom sample

& Replaymin

✓

TX

t=0

[

ˆQ(st, at|✓)� (rt + �max

a0ˆQ(st+1, a

0|✓))]2

ICML 2016 Tutorial: Deep Reinforcement Learning, David Silver, Google DeepMind

Page 20: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Solution 2: experience replay

Playing Atari with Deep Reinforcement Learning - University of Toronto by V Mnih et al.

Page 21: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Problem 2: correlations between samplesDeep Q-Networks (DQN): Experience Replay

To remove correlations, build data-set from agent’s own experience

s1, a1, r2, s2s2, a2, r3, s3 ! s, a, r , s 0

s3, a3, r4, s4...

s

t

, at

, rt+1, st+1 ! s

t

, at

, rt+1, st+1

Sample experiences from data-set and apply update

l =

✓r + � max

a

0Q(s 0, a0,w�) � Q(s, a,w)

◆2

To deal with non-stationarity, target parameters w� are held fixed

Page 22: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Problem 3: non-stationary targets

min

✓

TX

t=0

[

ˆQ(st, at|✓)� (rt + �max

a0ˆQ(st+1, a

0|✓))]2

Y = Q(st, at|✓) Y = rt + �max

a0ˆQ✓(st+1, a

0|✓)

Page 23: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Human-level control through deep reinforcement learning, Nature http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html

Solution 3: separate target network

min

✓

TX

t=0

[

ˆQ(st, at|✓)� (rt + �max

a0ˆQ(st+1, a

0|✓))]2

min

✓

TX

t=0

[

ˆQ(st, at|✓)� (rt + �max

a0ˆQ(st+1, a

0|¯✓))]2

Page 24: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Solution 3: separate target network

min

✓

TX

t=0

[

ˆQ(st, at|✓)� (rt + �max

a0ˆQ(st+1, a

0|¯✓))]2

(2)Ws(1)s

(1)s (2) Y (target)

Page 25: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Solution 3: copy network

(2)Ws(1)s

(1)s (2) Y (target)

Page 26: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Understanding Nature

Paper (2015)

Human-level control through deep reinforcement learning, Nature http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html

Page 27: Lecture 7: DQN - GitHub Pages · PDF fileLecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym ... Deep Q-Networks ... Deep Reinforcement Learning, David Silver,

Next

Lab: DQN


Human-level control through deep reinforcement learningir.hit.edu.cn/~jguo/docs/notes/dqn-atari.pdf · Human-level control through deep reinforcement learning Jiang Guo 2016.04.19

Human-level control through deep reinforcement learningir.hit.edu.cn/~jguo/docs/notes/dqn-atari.pdf · Human-level control through deep reinforcement learning Jiang Guo 2016.04.19

Averaged-DQN: Variance Reduction and Stabilization … Variance Reduction and Stabilization ... the target values. To un-derstand the effect of the ... and Stabilization for Deep Reinforcement

Averaged-DQN: Variance Reduction and Stabilization … Variance Reduction and Stabilization ... the target values. To un-derstand the effect of the ... and Stabilization for Deep Reinforcement

Leveraging Machine Learning for Advancing Smart-X Systems ... · 1.Deep Reinforcement Learning: Use deep neural networks to approximate Q (DQN) 2.Experience replay to reuse past data

Leveraging Machine Learning for Advancing Smart-X Systems ... · 1.Deep Reinforcement Learning: Use deep neural networks to approximate Q (DQN) 2.Experience replay to reuse past data

Asynchronous Methods for Deep Reinforcement Learning · Distributed version of DQN (Then) State-of-the-art results on the Atari Training takes days on 100+ machines Lavrenti Frobeen

Asynchronous Methods for Deep Reinforcement Learning · Distributed version of DQN (Then) State-of-the-art results on the Atari Training takes days on 100+ machines Lavrenti Frobeen

Hierarchical Deep Reinforcement Learning: Integrating ...papers.nips.cc/paper/6233-hierarchical-deep-reinforcement-learning... · work that integrates deep reinforcement learning

Hierarchical Deep Reinforcement Learning: Integrating ...papers.nips.cc/paper/6233-hierarchical-deep-reinforcement-learning... · work that integrates deep reinforcement learning

1 CNN - krnet.or.krBC%D5%B1%E2%B9%CE.pdf · 딥러닝(Deep learning)과 강화 학습(Reinforcement learning)을 접목한 DQN(Deep Q-Network)을 이용한 기계학습 방법론이

1 CNN - krnet.or.krBC%D5%B1%E2%B9%CE.pdf · 딥러닝(Deep learning)과 강화 학습(Reinforcement learning)을 접목한 DQN(Deep Q-Network)을 이용한 기계학습 방법론이

Learning 2048 with Deep Reinforcement Learningmli/zalevine-dqn-2048.pdf · 2017-08-01 · To the best of our knowledge, this is the first successful application of Deep Q-Learning

Learning 2048 with Deep Reinforcement Learningmli/zalevine-dqn-2048.pdf · 2017-08-01 · To the best of our knowledge, this is the first successful application of Deep Q-Learning

Deep Reinforcement Learning in a Handful of Trials …papers.nips.cc/paper/7725-deep-reinforcement-learning-in...Deep Reinforcement Learning in a Handful of Trials using Probabilistic

Deep Reinforcement Learning in a Handful of Trials …papers.nips.cc/paper/7725-deep-reinforcement-learning-in...Deep Reinforcement Learning in a Handful of Trials using Probabilistic

IntelliLight: A Reinforcement Learning Approach for ... · (DQN), for traffic light control problem. Figure 1 illustrates the basic idea of deep reinforcement learning framework.

IntelliLight: A Reinforcement Learning Approach for ... · (DQN), for traffic light control problem. Figure 1 illustrates the basic idea of deep reinforcement learning framework.

Introduction to Deep Reinforcement Learning · PDF fileIntroduction to Deep Reinforcement Learning Shenglin Zhao ... .

Introduction to Deep Reinforcement Learning · PDF fileIntroduction to Deep Reinforcement Learning Shenglin Zhao ... .

Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning ...

Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning ...

Intro to Deep Reinforcement Learning

Intro to Deep Reinforcement Learning

Deep Reinforcement Learning - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2017/Lecture/RL (v5).pdf · Dueling DQN Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt,

Deep Reinforcement Learning - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2017/Lecture/RL (v5).pdf · Dueling DQN Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt,

Reinforcement Learning - Deep Reinforcement Learningmmartin/URL/Lecture4.pdfReinforcement Learning Deep Reinforcement Learning Mario Martin CS-UPC May 17, 2018 ... Deep Q-Network algorithm

Reinforcement Learning - Deep Reinforcement Learningmmartin/URL/Lecture4.pdfReinforcement Learning Deep Reinforcement Learning Mario Martin CS-UPC May 17, 2018 ... Deep Q-Network algorithm

Introduction to Deep Reinforcement Learningwnzhang.net/teaching/cs420/slides/13-deep-rl.pdf · 2020-06-08 · Deep Reinforcement Learning •Deep Reinforcement Learning •leverages

Introduction to Deep Reinforcement Learningwnzhang.net/teaching/cs420/slides/13-deep-rl.pdf · 2020-06-08 · Deep Reinforcement Learning •Deep Reinforcement Learning •leverages

Deep Reinforcement Learning and Controlkatef/DeepRLFall2018/... · DQNs: Experience Replay ‣ DQN uses experience replay and fixed Q-targets ‣ Use stochastic gradient descent ‣

Deep Reinforcement Learning and Controlkatef/DeepRLFall2018/... · DQNs: Experience Replay ‣ DQN uses experience replay and fixed Q-targets ‣ Use stochastic gradient descent ‣

DQN-TAMER: Human-in-the-Loop Reinforcement Learning with ...DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback Riku Arakawa y, Sosuke Kobayashi , Yuya Unno

DQN-TAMER: Human-in-the-Loop Reinforcement Learning with ...DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback Riku Arakawa y, Sosuke Kobayashi , Yuya Unno

DEEP FEATURE EXTRACTION FOR SAMPLE-EFFICIENT REINFORCEMENT … · Deep Reinforcement Learning Il deep reinforcement learning (DRL) `e una branca dell’apprendimento per rinforzo

DEEP FEATURE EXTRACTION FOR SAMPLE-EFFICIENT REINFORCEMENT … · Deep Reinforcement Learning Il deep reinforcement learning (DRL) `e una branca dell’apprendimento per rinforzo

Ensemble Network Architecture for Deep Reinforcement Learningdownloads.hindawi.com/journals/mpe/2018/2129393.pdf · DQN Q-aue DQN Q-ae t 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 −40

Ensemble Network Architecture for Deep Reinforcement Learningdownloads.hindawi.com/journals/mpe/2018/2129393.pdf · DQN Q-aue DQN Q-ae t 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 −40

Deep Learning for Reinforcement Learning in · PDF fileDeep Learning for Reinforcement Learning in ... Deep Learning for Reinforcement Learning in Pacman Deep Learning für ... Während

Deep Learning for Reinforcement Learning in · PDF fileDeep Learning for Reinforcement Learning in ... Deep Learning for Reinforcement Learning in Pacman Deep Learning für ... Während