of 16

• date post

28-May-2020
• Category

## Documents

• view

0

0

Embed Size (px)

### Transcript of Composing graphical models with neural networks mlg. readinglist/slides/ 2016/11/01 ¢ ...

• Composing graphical models with neural networks for structured representations and fast

inference

Written by Matthew James Johnson, David Duvenaud, Alexander B. Wiltschko,

Sandeep R. Datta and Ryan P. Adams.

Published in NIPS 2016.

Presenter: Juho Lee

• Motivation

GMM:

• Motivation

VAE:

• Motivation

GMM + SVAE:

• Conjugate Exponential Families

Natural parameter

Sufficient Statistics

Log-partition function

• Conjugate Exponential Families

Compare to

• Variational Inference with Conjugate Exponential families

Assume that there exists a matrix such that

• Variational Inference with Conjugate Exponential families

• Variational Inference with Conjugate Exponential families • The objective function (ELBO):

• By the calculus of variations,

• Stochastic variational inference

• One can start by assuming,

and optimize w.r.t. the ELBO

• The gradient of is computed as

Hoffman et al, Stochastic variational inference, JMLR 2013

Fisher information matrix

• Stochastic variational inference

• Coordinate descent algorithm is a natural gradient descent

• Approximate it by stochastic (natural) gradient descent

Hoffman et al, Stochastic variational inference, JMLR 2013

• • Place conjugate exponential family prior on the latent variable

• Likelihood is an arbitrary (nonlinear) function

• Reparametrization + (stochastic) natural gradient descent

Structured Variational Autoencoder (SVAE)

• Structured Variational Autoencoder (SVAE)

• Mean-field approximation and the ELBO

• Intractable, consider the subproblem

• Structured Variational Autoencoder (SVAE)

• Now optimize the surrogate bound

• Optimizing : natural gradient descent

• Optimizing and : reparametrization trick

• Structured Variational Autoencoder (SVAE)

• Examples:

GMM + SVAE:

Latent switching linear dynamical systems:

• Structured Variational Autoencoder (SVAE)

• Some illustrations