Composing graphical models with neural networks mlg. readinglist/slides/ 2016/11/01 ¢ ...

download Composing graphical models with neural networks mlg. readinglist/slides/  2016/11/01 ¢  Composing

of 16

  • date post

    28-May-2020
  • Category

    Documents

  • view

    0
  • download

    0

Embed Size (px)

Transcript of Composing graphical models with neural networks mlg. readinglist/slides/ 2016/11/01 ¢ ...

  • Composing graphical models with neural networks for structured representations and fast

    inference

    Written by Matthew James Johnson, David Duvenaud, Alexander B. Wiltschko,

    Sandeep R. Datta and Ryan P. Adams.

    Published in NIPS 2016.

    Presenter: Juho Lee

  • Motivation

    GMM:

  • Motivation

    VAE:

  • Motivation

    GMM + SVAE:

  • Conjugate Exponential Families

    Natural parameter

    Sufficient Statistics

    Log-partition function

  • Conjugate Exponential Families

    Compare to

  • Variational Inference with Conjugate Exponential families

    Assume that there exists a matrix such that

  • Variational Inference with Conjugate Exponential families

  • Variational Inference with Conjugate Exponential families • The objective function (ELBO):

    • By the calculus of variations,

  • Stochastic variational inference

    • One can start by assuming,

    and optimize w.r.t. the ELBO

    • The gradient of is computed as

    Hoffman et al, Stochastic variational inference, JMLR 2013

    Fisher information matrix

    Natural gradient

  • Stochastic variational inference

    • Coordinate descent algorithm is a natural gradient descent

    • Approximate it by stochastic (natural) gradient descent

    Hoffman et al, Stochastic variational inference, JMLR 2013

  • • Place conjugate exponential family prior on the latent variable

    • Likelihood is an arbitrary (nonlinear) function

    • Reparametrization + (stochastic) natural gradient descent

    Structured Variational Autoencoder (SVAE)

  • Structured Variational Autoencoder (SVAE)

    • Mean-field approximation and the ELBO

    • Intractable, consider the subproblem

  • Structured Variational Autoencoder (SVAE)

    • Now optimize the surrogate bound

    • Optimizing : natural gradient descent

    • Optimizing and : reparametrization trick

  • Structured Variational Autoencoder (SVAE)

    • Examples:

    GMM + SVAE:

    Latent switching linear dynamical systems:

  • Structured Variational Autoencoder (SVAE)

    • Some illustrations