STICK-BREAKING CONSTRUCTIONS Patrick Dallaire June 10 th, 2011.
94
STICK-BREAKING CONSTRUCTIONS Patrick Dallaire June 10 th , 2011
-
Upload
beverly-welch -
Category
Documents
-
view
214 -
download
0
Transcript of STICK-BREAKING CONSTRUCTIONS Patrick Dallaire June 10 th, 2011.
STICK-BREAKING CONSTRUCTIONSPatrick Dallaire
June 10th, 2011
Outline
Introduction of the Stick-Breaking process
Outline
Introduction of the Stick-Breaking process
Presentation of fundamental representation
Outline
Introduction of the Stick-Breaking process
Presentation of fundamental representation The Dirichlet process The Pitman-Yor process The Indian buffet process
Outline
Introduction of the Stick-Breaking process
Presentation of fundamental representation The Dirichlet process The Pitman-Yor process The Indian buffet process
Definition of the Beta process
Outline
Introduction of the Stick-Breaking process
Presentation of fundamental representation The Dirichlet process The Pitman-Yor process The Indian buffet process
Definition of the Beta process A Stick-Breaking construction of Beta
process
Outline
Introduction of the Stick-Breaking process
Presentation of fundamental representation The Dirichlet process The Pitman-Yor process The Indian buffet process
Definition of the Beta process A Stick-Breaking construction of Beta
process Conclusion and current work
The Stick-Breaking process
The Stick-Breaking process
Assume a stick of unit length
The Stick-Breaking process
Assume a stick of unit length
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining
stick is broken by sampling the proportion to cut
The Stick-Breaking process
Assume a stick of unit length At each iteration, a part of the remaining stick
is broken by sampling the proportion to cut How should we sample these proportions?
Beta random proportions
Let be the proportion to cut at iteration
Beta random proportions
Let be the proportion to cut at iteration
The remaining length can be expressed as
Beta random proportions
Let be the proportion to cut at iteration
The remaining length can be expressed as
Thus, the broken part is defined by
Beta random proportions
Let be the proportion to cut at iteration
The remaining length can be expressed as
Thus, the broken part is defined by
We first consider the case where
Beta distribution
The Beta distribution is a density function on
Parameters and control its shape
The Dirichlet process
The Dirichlet process
Dirichlet processes are often used to produce infinite mixture models
The Dirichlet process
Dirichlet processes are often used to produce infinite mixture models
Each observation belongs to one of the infinitely many components
The Dirichlet process
Dirichlet processes are often used to produce infinite mixture models
Each observation belongs to one of the infinitely many components
The model ensures that only a finite number of components have appreciable weight
The Dirichlet process
A Dirichlet process, , can be constructed according to a Stick-Breaking process
Where is the base distribution and is a unit mass at
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
Construction demo
The Pitman-Yor process
The Pitman-Yor process
A Pitman-Yor process, , can be constructed according to a Stick-Breaking process
Where and
Evolution of the Beta cuts
The parameter controls the speed at which the Beta distribution changes
Evolution of the Beta cuts
The parameter controls the speed at which the Beta distribution changes
The parameter determines initial shapes of the Beta distribution
Evolution of the Beta cuts
The parameter controls the speed at which the Beta distribution changes
The parameter determines initial shapes of the Beta distribution
When , there is no changes over time and its called a Dirichlet process
Evolution of the Beta cuts
The parameter controls the speed at which the Beta distribution changes
The parameter determines initial shapes of the Beta distribution
When , there is no changes over time and its called a Dirichlet process
MATLAB DEMO
The Indian Buffet process
The Indian Buffet process
The Indian Buffet process was initially used to represent latent features
The Indian Buffet process
The Indian Buffet process was initially used to represent latent features
Observations are generated according to a set of unknown hidden features
The Indian Buffet process
The Indian Buffet process was initially used to represent latent features
Observations are generated according to a set of unknown hidden features
The model ensure that only a finite number of features have appreciable probability
The Indian Buffet process
Recall the basic Stick-Breaking process
The Indian Buffet process
Recall the basic Stick-Breaking process
The Indian Buffet process
Recall the basic Stick-Breaking process Here, we only consider the remaining
parts
The Indian Buffet process
Recall the basic Stick-Breaking process Here, we only consider the remaining
parts
The Indian Buffet process
Recall the basic Stick-Breaking process Here, we only consider the remaining
parts Each value corresponds to a feature
probability of appearance
Summary
Summary
The Dirichlet process induces a probability over infinitely many classes
Summary
The Dirichlet process induces a probability over infinitely many classes
This is the underlying de Finetti mixing distribution of the Chinese restaurant process
De Finetti theorem
It states that the distribution of any infinitely exchangeable sequence can be written
where is the de Finetti mixing distribution
Summary
The Dirichlet process induces a probability over infinitely many classes
This is the underlying de Finetti mixing distribution of the Chinese restaurant process
The Indian Buffet process induces a probability over infinitely many features
Summary
The Dirichlet process induces a probability over infinitely many classes
This is the underlying de Finetti mixing distribution of the Chinese restaurant process
The Indian Buffet process induces a probability over infinitely many features
Its underlying de Finetti mixing distribution is the Beta process
The Beta process
The Beta process
This process
Beta with Stick-Breaking
The Beta distribution has a Stick-Breaking representation which allows to sample from
Beta with Stick-Breaking
The Beta distribution has a Stick-Breaking representation which allows to sample from
The construction is
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
Beta with Stick-Breaking
The Beta distribution has a Stick-Breaking representation which allows to sample from
The construction is
The Beta process
A Beta process is defined as
as , and is a Beta process
Stick-Breaking the Beta process
The Stick-Breaking construction of the Beta process is such that
Stick-Breaking the Beta process
Expending the first terms
Conclusion
We briefly described various Stick-Breaking constructions for Bayesian nonparametric priors
These constructions help to understand the properties of each process
It also unveils connections among existing priors
The Stick-Breaking process might help to construct new priors
Current work
Applying a Stick-Breaking process to select the number of support points in a Gaussian process
Defining a stochastic process for unbounded random directed acyclic graph
Finding its underlying Stick-Breaking representation
