Hierarchical Dirichlet Process and Infinite Hidden Markov Model Duke University Machine Learning...
-
Upload
dulcie-pierce -
Category
Documents
-
view
218 -
download
1
Transcript of Hierarchical Dirichlet Process and Infinite Hidden Markov Model Duke University Machine Learning...
Hierarchical Dirichlet Process and Infinite Hidden Markov Model
Duke University Machine Learning Group
Presented by Kai Ni
February 17, 2006
Paper by Y. W. Teh, M. I. Jordan, M. J. Beal & D. M. Blei, NIPS 2004
Outline
• Motivation
• Dirichlet Processes (DP)
• Hierarchical Dirichlet Processes (HDP)
• Infinite Hidden Markov Model (iHMM)
• Results & Conclusions
Motivation
• Problem – “multi-task learning” in which the “tasks” are clustering problems.
• Goal – Share clusters among multiple, related clustering problems. The number of clusters are open-ended and inferred automatically by the model.
• Application– Genome pattern analysis
– Information retrieval of corpus
Hierarchical Model
• A single clustering problem can be analyzed as a Dirichlet process (DP). –
– Draws G from DP are discrete, generally not distinct.
• For J groups, we consider Gj for j=1~J is a group-specific DP.
• To share information, we link the group-specific DPs– If G(τ) is continuous, the draws Gj
have no atoms in common with probability one.
– HDP solution: G0 is itself a draw from a DP(, H)
0 01
~ DP( , ) kk
k
G G G
0 01
~ DP( , ) jkj j j j jk
k
G G G
0 0~ DP( , ( ))jG G
Dirichlet Process & Hierarchical Dirichlet Process
• Three different perspectives– Stick-breaking– Chinese restaurant– Infinite mixture models
• Setup
• Properties of DP– – –
0 0
0 0 0 0
DP HDP
~ ( , ) | , ~ ( , )
| , ~ ( , )j
G DP G G H DP H
G G DP G
Stick-breaking View
• A mathematical explicit form of DP. Draws from DP are discrete.
• In DP
• In HDP
1
~ ( ), ~ 0
withkk
k
Stick Gk
G
01
~ ( )
~
kkk
Stick
Hk
G
1
~ ( , )0
~ 0
kj jkk
DP
Gk
G
πj
DP – Chinese Restaurant Process
• Exhibit clustering property
• Φ1,…,Φi-1, i.i.d., r.v., distributed according to G; Ө1,…, ӨK to be the distinct values taken on by Φ1,…,Φi-1, nk be # of Φi’= Өk, 0<i’<i,
HDP – Chinese Restaurant Franchise
• First level: within each group, DP mixture–
– Φj1,…,Φj(i-1), i.i.d., r.v., distributed according to Gj; Ѱj1,…, ѰjTj to be the values taken on by Φj1,…,Φj(i-1), njk be # of Φji’= Ѱjt, 0<i’<i.
• Second level: across group, sharing clusters– Base measure of each group is a draw from DP:
– Ө1,…, ӨK to be the values taken on by Ѱj1,…, ѰjTj , mk be # of Ѱjt=Өk, all j, t.
)(~|,~|),,(~ 00 jijijijjjij FxGGGDPG
0 0 0| ~ , ~ ( , )jt G G G DP H
HDP – CRF graph
• The values of are shared between groups, as well as within groups. This is a key property of HDP.
Integrating out G0
DP Mixture Model
• One of the most important application of DP: nonparametric prior distribution on the components of a mixture model.
• G can be looked as an infinite mixture model.
)(~|
~|
),(~
1
00
iii
i
kkk
Fx
GG
G
GDPG
• HDP can be used as the prior distribution over the factors for nested group data.
• We consider a two-level DPs. G0 links the child Gj DPs and forces them to share components. Gj is conditionally independent given G0
HDP mixture model
Infinite Hidden Markov Model
• The number of hidden states is allowed to be countably infinite.
• The transition probabilities given in the ith row of the transition matrix A can be interpreted as mixing proportions = (ai1, ai2, …, aik, …)
• Thus each row of the A in HMM is a DP. Also these DPs must be linked, because they should have same set of “next states”. HDP provides the natural framework for the infinite HMM.
iHMM via HDP
• Assign observations to groups, where the groups are indexed by the value of the previous state variable in the sequence. Then the current state and emission distribution define a group-specific mixture model.
• Multiple iHMMs can be linked by adding an additional level of Bayesian hierarchy, letting a master DP couple each of the iHMM, each of which is a set of DPs.
HDP & iHMM
HDP (CRF aspect) iHMM
Group Restaurant J (fixed) By Si-1 (random)
Data Customer xji yi
Hidden factor
Table ji = k, k=1~
Dish k ~ H
Si = k, k=1~
B (Si , : )
DP weights Popularity jk, k=1~ A (Si-1, : )
Likelihood F(xji| ji ) B (Si, yi)
Non-trivialities in iHMM
• HDP assumes a fixed partition of the data into groups while HMM is for time-series data, and the definition of groups is itself random.
• Consider CRF aspect of HDP, the number of restaurant is infinite. Also in the sampling scheme, changing st may affect all subsequent data assignment.
• CRF is natural to describe the iHMM, however it is awkward for sampling. We need to use sampling algorithm from other respects for the iHMM.
HDP Results
iHMM Results
Conclusion
• HDP is a hierarchical, nonparametric model for clustering problems involving multiple groups of data.
• The mixture components are shared across groups and the appropriate number is determined by HDP automatically.
• HDP can be extended to infinite HMM model, providing effective inference algorithm.
Reference
• Y.W. Teh, M.I. Jordan, M.J. Beal and D.M. Blei, “Sharing Clusters among Related Groups: Hierarchical Dirichlet Processes”, NIPS 2004.
• Beal, M.J., Ghahramani, Z. and Rasmussen, C.E., “The Infinite Hidden Markov Model”, NIPS 2002
• Y.W. Teh, M.I. Jordan, M.J. Beal and D.M. Blei, “Hierarchical Dirichlet Processes”, Revised version to appear in JASA, 2006.