Introduction  to  Markov  Models

Kasthuri  Kannan,  PhDAssistant  Professor  of  Pathology

New  York  University

Overview  of  Topics

• Introduction  to  Markov  Processes

• Hidden  Markov  Models

• Forward  Algorithm

• Viterbi  Algorithm

• Tutorial

Introduction  to  Markov  Processes

• Interested  in  finding  patterns  appearing  over  time– Sequence  of  events

• Appear  in  many  areas  in  nature– Biological  sequences  (DNA,  RNA,  Proteins  etc.)– Sequences  of  words,  natural  language  processing– Weather  phenomenon  etc.

• Interested  in  knowing  if  the  sequences  makes  useful  patterns  which  can  be  modeled

• Sequence  of  traffic  lights  – red  -­‐ red/amber  –green  – amber  – red.– Can  be  viewed  as  a  trellis  diagram

• Each  state  is  dependent  solely  on  previous  state  (deterministic  system)  – observable  Markov  process

Introduction  to  Markov  Processes

Stop Prepare  to  go Go Prepare  to  stop

• Deducing  weather  from  a  piece  of  seaweed

• Soggy                          Sun

• Damp  means  can’t  be  sure

• Note:  State  of  the  weather  is  not  dependent  on  the  state  of  the  seaweed  so  we  can  say  something  (like  raining)

• Another  clue  would  be  the  state  in  the  preceding  day  – by  combining  this  knowledge  we  will  be  able  to  come  to  a  better  forecast

Introduction  to  Markov  Processes

• Consider  systems  which  generate  probabilistic  patterns  in  time  – such  as  weather  fluctuating  between  sunny  and  rainy,  or  signal  going  from  amber  to  red

• We  the  look  at  systems  where  what  we  wish  to  predict  is  not  what  we  observe  – the  underlying  hidden  system

• In  seaweed  example,  the  hidden  system  would  be  the  actual  weather  

• In  traffic  light  example,  the  hidden  system  could  be  the  actual  signal  while  observed  sequence  would  be  traffic  (cars,  busses)  stopping  or  moving

Introduction  to  Markov  Processes

• Some  problems  can  be  solved  once  the  system  is  modeled

• What  the  weather  was  for  a  week  given  each  day’s  seaweed  observation

• Given  a  sequence  of  seaweed  observations,  is  it  winter  or  summer?– Intuitively  if  the  seaweed  has  been  dry  for  a  while  it  may  be  summer,  else,  otherwise

Introduction  to  Markov  Processes

• Typically  we  build  trellis  diagram  – each  state  depending  only  on  the  previous  state

• Weather  example:  each  sequence  taking  a  single  day  (with  self-­‐transition)  – one  possible  sequence

Introduction  to  Markov  Processes

Introduction  to  Markov  Processes

All  possible  sequences

Introduction  to  Markov  ProcessesAll  possible  sequences  with  probabilities

0.5

0.6

0.4

0.2

0.1

0.3

0.4

0.2

0.3

Note:  Total  of  these  colored  probabilities  equals  1

Introduction  to  Markov  Processes

All  possible  sequences  with  probabilities

Note:  Total  of  these  colored  probabilities  equals  1

Sunny Cloudy Rainy

Sunny 0.5 0.2 0.3

Cloudy 0.1 0.6 0.3

Rainy 0.2 0.4 0.4

Transition  Matrix

• Many  cases  the  patterns  are  not  apparent

• A  crab  may  have  access  only  to  the  seaweed  and  not  the  weather  – although  weather  and  seaweed  states  are  closely  linked

• Two  set  of  states– Observable  states  (seaweed)– Hidden  states  (the  state  of  the  weather)  

• Devise  an  algorithm  for  the  crab  to  forecast  weather  from  the  seaweed  and  Markov  assumption  without actually  ever  seeing  the  weather

Introduction  to  Markov  Processes

Hidden  Markov  Models

• Observed  sequence  of  states  is  probabilistically  related  to  hidden  states

• Hidden  Markov  Models  – modeling  such  processes  where  set  of  observable  states  are  somehow  related  to  hidden  states  

• – with  Markov  assumption

• Number  of  hidden  states  may  be  different  from  number  of  observable  states

Sunny

Cloudy Humid

Rainy

Dry

Damp

Soggy

Seaweed  States

Weather  States

Hidden  Markov  Models

Hidden  States

Soggy Damp Dryish Dry

Observable  States

Hidden  Markov  Models• Hidden  states  are  modeled  by  simple  first  order  Markov  

process

• The  connections  between  hidden  states  and  observable  states  represent  the  probability  of  generating  a  particular  observed  state  given  that  the  Markov  process  is  in  a  particular  hidden  state

• Probabilities  “entering”  an  observable  state  will  sum  up  to  1,  since  it  would  be  the  sum  of  Pr(Obs|Sun),  Pr(Obs|Cloud)  and  Pr(Obs|Rain)

Hidden  Markov  Models

Dry Dryish Damp Soggy

Sunny 0.6 0.2 0.15 0.05

Cloudy 0.25 0.25 0.25 0.25

Rainy 0.05 0.10 0.35 0.5

Apart  from  the  transition  matrix  we  also  have  emission  matrix  which  contains  the  probabilities  of  the  observable  states  given  a  particular  hidden  state.Note:  sum  of  each  matrix  row  is  1

Probabilities  in  transition  and  emission  matrices  are  time  independent  – i.e.,  they  don’t  change  over  time

One  of  the  most  unrealistic  assumptionsof  Markov  models  when  applied  to  real  process.

Seaweed  State  Today

Weather  Yesterday

Emission  matrix/probabilities

Hidden  Markov  Models

• Evaluation  (pattern  recognition)  – Finding  the  probability  of  an  observed  sequence  given  a  HMM  

• Decoding  (pattern  recognition)  – Finding  the  sequence  of  hidden  states  that  most  probably  generated  an  observed  sequence

• Learning  – Generating  a  HMM  given  a  sequence  of  observations

Hidden  Markov  Models  -­‐ Evaluation

• Number  of  HMMs  describing  different  systems  and  sequence  of  observations

• Which  HMM  most  probably  generated  the  given  sequence– Example,  a  “summer”  and  “winter”  model  for  seaweed  behavior  to  determine  season

• Usually  carried  out  by  forward  algorithm  to  calculate  the  probability  of  an  observation  sequence  given  a  HMM  and  choose  the  most  probable  HMM

Hidden  Markov  Models  -­‐ Decoding

• Find  the  most  probable  sequence  of  hidden  states  given  some  observations

• In  several  instances  we  are  interested  in  identifying  the  hidden  states  since  they  represent  something  of  value– For  example,  chromatin  states,  CpG islands  etc.

• In  our  own  example,  identifying  the  weather  only  knowing  the  status  of  the  seaweed

• Viterbi’s  algorithm  is  handy  for  this  process.

Hidden  Markov  Models  -­‐ Learning

• Involves  generating  a  HMM  from  a  sequence  of  observations

• Take  a  sequence  of  observations  (from  a  known  set),  known  to  represent  a  set  of  hidden  states  and  fit  the  most  probable  HMM

• Baum-­‐Welch  expectation  maximization  (EM)  can  be  used  to  identify  local  optimal  parameters  for  the  HMM  for  the  Learning  problem

The  Forward  AlgorithmA  real  world  example

The  Forward  AlgorithmPredicting  the  state  after  n  observations

The  Forward  Algorithm

Predicting  the  state  after  n  observations

The  Forward  Algorithm

Predicting  the  state  after  n  observations

The  Forward  AlgorithmPredicting  the  state  after  n  observations

The  Forward  AlgorithmBuilding  computations  one  at  a  time

The  Forward  AlgorithmBuilding  computations  one  at  a  time

The  Forward  Algorithm

Trellis  for  the  states  of  the  dog  given  observations

The  Forward  AlgorithmTrellis  for  the  states  of  the  dog  given  observations

The  Forward  AlgorithmTrellis  for  the  states  of  the  dog  given  observations

The  Decoding  ProblemMost  probable  sequence  of  states  given  observations

https://www.youtube.com/watch?v=6JVqutwtzmo&t=178s

Viterbi’s  Algorithm