1

Chapter 5Continuous time Markov Chains

Learning objectives :Introduce continuous time Markov Chain

Model manufacturing systems using Markov Chain

Able to evaluate the steady-state performances

Textbook :C. Cassandras and S. Lafortune, Introduction to Discrete

Event Systems, Springer, 2007

2

Plan

• Basic definitions of continuous time Markov Chains • Characteristics of CTMC • Performance analysis of CTMC • Poisson process • Approximation of general distributions by phase type

distribution

3

Basic definitions of continuous time Markov Chains

4

Stochastic process

Discrete events

Continuous event

Discrete time

Continuous time

Memoryless

A CTMC is a continuous time and memoriless discrete event stochastic process.

Continuous Time Markov Chain (CTMC)

5

Continuous Time Markov Chain (CTMC)

Definition : a stochastic process with discrete state space and continuous time {X(t), t > 0} is a continuous time Markov Chain (CTMC) iff

P[X(t+s)= j X(u), 0≤u≤s] = P[X(t+s)= j X(s)], t, s, j

Memoryless:In a CTMC, the past history impacts on the future evolution of the system via the current state of the system

6

Continuous Time Markov Chain (CTMC)

Poisson Arrivals

Exponential service time

N(t) : number of customers at time t

Customer Arrivals

Customer departures

7

Homogenuous CTMC

Definition : A CTMC {X(t), t > 0} is homogeneous iff

P[X(t+s)= j X(t) = i] = P[X(t+s)= j X(t) = i] = pij(s)

Homogeneous memoryless:In reliability, we only say "a machine that does not fail at age t is as good as new"

Only homogeneous CTMC will be considered in this chapter.

8

Characteristics of CTMC

9

Behavior of a CTMC

X(t)

Two major components:

•Ti = sojourn time in state i (random variable)

•pij = probability of moving to state j when leaving state i

10

Sojourn time in a state

• Let Ti be the random variable corresponding to the time spent in state i

• The memoryless property of the homogenuous CTMC implies

• The exponential distribution is the only continuous probability distribution having this property.

In an CTMC, the sojourn time in any state is exponentially distributed.

¨ , ,i i iP T t x T t P T x t x

11

Exponential distribution

• Let T be a continuous random variable with an exponential distribution of parameter

• Distribution Function (figure) : FT(t) = P{T ≤ t}

• Probability density function : fT(t) = dFT(t)/dt

• Mean : E[T] = 1/• Standard deviation: [T] = 1/

• Coeficient of variation: Cv(T) = [T]/ E[T] = 1

• Parameter often corresponds to some event rate (failure rate, repair rate, production rate, ...)

1 , 0

0, 0

t

T

e tF t

t

, 0

0, 0

t

T

e tf t

t

12

Exponential distribution

• Memoryless :

¨

1t st

st

P t T t sP T t s T t

P T t

e ee P T s

e

• For a machine with exponentially distributed lifetime, we say that it is "as good as new" if it is not failed.

• The remaining lifetime of an used but UP machine has the same distribution as a new machine.

13

Transition probability

Whe a CTMC leaves state i, it jumps to state j with probability pij. This probability is:•independent of time as the CTMC is homogeneous•independent of sojourn time Ti as the process is markovian (memoryless)

14

1st characterization of a CTMC

An CTMC is fully characterized by the following parameters:•{i}iE with i as the parameter of the exponential distribution of sojourn time Ti

•{pij}i≠j , with pij as the transition probability from i to j when leaving state i

15

Classification of a CTMC

Each CTMC is associated an underlying DTMC by neglecting sojourn times.

A state i of a CTMC is said transient (resp. recurrent, absorbing) if it is transient (resp. recurrent, absorbing) in the underlying DTCM

A CTMC is irreducible if its underlying DTMC is irreducible.

Remark: the concept of periodicity is not relevant.

16

2nd characterization of a CTMC

Each state activates several potential events leading to different transitions.

A CTMC travels from state i to state j in Tij time, an exponentially distributed random variable with parameter ij.

i is called transition rate from i to j.

17

Equivalence of the two representation

Let •Ti = MINj{Tij}

•pij = P{Tij = Ti}

Result to prove: Ti = EXP(ij), pij is independent of Ti

Moment generating function MX(u) = E[exp(uX)]

18

Performance analysis of CTMC

19

Probability distribution

• State probability

i(t) = P{X(t) = i}

• state probability vector, also called probability distribution

(t) = (1(t), 2(t), ...)

20

Transient analysis

By conditionning on X(t),

With

21

Transient analysis

It can be shown,

Letting dt go to 0,

22

Infinitesimal generator

• Let

• The matrix Q = [qij] is called infinitesimal generator of the CTMC

• As a ressult,

23

Steady state distribution of a CTMC

Thereom: For an irreducible CTMC with postive recurrent states, the probability distribution converges to a vector of stationary probabilities (1, 2, ...) that is independent of the initial distribution (0). Further it is the unique solution of the following equation system:

normalization equation

flow balance equationorequilibrium eq

24

Flow balance equation

• The balance equation equivalent to : i≠jjji = i≠jiij

• Associate to each transition (i,j) a probability flow : iij

• i≠jjji : total flow into state i

• i≠jiij : total flow out of state i

• Interpretation : Total flow in = Total flow out

25

Flow balance equation of set of states

• Let E1 be a subset of states

• Flow balance equation : Total flow into E1 = Total flow out of E1

26

A manufaturing system

• Consider a machine which can be either UP or DOWN.

• The state of the machine is checked continuously.

• The average time to failure of an UP machine is 10 days.

• The average time for repair of a DOWN machine is 1.5 days.

• Determine the conditions for the state of the machine {X(t)} to be a Markov chain.

• Draw the Markov chain model.

• Find the transient distribution by starting from state UP and DOWN.

• Check whether the Markov chain is recurrent.

• Determine the steady state distribution.

• Determine the availability of the machine.

27

Poisson process

28

Poisson process

A Poisson process is a stochastic process N(t) such that•N(0) = 1•N(t) increments by +1 after a time T random distributed according to an exponential distribution of parameter .

An arrival process is said Poisson if the inter-arrival times are exponentially distributed.

29

Properties of Poisson process

A Poisson process is an irreducible CTMC

N(t) has a Poisson distribution with parameter t

30

Properties of Poisson process

A Poisson process is an irreducible CTMCP{N(t+dt) = k+1 | N(t) = k} = dt + o(dt)

Probability of 0 arrival in dtP{N(t+dt) = k | N(t) = k} = 1- dt + o(dt)

Probability of more than one arrival in dtP{N(t+dt) > k+1 | N(t) = k} = o(dt)

31

Properties of Poisson process

The superposition of n Poisson process of parameter i is a Poisson process of parameter i

Assume that a Poisson process is split into n processes with probabilities pi. These n process are independent Poisson process with parameter pi

32

Birth-Death process

33

Definition

• Consider a population of individuals

• Let N(t) be the size of the population with N(t) = 0, 1, 2, ...

• When N(t) = n, births arrive at according to a Poisson pocess of birth rate n > 0

• Deaths arrive also according to a Poisson process of death rate n > 0.

34

Key issues

• Graphic representation of the Markov chain

• Relation with the Poisson process (also called pure birth process)

• Condition for existence of steady state distribution

• Sufficient condition (larger death rate than birth rate)

• Steady state distribution n

0 1

11

...

...n

nn

S

1

1, *n

n

n n

35

Approximation of general distributions by phase type

distribution

36

Phase-type distribution

A probaiblity distribution that results from a system of one or more inter-related Poisson process occurring in sequence, or phases.

The sequence in which each of the phases occur may itself be a stochastic process.

Phase distribution = time until the absorption of a CTMC one absorbing state. Each of the states of the Markov process represents one of the phases.

Phase-type distributions can be used to approximate any positive valued distribution.

37

Definition

• A CTMC with m+1 states, where m ≥ 1, such that the states 1,...,m are transient states and state m+1 is an absorbing state.

• An initial probability of starting in any of the m+1 phases given by the probability vector (α, αm+1).

The continuous phase-type distribution is the distribution of time from the above process's starting until absorption in the absorbing state.

This process can be written in the form of a transition rate matrix,

where S is an m×m matrix and S0 = -S 1 with 1 represents an m×1 vector with every element being 1

0

0Q

S S

0

38

Characterization

Time X until the absorbing state is phase-type distributed PH(α,S).

The distribution function of X is given by,

F(x) = 1 - exp(Sx)1,

and the density function,

f(x) = exp(Sx)S0,

for all x > 0.

39

Erlang distribution

Ek : k-stage Erlang distribution with parameter

X = sum of k independent random variable of exponential distribution with parameter

E[X] = k/Var[X] = k/2

CX = X / E[X] = 1/k1/2

●●●

40

Hyper-exponential or mixture of exponential distribution

X = 1X1 + 2X2 ... + nXn

where •1 + 2 ... + n = 1,

•Xi = EXP(i)

E[X] = 1/1 + 2/2 ... + n/n

Var[X] = 1/12 + 2/2

2 ... + n/n2

41

Coxian distribution

n●●●p1 p2 pn-1

1-p1 1-p2

1

Coxian distribution can be used to approximate any distribution.

42

A manufaturing system

• Consider a machine which can be either UP or DOWN.

• The state of the machine is checked continuously.

• The average time to failure of an UP machine is 10 days.

• The average time for repair of a DOWN machine is 1.5 days.

• Assumed that UP time = E2 and DOWN time = E3.

• Draw the Markov chain model.