Markov models – lesson 6
Computation of system with repair or backup in general is very complex
 Can be simplified for some failure and repair distribution (exponential)  using Markov
models
Basic features of Markov models for system reliability:
 function of two random variables (state of the system and time or other variable on which
the state depends)
 two random variable can be discrete or continuous  4-type of models
basic models:
discrete time
discrete state of the system
Markov chain
continuous time
discrete state of the system
Markov process
 Markov model is define as:
set of probability transitions between states (initial state  consecutive state)
S1
p
S2
Where: p depends only on state S1 & S2 (it do not depend on old previous states,
sometimes Markov models are called models without memory)
p=p(S1,S2)
In Markov model are defined mutually exclusive states of the system:
Example:
1 component ~ 2 states (failure, correct working)
without repair
Definition:
initial state ~ t=0
final states ~ t>> 
 System of Markov equations ~ description of probability of transition from initial
state to final states
 System of Markov equations suppose:
1) Probability of transition from one state to another in time interval t is i(t).t, where i(t)
is probability rate of transition (so called hazard) between both states
2) Probability of more than one transition in time t is 0 (negligible error)
Remark: if i(t)=i  homogenous model
Example: suppose we have system with one component with two states and without repair.
The probability PS0(t+t) is the probability that the system will be in state without failure S0
in time t+t.
It holds:
PS0(t+t)= PS0(t).(1-(t).t);
where PS0(t) probability of working without failure in time t and
(1-(t).t) is probability that the failure was not in time t
 similar probability of failure state S1
PS1(t+t)= PS0(t).(t).t+ PS1(t);
where (t).t is probability of failure in time t
PS1(t) is probability that the system was in failure state S1 till time t
Therefore:
 probability of the system failure: (t).t
 probability of failure persistence: (~state S1) =1
(absorbing state)
Previous equations can be converted:
With limit for t  0 :
PS 0 t  t   PS 0 t 
   t PS 0 t 
t
PS1t  t   PS1t 
  t PS 0 t 
t
dPS 0 t 
  t PS 0 t   0
dt
dPS 1 t 
  t PS 0 t 
dt
 general init condition is: t=0, So PS0(0)=1 and PS1(0)=0
Solution of previous equation system
dPS 0 t 
   t t
PS 0 t 
t
ln PS 0 t      t dt  ln c
0
t
PS 0 t   ce
   t dt
0
; where c  PS 0 0   1
(1)
t
   t dt
 R t   PS 0 t   e 0
Solution equation(2), similar (can be computed by PS0(t)+ PS1(t)=1)
t
   t dt
 Q t   PS1t   1  e 0
 Markov model graph representation  oriented graph (previous example):

1-t
t
S0(=x1)
S1(=x1)
Necessary condition for every state:
 probabilities of all transitions from one state = 1
situation for two components without repair:
(4 states:
X1X 2 , X1X 2 , X1 X 2 , X1 X 2 )
13t
01t
[01+02]t
S1(=X1X2)
13t
1
this transition is vanished
S0(=X1X2)
(in practice do not arrive)
02t
23t
23t
S2(=X1X2)
The previous situation is described by equations for S0, S1, S2, S3
S0:
S1:
S2:
S3:
PS0(t+t)=[1-(01+02)t] PS0(t)
PS1(t+t)=01t PS0(t)+(1-13t)PS1(t)
PS2(t+t)=02t PS0(t)+(1-23t)PS2(t)
PS3(t+t)=13t PS1(t)+23t PS2(t)+1.Ps3(t)
S3(=X1X2)
With matrix:
[PS0(t+t) PS1(t+t) PS2(t+t) PS3(t+t)]=
02 t
0 
1  01  02 t 01t

0
1  13 t
0
13 t 

[PS0(t) PS1(t) PS2(t) PS3(t)]*

0
0 1   23 t  23 t 


0
0
0
1 

so: PS(t+t)= PS(t).p
matrix of transition probabilities
vector of probabilities of states in time t+t and t
Matrix of transition probabilities can be created directly from graph representation of Markov
model
Pkj denotes the probability of transition from state k  to state j
p=[Pkj]  rows ~ initial states in time t

columns ~ end states in time t+t
Remark:
element for k=j
zero element ~
P
kj
1 !!
~
probability of persist in state
transition cannot arrive
~
sum in rows =1, system has to be in next
k
step in some state from defined state set
(stochastic matrix)
Writing in differential type:
changing the equation for S0 (from previous example):
PS 0 t  t   PS 0 t 
 01  02 PS 0 t  and limit t 0 we receive:
t
PS 0 t   01  02 PS 0 t 
and similar for S1, S2, S3
P t    P t    P t 
S1
01 S 0
13 S 1
S3
13 S 1
23 S 2
PS 2 t   02 PS 0 t    23 PS 2 t 
P t    P t    P t 
Using matrix:
 01  02  01

0
 13
 t   P t .z  matrix of transition rate z  
P
S
S

0
0

0
0

02
0
  23
0
0 
13 
 23 

0 

sum in every row is = 0 !!!
 solution of previous equation system for previous example PSi(0)=0 ;
i=0…3
For i = constant we receive:
PS 0 t   e  01  02 t




01
e  13t  e  ( 01  02 ) t
01  02  13
01
PS 2 t  
e  23t  e  ( 01  02 )t
01  02  23
PS1t  
PS 3 t   1  PS1t   PS1t   PS 2 t 
probability of states S0…S3 computed independently on structure of the system
 it holds for every system configuration with two components without repair
So special case:
Serial system:
R(t)=PS0(t)
- both components working without failure
Parallel system:
R(t)=PS0(t)+ PS1(t)+ PS2(t)
-at least one component works without failure
System with backup:
02=0, 01=13, PS2(t)=0  R(t)=PS0(t)+PS1(t)=e-t+te-
Conclusion:
Complexity of Markov model
 depends on number of system’s states m

m – differential equations of 1-st order
 in general for n components with k-states m=kn !! increase very quickly
simplification
 distinction only states with different number of failure components (for n components
with 2 states the previous number of Markov states m=2n decrease to m=n-1)
Example: Previous system with two components can change by merging states S 1 and S2 and
introducing probability PS1'(t)=PS1(t)+ PS2(t)
Markov graph is:
1-'01t
1-'12t
'01t
S'0(=X1X2)
No failure
1
'12t
S'1(=X1X2+ X1X2)
One failure
S'3(=X1X2)
Two failures
Simplification of graph for 13=23 can be different
and
'01=01+02 , '12=13=23
Using Markov model for systems with repair
 How the repair changes the Markov graph?
It introduce the transition from states with w failed components to states with
w-1 failed components.
Example: Markov graph for system with one component with repair (constant failure rate 
and repair rate ).
PS 0 t   PS 0 t   PS1 t 
P t   P t   P t 
S1
S0
S1
1-t
1-t
t
S1(=x1)
S0(=x1)
t
With initial condition PS0(0)=1 and PS1(0)=0 


exp     t 
 


PS1 t  

exp     t 
 
in time t the system is working without failure = availability factor KP(t)=PS0(t)
PS 0 t  

The stable value of KP = lim ´K p t  
t 


1
1

Usually the mean values holds
2

1


3

1

 K P t  
4

1
1


5

 1 
t


 Solution of complex systems with repair:
 similar to system without repair (introduce the new transition that correspond to
repairs  modification of some matrix elements)
 for big number of states can simplify by merging some states
 constant failure rate and repair rate  homogenous differential equations
 simply solution