Markov model

A Markov model is a stochastic method for randomly changing systems where it is assumed that future states do not depend on past states. These models show all possible states as well as the transitions, rate of transitions and probabilities between them. 

Markov models are often used to model the probabilities of different states and the rates of transitions among them.  The method is generally used to model systems. Markov models can also be used to recognize patterns, make predictions and to learn the statistics of sequential data.

There are four types of Markov models that are used situationally:

• Markov chain - used by systems that are autonomous and have fully observable states
• Hidden Markov model - used by systems that are autonomous where the state is partially observable. 
• Markov decision processes - used by controlled systems with a fully observable state.
• Partially observable Markov decision processes - used by controlled systems where the state is partially observable.

Markov models can be expressed in equations or in graphical models. Graphic Markov models typically use circles (each containing states) and directional arrows to indicate possible transitional changes between them. The directional arrows are labeled with the rate or the variable one for the rate. Applications of Markov modeling include modeling languages, natural language processing (NLP), image processing, bioinformatics, speech recognition and modeling computer hardware and software systems.

Markov models are named after their creator, Andrey Markov, a Russian mathematician in the late 1800s to early 1900s.

An introduction to Markov models: