Book description
Markov processes are processes that have limited memory. In particular, their dependence on the past is only through the previous state. They are used to model the behavior of many systems including communications systems, transportation networks, image segmentation and analysis, biological systems and DNA sequence analysis, random atomic motion and diffusion in physics, social mobility, population studies, epidemiology, animal and insect migration, queueing systems, resource management, dams, financial engineering, actuarial science, and decision systems.
Covering a wide range of areas of application of Markov processes, this second edition is revised to highlight the most important aspects as well as the most recent trends and applications of Markov processes. The author spent over 16 years in the industry before returning to academia, and he has applied many of the principles covered in this book in multiple research projects. Therefore, this is an applicationsoriented book that also includes enough theory to provide a solid ground in the subject for the reader.
 Presents both the theory and applications of the different aspects of Markov processes
 Includes numerous solved examples as well as detailed diagrams that make it easier to understand the principle being presented
 Discusses different applications of hidden Markov models, such as DNA sequence analysis and speech analysis.
Table of contents
 Cover image
 Title page
 Table of Contents
 Copyright
 Acknowledgments
 Preface to the Second Edition
 Preface to the First Edition
 1. Basic Concepts in Probability
 2. Basic Concepts in Stochastic Processes
 3. Introduction to Markov Processes

4. DiscreteTime Markov Chains
 4.1 Introduction
 4.2 StateTransition Probability Matrix
 4.3 StateTransition Diagrams
 4.4 Classification of States
 4.5 LimitingState Probabilities
 4.6 Sojourn Time
 4.7 Transient Analysis of DiscreteTime Markov Chains
 4.8 First Passage and Recurrence Times
 4.9 Occupancy Times
 4.10 Absorbing Markov Chains and the Fundamental Matrix
 4.11 Reversible Markov Chains
 4.12 Problems
 5. ContinuousTime Markov Chains
 6. Markov Renewal Processes

7. Markovian Queueing Systems
 7.1 Introduction
 7.2 Description of a Queueing System
 7.3 The Kendall Notation
 7.4 The Little’s Formula
 7.5 The PASTA Property
 7.6 The M/M/1 Queueing System
 7.7 Examples of Other M/M Queueing Systems
 7.8 M/G/1 Queue
 7.9 G/M/1 Queue
 7.10 M/G/1 Queues with Priority
 7.11 Markovian Networks of Queues
 7.12 Applications of Markovian Queues
 7.13 Problems

8. Random Walk
 8.1 Introduction
 8.2 Occupancy Probability
 8.3 Random Walk as a Markov Chain
 8.4 Symmetric Random Walk as a Martingale
 8.5 Random Walk with Barriers
 8.6 Gambler’s Ruin
 8.7 Random Walk with Stay
 8.8 First Return to the Origin
 8.9 First Passage Times for Symmetric Random Walk
 8.10 The Ballot Problem and the Reflection Principle
 8.11 Returns to the Origin and the ArcSine Law
 8.12 Maximum of a Random Walk
 8.13 Random Walk on a Graph
 8.14 Correlated Random Walk
 8.15 ContinuousTime Random Walk
 8.16 SelfAvoiding Random Walk
 8.17 Nonreversing Random Walk
 8.18 Applications of Random Walk
 8.19 Summary
 8.20 Problems

9. Brownian Motion
 9.1 Introduction
 9.2 Mathematical Description
 9.3 Brownian Motion with Drift
 9.4 Brownian Motion as a Markov Process
 9.5 Brownian Motion as a Martingale
 9.6 First Passage Time of a Brownian Motion
 9.7 Maximum of a Brownian Motion
 9.8 First Passage Time in an Interval
 9.9 The Brownian Bridge
 9.10 Geometric Brownian Motion
 9.11 Introduction to Stochastic Calculus
 9.12 Solution of Stochastic Differential Equations
 9.13 Solution of the Geometric Brownian Motion
 9.14 The Ornstein–Uhlenbeck Process
 9.15 MeanReverting OU Process
 9.16 Fractional Brownian Motion
 9.17 Fractional Gaussian Noise
 9.18 Multifractional Brownian Motion
 9.19 Problems
 10. Diffusion Processes
 11. Levy Processes
 12. Markovian Arrival Processes
 13. Controlled Markov Processes
 14. Hidden Markov Models

15. Markov Point Processes
 15.1 Introduction
 15.2 Temporal Point Processes
 15.3 Specific Temporal Point Processes
 15.4 Spatial Point Processes
 15.5 Specific Spatial Point Processes
 15.6 Spatial–Temporal Point Processes
 15.7 Operations on Point Processes
 15.8 Marked Point Processes
 15.9 Introduction to Markov Random Fields
 15.10 Markov Point Processes
 15.11 Markov Marked Point Processes
 15.12 Applications of Markov Point Processes
 15.13 Problems
 References
Product information
 Title: Markov Processes for Stochastic Modeling, 2nd Edition
 Author(s):
 Release date: May 2013
 Publisher(s): Elsevier
 ISBN: 9780124078390
You might also like
book
HandsOn Machine Learning with ScikitLearn, Keras, and TensorFlow, 3rd Edition
Through a recent series of breakthroughs, deep learning has boosted the entire field of machine learning. …
book
Probabilistic Deep Learning
Probabilistic Deep Learning is a handson guide to the principles that support neural networks. Learn to …
book
Fluent Python, 2nd Edition
Don't waste time bending Python to fit patterns you've learned in other languages. Python's simplicity lets …
book
Deep Learning with Python, Second Edition
Deep Learning with Python, Second Edition introduces the field of deep learning using Python and the …