book

Markov Processes for Stochastic Modeling, 2nd Edition

Name: Markov Processes for Stochastic Modeling, 2nd Edition
Author: Oliver Ibe
ISBN: 9780124078390

by Oliver Ibe

May 2013

Intermediate to advanced

514 pages

12h 4m

English

Elsevier

Read now

Unlock full access

Cover image
Title page
Table of Contents
Copyright
Acknowledgments
Preface to the Second Edition
Preface to the First Edition
1. Basic Concepts in Probability
1.1 Introduction1.2 Random Variables1.3 Transform Methods1.4 Bivariate Random Variables1.5 Many Random Variables1.6 Fubini’s Theorem1.7 Sums of Independent Random Variables1.8 Some Probability Distributions1.9 Limit Theorems1.10 Problems
2. Basic Concepts in Stochastic Processes
2.1 Introduction2.2 Classification of Stochastic Processes2.3 Characterizing a Stochastic Process2.4 Mean and Autocorrelation Function of a Stochastic Process2.5 Stationary Stochastic Processes2.6 Ergodic Stochastic Processes2.7 Some Models of Stochastic Processes2.8 Problems
3. Introduction to Markov Processes
3.1 Introduction3.2 Structure of Markov Processes3.3 Strong Markov Property3.4 Applications of Discrete-Time Markov Processes3.5 Applications of Continuous-Time Markov Processes3.6 Applications of Continuous-State Markov Processes3.7 Summary

4. Discrete-Time Markov Chains
4.1 Introduction4.2 State-Transition Probability Matrix4.3 State-Transition Diagrams4.4 Classification of States4.5 Limiting-State Probabilities4.6 Sojourn Time4.7 Transient Analysis of Discrete-Time Markov Chains4.8 First Passage and Recurrence Times4.9 Occupancy Times4.10 Absorbing Markov Chains and the Fundamental Matrix4.11 Reversible Markov Chains4.12 Problems
5. Continuous-Time Markov Chains
5.1 Introduction5.2 Transient Analysis5.3 Birth and Death Processes5.4 First Passage Time5.5 The Uniformization Method5.6 Reversible CTMCs5.7 Problems
6. Markov Renewal Processes
6.1 Introduction6.2 Renewal Processes6.3 Renewal-Reward Process6.4 Regenerative Processes6.5 Markov Renewal Process6.6 Semi-Markov Processes6.7 Markov Regenerative Process6.8 Markov Jump Processes6.9 Problems
7. Markovian Queueing Systems
7.1 Introduction7.2 Description of a Queueing System7.3 The Kendall Notation7.4 The Little’s Formula7.5 The PASTA Property7.6 The M/M/1 Queueing System7.7 Examples of Other M/M Queueing Systems7.8 M/G/1 Queue7.9 G/M/1 Queue7.10 M/G/1 Queues with Priority7.11 Markovian Networks of Queues7.12 Applications of Markovian Queues7.13 Problems
8. Random Walk
8.1 Introduction8.2 Occupancy Probability8.3 Random Walk as a Markov Chain8.4 Symmetric Random Walk as a Martingale8.5 Random Walk with Barriers8.6 Gambler’s Ruin8.7 Random Walk with Stay8.8 First Return to the Origin8.9 First Passage Times for Symmetric Random Walk8.10 The Ballot Problem and the Reflection Principle8.11 Returns to the Origin and the Arc-Sine Law8.12 Maximum of a Random Walk8.13 Random Walk on a Graph8.14 Correlated Random Walk8.15 Continuous-Time Random Walk8.16 Self-Avoiding Random Walk8.17 Nonreversing Random Walk8.18 Applications of Random Walk8.19 Summary8.20 Problems
9. Brownian Motion
9.1 Introduction9.2 Mathematical Description9.3 Brownian Motion with Drift9.4 Brownian Motion as a Markov Process9.5 Brownian Motion as a Martingale9.6 First Passage Time of a Brownian Motion9.7 Maximum of a Brownian Motion9.8 First Passage Time in an Interval9.9 The Brownian Bridge9.10 Geometric Brownian Motion9.11 Introduction to Stochastic Calculus9.12 Solution of Stochastic Differential Equations9.13 Solution of the Geometric Brownian Motion9.14 The Ornstein–Uhlenbeck Process9.15 Mean-Reverting OU Process9.16 Fractional Brownian Motion9.17 Fractional Gaussian Noise9.18 Multifractional Brownian Motion9.19 Problems
10. Diffusion Processes
10.1 Introduction10.2 Mathematical Preliminaries10.3 Models of Diffusion10.4 Examples of Diffusion Processes10.5 Correlated Random Walk and the Telegraph Equation10.6 Introduction to Fractional Calculus10.7 Anomalous (or Fractional) Diffusion10.8 Problems
11. Levy Processes
11.1 Introduction11.2 Generalized Central Limit Theorem11.3 Stable Distribution11.4 Levy Distribution11.5 Levy Processes11.6 Infinite Divisibility11.7 Jump-Diffusion Processes
12. Markovian Arrival Processes
12.1 Introduction12.2 Overview of Matrix-Analytic Methods12.3 Markovian Arrival Process12.4 Batch Markovian Arrival Process12.5 Markov-Modulated Poisson Process12.6 Markov-Modulated Bernoulli Process12.7 Sample Applications of MAP and Its Derivatives12.8 Problems
13. Controlled Markov Processes
13.1 Introduction13.2 Markov Decision Processes13.3 Semi-MDPs13.4 Partially Observable MDPs13.5 Problems
14. Hidden Markov Models
14.1 Introduction14.2 HMM Basics14.3 HMM Assumptions14.4 Three Fundamental Problems14.5 Solution Methods14.6 Types of HMMs14.7 HMMs with Silent States14.8 Extensions of HMMs14.9 Other Extensions of HMM14.10 Problems
15. Markov Point Processes
15.1 Introduction15.2 Temporal Point Processes15.3 Specific Temporal Point Processes15.4 Spatial Point Processes15.5 Specific Spatial Point Processes15.6 Spatial–Temporal Point Processes15.7 Operations on Point Processes15.8 Marked Point Processes15.9 Introduction to Markov Random Fields15.10 Markov Point Processes15.11 Markov Marked Point Processes15.12 Applications of Markov Point Processes15.13 Problems
References

Content preview from Markov Processes for Stochastic Modeling, 2nd Edition

Markov Renewal Processes

6.1 Introduction

A Markov renewal process is a stochastic process that is a combination of Markov chains and renewal processes. It can be described as a vector-valued process from which processes, such as the Markov chain, semi-Markov process (SMP), Poisson process, and renewal process, can be derived as special cases of the process. Before we discuss Markov renewal process, we first provide basic introduction to renewal processes and regenerative processes.

6.2 Renewal Processes

Consider an experiment that involves a set of identical light bulbs whose lifetimes are independent. The experiment consists of using one light bulb at a time, and when it fails it is immediately replaced by another light bulb from the set. Each ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Bayesian Analysis of Stochastic Process Models

Publisher Resources

ISBN: 9780124077959

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design