ACM/IDS 216 - Markov Chains, Discrete Stochastic Processes and Applications

Introduction to Markov chains and processes covering discrete and continuous state-spaces in both discrete and continuous time settings. Topics include irreducibility, aperiodicity, stationary and equilibrium distributions, convergence behavior, transience and recurrence, and the Ergodic Theorem. Emphasis on Markov Chain Monte Carlo (MCMC) algorithms, particularly Metropolis-Hastings and Simulated Annealing, with practical applications in scientific computing. Additional topics include coupling from the past, convergence rates, and an introduction to Markov Decision Processes.