STA2311: Advanced Computational Methods for Statistics I
PhD course, University of Toronto, 2023
STA2311 is a new course required for most first-year students in the Statistical Theory and Applications PhD stream at the University of Toronto. The course, which examines optimization and sampling techniques (focusing on both underlying motivation and theoretical justification), was fully designed by Prof. Radu V. Craiu and me.
Syllabus
Classes
The course included eleven classes; the first eight are given here.
Class 1: Validation
Slides
Optional Readings and Resources
- Burns (2012) - The R Inferno
- Wickham (2019) - Advanced R
- Arlot and Celisse (2010) - A Survey of Cross-Validation Procedures for Model Selection
- Stone (1977) - An Asymptotic Equivalence of Choice of Model by Cross-Validation and Akaike’s Criterion
- Celisse (2014) - Optimal Cross-Validation in Density Estimation with the l^2-Loss
Class 2: Classical Optimization Methods
Slides
Practice Problems
Optional Readings and Resources
- Ruder (2016) - An overview of gradient descent optimization algorithms
- Petersen and Pedersen (2012) - The Matrix Cookbook
Class 3: The EM Algorithm
Slides
Practice Problems
Optional Readings and Resources
- Dempster et al (1977) - Maximum Likelihood from Incomplete Data via the EM Algorithm
- Liu and Rubin (1994) - The ECME Algorithm: A Simple Extension of EM and ECM with Faster Monotone Convergence
- Meng and Rubin (1993) - Maximum Likelihood Estimation via the ECM Algorithm: A General Framework
- Meng and Rubin (1994) - On the Global and Componentwise Rates of Convergence of the EM Algorithm
- Meng and Rubin (1991) - Using EM to Obtain Asymptotic Variance-Covariance Matrices: The SEM Algorithm
- Meng and van Dyk (1997) - The EM Algorithm–An Old Folk-Song Sung to a Fast New Tune
- Meng and van Dyk (1999) - Seeking Efficient Data Augmentation Schemes via Conditional and Marginal Augmentation
Class 4: Stochastic Optimization
Slides
Practice Problems
Optional Readings and Resources
- Bertsimas and Tsitsiklis (1993) - Simulated Annealing
- Whitley (1994) - A Genetic Algorithm Tutorial
- Chatterjee et al (1995) - Genetic Algorithms and their Statistical Applications: An Introduction
Class 5: Variational Inference
Slides
Practice Problems
Optional Readings and Resources
- The Open University - Introduction to the Calculus of Variations
- Zhang et al (2018) - Advances in Variational Inference
- Bishop (2008) - Pattern Recognition and Machine Learning
- Blei et al (2016) - Variational Inference: A Review for Statisticians
Class 6: Simulation and Monte Carlo
Slides
Practice Problems
Optional Readings and Resources
Class 7: MCMC Basics
Slides
Practice Problems
Optional Readings and Resources
- He at al (2016) - Scan Order in Gibbs Sampling: Models in Which it Matters and Bounds on How Much
- Holms and Jasra (2009) - Antithetic Methods for Gibbs Samplers
- Liu et al (2000) - The Multiple-Try Method and Local Optimization in Metropolis Sampling
Class 8: MCMC Tuning and Diagnostics
Slides
Practice Problems
Optional Readings and Resources
- Vats et al - SimTools: Variability assessment for simulation methods in R
- Plummer et al - coda: Output Analysis and Diagnostics for MCMC