Course 02407:
Stochastic
processes Fall 2026
Lecturer and instructor: Professor Bo Friis Nielsen
Instructor:
Clara Brimnes Gardner, Jonas Bruun Hubrechts
Contact: bfni@dtu.dk
Textbook: Gabriel J. Lord and Cónall Kelly
An Introduction to Stochastic Modelling fifth edition
- can maybe be bought at Polyteknisk Boghandel,
DTU. The bookstore offers a 10% discount off the announced price.
In my last communication with the bookstore there was still negotiations with the publisher.
Supplementary text on conditional expectation: Zdzisław Brzeźniak and Tomasz Zastawniak,
Basic stochastic processes : a course through exercises
Lectures are held in
Building 421, Room 071
Lectures will be held tuesdays between 8 and 10 (E3A).
Exercises and Problems sessions are held in the same room from 10 to 12.
Course plan:
- Lecture 1 - September 1: Markov Chains: Introduction
- Reading: Lord & Kelly 3.1-3.5
- A set of slides recapitulating basic probability
here, for printing
here.
- Slide show (draft) in pdf, printable
slides (draft) in pdf
-
Week 1.
Teaching notes (only Markov chain part).
-
Exercises and problems from Lord & Kelly .
- In class exercises: LK 3.1.1, LK 3.2.1, LK 3.4.1, LK 3.4.2, (LK 3.1.4), (LK 3.2.4), (LK 3.4.6).
LK exercises 3.1 ,
LK problems 3.1 ,
LK exercises and problems 3.2 and 3.4 ,
- In class problems:
LK 3.1.4 ,
LK 3.2.1 ,
LK 3.4.4 ,
(LK 3.2.5) ,
(LK 3.4.7) .
Zip file with hand written solutions for exercises and problems.
- Problems for self study:
LK 3.1.1 ,
LK 3.2.3 ,
LK 3.4.12 ,
(LK 3.5.2) .
- Lecture 2 - September 8: Markov Chains: Random walk, First Step Analysis, Random sums, Branching Processes
- Reading: Lord & Kelly 3.6 up to 3.6.1 you only need to understand the setup and the results, 3.7, 2.3, 3.9.1-3.9.2, 3.8 can be read superficially.
You can omit: 3.6.1,3.6.2,3.6.3, 3.8.1, 3.9.3
- Slide show (draft) in pdf, printable
slides (draft) in pdf
- Week 2.
Teaching notes.
- In class exercises: LK 3.6.1, LK 3.7.1, LK 2.3.3, LK 3.9.2.
- In class problems:
LK 3.7.1, LK 2.3.4, LK 3.9.4, (LK 3.6.7), (LK 3.8.3).
LK exercises and problems 3.6 ,
LK exercises and problems 3.7 ,
LK exercises and problems 3.8 ,
LK exercises and problems 3.9 ,
LK 2.3.4,
- Problems for self study:
LK 3.6.2 ([derive equations 3.65
only and discuss how you think the non-zero value of r will effect
the mean time to absorption]),
LK 3.7.4 ,
LK 3.8.2 ,
LK 3.9.7 .
- Lecture 3 - September 15: Markov Chains: Limiting and invariant distributions, classification of states and chains
- Reading: Lord & Kelly 4.1,(4.2),4.3,4.4,(4.5)
- Slide show (draft) in pdf, printable
slides (draft) in pdf
- Week 3.
Teaching notes.
- In class exercises: LK 4.1.1, LK 4.3.1, LK 4.4.2, (LK 4.3.2).
- In class problems:
LK 4.1.1 ,
LK 4.3.3 ,
(LK 4.1.13) ,
(LK 4.4.7 - to be simplified) .
Zip file with hand written solutions for exercises and problems.
- Problems for self study:
LK 4.1.12 ,
LK 4.3.2 ,
LK 4.4.3 .
- Lecture 4 - September 22:
Poisson Processes
- Reading: Lord & Kelly Chapter 5 - for details see weekly note
- Slide show (draft) in pdf, printable
slides (draft) in pdf
- Week 4 .
Teaching notes.
- In class exercises: LK 5.1.1, LK 5.3.2, LK 5.4.1, LK 5.4.3
- In class problems:
LK 5.3.7 ,
LK 5.1.9 ,
LK 5.2.6 ,
(LK 5.4.8) ,
(LK 5.3.9) ,
(LK 5.6.3) .
Zip file with hand written solutions for exercises and problems.
- Problems for self study:
LK 5.1.7 ,
LK 5.4.7 ,
LK 5.5.6 .
- Lecture 5 - September 29:
Birth and Death Processes
- Reading: Lord & Kelly 6.1-6.4 The last example of 6.4 can be omitted.
- Additional Reading:
note in time reversed Markov chains (discrete time)
- Slide show (draft) in pdf, printable
slides (draft) in pdf
- Week 5 .
Teaching notes.
- In class exercises: LK 6.1.2, LK6.4.4.
- In class problems:
(LK 6.1.3), LK 6.1.4, LK 6.2.1, (LK 6.3.1), LK 6.3.3, LK6.4.2, LK6.4.6.
- Problems for self study:
LK 6.1.2 ,
LK 6.2.4 ,
(LK 6.3.4) ,
LK 6.4.4 .
- Lecture 6 - October 6:
Finite State Continuous Time Markov Chains,
Queueing systems; basic and intermediate queueing theory
- Reading: Lord & Kelly 6.6, 8.1-8.2, 8.3.
-
Slide show (draft) in pdf,
printable slides (draft) in pdf.
- Week 6.
Teaching notes.
- In class exercises:
LK 6.6.2.
- In class problems:
Exercise 27,
Exercise 27,
(solution),
LK 9.2.2 ,
LK 9.2.6 ,
LK 9.2.4 ,
LK 9.3.2 .
Zip file with hand written solutions for exercises and problems.
- Problems for self study:
Exercise 30, LK 9.2.5, LK 9.2.7, (LK 9.3.1).
- Lecture 7 - October 20: Lord & Kelly 6.5, Phase-type distributions
- Lecture 8 - October 27:
Renewal Phenomena
The rest of the course plan is tentative. Updates and material will be provided.
The following material is from the fall semester of 2025.
Some adjustments are likely.
You can particularly expect updates due to the changes of textbook from the fourth to the fifth edition.
- Lecture 9 - November 3: Markov Decision Processes
- The lecture will be held by Clara Brimnes Gardner
- Reading:
Martin L. Puterman: Markov Decision Processes: 1,1, (1.6), - 2.1 This section is notationally heavy, but we will only focus on the simplest type of model, which has;
- in 2.1.1: Finite horizon,
- in 2.1.2: finite state and action sets,
- in 2.1.4 and 2.1.5: Markovian and deterministic (MD) decision rules and policies
- 2.1.6 can be read superficially, but note the connection to Markov chains on the bottom of page 24.
- 2.2 (except page 27), 4.1.2, 4.5 (Page 92), 4.6.4.
- printable slides (draft) in pdf, Nikolaj Primault's slides in pdf
- A precorona student relevant example of the secretary problem (youtube video).
- In class problems:
MDP exercise
Puterman 3.9 - Optimal parking,
Puterman 3.26 - The lazy adaptable lion,
(Puterman 2.5 (only part a)).
Zip file with hand written solutions for exercises and problems.
- Problems for self study: Puterman 3.10 & 3.11 Variants of secretary problem
- Lecture 10 - November 10:
Brownian Motion
- Lecture 11 - November 17:
Brownian Motion with Drift and Variations and Extensions 8.3-8.4.1
- Reading: Lord & Kelly 8.3-8.4.1 (to Page 424)
- Slide show (draft) in pdf, printable
slides (draft) in pdf
-
Week 11.
Teaching notes.
- In class exercises:
- In class problems:
LK8.3.3, (LK8.3.6), LK8.4.1, LK8.4.2, LK8.4.5.
- Problems for self study:
LK8.3.1, LK8.4.4 there is a typo in this exercise you should replace 1/2 with t/2, LK8.4.10,
- Lecture 12 - November 24:
Black-Scholes Option Pricing Formula and Ohrnstein-Uhlenbeck Process
- Reading: Lord & Kelly 8.4.1 from P.424 (Geometric Brownian Motion and Black Scholes), 8.5 (we might not make it to 8.5.4)
- Short note on stationary Ornstein-Uhlenbeck process .
Week 12 (draft only).
Teaching notes.
- In class exercises:
LK 8.4.5, LK 8.5.1, LK 8.5.2, LK 8.5.3
- In class problems:
( LK 8.4.6 ).
LK 8.4.7 ,
LK 8.4.8 ,
LK 8.4.10 ,
LK 8.5.2 ,
LK 8.5.3 .
Zip file with hand written solutions for exercises and problems.
- Problems for self study:
Brownian motion .
LK 8.3.1 ,
LK 8.4.4 ,
LK 8.4.10 .
- Lecture 13 - December 1:
General course overview
Zip file with selected exercises. The style of these exercises are somewhat similar to the type of questions you can expect in the take home exam..
Zip file with selected solutions..
Last change: 27/10 2026, by bfn