ECE 555 Control of Stochastic Systems Fall, 2008

Sean Meyn Wednesday & Friday, 11:30 -12:50 p.m. Room 170 EL

This course introduces both the analytical and computational aspects of stochastic control and performance evaluation, with applications to engineering, computer science, economics, and management science. Topics include Markov models and stochastic stability,development of controllawsbydynamicprogramming, complete andpartialinformation,Kalman filtering,and anintroductiontomachinelearning. It is intended for graduate students who have some background in control and stochastic processes. Experience with Matlab is also desirable.

References: The following textbooks are on reserve in the Engineering Library.

P. R Kumar, Stochastic systems: Estimation, identification, and adaptive control.

Torsten Soderstrom, Discrete-time stochastic systems: estimation and control.

Thefollowing are availablefree on-line(sendyourthankstoCambridgeUniversityPress):

S. P. Meyn and R. L. Tweedie, Markov Chains and Stochastic Stability.

S. P. Meyn, Control Techniques for Complex networks.

The course TA is Shankar Rao - srrao at illinois dot edu

Office hours Sean Meyn: Tuesdays, 4-5 p.m., 162 CSL. Shankar Rao: Wednesdays, 1-2:30 p.m., 154 CSL

Exams, homework, and grading Homework problems will be assigned on a semi-weekly basis, to be handed in at the beginning of class on the date due. They will be graded and returned the following week. Late homework cannot be accepted.

There will be two evening midterm exams, October 9 and November 20, from 7:00 -8:30 p.m. You will be allowed one sheet of notes(8¹ × 11; both sides) in the first exam, and two in the

2

second. Otherwise, the exams are closed-book and closed-notes.

Tentative grading scheme: Homework problems will count 20%, the midterm exam 25%, and the final will count 30% towards the final grade in the course.

In addition to the University-wide scheduled holidays, class is cancelled on

Wednesdays of October 22 & December 10, and September 24 & 26 Allerton Conference http://www.csl.uiuc.edu/allerton/

 

 

Course Outline

I. Markov Models

1) Overview and examples. 2) Linear and non-linear models. 3) Representations of π and value functions. 4) Lyapunov theory.

Allerton Conference: September 28–30.

II. Optimal Control

1) Controlled Markov chain models. 2) Markov and stationary policies. 3) Numerical techniques: Policy and value iteration, LP methods. 4) Partial information

Exam (October 9 )

III. Linear Theory

1) Linear Gaussian systems. 2) Optimal linear-quadradic and minimum-variance control. 3) Partial information and the Kalman filter.

Exam (November 20 )

Fall Break: November 24–28.

IV. Adaptation and Learning

1) Stochastic approximation. 2) Approximate dynamic programming. 3) Introduction to adaptation and machine learning.

Final Exam (1:30-4:30 p.m., Wednesday, December 17)