CS 294: Deep Reinforcement Learning, Fall 2017

IMPORTANT: If you are a UC Berkeley undergraduate student or non-EECS graduate student and want to enroll in the course for fall 2018, please do not email the instructors. We will have an application form for you to fill out in late April or early May, and we will select students from this list in August based on space availability and prerequisites.
Instructor: Sergey Levine
GSIs: Abhishek Gupta, Joshua Achiam
Lectures:Monday and Wednesday 10:30am-12pm in 306 Soda Hall
Office Hours: MW 1-2pm, F 9-10am, by appointment (see signup sheet on Piazza)
Communication: Piazza will be used for announcements, general questions and discussions, clarifications about assignments, student questions to each other, and so on. To sign up, go to Piazza and sign up with “UC Berkeley” and “CS294-112”. We request that enrollment be restricted to students enrolled in the class or on the waitlist.
If you are not enrolled, but are interested in following and discussing the course, there is a subreddit forum here: reddit.com/r/berkeleydeeprlcourse/


Table of Contents



Fall 2017 Materials

Lecture Videos

The course lectures are available below. The course is not being offered as an online course, and the videos are provided only for your personal informational and entertainment purposes. They are not part of any course requirement or degree-bearing university program.
For all videos, click here.
For live stream, click here.

Lectures, Readings, and Assignments

Below you can find an outline of the course. Slides and references will be posted as the course proceeds.

  • Aug 23: Introduction and course overview (Levine)
  • Aug 28: Supervised learning and imitation (Levine)
  • Aug 30: Reinforcement learning introduction (Levine)
  • Sep 4: Holiday - no class.
  • Sep 6: Policy gradients introduction (Levine)
    • Slides
    • Homework 1 milestone due
  • Sep 8: Neural networks review (Achiam)
  • Sep 11: Actor-critic introduction (Levine)
  • Sep 13: Value functions introduction (Levine)
  • Sep 18: Advanced Q-learning algorithms (Levine)
  • Sep 20: Optimal control and planning (Levine)
    • Slides
    • Homework 2 is due, Homework 3 is out: DQN
  • Sep 25: Learning dynamical systems from data (Levine)
  • Sep 27: Learning policies by imitating optimal controllers (Levine)
  • Oct 2: Advanced model learning and images (Guest lecture: Chelsea Finn)
  • Oct 4: Connection between inference and control (Levine)
  • Oct 9: Inverse reinforcement learning (Levine)
    • Slides
    • Project proposal is due
  • Oct 11: Advanced policy gradients (natural gradient, importance sampling) (Achiam)
  • Oct 16: Exploration (Levine)
  • Oct 18: Exploration (part 2) and transfer learning (Levine)
  • Oct 23: Multi-task learning and transfer (Levine)
  • Oct 25: Meta-learning and parallelism (Levine)
  • Oct 30: Advanced imitation learning and open problems (Levine)
  • Nov 1: Guest lecture: Igor Mordatch, OpenAI
  • Nov 6: Guest lecture: Mohammad Norouzi, Google
    • Project milestone
  • Nov 8: Guest lecture: Pieter Abbeel, UC Berkeley and OpenAI
  • Nov 13: Guest lecture: Emma Brunskill, Stanford University
  • Nov 15: Guest lecture: Aviv Tamar, UC Berkeley
  • Nov 20: Guest lecture: John Schulman, OpenAI
  • Nov 22: Holiday - no class
  • Nov 27: Final project presentations 1
  • Nov 29: Final project presentations 2

Previous Offerings

A full version of this course was offered in Spring 2017. Lecture videos from Spring 2017, are available here

An abbreviated version of this course was offered in Fall 2015.

Prerequisites

CS189 or equivalent is a prerequisite for the course. This course will assume some familiarity with reinforcement learning, numerical optimization and machine learning. Students who are not familiar with the concepts below are encouraged to brush up using the references provided right below this list. We’ll review this material in class, but it will be rather cursory.

  • Reinforcement learning and MDPs
    • Definition of MDPs
    • Exact algorithms: policy and value iteration
    • Search algorithms
  • Numerical Optimization
    • gradient descent, stochastic gradient descent
    • backpropagation algorithm
  • Machine Learning
    • Classification and regression problems: what loss functions are used, how to fit linear and nonlinear models
    • Training/test error, overfitting.

For introductory material on RL and MDPs, see

For introductory material on machine learning and neural networks, see

John's lecture series at MLSS

  • Lecture 1: intro, derivative free optimization
  • Lecture 2: score function gradient estimation and policy gradients
  • Lecture 3: actor critic methods
  • Lecture 4: trust region and natural gradient methods, open problems

Courses

Relevant Textbooks