Robot Learning (M2608.002700) Spring 2026

Course Description

An intelligent robotic agent is required to be able to adapt and learn from new environments, tasks, and situations through interactions with its surroundings. Robot learning is a field that combines robotics and artificial intelligence (AI) to develop technologies for robotic agents that can acquire new skills and knowledge through experience, similar to humans. Robot learning encompasses various techniques and approaches that enable robots to become more autonomous and capable of performing tasks without explicit programming for all possible scenarios. In this course, we will review recent advances in robot learning, including imitation learning and deep reinforcement learning. We will first review Markov decision processes (MDP) and reinforcement learning. Then we will discuss recent developments in imitation learning, deep learning, and deep reinforcement learning, including topics such as behavior cloning, inverse reinforcement learning, policy gradient, deep Q-network (DQN), generative adversarial imitation learning, maximum entropy reinforcement learning, safe reinforcement learning, and offline reinforcement learning. Recent advances in robot foundation models will be discussed as well. This is an advanced graduate course and substantial reading and programming projects will be assigned. Students are expected to participate actively in class. Lectures will be in English.

Announcements

• [02/23] Please read Ethics of Learning.

Schedule

• Week 1:

• 03/03: Introduction

• Ch. 3 from Reinforcement Learning: An Introduction; Ch. 16 from AIMA

• Week 2:

• 03/09: Review on MDPs and POMDPs

• Deep Learning Short Course

• Ch. 3 from Reinforcement Learning: An Introduction; Ch. 16 from AIMA

• 03/11: Reinforcement learning

• Ch. 4, Ch. 6 from Reinforcement Learning: An Introduction; Ch. 23 from AIMA

• Week 3:

• HW1: Deep learning tutorial (03/16 ~ 03/23 23:59KST)

• 03/16: Review of probability theory; Gaussian process regression

• Gaussian process regression (Ch. 2 from Gaussian Processes for Machine Learning)

• Sungjoon Choi, Eunwoo Kim, Kyungjae Lee, Songhwai Oh, "Real-Time Nonparametric Reactive Navigation of Mobile Robots in Dynamic Environments," Robotics and Autonomous Systems, vol. 91, pp. 11–24, May 2017.

• 03/18: Behavior cloning (leveraged Gaussian process regression)

• Sungjoon Choi, Kyungjae Lee, and Songhwai Oh, "Robust Learning from Demonstrations with Mixed Qualities Using Leveraged Gaussian Processes," IEEE Transactions on Robotics, vol. 35, no. 3, pp. 564-576, Jun. 2019.

• Sungjoon Choi, Eunwoo Kim, Kyungjae Lee, and Songhwai Oh, "Leveraged Non-Stationary Gaussian Process Regression for Autonomous Robot Navigation," in Proc. of the IEEE International Conference on Robotics and Automation (ICRA), May 2015.

• Sungjoon Choi, Kyungjae Lee, and Songhwai Oh, "Robust Learning from Demonstration Using Leveraged Gaussian Processes and Sparse-Constrained Optimization," in Proc. of the IEEE International Conference on Robotics and Automation (ICRA), May 2016.

• Sungjoon Choi, Kyungjae Lee, and Songhwai Oh, "Scalable Robust Learning from Demonstration with Leveraged Deep Neural Networks," in Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Sep. 2017.

• Week 4:

• HW2: Imitation learning (03/23 ~ 03/30 23:59KST)

• 03/23: DAgger, Behavior cloning applications

• S. Ross, G. Gordon, and D. Bagnell, "A reduction of imitation learning and structured prediction to no-regret online learning," in Proc. of the international conference on artificial intelligence and statistics, 2011.

• Giusti, Alessandro, Jérôme Guzzi, Dan C. Cireşan, Fang-Lin He, Juan P. Rodríguez, Flavio Fontana, Matthias Faessler et al. "A machine learning approach to visual perception of forest trails for mobile robots." IEEE Robotics and Automation Letters 1, no. 2 (2016): 661-667. [Project Page with Datasets]

• Loquercio, Antonio, Ana Isabel Maqueda, Carlos R. Del Blanco, and Davide Scaramuzza. "DroNet: Learning to Fly by Driving." IEEE Robotics and Automation Letters (2018). [Project Page with Code and Datasets]

• 03/25: Deep Q Learning

• Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., and Petersen, S. "Human-level control through deep reinforcement learning," Nature, 2015.

• H. van Hasselt, A. Guez, and D. Silver, "Deep reinforcement learning with double q-learning," in Proc. of the AAAI Conference on Artificial Intelligence (AAAI), Feb. 2016.

• Z. Wang, N. de Freitas, and M. Lanctot, "Dueling Network Architectures for Deep Reinforcement Learning," in Proc. of the International Conference on Machine Learning (ICML), Jun. 2016.

• T. Schaul, J. Quan, I. Antonoglou, and D. Silver, "Prioritized Experience Replay," arXiv, 2015. 

• Week 5:

• HW3: Deep Q learning (03/30 ~ 04/06 23:59KST)

• 03/30: Sparse MDPs

• Kyungjae Lee, Sungjoon Choi, and Songhwai Oh, "Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learning," IEEE Robotics and Automation Letters, vol. 3, no. 3, pp. 1466-1473, Jul. 2018. [Supplementary Material | Video | arXiv preprint]

• 04/01: Policy gradient

• R. J. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning," Machine Learning, 1992.

• R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, "Policy gradient methods for reinforcement learning with function approximation," Advances in Neural Information Processing Systems (NIPS), Nov. 2000.

• Baxter, Jonathan, and Peter L. Bartlett. "Reinforcement learning in POMDP's via direct gradient ascent." ICML. 2000.

• J. Baxter and P. L. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research, 15:319--350, 2001.

References

• Reinforcement Learning: An Introduction (2018, 2nd Edition) Richard S. Sutton, Andrew G. Barto

• Artificial Intelligence: A Modern Approach (4th edition), Stuart Russell and Peter Norvig, Prentice Hall, 2022. (AIMA Website)

• Gaussian Processes for Machine Learning, Carl Edward Rasmussen and Christopher K. I. Williams, The MIT Press, 2006.

Prerequisites

• (430.457) Introduction to Intelligent Systems (지능시스템개론).

• Also requires strong background in algorithms, linear algebra, probability, and programming.