Research

Interests: Reinforcement Learning, Continual learning, Deep learning.

My research objective is to understand the computational principles underlying intelligent decision-making. To this end, most of my research is in reinforcement learning (RL). Within RL, my PhD research has been at the intersection of temporal difference learning and function approximation, specifically deep RL. In addition to deep RL, I have spent time researching inverse RL, goal-conditioned RL, temporal abstraction, and RL for language model reasoning.

Talks | Code

Talks

  • Accelerating Deep Q-learning with the Mean-expansion Layer (Video)
    Amii AI Seminar introducing the mean-expansion layer for Deep Q-learning.

  • Recent Insights in Value-based Deep Reinforcement Learning (Slides) | (Video)
    Amii AI Seminar surveying the Policy Churn, The Tandem effect, and the curse of diversity.

  • Revisiting Overestimation in Value-based Deep Reinforcement Learning (Slides) | (Video)
    Amii AI Seminar on a preliminary version of Deep Double Q-learning. Includes a tutorial on value overestimation.

  • Trajectory-Ranked Reward Extrapolation (Slides)
    Internal talk I gave on our ICML 2019 work at Preferred Networks.

  • Deterministic Implementations for Reproducibility in Deep Reinforcement Learning (Slides)
    Talk given at the AAAI 2019 Workshop on Reproducibility in AI for our paper.

Code

  • PFRL
    PFRL is a PyTorch deep reinforcement learning library. Part of my work at Preferred Networks was devoted to developing algorithms and infrastructure for PFRL.
  • Table-RL
    Table-rl is a library for tabular reinforcement learning in Python.
  • MAML
    A PyTorch implementation of Model Agnostic Meta Learning (MAML) on the Sinusoid task.
  • Atari Behavioral Cloning
    A Behavioral cloning implementation on Atari. This was adapted for our T-REX paper and modeled after DQfD's Behavioral cloning.
  • Bayesian Inverse RL
    A basic implementation of Bayesian Inverse Reinforcement Learning.