Reinforcement learning
Mnih et al. · 2013 (NIPS Deep Learning Workshop)
Fetching research paper
Downloading PDF from the archive
Original source not responding
We could not fetch or display this PDF. The host may be down, blocking embedding, or your connection may have dropped.
A button will appear below to pick another paper from the lab.
Continue reading
Choose another paper from the research lab.
These notes are written in plain language for this specific paper—so you can grasp the ideas before you wrestle with the authors’ formal wording. Use the button to open the PDF near the matching section (approximate page; Chromium-style viewers support #page=, otherwise we open a new tab).
Classic reinforcement learning struggled when the “state” was raw pixels and the action space was large. The question: can a single deep network learn to play Atari from screen input only, using rewards as the only supervision?
A CNN approximates the Q-function (expected future reward per action). Experience replay stores past transitions and samples them randomly so training doesn’t chase correlated frames. A target network updates slowly to stabilize the moving target.
The Arcade Learning Environment provides dozens of Atari games—each a benchmark with the same interface. The agent sees stacks of frames, skips frames for speed, and clips rewards to keep scales stable.
They show human-level or near-human play on several games and learning curves that improve with training. Not every game is solved—some stay hard—but the paper proves deep RL from pixels is possible at all.
Sample efficiency is poor by today’s standards: millions of frames per game. Training can be unstable; hyperparameters matter. These limits spurred Double DQN, Rainbow, and many improvements.
They contrast with linear function approximators and tabular Q-learning that couldn’t scale to vision. The story: function approximation + replay + targets makes deep Q-learning feasible.
The paper spells out network shape, replay buffer, reward clipping, and frame preprocessing. Researchers reproduced DQN quickly; it’s a standard homework baseline—though full Atari runs need GPU time.
Eight highlights per paper—why each part matters before you read dense notation and proofs.
Raw frames go through a conv stack to Q-values per action—no engineered game-specific features. Representation learning sits inside RL, not bolted on.
Bellman backups estimate expected return per (state, action). The net predicts those values; argmax over actions yields a greedy policy. Ground the math before the hacks.
Consecutive frames correlate strongly; i.i.d. SGD assumes break that correlation. Shuffling past transitions stabilizes gradients like shuffling a dataset in supervised learning.
A lagged copy of weights defines stable targets for the Bellman update. Without it, chasing a moving target causes divergence—a common failure in naive deep Q-learning.
Frame stacking, downsampling, and reward clipping normalize difficulty across games. Know which choices are algorithmic vs. engineering convenience.
Random actions early encourage coverage of the state space. Too little exploration misses good policies; too much slows exploitation of what was learned.
One setup across diverse games tests robustness. Compare human-normalized scores to see which titles remain hard (sparse reward, long horizons).
Double DQN, dueling heads, prioritized replay, and policy-gradient methods address DQN weaknesses. DQN remains the canonical intro to function approximation in value-based RL.