Deep Reinforcement Learning
Understanding how AI agents learn to interact with dynamic environments through trial, error, and probability.
Moving Beyond Supervised Learning
Unlike traditional deep learning paradigms where models learn to map static inputs to outputs, Deep Reinforcement Learning (RL) involves an active model interacting dynamically with its environment. The ultimate goal is to learn a policy that maximizes future rewards over many continuous time steps.
The Five Pillars of RL (Bicycle Example)
1. Agent
The actor making decisions.
3. Action (at)
A specific move executed in the world.
4. State (st)
Observation of the world.
5. Reward (rt)
Feedback measuring success.
Reward Optimization Approaches
2. Environment
The world the agent acts within.
Value Learning & DQN
The Q-Function captures the expected total future reward an agent can receive by executing a certain action given a state. A Deep Q-Network (DQN) uses a neural network to estimate these values for highly complex spaces.
If we attempted to build a traditional Q-table and mathematically calculate the true expected future reward for every possible raw pixel combination and every continuous distance measurement in our bicycle environment, the massive amount of calculation required would instantly freeze our systems—the computer would simply "die to compute." This explosion of states is exactly why this interactive simulator is a sampled approximation, and why Deep Neural Networks are strictly required to estimate Q-functions in real-world scenarios.
Downsides: Q-Learning cannot handle continuous action spaces, as it requires a distinct, predefined output list (e.g., exclusively Left, Right, or Straight). It is also entirely deterministic, failing to learn stochastic, adaptable policies.
Policy Learning & Continuous Actions
Instead of predicting static mathematical values for distinct choices, Policy Gradients output a probability distribution. This enables the modeling of infinite continuous actions (like precise steering wheel angles) by parameterizing a Gaussian curve defined by a Mean and Variance. The agent then samples from this curve to act.
Policy Gradient Update (Gradient Ascent) w' = w + ∇ log P(at | st) * Rt
The 5-Step Training Algorithm
Watch how a policy network optimizes its continuous actions. The agent learns from its mistakes iteratively in a continuous feedback loop.
Agent steers continuously until crash.
Log all (st, at, rt) triplets.
Decrease P(bad actions) via negative return. Increase P(good actions).
Interactive RL Simulator
🕹️ DQN (Discrete Actions)
The agent is restricted to rigid choices: strictly left (-2.5), straight (0.0), or strictly right (+2.5). Watch the dashboard and the bike: DQN agents abruptly jerk back and forth, failing to weave smoothly.
🌊 Policy Gradients (Continuous Actions)
The agent samples an exact, precise decimal angle (e.g., -1.24) from a Gaussian distribution. Watch the dashboard continuously update with precise floats as the bike fluidly weaves around obstacles.
Simulation Controls
Run the automated Agent. You can manually intervene at any time by dragging sliders or holding buttons.
1. Select Action Space
2. Manual Intervention
Drag sliders to manually override the AI's distribution probability.
3. Simulation Execution
This simulator is built in plain JavaScript to provide a visual intuition of the algorithms. While real-world RL requires massive neural networks and heavy compute, the underlying mathematical principles shown here remain exactly the same.