State Novelty: Random Network Distillation (RND)

While methods like Intrinsic Curiosity Motivation (ICM) leverage prediction errors about the environment's dynamics to generate intrinsic rewards, they can sometimes be distracted by aspects of the environment that are inherently stochastic or hard to predict but not necessarily interesting for exploration (the "noisy TV problem"). Random Network Distillation (RND) offers an alternative approach to generating intrinsic rewards based purely on state novelty, sidestepping the need to predict environment dynamics.

The core idea behind RND is elegant: measure how familiar a state is by checking how accurately a neural network, trained on visited states, can predict the output of a fixed, randomly initialized target network for that same state.

The RND Mechanism

RND employs two neural networks that process the observed state $s_t$:

1. Target Network ($f$): This network is initialized randomly at the beginning of training, and its parameters $\theta_f$ are kept fixed throughout the entire learning process. It takes the state $s_t$ as input and produces a feature embedding $f(s_t; \theta_f)$. Think of it as a fixed, random projection of the state space.
2. Predictor Network ($\hat{f}$): This network is trained concurrently with the agent's policy. Its goal is to predict the output of the target network for the states the agent encounters. It takes $s_t$ as input and outputs $\hat{f}(s_t; \theta_{\hat{f}})$, where $\theta_{\hat{f}}$ are the parameters being learned.

The predictor network $\hat{f}$ is trained by minimizing the Mean Squared Error (MSE) between its prediction and the target network's output, using the states visited by the agent:

$$L(\theta_{\hat{f}}) = \mathbb{E}_{s \sim D} \left[ || \hat{f}(s; \theta_{\hat{f}}) - f(s; \theta_f) ||^2 \right]$$

where $D$ represents the distribution of states visited by the agent.

The RND architecture uses a trained predictor network to approximate the output of a fixed random target network for a given state. The prediction error serves as the intrinsic reward.

Calculating the Intrinsic Reward

The intrinsic reward $r_t^i$ generated by RND at time step $t$ is simply the prediction error (typically squared error) of the predictor network for the current state $s_t$:

$$r_t^i = || \hat{f}(s_t; \theta_{\hat{f}}) - f(s_t; \theta_f) ||^2$$

The intuition is straightforward:

• Novel States: When the agent encounters a state $s_t$ that it hasn't seen often, the predictor network $\hat{f}$ will likely make a poor prediction of the fixed target $f(s_t)$. This results in a large prediction error and, consequently, a high intrinsic reward $r_t^i$. This high reward encourages the agent to explore this novel state and similar states further.
• Familiar States: As the agent repeatedly visits a particular state or region of the state space, the predictor network $\hat{f}$ gets trained on these states and becomes better at predicting the target network's output for them. The prediction error decreases, leading to a lower intrinsic reward. This reduction in reward naturally guides the agent away from already well-explored areas towards potentially more rewarding, unexplored parts of the environment.

This process "distills" the knowledge of visited states into the predictor network, and the error acts as a signal for novelty.

Normalization and Reward Combination

The raw prediction errors can vary significantly in scale depending on the state, the network architectures, and the stage of training. It's common practice to normalize the intrinsic rewards to stabilize learning. A typical approach is to divide the raw error by a running estimate of the standard deviation of the intrinsic rewards encountered so far.

The final reward signal used to train the RL agent (e.g., using PPO or A2C) is often a weighted sum of the extrinsic reward $r_t^e$ from the environment and the normalized intrinsic reward $r_t^i$:

$$r_t = r_t^e + \beta r_t^i$$

The hyperparameter $\beta$ controls the influence of the exploration bonus relative to the task reward.

Advantages of RND

1. Robustness to Stochasticity: Because the target $f(s_t)$ depends only on the current state $s_t$ and the fixed parameters $\theta_f$, RND is inherently less sensitive to unpredictable elements in the environment dynamics compared to methods trying to predict the next state or the consequences of actions. It focuses purely on state novelty.
2. Simplicity and Stability: RND avoids the complexities of learning an inverse or forward dynamics model of the environment. The target network is fixed, making the prediction task potentially more stable than predicting environment dynamics, which might themselves be changing if the environment contains other agents or complex non-stationarities.
3. Effective Exploration: RND has demonstrated strong performance in challenging exploration tasks, especially those with sparse extrinsic rewards where intrinsic motivation is essential for guiding the agent.

Considerations

While effective, RND has aspects to consider during implementation:

• Network Architecture: The choice of architecture for both the predictor and target networks can impact performance. Deeper or more complex target networks might provide more unique signatures per state but could also be harder to predict.
• Initialization: The random initialization of the target network matters. Different initializations can lead to different random projections and potentially affect learning dynamics.
• Normalization: Proper normalization of the intrinsic reward is important for stable training. The method used (e.g., running mean/std) and scaling factor $\beta$ are hyperparameters requiring tuning.
• Computational Cost: RND adds the forward pass computation for two networks (predictor and target) and the backward pass for training the predictor network at each step.

In summary, Random Network Distillation provides a practical and effective mechanism for generating exploration bonuses based on state novelty. By training a network to predict the output of a fixed random network on experienced states, RND generates high intrinsic rewards for unfamiliar states, effectively driving exploration in complex environments, particularly where predicting environment dynamics might be unreliable or overly complex.