States, Actions, and Rewards

Okay, let's explore the specifics of states, actions, and rewards. These three elements form the core feedback loop in nearly every Reinforcement Learning problem. Understanding them clearly is essential before we move on to how an agent actually learns. Building on the idea of an agent interacting with an environment, we now formalize what information flows between them.

States ($S$)

A state is a snapshot of the environment at a particular moment in time. It contains the information the agent needs (or is allowed) to perceive to make a decision. Think of it as the current context or situation.

Formally, we denote a specific state at time step $t$ as $s_t$. The set of all possible states the environment can be in is called the state space, denoted by $S$. The nature of a state can vary widely depending on the problem:

• Simple Grid: The state could simply be the agent's coordinates, like $(x, y)$. The state space is finite and discrete.
• Board Games (Chess, Go): The state is the position of all pieces on the board. While discrete, the number of possible states is enormous.
• Robotics: The state might include joint angles, velocities, sensor readings (like lidar or camera data), and target location. This often involves continuous values.
• Video Games (Atari): The state could be the raw pixel values from the screen, possibly combined with internal game information like score or lives remaining. This is often a high-dimensional state space.

An important concept related to states is observability.

• Fully Observable: The agent's observation directly corresponds to the true state of the environment. The agent knows everything relevant to make an optimal decision. Many classic RL problems assume full observability.
• Partially Observable: The agent only receives an observation that gives incomplete information about the true underlying state. For example, a robot with only a forward-facing camera doesn't know what's behind it. Dealing with partial observability often requires different techniques, like remembering past observations.

For much of this course, especially when introducing core concepts like Markov Decision Processes (MDPs) in the next chapter, we'll assume the environment is fully observable or that the state representation we use captures all relevant information (satisfying the Markov property).

Actions ($A$)

Based on the current state $s_t$, the agent chooses an action, denoted as $a_t$. An action is simply a decision the agent makes, one of the choices available to it.

The set of all possible actions the agent can take is the action space, denoted by $A$. Sometimes, the available actions depend on the current state, in which case we might write $A(s_t)$ for the set of valid actions in state $s_t$. Like states, actions can be:

• Discrete: A finite set of distinct choices.
  • Grid: move_north, move_south, move_east, move_west.
  • Chess: Move pawn e2 to e4, move knight g1 to f3, etc. (a large but finite set).
  • Atari Game: joystick_left, joystick_right, button_fire.
• Continuous: Actions represented by real-valued numbers, often within certain bounds.
  • Robotics: Apply a specific torque (e.g., 1.5 Nm) to a motor, set steering angle (e.g., -0.2 radians).
  • Thermostat Control: Set target temperature (e.g., 21.5 degrees Celsius).

The nature of the action space significantly influences which RL algorithms are most suitable.

Rewards ($R$)

After the agent takes action $a_t$ in state $s_t$, the environment transitions to a new state $s_{t+1}$ and provides a numerical reward, denoted $r_{t+1}$. This reward signal is important; it's the primary feedback mechanism that tells the agent how well it's doing.

• Purpose: Rewards define the goal of the RL problem. The agent's objective isn't necessarily to get the highest immediate reward, but to maximize the total cumulative reward it receives over the long run.
• Nature: Rewards are scalar values (single numbers). They can be positive, negative, or zero.
  • Grid: +10 for reaching a goal state, -5 for falling into a pit, -0.1 for each step taken (to encourage efficiency).
  • Game Playing: +1 for winning, -1 for losing, 0 for all intermediate moves (sparse reward). Alternatively, small positive rewards for capturing opponent pieces or achieving advantageous positions.
  • Robot Control: Positive reward for reaching the target, negative reward for hitting obstacles or consuming too much energy.

The Reward Hypothesis is a fundamental concept in RL: It states that all goals and purposes can be thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (the reward). Designing an effective reward function is often one of the most challenging aspects of applying RL in practice. A poorly designed reward function can lead to unintended or suboptimal agent behavior. For instance, rewarding a cleaning robot solely for collecting dust might lead it to dump the dust just to collect it again!

The Cycle

These three components form the basis of the agent-environment interaction loop:

1. The agent observes the current state $s_t$.
2. The agent selects an action $a_t$.
3. The environment processes the action, transitions to the next state $s_{t+1}$, and returns a scalar reward $r_{t+1}$ to the agent.
4. This cycle repeats (increment $t$).

The agent observes the state, selects an action which influences the environment. The environment then provides the next state and a reward back to the agent, completing one step of the interaction loop.

Understanding states, actions, and rewards is the first step in formally defining an RL problem. In the next chapter, we will see how these elements, along with the environment's dynamics, are captured within the framework of Markov Decision Processes (MDPs).