Dynamic Treatment Regimes and Estimation

Many real-world problems, particularly in healthcare, personalized education, and adaptive system control, involve sequences of decisions made over time. The optimal decision at any given point often depends on the history of the individual or system, including past treatments, covariates, and outcomes. Static interventions, where the treatment is fixed beforehand, are insufficient in these dynamic settings. This necessitates the framework of Dynamic Treatment Regimes (DTRs).

A DTR is a sequence of decision rules, one for each decision point or stage, that map the available history information to a recommended action or treatment. The goal is typically to optimize a long-term outcome by adapting interventions based on the evolving state of the system or individual.

Formalizing Dynamic Treatment Regimes

Let's consider a setting with $K$ decision stages, indexed by $k = 1, ..., K$. At each stage $k$:

• $S_k$ represents the state of the system (e.g., patient health indicators, system metrics).
• $A_k$ represents the action or treatment chosen at stage $k$.
• $R_{k+1}$ represents the reward or outcome observed after taking action $A_k$ in state $S_k$.
• $H_k = (S_1, A_1, R_2, S_2, ..., A_{k-1}, R_k, S_k)$ denotes the history available up to the point of decision at stage $k$. Note that $H_1 = S_1$.

A DTR is formally defined as a sequence of decision rules $d = (d_1, ..., d_K)$, where each rule $d_k$ is a function mapping the history $H_k$ to a specific action from the set of available actions $\mathcal{A}_k$:

$$d_k: H_k \mapsto A_k \in \mathcal{A}_k$$

The value of a specific DTR $d$, denoted $V(d)$, is the expected cumulative outcome (sum of rewards) obtained if treatments are assigned according to this regime:

$$V(d) = E \left[ \sum_{k=1}^{K} R_{k+1} \mid A_k = d_k(H_k) \text{ for all } k \right]$$

The objective is to find the optimal DTR, $d^{opt} = (d_1^{opt}, ..., d_K^{opt})$, which maximizes this expected cumulative outcome:

$$V(d^{opt}) = \max_{d} V(d)$$

Estimating $d^{opt}$ from data requires tackling sequential causal inference challenges, particularly when using observational data where treatments were not randomly assigned. We need methods to estimate the counterfactual outcomes under different sequences of treatments specified by potential DTRs. A common identifying assumption is the Sequential Conditional Ignorability (or sequential randomization) assumption, which states that at each stage $k$, the treatment $A_k$ is conditionally independent of the future potential outcomes given the observed history $H_k$. Formally:

$$\{Y_k(a_k, ..., a_K), ..., Y_K(a_k, ..., a_K)\} \perp A_k \mid H_k$$

for all possible treatment sequences $(a_k, ..., a_K)$, where $Y_j(\cdot)$ denotes the potential outcome at stage $j$. We also require the Positivity assumption: $P(A_k = a_k | H_k = h_k) > 0$ for all $k$, $h_k$, and $a_k$ with positive probability in the population.

A simplified representation of a two-stage Dynamic Treatment Regime. Decisions (diamonds) at each stage depend on the accumulated history and influence subsequent states and the final cumulative outcome.

Estimation via Q-Learning

Q-learning, adapted from reinforcement learning, is a popular method for estimating the optimal DTR. It works via backward recursion, starting from the last stage. The core idea is to estimate the state-action value function, or Q-function, $Q_k(h_k, a_k)$, which represents the expected cumulative outcome from stage $k$ onwards, given history $h_k$ and having chosen action $a_k$.

The procedure unfolds as follows:

1. Stage K (Final Stage): Model the expected final reward conditional on the history $H_K$ and action $A_K$. This directly gives the Q-function for the last stage:

   $$Q_K(H_K, A_K) = E[R_{K+1} | H_K, A_K]$$

   This expectation is estimated using a regression model (e.g., linear regression, random forest, neural network) fitted to the observed data $(H_{Ki}, A_{Ki}, R_{K+1, i})$ for all subjects $i$. Let $\hat{Q}_K(h_K, a_K)$ be the fitted model. The optimal decision rule at stage K is then:

   $$d_K^{opt}(h_K) = \arg\max_{a_K \in \mathcal{A}_K} \hat{Q}_K(h_K, a_K)$$

2. Stage k (k = K-1 down to 1): Assume we have already estimated $\hat{Q}_{k+1}(h_{k+1}, a_{k+1})$. We can estimate the optimal value starting from stage $k+1$, given history $H_{k+1}$, as $V_{k+1}(H_{k+1}) = \max_{a_{k+1} \in \mathcal{A}_{k+1}} \hat{Q}_{k+1}(H_{k+1}, a_{k+1})$. The Q-function at stage $k$ is defined by the Bellman equation:

   $$Q_k(H_k, A_k) = E[R_{k+1} + V_{k+1}(H_{k+1}) | H_k, A_k]$$

   To estimate this, we fit a regression model for $Q_k(H_k, A_k)$ using the pseudo-outcome $Y_{k,i} = R_{k+1, i} + \max_{a_{k+1}} \hat{Q}_{k+1}(H_{k+1, i}, a_{k+1})$ as the response variable and $(H_{ki}, A_{ki})$ as predictors. Let the fitted model be $\hat{Q}_k(h_k, a_k)$. The optimal decision rule at stage k is:

   $$d_k^{opt}(h_k) = \arg\max_{a_k \in \mathcal{A}_k} \hat{Q}_k(h_k, a_k)$$

This backward iterative process yields estimates of the optimal decision rules $(\hat{d}_1^{opt}, ..., \hat{d}_K^{opt})$ for all stages.

Implementation Aspects:

• The flexibility of the regression models used at each stage ($Q$-models) is important for capturing complex relationships between history, actions, and outcomes.
• A significant practical challenge is potential model misspecification. Errors in the $Q$-model at stage $k+1$ can propagate backward and affect the estimation at stage $k$.

Estimation via A-Learning (Augmented Inverse Probability Weighting)

A-Learning offers an alternative approach that directly models the advantage or contrast of taking one action versus a baseline or reference action at each stage, rather than modeling the full expected outcome trajectory like Q-learning. It often leads to estimating equations that are doubly robust, offering protection against misspecification of either the outcome model or the propensity score model (but not both).

Let's focus on a single stage $k$ and simplify notation slightly. Suppose we want to estimate parameters $\psi_k$ that define the optimal treatment rule $d_k(H_k; \psi_k)$. A-learning focuses on the "blip function" or stage-k treatment effect among individuals following the optimal regime until stage $k-1$ and receiving treatment $A_k$ at stage $k$.

The core estimating equation for A-learning at stage $k$ often takes a form related to:

$$E \left[ \frac{I(A_k = a_k)}{\pi_k(H_k)} (Y - Q_k^*(H_k, A_k)) \cdot \frac{\partial}{\partial \psi_k} C_k(H_k, A_k; \psi_k) \right] = 0$$

where:

• $Y$ is the cumulative outcome from stage $k$.
• $\pi_k(H_k) = P(A_k = a_k | H_k)$ is the propensity score for the observed action $a_k$.
• $Q_k^*(H_k, A_k)$ is a model for the expected outcome conditional on history and action (an outcome regression model).
• $C_k(H_k, A_k; \psi_k)$ is the "contrast" function, parameterized by $\psi_k$, capturing the benefit of treatment $A_k$ relative to a baseline, potentially tailored by history $H_k$. Often $C_k(H_k, A_k; \psi_k) = A_k \cdot \beta_k^T X_k$ for binary $A_k \in \{0, 1\}$, where $X_k$ are features derived from $H_k$.

A-learning proceeds stage-by-stage, typically backward or forward, solving these estimating equations.

Implementation Aspects:

• Requires fitting propensity score models $\pi_k(H_k)$ at each stage.
• Requires fitting stage-specific outcome models $Q_k^*(H_k, A_k)$.
• The primary target is the parameter $\psi_k$ defining the optimal rule, making it more direct than Q-learning for identifying the optimal policy parameters.
• It can be more effective than Q-learning if the propensity score model is correct, even if the outcome model $Q_k^*$ is misspecified (under certain conditions). Conversely, it relies heavily on the correctness of the propensity score models.

Connections and Considerations

Both Q-learning and A-learning provide powerful frameworks for estimating optimal DTRs, fundamentally addressing a sequential causal inference problem. They are closely related to methods in off-policy evaluation and learning in reinforcement learning. Q-learning directly implements value iteration based on the Bellman equation. A-learning methods are related to policy gradient and advantage-based methods in RL.

Choosing between Q-learning and A-learning often depends on the specifics of the problem, the quality of available data, and assumptions one is willing to make about outcome or propensity score models. Evaluating the performance of estimated DTRs is also a significant challenge, often requiring simulation studies or separate validation datasets. Careful consideration of the sequential ignorability and positivity assumptions is necessary for the validity of the estimated regimes. These methods provide essential tools for optimizing sequences of interventions in complex dynamic systems.