Federated Learning Principles: A Recap

Federated Learning (FL) represents a significant shift from traditional centralized machine learning. Instead of collecting vast amounts of user data into a central repository for model training, FL enables collaborative model building directly on distributed devices or within separate organizational silos, all while keeping the raw data localized. This approach intrinsically enhances data privacy, as sensitive information never leaves the client's control. Before we examine advanced techniques, let's solidify our understanding of the core principles that underpin this distributed learning approach.

The Federated Learning Workflow

The most common and foundational approach to FL is Federated Averaging (FedAvg). It follows an iterative process coordinated by a central server, typically involving these steps:

1. Initialization: The server starts with an initial global model, often randomly initialized or pre-trained on a public dataset.
2. Client Selection: In each communication round $t$, the server selects a subset of available clients (e.g., mobile devices, hospitals) to participate in training. Selection strategies can vary, often involving random sampling.
3. Model Distribution: The server transmits the current global model parameters, $w_t$, to the selected clients.
4. Local Training: Each selected client $k$ updates the received model using its local data $D_k$. This typically involves running multiple steps of gradient descent (or a variant) on its local objective function $F_k(w)$, derived from its private data. Let $w_{t+1}^k$ be the updated local model parameters on client $k$.
5. Model Update Transmission: Clients send their computed updates back to the server. This could be the full updated model parameters $w_{t+1}^k$, or more commonly, the difference $\Delta_k = w_{t+1}^k - w_t$, or gradients $\nabla F_k(w_t)$. This transmission is a potential communication bottleneck and privacy risk area.
6. Aggregation: The server aggregates the received updates from the participating clients to compute a new global model $w_{t+1}$. In FedAvg, this is typically a weighted average based on the amount of data each client used for training: $$w_{t+1} = \sum_{k \in S_t} \frac{n_k}{n} w_{t+1}^k$$ where $S_t$ is the set of selected clients in round $t$, $n_k = |D_k|$ is the number of data points on client $k$, and $n = \sum_{k \in S_t} n_k$ is the total number of data points across selected clients. Alternatively, if updates $\Delta_k$ are sent: $$w_{t+1} = w_t + \sum_{k \in S_t} \frac{n_k}{n} \Delta_k$$
7. Iteration: The process repeats from Step 2 for a predefined number of communication rounds or until a convergence criterion is met.

This cyclical process allows the global model to learn from the collective knowledge embedded within the distributed datasets without centralizing them.

A diagram illustrating the standard synchronous Federated Learning cycle involving the server and a representative client.

Entities: Server and Clients

The FL ecosystem primarily consists of two types of entities:

• Clients: These are the devices or organizations (e.g., smartphones, laptops, hospitals, banks) holding the local data. They possess computational resources to perform local model training based on instructions from the server. Their participation might be intermittent, and their resources (CPU, network bandwidth, data size) can vary significantly, leading to systems heterogeneity.
• Server: This central entity orchestrates the learning process. It initializes the model, selects clients, distributes the model, aggregates updates, and maintains the global state. While the server does not access raw client data, its role is critical for coordination and convergence. The server itself can be a potential point of failure or attack.

The Optimization Goal

As mentioned in the chapter introduction, the overarching goal is typically to minimize a global objective function $F(w)$ which represents the aggregate performance across all $N$ clients:

$$\min_{w} F(w) \quad \text{where} \quad F(w) = \sum_{k=1}^N p_k F_k(w)$$

Here, $F_k(w) = \mathcal{L}(w; D_k)$ is the local loss function for client $k$ computed over its local dataset $D_k$, and $p_k$ is a weight assigned to client $k$, usually $p_k = n_k / \sum_{j=1}^N n_j$, where $n_k = |D_k|$. This formulation highlights that we aim for a model that performs well on average across the data distribution represented by all clients.

Contrasting with Other Learning Paradigms

It's important to distinguish FL from other approaches:

• Centralized Learning: Requires pooling all data in one location, which FL explicitly avoids due to privacy, communication, or regulatory constraints.
• Classical Distributed Learning: Often assumes data is distributed across nodes in a cluster (e.g., using parameter servers), but typically within a single trusted domain, with more homogeneous node capabilities and often IID (Independent and Identically Distributed) data partitions. FL specifically targets scenarios with higher heterogeneity, untrusted environments (potentially), and non-IID data.

This recap of the basic FL principles and the standard FedAvg workflow sets the stage. While straightforward, this foundational model operates under several simplifying assumptions. Real-world federated environments introduce significant challenges related to data and systems heterogeneity, privacy vulnerabilities, communication costs, and potential adversarial behavior, motivating the advanced techniques we will explore throughout this course.