From Centralized to Federated
Recap: how models are trained
| Setting | Slide definition |
|---|---|
| Subject-independent | Trained on data from several subjects, with the objective of generalizing on unseen subjects — the most challenging setting |
| Subject-dependent | Trained and tested on data from the same subjects — the most accurate |
| Hybrid | Combines the strengths of the two above — a bit less accurate than subject-dependent |
Collaborative learning
The most accurate solutions are subject-dependent, but the real bottleneck is data acquisition and labeling. Collaborative learning distributes labeled data collection over a large number of users to build a single global model (subject-independent or hybrid, depending on the scenario). There are three ways to organise it:
Centralized vs Distributed vs Federated
| Approach | How it works & its problems |
|---|---|
| Centralized | Each client transmits its labeled data to a central server, which combines all of it into one training set and builds a single collaborative model. Problems: privacy (sensor data may reveal personal habits / health conditions) and scalability (communication latency from transmitting lots of data + computational cost of training on huge labeled sets) |
| Distributed on-site | Training and inference are done only on locally collected data; the server distributes a pre-trained model and each device personalizes it (privacy mitigated). Problems: nodes may have limited labeled samples, get no benefit from peers' data, and the local models can not generalize |
| Federated | Each client trains a local model on its own labeled data; only the local model parameters (not the data!) are shared with the server, which aggregates them in a privacy-preserving fashion to generate a global model |
An early application: GBoard
One of the first uses of FL: suggesting the next word in Android keyboards. When users "click" or "ignore" a suggestion they implicitly provide labels; a personal model is trained locally, and the local models are merged by a server into a stronger model thanks to collaborative learning. FL has since been applied to federated HAR, smart energy, pervasive healthcare, smart-city pollution sensing, and autonomous driving.
The Basic Federated Learning Approach
The FL mechanism — a communication round
Periodically the server updates the global model: it selects a number of available clients, each selected client receives the global model, trains it on its available labeled data, and sends the resulting local model parameters back. The server aggregates them into an updated global model. The process repeats until convergence — each iteration is a communication round.
FedSGD vs FedAvg
FedSGD (naïve)
- Each client computes the average gradient on its local data at the current model and sends the gradient
- The server does a weighted aggregation of gradients and updates the model with one step of SGD
FedAvg (standard)
- Each client performs several SGD steps locally and sends the resulting weights
- The server does a weighted aggregation of the weights → the new global model
FedAvg requires more computation on each client (several SGD steps), but it lets the model converge with a significantly reduced number of communication rounds — it is the standard approach generally used in FL. The weighted aggregation uses each client's data size: nk/n.
Client selection
FL targets scenarios with a large number of clients (e.g., thousands); considering all of them each round is not scalable, and some may be unavailable.
- Random sampling from the available devices — local training is costly (energy, usability), so a device may be available e.g. at night while charging; the server asks for availability and samples a random subset.
- Selection through utility — pick the clients with the best utility. Statistical utility: the usefulness of a client's local update to the global model (e.g., number of training samples, local cumulative loss, difference between local and global model). System utility: different hardware leads to different overheads (training/transmission time) — slow devices (stragglers) prolong the round, so a threshold on response time can be used.
Asynchronous Federated Learning
When the global model must stay constantly updated, asynchronous FL is considered — the goal is reaching convergence quicker than FedAvg. Each client transmits its local update as soon as it is available; the server tracks the timestamps at which each global model was created to weight the updates — intuitively, a local model that arrives "late" may contain old information and should impact the global model less.
FedAsync & staleness
- The server periodically schedules a global update and contacts each client separately, transmitting the current model xτ (τ = the timestamp at which the global model was created).
- As soon as possible the client submits (xnew, τ); the server immediately updates the model: xt ← (1 − αt) xt−1 + αt xnew.
- Staleness is assessed via the difference t − τ: larger staleness → greater error when updating (using old information). Hence αt ← α × s(t − τ) controls the update considering staleness (one possible formula: s(t − τ) = 1/(t − τ + 1)).
Heterogeneity & the non-IID Problem
Clients may be heterogeneous
FL clients differ along several axes: communication heterogeneity (4G/Wi-Fi/2G), model heterogeneity, statistical heterogeneity, and device heterogeneity.
Model heterogeneity → HeteroFL
A widely accepted assumption is that local models share the same architecture as the global model — but that forces limiting the global model's complexity for resource-constrained devices, whose capabilities may vary (even dynamically). HeteroFL provides "smaller" versions of the global model to constrained devices by reducing the width of hidden channels (hence the number of parameters); local and global architectures stay within the same model class to stabilize aggregation, and each client contributes to a different number of parameters based on its model size.
Task heterogeneity → multi-task learning
By keeping part of the parameters private and part public, FL can rely on transfer learning to collaboratively learn multiple tasks: FL learns the shared layers that may be general across tasks, and each client fine-tunes its local model on a specific task.
Statistical heterogeneity: the non-IID problem
The global model should generalize over many clients, but each client's local model typically refers to a specific user (physical characteristics, age, habits). So data from different clients are non identically and independently distributed (non-IID), and FL solutions based only on local optimization (e.g., FedAvg) are not adequate.
| Skew | Slide definition |
|---|---|
| Feature distribution skew | Each subject may generate peculiar sensor data (e.g., walking inertial patterns of a young vs an elder subject) |
| Label distribution skew | Different subjects may have different routines (e.g., a sporty subject spends more time running than a sedentary one) |
| Quantity distribution skew | Different subjects may have significantly different availability of labeled data |
Tackling non-IID with transfer learning
The layers closest to the input represent features that may be general across clients (feature-extraction component); the layers closest to the output represent client-specific characteristics (fully-connected output layers). So after the server generates a global model, each client can locally fine-tune the personalized layers — sharing the feature extractor, personalizing the head. (This is exactly sim Q17.)
Federated Clustering
The above fixes use a single global model fine-tuned per client, but trading off personalization vs generalization is hard. Federated Clustering tackles non-IID by grouping clients by similarity, giving a global model per group — within a group the clients are similar, so they look IID. A general global model trained on all clients is still kept for generalization and for non-clustered clients.
- Similarity between users: raw data isn't available in FL, but two local models with similar weights were likely trained on similar data → similar users. Usually the cosine similarity between model parameters is computed, considering only the personal layers (they reflect the most client-specific characteristics).
- Hierarchical clustering: compute pairwise similarity; start with one cluster per user; repeatedly merge the two most similar clusters, generating a new specialized model (e.g., via FedAvg) until convergence (threshold on similarity). Clusters with only one client are non-clustered.
- Server side: for the first r−1 rounds a single global model is trained; at the r-th round hierarchical clustering produces specialized models; in the remaining rounds the server updates the specialized global models. The clustering round must be tuned: too soon → local models not trained enough; too late → local models too similar.
- ProtoHAR (for label/quantity skew): each client sends its feature-extractor parameters plus a prototype of each activity (latent space); the server builds a global feature extractor + global prototypes; during local training each client minimizes the distance of its local prototypes to the global ones, performing better on locally under-represented classes.
Labeled Data Scarcity in FL
FL requires participating clients to have some labeled data for local training — an assumption that is not realistic in Ambient Intelligence. Several strategies mitigate this:
Semi-supervised FL
- The global model is initialized with a limited labeled training set; each client receives this pre-trained global model.
- Locally, each client uses semi-supervised strategies to obtain pseudo-labels; the pseudo-labels are used for local training, so the local parameters can be transmitted to the server.
Combining active learning & label propagation
- Each client starts from a pre-trained model; each unlabeled sample is stored locally.
- Active learning labels a small number of unlabeled points (querying the user); periodically (e.g., at the end of each day) label propagation spreads those labels to a larger pool of unlabeled points; the resulting pseudo-labeled dataset trains the local model.
Combining clients with and without labels
Active learning isn't always possible (it needs human interaction). Sometimes some clients have labels (or pseudo-labels) while others have only unlabeled data → aggregation strategies transfer knowledge from "labeled clients" to "unlabeled clients": compute supervised model weights from labeled clients, unsupervised weights from unlabeled clients (e.g., self-supervised learning), and aggregate the two types.
Self-supervised FL (worst case: no labels anywhere)
Each client leverages large amounts of unlabeled data to perform self-supervised learning locally (a local feature representation); the server aggregates the locally trained feature extractors into a global self-supervised model, which is finally fine-tuned with a small amount of labeled data (e.g., publicly available datasets).
Is FL Really Protecting Privacy?
The four attacks
| Attack | Objective & mechanism |
|---|---|
| Model Inversion | Reconstruct typical samples of a class the attacker only knows by label. Idea: feed random noise to the target model, backpropagate the loss, but optimize the input (not the weights) to minimize it — generating the input the model thinks is the most likely sample of a class (notably used for face recognition) |
| Property Inference | Infer a property of the training set the owner didn't intend to share (e.g., the men/women ratio). The attacker builds shadow datasets (with or without property P), trains a shadow model on each, and trains a meta-classifier (binary: has P or not) to attack the target |
| Membership Inference | Infer whether a specific sample was used to train the target model. A binary attack model is trained by observing the behaviour of a shadow model on member vs non-member data; outputs a Membership Probability (≈1 member, ≈0 non-member). Can be black-box or white-box |
| Model Extraction | Reconstruct a black-box model f by creating a substitute model f′ that behaves similarly, to obtain a white-box version attackable by other methods. Reconstructed from input/response pairs with few queries (e.g., via distillation) |
A countermeasure: Local Differential Privacy (LDP)
Before uploading, each client adds noise to its gradients (a noisy gradient) so the server can't recover sensitive information from the updates while still aggregating a useful global model. Defenses more broadly include differential privacy, adversarial ML, watermarking and cryptography techniques.
Final Quiz — Exam Style
15 MCQs + 2 open-ended. Three are the actual simulation questions (Q2, Q17, Q18) reproduced verbatim; the rest follow the professor's recipe — sibling distractors, one-word swaps, a 2×2 grid, and a FALSE question hiding an absolutizer.
Which of these sentences better describes the asynchronous aggregation process "FedAsync" in Federated Learning?
In Federated Learning, how to use transfer learning to mitigate the non-IID problem?
How does the "membership inference attack" train the attack model?
a) Centralized: every client sends its labeled data to a server that merges it into one training set and trains a single model — but this raises privacy (sensor data reveals habits/health) and scalability issues (communication latency from large data transfers, computational cost of training on huge labeled sets). Distributed on-site: training and inference happen only on locally collected data (the server distributes a pre-trained model that each device personalizes), mitigating privacy — but nodes may have limited labels, get no benefit from peers' data, and the local models cannot generalize. Federated: each client trains a local model and shares only the parameters (not the data); the server aggregates them privacy-preservingly into a global model — combining peers' knowledge without moving data. Communication round: the server selects a subset of available clients, broadcasts the global model, each client trains locally on its labeled data and returns its local parameters, and the server aggregates them into an updated global model; this repeats until convergence.
b) FedSGD: each client computes the average gradient on its local data and sends the gradient; the server does a weighted aggregation and applies one SGD step. FedAvg: each client performs several local SGD steps and sends the resulting weights; the server does a weighted aggregation of the weights (by data size nk/n). FedAvg costs more local computation but converges in far fewer communication rounds, so it is the standard. Client selection: FL targets thousands of clients, so not all participate each round; the simplest strategy is random sampling of available devices (a device may be available at night while charging); alternatively, utility-based selection picks the best clients by statistical utility (usefulness of the update — sample count, local loss, local-vs-global difference) or system utility (hardware overhead; a response-time threshold avoids stragglers). Asynchronous FL: synchronous FL aggregates only after every selected client returns, which stalls on unresponsive or slow straggler devices; async FL lets each client transmit as soon as ready and the server updates immediately, weighting each update by its staleness t − τ (older updates impact less, e.g. αt = α·s(t−τ)).
a) non-IID: the global model must generalize over many clients, but each local model reflects a specific user (characteristics, age, habits), so client data are not identically and independently distributed and FedAvg-style local-only optimization is inadequate. The three skews: feature distribution skew (same activity, different sensor patterns — young vs elder walking), label distribution skew (different routines — sporty vs sedentary), quantity distribution skew (different amounts of labeled data). Transfer-learning personalization: the input-side layers are general (feature extractor) and are federated, while the output-side personalized layers (head) are fine-tuned locally on each client. Federated clustering: group clients by similarity (cosine similarity of their personal layers, since raw data is unavailable) via hierarchical clustering — start one cluster per user, merge the two most similar repeatedly (new specialized model via FedAvg) until a similarity threshold; this yields a global model per group (clients within a group look IID), keeping a general model for non-clustered clients. (ProtoHAR additionally shares class prototypes and pulls local prototypes toward global ones to fight label/quantity skew.)
b) In FL only weights are shared, but the weights of a deep model can still leak information about participants. Model inversion: feed random noise and optimize the input (not the weights) to reconstruct a typical sample of a class (e.g., a face). Property inference: infer an unintended property of the training set (e.g., gender ratio) by training shadow models on datasets with/without property P and a binary meta-classifier. Membership inference: decide whether a specific sample was in the training set by training an attack model on a shadow model's behaviour over member vs non-member data, outputting a Membership Probability (≈1 member, ≈0 non-member); black-box or white-box. Model extraction: reconstruct a black-box model as a substitute model from input/response pairs (few queries, e.g. distillation) to obtain a white-box version attackable by the others. Countermeasure — Local Differential Privacy: each client adds noise to its gradients before uploading, so the server can't recover sensitive info while still aggregating a useful model (other defenses: differential privacy, adversarial ML, watermarking, cryptography).