What is Transfer Learning?
Deep models need huge labeled datasets — but in Ambient Intelligence labels are scarce and expensive. Transfer Learning (TL) reuses knowledge learned on a data-rich source to boost a data-poor target.
The formal definitions — know the pieces
| Concept | Slide definition |
|---|---|
| Domain D | D = {𝒳, P(X)} — a feature space 𝒳 and a marginal probability distribution P(X) over the data X |
| Task T | T = {𝒴, f(·)} — a label space 𝒴 and an objective predictive function f(·) (learned from training data, used to predict the label of new instances) |
| Transfer learning | Given a source domain DS / task TS and a target domain DT / task TT, improve the learning of fT(·) using the knowledge in DS, TS, where DS ≠ DT or TS ≠ TT |
When does it work? — necessary factors
- Sufficient domain similarity: source and target must be related — the knowledge has to be transferable (a connection between the tasks must exist).
- Sufficient target data: at least a small amount of target data is needed to adapt the transferred knowledge.
The Transfer Learning Taxonomy
TL can be categorized from two complementary perspectives: what changes between source and target (problem perspective), and how the knowledge is transferred (solution perspective).
Problem perspective — what is transferred
| Setting | Definition |
|---|---|
| Transductive TL | Source and target tasks are the same, but the domains differ (labels available only in the source). → Domain Adaptation |
| Inductive TL | The target task is different from the source task (labels available in the target). → Multi-Task Learning (labels also in source) & Self-Taught Learning (no labels in source) |
| Unsupervised TL | Focus on unsupervised tasks in the target domain (e.g., clustering, dimensionality reduction); no labels in source nor target |
Solution perspective — how it is transferred
| Category | What is transferred |
|---|---|
| Instance-based transfer | Re-weight labeled source instances so they fit the target distribution and can be reused for target training |
| Feature-based transfer | Find a good feature representation (latent features) that reduces the gap between source and target domains |
| Parameter-based transfer | Transfer knowledge at the model/parameter level (shared parameters or priors between source and target models) |
| Knowledge-based transfer | Transfer relational knowledge / rules among the data (a knowledge base relating source and target) |
These can be supervised or unsupervised depending on label availability in the target.
TL Approaches: Data, Model, Domain Adaptation
Data-based transfer
Instance-based
- Re-weight source instances by their relevance to the target, so labeled source data can be reused in target training
Feature-based
- Learn latent features shared across domains. Symmetric: map both source and target into one common space. Asymmetric: transform the source features to match the target
Model-based transfer
- Model control: add terms acting on the model — regularizers, knowledge distillation, domain adaptation — to steer the target model with source knowledge.
- Parameter control: directly control the parameters — the canonical case is fine-tuning.
Fine-tuning
The early layers learn generic, transferable features and are kept frozen; the final layers (closer to the output) are fine-tuned on the small target dataset. It is the simplest, most widely used form of parameter-based transfer.
Unsupervised Domain Adaptation (UDA)
- A binary domain classifier D tries to predict whether a feature comes from the source or the target domain.
- Training combines a Discrepancy Loss (push features to be domain-invariant — fool D) and a Classification Loss (keep features discriminative on the labeled source task).
- In AmI: cross-subject / cross-environment / cross-position / cross-device / cross-domain transfer, e.g. GAN-based UDA for cross-subject HAR and domain adaptation in load disaggregation.
Self-Supervised Learning
Pretext task → downstream task
- The pretext task is a supervised task — but the supervision (the labels) is extracted automatically from the unlabeled data, not provided by a human.
- Its only role is to learn a good representation: a reliable feature extractor that leverages a large pool of unlabeled data.
- Three big families (next section): AutoEncoder (generative), predictive, contrastive.
The Three Self-Supervised Families
| Family | Pretext idea & notes |
|---|---|
| AutoEncoder | The simplest: reconstruct the input x from a compressed latent code (dimensionality reduction). The least effective at learning rich representations |
| Predictive | Mask part of the input and predict it (à la NLP / GPT). Cons: reasoning stays at the data level and the model learns a conservative distribution. Example: Masked AutoEncoder (MAE) |
| Contrastive | Build similar (positive) and dissimilar (negative) pairs and use a distance-based loss to pull positives together and push negatives apart. Example: SimCLR |
Contrastive learning — SimCLR
- Apply two random augmentations to a sample → the two views form a positive pair; views of different samples form negative pairs.
- An encoder (shared weights) maps views to features; a projection head maps them to the space where the contrastive loss (with temperature τ) is computed.
- HAR-specific augmentations for sensor time series: Gaussian noise, scaling, 3D rotation, negation, time-flip, section scramble, time-warp, channel shuffle.
Triplet loss — another contrastive loss
Given three points — an anchor, a positive (a transformation of the anchor) and a negative (from another data point) — minimize the anchor↔positive distance and maximize the anchor↔negative distance, with a margin α enforcing a minimum gap.
SSL in Ambient Intelligence & Foundation Models
Self-supervised methods for sensor data
Wavelet contrastive
Positive = a raw signal & its wavelet scalogram; negative = scalogram of another window. A Siamese signal+scalogram network; only the signal net is kept for feature extraction.
Multi-device contrastive
Several devices observe the same activity. Time-synchronized windows are positives; non-synchronized ones are negatives (CelloSSL-style).
Contrastive Predictive Coding
Pretext = predict future sensor points from past: an encoder builds z-vectors, an autoregressive model (LSTM) builds context vectors; future predictions minimize the L2 error.
Masked Reconstruction / MAE
Mask x% of samples and force the model to reconstruct them, learning temporal patterns from context; Spatio-Temporal MAE adds a device dimension.
Towards Foundation Models
Broad pretraining
- Trained on large and diverse datasets, using a general-purpose objective
Wide adaptability
- Adaptable to a wide set of downstream tasks without re-training from scratch
The AmI arc: handcrafted features (2004) → deep learning (2013) → transfer & domain generalization (2016) → self-supervised learning (2019) → foundation models (2023). A recent example is Google's SensorLM, a FM over daily multimodal wearable data.
Bootstrapping models: SSL + active learning
- When even a small labeled set for fine-tuning is unavailable, use the self-supervised model to cluster unlabeled target data — but the clusters still need labels.
- SelfAct: labels for those clusters are obtained via active learning — a query is triggered when a new sample significantly changes its closest cluster. SSL pre-training + active learning bootstraps a personalized classifier with minimal user effort.
Final Quiz — Exam Style
13 MCQs + 2 open-ended. One is the actual simulation question (Q10) reproduced verbatim; the rest are built with the professor's recipe — sibling distractors, one-word swaps, a 2×2 grid, and a FALSE question hiding an absolutizer.
What is a pretext task in Self-Supervised Learning?
a) Transfer learning uses knowledge gathered solving one problem to improve a target learner on a different but related problem — transferring knowledge from data-rich source domains/tasks to a data-poor target. Formally, a domain is D = {𝒳, P(X)} (feature space + marginal distribution) and a task is T = {𝒴, f(·)} (label space + predictive function); TL applies when DS ≠ DT or TS ≠ TT. Problem perspective (what changes): transductive (same task, different domains, source-only labels → domain adaptation), inductive (different target task, target labels → multi-task & self-taught learning), unsupervised (unsupervised target task, no labels anywhere). Solution perspective (what to transfer): instance-based (re-weight source instances), feature-based (shared latent features), parameter-based (shared parameters/priors), knowledge-based (relational knowledge). Conditions: sufficient domain similarity and sufficient target data. If they don't hold, knowledge transfer can hurt the target — negative transfer — so TL is not always beneficial.
b) Data-based. Instance-based: re-weight labeled source instances by their relevance so they can be reused for target training. Feature-based: learn latent features that reduce the domain gap — symmetric maps both source and target into one common space, asymmetric transforms the source to match the target. Model-based. Model control adds regularizers / distillation / domain adaptation; parameter control = fine-tuning: pre-train on a large source, freeze the early (generic) layers and fine-tune the final layers on the small target dataset. Unsupervised Domain Adaptation: when the target has no labels, learn features that are domain-invariant (a binary domain classifier can't tell source from target — driven by a discrepancy loss) yet class-discriminative (a classification loss keeps them useful for the task). In AmI this enables cross-subject / cross-environment / cross-device transfer (e.g., GAN-based UDA for cross-subject HAR, domain adaptation in load disaggregation).
a) Self-supervised learning is a special case of transfer learning where the source data has no labels. The model is first trained on a pretext (upstream) task whose labels are automatically derived from the data itself; this builds a reliable feature extractor over a large unlabeled pool, which is then transferred to the real downstream task (e.g., fine-tuned with a small labeled set). The pretext task is a supervised learning problem — it has inputs and targets and is trained with a supervised loss — but the supervision is generated from the unlabeled data (e.g., the masked value to predict, or whether two views match) rather than provided by a human annotator. That is the subtle exam point: SSL uses unlabeled data, yet the pretext task itself is supervised (the sim-Q10 supervised↔unsupervised trap).
b) AutoEncoder (generative): reconstruct the input from a compressed latent code (dimensionality reduction) — the simplest and least effective at rich representations. Predictive: mask part of the input and predict it (NLP/GPT-style, e.g., the Masked AutoEncoder); cons: reasoning stays at the data level and the model learns a conservative distribution. Contrastive: form positive (similar) and negative (dissimilar) pairs and use a distance-based loss to pull positives together and push negatives apart — e.g. SimCLR (two augmented views of a sample are positives; an encoder + projection head feed a contrastive loss with temperature τ) and the triplet loss (anchor/positive/negative with a margin α). Challenge: choosing negative samples is hard — random sampling from large unlabeled data is expensive, and the most useful ones are hard negatives (look close in the original space but should be far apart). In AmI, SSL methods include wavelet/scalogram Siamese networks, multi-device time-synchronous contrastive learning, CPC and masked reconstruction. These point toward foundation models — trained on broad data with self-supervision and adaptable to many downstream tasks (e.g., SensorLM). When not even a small labeled set exists, SSL is combined with active learning to bootstrap a model (SelfAct: cluster the unlabeled data, then query labels when a sample significantly changes its closest cluster).