AmI · Lecture 14 — Generative AI

GANs & Conditional GANs

Generative AI has exploded (deepfakes, image generators, ChatGPT). Several research groups apply the newest generative approaches to Ambient Intelligence — this lecture surveys the main families.

GANs — Generative Adversarial Networks

One of the first generative approaches (2014); mainly adopted for data augmentation.
Adversarial training: the discriminator learns which inputs are real and which are synthetic; the generator learns to "fool" the discriminator.
A known problem — mode collapse: the generator produces only a limited variety of samples.

GAN loss functions (note: D(x) = probability that x is fake)

The discriminator has high loss when it misclassifies a real instance as fake, or a generated (fake) instance as real. The generator has high loss when the discriminator is confident in classifying generated points as fake.

Conditional GANs (CGAN)

With classic GANs you may need to train a model for each class of interest.
CGANs condition the generation on specific classes, training a single model (the class label y is fed to both generator and discriminator).
Time-series example: ActivityGAN — a generator of 1D-transposed + 1D-conv layers produces a 100-point activity segment; a 2D-conv discriminator decides fake/real.

From AutoEncoders to Variational AutoEncoders

Can we generate with a plain AutoEncoder?

Using the Decoder of an AE to generate data looks tempting — draw random samples from the latent space and decode them. But this usually does not work: it is hard to ensure a "regularity" in the latent space, because the classic AE is trained only to encode and decode, so the latent organization typically overfits the training data. Without explicit regularization, some latent points are "meaningless" once decoded.

Variational AutoEncoder — sim Q9 lives here

A VAE is an AutoEncoder whose training is regularised to (1) avoid overfitting and (2) ensure the latent space has good properties that enable the generative process. The encoder maps the input into a normal distribution over the latent space (it outputs a mean and standard deviation); a sample from that distribution is drawn and given to the decoder.

loss = ‖x − d(z)‖² + KL[ N(μₓ, σₓ) , N(0, I) ]

Besides the reconstruction loss, a VAE adds the Kullback-Leibler loss to minimize the distance between the estimated distribution and a Gaussian N(0,I) — this is what regularizes the latent space.

Simple AutoEncoder

input x → encoding → latent representation z = e(x) → decoding → reconstruction d(z)

Variational AutoEncoder

input x → encoding → latent distribution p(z|x) → sampling → z ~ p(z|x) → decoding → reconstruction d(z)

Regular vs irregular latent space: in an irregular space, close points may be dissimilar once decoded and some points are meaningless; with regularization we want a regular space where points close in the latent space are similar once decoded.

Conditional VAE

It is possible to condition the generation with different input modalities (e.g., generating images from a text prompt). The encoder maps the input into a latent representation also considering the conditioning information c, and the decoder reconstructs from the latent space also using that additional information.

Diffusion Models

The model is trained to denoise the input data in several steps. Generation: starting from complete noise, generate a synthetic sample step by step.
The most effective approach in Computer Vision; systems like MidJourney (images) and Suno (music) are based on advanced diffusion models.

Sibling set — forward vs backward diffusion

Don't swap them: one destroys information, one learns to restore it.

Forward Diffusion

Adding noise to the image at different steps until reaching pure noise (no information)

Backward Diffusion

At each step t, the model learns how to remove noise to reach step t−1 (the tricky part). After training, generation starts from pure noise by denoising step by step

Denoising with a U-net

Diffusion models usually rely on a U-net for denoising at each step (originally proposed for medical image segmentation). While it looks like an AutoEncoder, it also leverages skip connections: the decoder also uses information from the input, enabling high-quality generation — and skip connections make separating the encoder from the decoder impossible.

Latent Diffusion Models

The diffusion process is run in a latent space: a variational encoder maps the input to a latent z, the diffusion model operates there, and a variational decoder reconstructs the output in pixel space. Generation can be conditioned with other modalities (e.g., text) by (1) enhancing the U-net with encoder-decoder attention (cross-attention) to focus on multiple input types, and (2) using a conditional VAE after denoising. For time series, diffusion serves imputation, generation, and forecasting.

GAN vs VAE vs Diffusion

Family	Trade-offs
GAN	High-quality samples, low diversity (mode collapse), hard to train
VAE	Low-fidelity samples, high diversity, easy to train
Diffusion	High-fidelity samples, high diversity, easy to train, but slow in sample generation

Using LLMs in Ambient Intelligence (prompting)

Transformers are "advanced" seq2seq models, so they can be trained to generate sequences (text, time series); predictive self-supervised learning pre-trains them on large unlabeled corpora. LLMs are Transformers pre-trained with self-supervision on massive text (e.g., next-word prediction), usually fine-tuned on a labeled corpus to learn human-like conversations so they can be prompted. LLMs are being applied across time-series tasks (forecasting, classification, interpolation, anomaly detection, generation) and domains (EEG, finance, ECG, traffic, audio, IoT, robotics).

Leveraging LLMs to annotate unlabeled data

Smart-home sensor events can be translated into human-readable sentences (e.g., "the fridge has been opened"), so a window of sensor data maps to a sentence.
Goal: ask an LLM to determine the activity from the textual description of a window. Useful additions to the prompt: temporal information (morning/afternoon/evening), room-layout descriptions (rooms, spatial arrangement, sensors per room), and sensor activations.

ADL-LLM

A window of sensor data is transformed to text, then the LLM is given a system prompt ("act as a HAR system…", listing locations, interactions and possible activities) and returns the most likely activity (e.g., ACTIVITY = preparing a hot meal) — a zero-shot recognizer.

Other prompting-based methods

🏠

AgentSense

An LLM generates personalities, high-level schedules and low-level daily routines; mapped to a VirtualHome simulator → virtual motion/door/object sensor data to train a classifier.

🔤

TDOST

Transform sensor-data windows into natural-language sentences encoding semantics rather than sensor patterns → train on the labeled source home and transfer weights to a target home (cross-home transfer).

⌚

HARGPT

Feed raw IMU (accelerometer/gyroscope) readings to an LLM with a prompt template → the LLM reasons step-by-step and predicts the activity (zero-shot).

Cons of prompting-based approaches

Positive preliminary results, but scalability is questionable — especially for continuous sensor data (accelerometers), since LLMs are typically not trained to understand raw sensor data. Possible fixes: few-shot prompting or, better, fine-tuning; an even more promising direction is foundation models for time series.

Adapting LLMs to Sensor Data

Fine-tuning & PromptCast

Google (2023) showed that fine-tuning LLMs on health data (activity recognition, atrial-fibrillation classification, stress) is a promising direction.
PromptCast: fine-tune an LLM for time-series forecasting by phrasing the series and target as text ("the temperature were 77, 68, 66, 73 degrees… 76 degrees tomorrow").

LLM reprogramming (Time-LLM)

The idea

Prompting time series as text may not be optimal. LLM reprogramming adapts an LLM to another task (e.g., forecasting) while keeping the backbone model intact (frozen): the input time series is converted into text prototype representations (tokens), and only the head of the LLM is fine-tuned with few-shot training.

Time-LLM implements this: time-series patches are reprogrammed against pre-trained word embeddings (text prototypes) via multi-head attention, passed through the frozen pre-trained LLM body, and an output projection produces the forecast.

Natural Language Supervision (NLS) for HAR

Goal: learn a shared latent representation common to sensor data and text. HAR is then done by computing cosine-similarity scores between the sensor embedding and all activity-label embeddings.
This can enable zero-shot classification: since sensor data is mapped into the text space, we can classify activities unseen during training.
Built with cross-modal contrastive pre-training (a text encoder + an IMU encoder, each with a projection head, trained so matching sensor/text pairs align).

Generating Large-Scale HAR Datasets

Because of labeled data scarcity, it would be great to generate massive labeled inertial data — but generative methods themselves need a lot of data. Idea: leverage generative models trained on other modalities where large labeled datasets exist — here, the 3D motion domain.

Generating 3D human motion from text

Generating motion from textual descriptions is useful (games, film-making, animating robots), and massive labeled datasets are available. Since synthetic sensor patterns can be produced from synthetic motion (simulating devices placed on the body), these pre-trained models can automatically generate labeled inertial datasets.

VQ-VAE → T2M-GPT

VQ-VAE (Vector Quantized VAE): learns a discrete representation of human motion — quantization maps motion sequences into sequences of symbols from a codebook C of K symbols. The encoder maps motion → symbols; the decoder reconstructs motion from the discrete sequence.
T2M-GPT: once the VQ-VAE is trained, the encoder maps each 3D motion to a sequence of symbols. A Transformer is trained with text embeddings as input (e.g., CLIP) and an autoregressive task to predict the most likely sequence of symbols from text. At inference, it generates the symbol sequence from text, and the decoder turns it into 3D motion.

IMUGPT — from text to inertial sensor data

Activity name→ LLM → motion description→ T2M-GPT → 3D motion→ Map motion → synthetic IMU→ Train HAR classifier

An LLM turns the activity name into motion descriptions, T2M-GPT generates 3D motions from that text, and the 3D motion is mapped (via inverse kinematics / IMUSim, with a motion filter & calibration) to synthetic inertial sensor data — a full labeled HAR dataset.

Cons of such approaches

It leverages models pre-trained on a significantly different task and then uses ad-hoc methods to reconstruct IMU data — this reconstruction is an approximation and may not capture all the subtle movements real IMU sensors capture. Also, not all activity types may be captured — it depends on the motions used to train T2M-GPT.

ThinkYou need a labeled accelerometer dataset for "jumping" but have no wearable recordings. Walk through how IMUGPT produces it, naming every generative component and where the labels come from — and state one reason the result is only an approximation.

Answer: Start from the activity name "jumping". (1) An LLM expands it into natural-language motion descriptions (capturing several variations). (2) T2M-GPT generates 3D human motion from that text — internally, a VQ-VAE gives a discrete codebook of motion symbols, and a Transformer (fed CLIP text embeddings, trained autoregressively) predicts the symbol sequence, which the VQ-VAE decoder turns into motion. (3) The 3D motion is mapped to synthetic IMU data via inverse kinematics / IMUSim with filtering and calibration. The label ("jumping") is known from the start — it's the seed prompt — so every generated sample is automatically labeled. Approximation: the motion→IMU reconstruction is ad-hoc and may miss subtle movements real accelerometers capture, and T2M-GPT can only produce activity types present in its training motions.

✓

Final Quiz — Exam Style

16 MCQs + 2 open-ended. One is the actual simulation question (Q9) reproduced verbatim; the rest follow the professor's recipe — sibling distractors, one-word swaps, and a FALSE question hiding an absolutizer.

In a GAN's adversarial training, what is the generator's objective?

ATo learn which inputs are real and which are synthetic

BTo "fool" the discriminator into classifying its synthetic samples as real

CTo regularize the latent space so that close points decode to similar outputs

DTo remove noise from the input step by step until reaching the original image

B — the generator learns to fool the discriminator; A is the discriminator's job (real vs synthetic). C is the VAE's goal, D is backward diffusion. GANs were the first generative approach (2014), mainly for data augmentation.

What is mode collapse in GANs?

AThe discriminator becomes unable to distinguish real from fake samples

BThe latent space loses its regularity and some points become meaningless

CThe generator produces only a limited variety of samples

DThe generation becomes extremely slow at sampling time

C — mode collapse = the generator generates only a limited variety of samples. B is the AE latent-space problem, D is the diffusion drawback (slow sampling). Mode collapse is exactly why GANs are "low-diversity".

What does a Conditional GAN (CGAN) add over a classic GAN?

AIt conditions the generation on specific classes, training a single model

BIt removes the discriminator and trains the generator alone

CIt replaces adversarial training with a reconstruction + KL loss

DIt generates samples by denoising pure noise step by step

A — with classic GANs you may need a model per class; CGANs condition generation on the class (label fed to both generator and discriminator), so a single model suffices. B breaks the GAN; C describes a VAE; D is diffusion.

Why does generating data by sampling the latent space of a classic AutoEncoder usually fail?

ABecause the AE has no decoder to reconstruct the sampled points

BBecause the AE can only be trained with adversarial losses

CBecause sampling requires conditioning on a class label

DBecause the latent space is irregular and overfits, so some points are meaningless once decoded

D — a classic AE is trained only to encode/decode, so the latent space has no enforced "regularity" and overfits; sampled points can be meaningless once decoded. A is false (the AE does have a decoder), B/C are unrelated. This is precisely what the VAE fixes.

Simulation exam · Q9
Why is a Variational AutoEncoder better than a "classic" AutoEncoder for Generative AI?

AIts decoder generates multiple variations of the output given the same input in the latent space

BIt is significantly faster during inference

CIt is significantly faster during training

DIt learns a regularized latent space while mitigating overfitting

D — the slide: a VAE's training "is regularised to avoid overfitting" and to ensure the latent space has good properties for generation (via the KL term). B and C (faster inference/training) are false — a VAE isn't about speed. A sounds plausible but is not the defining advantage the deck gives; the regularized latent space is the point.

What does the encoder of a VAE output, and what extra loss term does it use?

AA single latent vector z = e(x), trained only with a reconstruction loss

BA normal distribution (mean and standard deviation), with a KL loss toward N(0, I)

CA discrete sequence of codebook symbols, with a quantization loss

DA heatmap of important input regions, with a contrastive loss

B — the VAE encoder maps the input into a normal distribution (it produces μ and σ); a sample is drawn and decoded, and the loss adds the KL divergence to N(0,I) on top of reconstruction. A is the plain AE; C is VQ-VAE; D is unrelated (XAI).

What characterizes a Conditional VAE?

AThe discriminator is conditioned on the class label fed alongside the real data

BThe latent space is quantized into a finite codebook of symbols

CBoth encoder and decoder also use conditioning information (e.g., a text prompt)

DThe model removes noise step by step conditioned on the diffusion timestep

C — a Conditional VAE conditions generation on another modality: the encoder maps the input considering the conditioning c, and the decoder reconstructs also using c. A describes a CGAN, B is VQ-VAE, D is conditional diffusion.

In a diffusion model, what happens during backward (reverse) diffusion?

AAt each step, the model learns to remove noise to reach the previous, less noisy step

BNoise is progressively added to the image until it becomes pure noise

CThe image is mapped into a discrete codebook of symbols

DThe generator is trained to fool a discriminator on real vs fake images

A — backward diffusion learns to remove noise (step t → t−1), the tricky part; after training, generation starts from pure noise and denoises step by step. B is forward diffusion (adds noise). C is VQ-VAE, D is a GAN. Don't swap forward/backward.

Why do diffusion models use a U-net with skip connections for denoising?

ASo the encoder and decoder can be cleanly separated and used independently

BSo the decoder also uses information from the input, enabling high-quality generation

CSo the model can quantize the latent space into discrete symbols

DSo the model trains adversarially against a discriminator

B — skip connections let the decoder reuse input information, producing high-quality output. They actually make separating encoder from decoder impossible (so A is the inverted, false claim). C/D belong to VQ-VAE and GANs. The U-net came from medical image segmentation.

What defines a Latent Diffusion Model?

AIt runs the diffusion process directly in pixel space without any encoder

BIt replaces the U-net with a GAN generator and discriminator

CIt conditions generation only on class labels, never on text

DA variational encoder maps to a latent space, the diffusion runs there, and a variational decoder reconstructs

D — a latent diffusion model performs diffusion in a compressed latent space (variational encoder → diffusion → variational decoder), which is far cheaper than pixel space (A is the opposite). Conditioning can use cross-attention with text (so C is false); it still uses a U-net (B is false).

In the GAN vs VAE vs Diffusion comparison, which family gives high-quality but low-diversity samples and is hard to train?

AGAN

BVAE

CDiffusion model

DVector-Quantized VAE

A — GANs: high-quality, low-diversity (mode collapse), hard to train. VAEs: low-fidelity, high-diversity, easy to train. Diffusion: high-fidelity, high-diversity, easy to train but slow sampling. Each family trades off quality, diversity and training difficulty differently.

Which of these statements about prompting LLMs on sensor data is FALSE?

ASmart-home sensor events can be translated into human-readable sentences

BTheir scalability is questionable, especially for continuous accelerometer data

CLLMs are inherently trained to understand raw sensor data such as accelerometer signals

DFew-shot prompting or fine-tuning can improve them on specific tasks

C is FALSE — the slide states LLMs are typically NOT trained to understand raw sensor data, which is exactly why scalability is questionable for accelerometer streams. A, B, D are true. The absolutizer "inherently trained" flips the slide's point.

In ADL-LLM, how does an LLM recognize the activity from smart-home data?

AIt is fine-tuned on the raw sensor stream with a reconstruction loss

BA window of sensor data is transformed into a text description and the LLM is prompted to output the activity

CThe sensor data is mapped to a semantic image and a CNN classifies it

D3D motion is generated from the activity name and converted to IMU data

B — ADL-LLM turns a sensor window into a textual description, then prompts the LLM (system prompt listing locations, interactions, possible activities) to return the most likely activity — zero-shot, no fine-tuning. C is DeXAR (L13), D is IMUGPT (generation, not recognition).

What characterizes LLM reprogramming (e.g., Time-LLM) for time series?

AThe entire LLM is retrained from scratch on the time-series task

BThe time series is fed as raw numbers and the whole backbone is fine-tuned

CThe backbone stays frozen; the input is converted into text-prototype tokens and only the head is fine-tuned

DThe LLM is replaced by a U-net that denoises the series step by step

C — reprogramming keeps the backbone intact (frozen), converts the input series into text-prototype representations (tokens), and fine-tunes only the head with few-shot training. A and B contradict "keeping the backbone intact"; D is diffusion. Time-LLM reprograms patches against pre-trained word embeddings.

What is the goal of VQ-VAE as used in T2M-GPT?

ALearning a discrete representation of human motion via a codebook of K symbols

BMapping the input into a continuous normal distribution over the latent space

CDenoising pure noise into a 3D motion step by step

DConditioning a GAN generator on the activity class label

A — VQ-VAE learns a discrete motion representation: quantization maps motion sequences to sequences of symbols from a codebook C of K symbols; the decoder reconstructs motion from them. B is a (continuous) VAE, C is diffusion, D is a CGAN. T2M-GPT then predicts these symbols from text.

In IMUGPT, how is a labeled inertial HAR dataset produced from an activity name?

AThe activity name is fed to a diffusion model that directly outputs accelerometer signals

BAn LLM expands the name to a motion description, T2M-GPT generates 3D motion, which is mapped to synthetic IMU data

CA CGAN conditioned on the activity name generates the sensor segment in one step

DThe LLM is reprogrammed so its frozen backbone outputs IMU readings

B — IMUGPT chains: LLM (activity name → motion description) → T2M-GPT (text → 3D motion) → mapping (inverse kinematics / IMUSim) to synthetic IMU data, with the label known from the seed name. A/C are single-step generators the deck doesn't use here; D is reprogramming, a different idea.

Open-ended · Generative families

a) Compare GANs, VAEs and diffusion models — how each generates, their characteristic trade-offs, and the role of the U-net and latent space in diffusion.

b) Why does a plain AutoEncoder fail at generation, and how does the VAE fix it (encoder output, loss, regular vs irregular latent space, conditional variant)?

Model answer

a) GAN: a generator and a discriminator trained adversarially — the discriminator separates real vs synthetic, the generator fools it; gives high-quality but low-diversity samples (mode collapse) and is hard to train; mainly used for data augmentation (CGANs condition on a class to train one model). VAE: an encoder maps to a latent distribution, a sample is decoded; low-fidelity, high-diversity, easy to train. Diffusion: trained to denoise in steps — forward diffusion adds noise to pure noise, backward diffusion learns to remove it; generation starts from noise and denoises step by step; high-fidelity, high-diversity, easy to train, but slow at sampling; the most effective in CV (MidJourney, Suno). Diffusion denoises with a U-net whose skip connections let the decoder reuse input info (high quality, and they make encoder/decoder inseparable). Latent Diffusion Models run the diffusion in a latent space (variational encoder → diffusion → variational decoder), and can be conditioned by other modalities via cross-attention.

b) A classic AE is trained only to encode and decode, so its latent space is irregular and overfits — sampling random latent points often yields meaningless reconstructions. The VAE fixes this by regularizing training: the encoder outputs a normal distribution (mean μ and standard deviation σ) instead of a single point, a sample z ~ p(z|x) is drawn and decoded, and the loss adds a KL divergence term pulling the latent distribution toward N(0,I) on top of the reconstruction loss. This yields a regular latent space where points close together decode to similar outputs (avoiding meaningless gaps) and overfitting is mitigated — enabling generation. A Conditional VAE additionally conditions both encoder and decoder on extra information c (e.g., text), so generation can be steered by another modality.

Open-ended · LLMs and generating HAR data

a) Describe the ways LLMs are used in AmI: prompting to annotate/recognize (ADL-LLM, TDOST, HARGPT), fine-tuning/PromptCast, reprogramming (Time-LLM), and Natural Language Supervision (zero-shot HAR).

b) Explain how generative models from the 3D-motion domain (VQ-VAE, T2M-GPT, IMUGPT) generate labeled inertial datasets, and the limitations.

Model answer

a) Prompting: smart-home events translate to sentences ("the fridge has been opened"), so a window maps to text; ADL-LLM prompts an LLM (with locations, interactions, candidate activities, plus temporal/layout/sensor info) to output the activity zero-shot. TDOST turns windows into sentences encoding semantics to enable cross-home transfer (train on source, transfer weights to target). HARGPT feeds raw IMU readings to an LLM with a prompt template for step-by-step zero-shot recognition. Fine-tuning: Google (2023) fine-tuned LLMs on health data (activity, atrial fibrillation, stress); PromptCast fine-tunes for time-series forecasting via text. Reprogramming (Time-LLM): keep the backbone frozen, convert the series into text-prototype tokens, fine-tune only the head (few-shot). Natural Language Supervision: learn a shared latent space for sensor data and text via cross-modal contrastive pre-training (IMU encoder + text encoder); classify by cosine similarity to activity-label embeddings, enabling zero-shot recognition of unseen activities. Cons of prompting: scalability is questionable since LLMs aren't trained on raw sensor data.

b) Labeled inertial data is scarce and generative models need lots of data, so we borrow from the 3D-motion domain, where massive labeled datasets exist. VQ-VAE learns a discrete motion representation: motion sequences are quantized into sequences of symbols from a codebook of K symbols (encoder → symbols, decoder → motion). T2M-GPT then trains a Transformer with CLIP text embeddings and an autoregressive objective to predict the symbol sequence from text, which the VQ-VAE decoder turns into 3D motion. IMUGPT chains it end-to-end: an LLM expands an activity name into motion descriptions, T2M-GPT generates 3D motion, and the motion is mapped (inverse kinematics / IMUSim, motion filter, calibration) to synthetic IMU data — automatically labeled by the seed activity. Limitations: it reuses models trained on a significantly different task and reconstructs IMU data with ad-hoc methods — an approximation that may miss subtle movements, and it can only cover activity types present in T2M-GPT's training motions.

0/16