AmI · Lecture 7a — Human Activity Recognition: Intro

What is HAR?

Definition

Human Activity Recognition (HAR): sensing technologies + AI methods continuously and unobtrusively analyze the activities humans perform in daily life. Human activities are a crucial context information for Ambient Intelligence systems (back to Lecture 1!).

Two types of activities

Physical activities

Low-level, tied to specific body movements
Captured by mobile/wearable devices (smartphone, smartwatch)
e.g., running, walking, cycling, stairs

Activities of Daily Living (ADLs)

High-level, performed in everyday life
Monitored with environmental sensors (smart homes), sometimes + wearables
e.g., eating, cooking, taking medicines, sleeping

Applications

🏥

Healthcare

Remotely monitor fragile subjects (elderly): support daily life, alert caregivers, detect early pathology symptoms (cognitive decline).

🏃

Well-being

Track physical activities with mobile devices to promote a healthy lifestyle.

🏭

Industry

Worker safety, training on critical operations, process optimization.

⚽

Sports

Training assistance, injury prevention, performance analysis, rehabilitation.

📹

Surveillance

Prevent risky situations in waiting rooms, malls, hospitals.

🎮

HMI & Domotics

Gesture recognition for interaction; reminders, automations and energy saving in smart homes (Lecture 6!).

The Challenges

Data distribution heterogeneity

Inter-variability

Variability among different subjects
e.g., walking patterns differ between an elderly and a young subject

Intra-variability

Execution variability for the same subject
e.g., walking differently indoors vs outdoors

Plus: some activities are intrinsically hard to discriminate — walking and going upstairs have very similar acceleration patterns.

Sensing heterogeneity

Different users → different device models, different positioning of wearables, different smart-home settings. Building one generalized model for every sensor setting is challenging.

Class imbalance

Some activities are more frequent / longer than others; habits differ per subject (a sporty subject runs more).
Mitigations: data augmentation (e.g., SMOTE) and synthetic data generation (e.g., GANs).

The "OTHER" activity class

A real-world necessity

HAR is a classification problem, but you can't foresee every possible activity. Real deployments need an "OTHER" class for when the user isn't performing any considered activity — challenging because it's a theoretically infinite space of arbitrary activities, and it risks becoming a dominating class that absorbs everything.

Transitions & labeled data scarcity

Transitions: humans don't switch activities instantly — a window capturing a transition (standing → sitting) represents none of the activities. Slightly different from the OTHER problem.
Labeled data scarcity: a major challenge — collecting labeled activity data is difficult, so methods for limited labels are needed (covered in later lectures).

ThinkA HAR model trained on data from 20 university students performs poorly when deployed to elderly users. Which challenge is this, and is it inter- or intra-variability?

Answer: This is data distribution heterogeneity, specifically inter-variability — variability among different subjects (elderly perform the same activities with different patterns than young students). Intra-variability would be the same person performing an activity differently in different contexts (e.g., indoors vs outdoors).

Personalized vs Generalized Models

Model type	How	Pro	Cons
Personalized	Trained directly with target user data	Very accurate	Prohibitive to collect labeled data for each user
Generalized	Trained with data from many subjects, then used directly on unseen users	Trained once, then deployed	Often not accurate — generalization is hard due to intra/inter-variability + sensing heterogeneity
Hybrid	Train on many users, then fine-tune with a small portion of labeled target-user data	Combines both advantages	Still needs some labeled data from the target user

Echo from Lecture 6: this is the same pre-train + fine-tune idea used to avoid harmful RL exploration in smart homes.

Sensing Modalities

Modality	How it works	Main limit
Mobile / Wearable	Wide variety of on-body sensors; inertial sensors (accelerometer) are the most adopted	Position/device heterogeneity
Environmental	IoT devices in the home capture interaction with the environment; can include indoor localization (Lecture 5)	Fixed infrastructure needed
Camera-based	Fixed environmental cameras or egocentric (worn) cameras; very accurate	Very privacy-intrusive — accepted only in limited domains
Audio-based	Each activity generates specific audio patterns (device or home microphones)	Continuously recording audio is also very privacy-intrusive
Wireless sensing	Human movement perturbs wireless signals (e.g., WiFi CSI — Lecture 5!); unique patterns → spectrogram → CNN	Poor generalization on unseen environments; environmental changes degrade recognition

Multi-modal HAR: modalities can be combined to merge their strengths. This course focuses on two settings: HAR with mobile/wearables (Lecture 7b) and HAR in smart homes with environmental sensors (Lecture 8).

HAR Pipeline & Collecting Data

The pipeline

Sensing devices→ Data cleaning→ HAR model→ Probability distribution over activities→ Most likely activity

"Classic" pipeline

Handcrafted features (time domain: mean, variance, max…; frequency domain: amplitude, energy, kurtosis…)
Classic ML: KNN, Naïve Bayes, decision trees, SVM, HMM, Gaussian Mixtures, shallow NNs

"Deep" pipeline

Deep feature extraction + model building in one: CNN, DNN, RNN, DBN
Learns features from raw signals (Lecture 3's automatic feature extraction)

Why there are no big pre-trained models for HAR

Unlike Computer Vision and NLP (data everywhere), collecting labeled HAR data needs volunteers actually performing activities, complex collection/annotation infrastructure, and accurate annotators. Public HAR datasets are significantly smaller than CV/NLP ones.

Scripted vs In-the-wild collection

Scripted

Volunteers perform a predetermined set of activities while observed and annotated
Problems: limited realism, few subjects, limited activity types/contexts
The majority of public datasets use this approach

In-the-wild

Users live freely and self-annotate their activities
Realistic, long-term, wide variety of data
Problems: complex software infrastructure, noisy/erroneous annotations, intrusive

Annotation by observation: an expert team watches videos / observes users — intrusive, error-prone, time-consuming, scripted-only.
Self-annotation tools: users label their own activity (good for realistic scenarios) — intrusive, annoying, possibly unreliable.
Result of collection: time-stamped multidimensional sensor time series per user + temporal start/end of activities (the ground truth).
Design choices to plan: activities, devices & positions, sensors & sampling rate, number of subjects, scripted vs in-the-wild, annotation method, duration.

Evaluation

Splitting strategies

Strategy	How	Problem for HAR
"Naive" split	Train / validation / test (e.g., 70/10/20)	HAR datasets are small → the split may not be representative
Cross-validation	k equal partitions; each fold uses one block as test; average the results	Data from the same user can be in both training and test → overfitting
Leave-one-subject-out	At each fold, one user's data is the test set, all others train	— (this is the fix: it assesses generalization on unseen users)

Metrics — per activity

True Positive: the inferred activity is correct. False Positive: the model inferred an activity not actually performed. False Negative: the activity was performed but not recognized.
Precision — inversely proportional to false positives. Recall — inversely proportional to false negatives. F1 — harmonic mean of the two.

Why not accuracy?

Due to the high class imbalance of HAR datasets, accuracy is not a good metric — a model predicting only the majority class can score high. Metrics are computed per activity and globally; the confusion matrix shows which activities the model confuses ("debugging" the classifier).

ThinkYour 5-fold cross-validation gives 95% F1, but the deployed model scores 60% on new customers. What went wrong methodologically, and which evaluation should you have used?

Answer: In plain k-fold CV, data from the same user ends up in both training and test sets — the model partially "memorizes" each user, inflating the score (overfitting to subjects, not generalizing). To estimate performance on unseen people you need leave-one-subject-out cross-validation: each fold tests on one entirely held-out user, exactly simulating deployment to new customers.

✓

Final Quiz — Exam Style

9 MCQs + 2 open-ended, all in the simulation exam's style. (No sim question maps directly to this intro lecture — its content feeds Q7, Q14 and Q24 in the next lectures.)

What is the difference between inter-variability and intra-variability in HAR?

AInter is variability across sensors; intra is variability across activities

BInter is variability among different subjects; intra is variability for the same subject

CInter is variability for the same subject; intra is variability among different subjects

DInter is noise in the training set; intra is noise in the test set

B — inter: different subjects perform the same activity differently (elderly vs young walking). Intra: the same subject varies their execution (walking indoors vs outdoors).

Why is the "OTHER" activity class challenging in real-world HAR deployments?

ABecause it can only be detected with camera-based sensing

BBecause it never appears in real data, only in scripted datasets

CBecause it covers a theoretically infinite space of activities and may become a dominating class

DBecause the softmax layer cannot output more than ten classes

C — you can't foresee all possible activities; OTHER captures "none of the considered ones", but it has no limited pattern and risks absorbing many activities.

Which statement about generalized HAR models is correct?

ATrained once on many subjects, but often inaccurate on unseen users

BTrained on the target user, very accurate but costly to label

CThey always outperform personalized models on the target user

DThey require fine-tuning on every user before any deployment

A — generalized: train once on many subjects, deploy directly; recognition is often inaccurate due to intra/inter-variability and sensing heterogeneity. B describes personalized models; D describes the hybrid approach.

What is the goal of leave-one-subject-out cross-validation?

AReducing the training time by using fewer folds

BBalancing the classes before computing the F1 score

CMaximizing accuracy by mixing each user's data in all folds

DAssessing model generalization on users unseen during training

D — if the same user's data is in both training and test (as in plain k-fold CV), the evaluation overestimates performance. LOSO holds out one entire subject per fold to test generalized models honestly.

Why is accuracy not a good metric for HAR?

ABecause HAR is a regression problem, not classification

BBecause HAR datasets are highly class-imbalanced

CBecause accuracy cannot be computed on time series

DBecause accuracy ignores true positives entirely

B — with imbalanced classes, predicting only the majority class yields high accuracy. Use precision (↓ with FP), recall (↓ with FN) and F1 (harmonic mean), per activity and globally, plus the confusion matrix.

What characterizes the in-the-wild data-collection setting?

AVolunteers perform predetermined activities while experts observe them

BData is generated synthetically with GANs instead of being collected

CUsers live freely and self-annotate, yielding realistic but noisier labels

DActivities are simulated by robots in a controlled laboratory

C — in-the-wild: realistic, long-term, varied data, but complex infrastructure and noisy/erroneous self-annotations. A describes the scripted setting (used by most public datasets).

What is the main limitation of camera-based and audio-based HAR?

AThey are very intrusive in terms of privacy

BThey cannot recognize low-level physical activities

CTheir accuracy is far lower than inertial sensors

DThey only work outdoors with good lighting

A — cameras are very accurate and audio patterns are informative, but continuously recording video/audio of people's lives is highly privacy-intrusive; cameras are accepted only in limited domains.

What is the main problem of wireless sensing (e.g., WiFi CSI) for HAR?

AIt requires the user to wear a dedicated receiver tag

BIt cannot distinguish more than two activities

CIt only works when the user carries a smartphone

DIt generalizes poorly to unseen or changed environments

D — human movement perturbs the wireless signal in unique, environment-specific patterns; recognition degrades on unseen environments or when the environment changes. (CSI is the same signal we met in Lecture 5.)

Why are there no large pre-trained models for sensor-based HAR, unlike CV and NLP?

ABecause sensor data cannot be processed by deep neural networks

BBecause collecting labeled activity data is hard, so public datasets are small

CBecause HAR models are forbidden from being shared for privacy reasons

DBecause transformers do not work on inertial signals

B — labeling needs volunteers actually performing activities, complex infrastructure and accurate annotators; HAR datasets are far smaller than CV/NLP ones, hence the labeled-data-scarcity challenge.

Open-ended · HAR challenges

a) Describe the main challenges of HAR (heterogeneity, class imbalance, the OTHER class, transitions, data scarcity).

b) Compare personalized, generalized and hybrid HAR models, with pros and cons.

Model answer

a) Data distribution heterogeneity: the same activity is performed in many ways — inter-variability (across subjects, e.g., elderly vs young walking) and intra-variability (same subject in different contexts, e.g., indoors vs outdoors); some activities also have very similar patterns (walking vs going upstairs). Sensing heterogeneity: different device models, sensor positioning and smart-home settings make one generalized model hard. Class imbalance: activities differ in frequency/duration per subject; mitigated with data augmentation (SMOTE) or synthetic data (GANs). The OTHER class: real deployments must capture "no considered activity", but it spans a theoretically infinite space and risks dominating. Transitions: windows spanning an activity change represent none of the classes. Labeled data scarcity: collecting annotated activity data is costly (volunteers, infrastructure, annotators), so datasets are small.

b) Personalized: trained directly with target-user data — very accurate, but collecting labeled data for each user is prohibitive. Generalized: trained once on many subjects and deployed directly to unseen users — convenient, but often inaccurate due to intra/inter-variability and sensing heterogeneity. Hybrid: train on many users, then fine-tune with a small amount of labeled data from the target user — combines the advantages of both (good starting accuracy + personalization at low labeling cost).

Open-ended · Evaluating HAR models

a) Why are the naive split and standard cross-validation problematic for HAR, and how does leave-one-subject-out fix this?

b) Which metrics are appropriate for HAR and why is accuracy not one of them?

Model answer

a) The naive split (train/validation/test, e.g. 70/10/20) is unreliable because HAR datasets are small — the split may not be representative. k-fold cross-validation averages over k folds, but data from the same user can appear in both training and test sets, so the model partially memorizes subjects and the score overestimates real performance (overfitting). Leave-one-subject-out CV holds out one entire user per fold as the test set, training on all others: it assesses generalization on users unseen by the model — exactly what matters for generalized HAR deployments.

b) For each activity, compute TP (correct inference), FP (inferred activity not actually performed), FN (performed but not recognized). From these: precision (inversely proportional to FP), recall (inversely proportional to FN), and F1 (harmonic mean), computed per activity and globally. Accuracy is not appropriate because HAR datasets are highly class-imbalanced — a model predicting only the majority class would still score high. The confusion matrix complements the metrics by showing which activities are confused with each other.

0/9