HAR on Mobile & Wearables
Mobile and wearable devices are ubiquitous and packed with sensors that can track our activities — powering healthcare, well-being and sport applications.
Existing commercial solutions
Fitbit, Garmin, Suunto, Jawbone, Withings… offering lifestyle monitoring, sleep quality tracking, and activity recognition based on visited places.
Activity Recognition APIs in mobile OSes
Both Android and iOS have included activity-recognition APIs for years — usable to create context-aware applications (Lecture 1!).
🤖 Android
IN_VEHICLE,ON_BICYCLE,ON_FOOTWALKING,RUNNING,TILTINGSTILL,UNKNOWN
🍎 iOS
- stationary, walking, running
- automotive, cycling
- unknown
Sensors & Placement
Inertial sensors — the workhorses of mobile HAR
| Sensor | What it measures | Why it matters |
|---|---|---|
| Accelerometer | Acceleration values on three axes | The most adopted sensor for HAR |
| Gyroscope | Angular speed on three axes | Reveals changes in direction |
| Magnetometer | Magnetic field on three axes | Field changes may hint about the activity |
Other useful sensors (context!)
Localization
e.g., GPS — where the activity happens
Light sensor
Indoor/outdoor, pocket vs hand
Microphone
Activity-specific audio patterns
Proximity
Device near body or surface
Physiological
e.g., heart-rate sensors
Device placement
Wearables can sit on the ear, necklace, chest, arm, wrist, hip, thigh, knee, foot… The most common HAR setting: smartphone in the pocket + smartwatch on the wrist. (Remember sensing heterogeneity from 7a: different placements → different signals!)
Pre-processing
Data cleaning
- Inertial data are noisy → smoothing filters keep only the informative part: median, low-pass, gaussian filters (same smoothing idea as Lecture 3!).
- Linear acceleration: the accelerometer also captures gravity. Combining acceleration with orientation (from the gyroscope), the gravitational component is subtracted from each axis — the result better represents the user's actual movements/gestures.
Segmentation
Performed on a multivariate time series: multiple sensors per device, multi-channel data per sensor (3 axes), possibly multiple devices per user. Each sensor may have its own sampling rate → data must be temporally aligned and device clocks synchronized.
Sliding window segmentation
- Two parameters to tune: segment length and overlap — the choice strictly depends on the target activities.
- Overlap is crucial to capture transitions between activities.
- Simple physical activities (running, walking, standing) → short windows, e.g. 3 seconds with 50% overlap. More complex activities → longer windows.
- Real-time requirements also drive the choice: trade-off between the system's response delay and accuracy (literature reports a Wmin_size to prioritize speed and a Wmax_perf to prioritize accuracy per activity category).
Dynamic segmentation
- Drawback of sliding windows: a segment may not fully represent an activity pattern, and choosing the size is not trivial.
- Idea: unsupervised AI methods detect change points in the sensor stream — changes intuitively indicate transitions between activities.
- Framed as an optimization problem: each segment has a cost (e.g., avg, std, slope); find the change points minimizing the total cost, plus a penalty per added change point — high penalty → too few change points, low penalty → too many.
- Generally computationally expensive → not adequate for real-time requirements.
Segments labeling (training time)
Handcrafted Features
The "classic" pipeline (7a!) extracts handcrafted features from each segment and feeds them to classic ML models.
Time-based features
- Statistics computed directly on sensor data; they capture temporal information
- Mean, standard deviation, variance, IQR, mean absolute deviation (MAD), correlation between axes, entropy, kurtosis
Frequency-based features
- Statistics capturing the frequency properties of sensor data
- Fourier Transform (FT), Discrete Cosine Transform (DCT)
- Others: PCA, LDA, Autoregressive models, HAAR filters.
- Important: features must be computed on each axis of each sensor — feature vectors grow fast!
Automatic Feature Extraction
CNN
- Designed for vision, but good for time series: local dependency (close signals are likely correlated) and scale-invariance (robustness to frequency changes).
- Drawback: CNNs do not fully capture temporal relationships.
- Input adaptation (HAR input = multivariate time series, a set of 1D vectors): single-head 1D CNN · multi-head 1D CNN (each axis a separate channel) · transform the input into a 2D image + 2D convolution.
- Sensor-to-image example: stack the time series row-by-row (each repeated to be adjacent to every other) into a Signal Image, then apply 2D DFT and keep the magnitude as the Activity Image — letting the CNN find hidden correlations between neighboring signals.
LSTM & friends
| Model | Strength | Watch out |
|---|---|---|
| LSTM | Automatically extracts strong temporal features — widely used for HAR (Lecture 3!) | May overfit on small datasets |
| BiLSTM | Adds a second LSTM processing the series backwards; forward + backward outputs are aggregated → better temporal properties, reduces overfitting | Higher cost |
| LSTM ensembles | A pool of LSTMs trained at once — epoch-wise bagging (random training subsets) + different loss functions; best-M models fused at score level. Goal: robustness to noise and unbalanced data | Expensive to train |
Hybrid CNN-LSTM & Transformers
- Hybrid approaches are among the most effective for supervised HAR: convolutional layers first extract meaningful features from raw data, recurrent layers then capture temporal relationships.
- Transformer for HAR: an encoder-only transformer — the architecture typically adopted for classification tasks (even in NLP). (Compare with Lecture 4: full encoder-decoder is for seq2seq generation.)
Deployment on Mobile Devices
Offline vs Online recognition
Offline
- Devices only collect data, processed in another phase
- Fine when real-time is not required, in favor of accuracy (e.g., behavior analysis)
Online
- Recognizing activities in real-time
- e.g., to support context-aware applications
Online: where should it run?
Server-side
- Pro: high computational capabilities
- Cons: network communication overhead, reduced privacy
Client-side
- Pro: good real-time support, better privacy
- Cons: complex models can't run efficiently on resource-constrained devices
How to deploy
- SDKs: Weka (Android), CoreML (iOS), TensorFlow Lite (both). The same SDK should be used to create and deploy the model.
- After deployment, the device collects the stream and performs segmentation/pre-processing with the same parameters used during training! Each segment is classified to obtain the activity label.
Efficiency: making models fit
Mobile/wearables have computational constraints, and deep models hurt energy usage (and usability). Either design simpler models from the beginning, or reduce the size of existing complex models:
Compression
- Pruning unimportant parts (neurons/weights with minor impact on accuracy)
- Merging components (sharing weights for similar edges)
- Knowledge distillation
Quantization
- Reducing the resolution of weights storage — the numerical precision
- 32-bit float → 16-bit; or integers/fixed point (8-bit)
- Advanced: sub-byte quantization (<8 bits, hard since HW stores at byte level), mixed precision (different bit-widths per layer — better accuracy/size trade-off, much harder to design)
Knowledge distillation
- Train a small "student" model relying on the knowledge of a bigger "teacher".
- The teacher's softmax output gives soft labels; the distillation loss forces the student to generate a similar probability distribution over activities on the same data.
- The student loss is the target HAR task itself (e.g., cross-entropy with the ground truth) — the two losses are jointly learned.
Final Quiz — Exam Style
10 MCQs + 2 open-ended. Question 4 is the actual simulation-exam Q14, reproduced verbatim.
Which of the following options describes the "model quantization" approach to deploy HAR models on resource-constrained devices?
a) The pipeline is data cleaning → segmentation → feature extraction → feature vector. Cleaning: inertial data are noisy, so smoothing filters (median, low-pass, gaussian) keep the informative part; often the gravity component is removed from acceleration (using the orientation from the gyroscope) obtaining linear acceleration, which better represents actual movements. Segmentation works on a multivariate time series (multiple sensors, multiple axes, possibly multiple devices; data must be temporally aligned and clocks synchronized). Sliding windows have two parameters: window length L and overlap M, chosen based on the target activities — simple physical activities need short windows (e.g., 3 s with 50% overlap), complex activities need longer ones. Overlap is crucial to capture transitions. With real-time requirements there is a trade-off between response delay and accuracy (short windows → fast but less accurate; longer windows → more accurate but delayed). At training time each window is labeled with the prevalent activity.
b) Sliding windows may not fully represent an activity pattern and the size choice is non-trivial. Dynamic segmentation uses unsupervised methods to detect change points in the sensor stream — intuitively, transitions between activities. It is framed as an optimization problem: each segment is associated with a cost (e.g., avg, std, slope) and the goal is to find the change points minimizing the total cost; a penalty is added per change point (high penalty → too few change points; low penalty → too many). Main limit: these methods are computationally expensive, hence inadequate when there are real-time requirements.
a) Offline recognition: devices only collect data, which are stored and processed in another phase — useful when real-time is not required, favoring accuracy (e.g., behavior analysis). Online recognition: activities recognized in real-time, e.g., to support context-aware applications. Online can run server-side (pro: high computational capabilities; cons: network communication overhead and reduced privacy, since sensor data leave the device) or client-side (pro: good real-time support and better privacy; cons: complex models cannot run efficiently on resource-constrained devices). Deployment workflow: train offline (e.g., Python), convert to the format of the mobile SDK (Weka for Android, CoreML for iOS, TensorFlow Lite for both — the same SDK should create and deploy the model), then on-device the stream is segmented and pre-processed with the same parameters used during training before classification.
b) Either design simpler models from the start, or reduce existing models. Compression shrinks the model structure: pruning unimportant neurons/weights (minor impact on accuracy), merging components (sharing weights for similar edges), or knowledge distillation. Quantization instead reduces the numerical precision of the weights: 32-bit floats → 16-bit, or 8-bit integers/fixed point; advanced variants are sub-byte quantization (<8 bits, hard because hardware stores data at byte level) and mixed precision (different bit-widths per layer — better accuracy/size trade-off, harder to design). They are two different processes but can be combined; the key problem is the trade-off between recognition rate and efficiency. Knowledge distillation: a small student learns from a large teacher — the teacher's softmax output provides soft labels, and the distillation loss forces the student to produce a similar probability distribution on the same data, while the student loss (e.g., cross-entropy with ground truth) optimizes the actual HAR task; the two losses are learned jointly.