Sensing in Smart Homes
Smart homes monitor the high-level activities (ADLs) of residents — crucial for ambient assisted living: monitoring elderly subjects for early symptoms of cognitive decline, emergency monitoring.
The binary sensor zoo
Magnetic
Door/drawer open-close via a magnet: magnet close → OFF (closed), otherwise ON (open). Fridge, closets, medicine drawers…
PIR (Passive InfraRed)
Detects motion/proximity in areas. Senses motion only — a still user is invisible to it!
Pressure mats
Detect sitting on a chair or standing up.
Sleep mats
Unobtrusive sleep quality: phases (light, REM), snoring, heart rate.
Smart plugs
Detailed power usage (Lecture 6!) — but thresholded into binary ON/OFF appliance usage.
The raw output is a binary sensors log: timestamped lines like 2010-11-04 05:43:45 M003 ON — sensor ID + state change.
Enriching the picture
- Indoor localization (Lecture 5): binary sensors sit in fixed positions, are not everywhere, and cannot identify the resident — localization gives richer position/trajectory context. Drawback: the resident must wear a device.
- Mobile/wearables (Lecture 7b): alone they can't capture complex high-level activities, but combined with environmental sensors they add gestures and posture; some applications (e.g., fall detection) rely mainly on wearables.
- Audio/video: possible context sources, but perceived as too privacy-intrusive (especially by elderly subjects) — not considered in this course.
Segmentation
Binary environmental sensors behave very differently from inertial sensors — and that changes how we window the stream.
Time-based windowing
- Fixed time sliding windows: simple, efficient, great for continuous-value sensors (accelerometer — Lecture 7b)
- Problem here: binary sensors have no fixed sampling rate → event distribution across windows is inconsistent
- Many windows may contain no events at all when the subject is idle
Event-based windowing
- Each window contains a fixed number of sensor events; the time interval covered is variable
- The number of events is a parameter requiring proper tuning
- Why better: events are dense during activities, sparse/absent in silent periods → avoids unrepresentative windows
Handcrafted Features
Basic feature extraction
Given a window of the latest k sensor events, build a fixed-dimensional feature vector with: timestamp of the first event, timestamp of the last event, temporal span of the window, and the activation count per sensor event type.
Classify the last event
Common strategy: each time a sensor event occurs, create a window with it and the previous k−1 events as context; the ground truth is the activity performed on the last event. This is a sliding window shifting by a single event — it also improves real-time recognition.
Time-dependency features — the discounted count
- Each activation of sensor j is weighted by its time distance from the latest event in the window, with temporal decay rate χ; the feature for sensor type j is the sum of its contributions.
- χ too low → temporally distant events impact the recognition rate. χ too high → even close events are considered unimportant. The trade-off is found empirically.
Sensor-dependency features — Mutual Information
- During transitions, a window may contain temporally close events that are poorly related. Solution: a Mutual Information matrix computed on the training set — MI(i, j) is the chance of sensor events i and j occurring consecutively in the stream.
- The MI w.r.t. the last event weights the activation count (like time dependency does). Drawback: the matrix may overfit the training set. Sensor and time dependency can be combined.
Past activities as features
Previously recognized activities hint at the current one (washing dishes likely follows eating) → the classifier's outputs on previous windows can become extra features. Not trivial: past classifications may be wrong, so their confidence should be used.
Automatic Feature Extraction
Deep models want to generate embeddings from raw data, and they usually require fixed-size input — so we are back to time-based segmentation, and we must encode binary events as time series.
Three encodings of sensor events
| Encoding | What gets a 1 |
|---|---|
| Raw data | At each instant of the window (e.g., each second), 1 if the sensor event is active, 0 otherwise |
| Change point | 1 only at the instants where the sensor's status changes (e.g., ON → OFF) |
| Last-fired | At each instant, 1 only for the latest sensor event that fired |
Multiple encodings can be combined by creating more than one time series per sensor.
Models
- LSTMs: stacked LSTM layers over the encoded temporal data, flattened into a dense classifier (Lecture 3!).
- LSTM-CNN: 1D CNN with ReLU + pooling first, LSTM after — the same hybrid logic as Lecture 7b.
- CNNs with image representation: windows become activity images — a binary matrix where the x-axis encodes time (with a chosen granularity, e.g., 1 cell = 1 second), each row is a sensor, and white pixels mark the instants where that sensor was active. A DCNN classifies the images (eating vs work vs sleeping look visually different!).
GRU — the small-data workhorse
Why GRU here?
- With small datasets LSTM may overfit → GRUs often preferred in this domain
- A light LSTM: only two gates and one memory (the long-term one, which is also the cell's output)
- Faster to train, less overfitting on small datasets
- Not adequate for long sequences
The two gates
- Reset gate: determines the % of long-term memory to forget based on the input
- Update gate: determines how to combine the input and the long-term memory to generate the output
- Compare with LSTM (Lecture 3): three gates and two memories (cell state + hidden state)
Detecting Behavioral Changes
Why recognize ADLs at all? Analyzing behavior long-term can reveal significant changes in activity execution — possible early indicators of cognitive decline.
Activity curves
- An activity curve models an individual's generalized activity routine, at different time granularities (daily, weekly, monthly).
- How to build: segment an observation period into equal-size consecutive time windows and define a probability distribution over activities for each window (e.g., 1:00–2:00 AM: Sleep 0.95, Bed-to-toilet 0.05).
- Aggregated activity curves average distributions over months of data (e.g., at 5-minute intervals).
Comparing routines — KL divergence
- Changes are detected by computing the distance between probability distributions: a high distance may indicate a behavioral change (e.g., year-1 routine vs year-2 routine).
- KL divergence is not symmetric → use the symmetric version: SDKL(D₁ ∥ D₂) = DKL(D₁ ∥ D₂) + DKL(D₂ ∥ D₁).
- Two whole activity curves are compared by summing SDKL over all time intervals covered by the curve.
Wandering Behavior
One symptom of cognitive decline is wandering: moving around aimlessly or without a clear purpose, often in repetitive patterns — usually characterized by loops in the trajectories.
Locomotion categorization
| Pattern | Definition |
|---|---|
| Direct | A single straightforward path to a destination, not diverging significantly from the most efficient path |
| Pacing | At least three consecutive repeated back-and-forth movements between two locations |
| Lapping | A circular repeated movement across at least 3 distinct points, repeated at least twice |
| Random | An aimless movement across numerous locations, which is not direct |
Where do trajectories come from?
- Smart-home sensors: environmental sensors have fixed positions → map each to relative (x, y) coordinates; a sequence of triggered events yields a sequence of timestamped positions — an approximated trajectory.
- Indoor localization (Lecture 5): periodically compute the position of a user wearing a tag (e.g., BLE anchors).
Detecting loops
- Beyond finding loops, clinically relevant wandering also considers: the loop's area, length, the relative coordinates of its centroid, and the time taken to walk it.
- Supervised learning approaches classify the cognitive status from trajectories labeled with the subject's status (healthy, mild cognitive impairment, cognitively impaired).
Final Quiz — Exam Style
10 MCQs + 2 open-ended in the simulation's style. (No simulation question targets Lecture 8 directly — but this lecture's smart-home setting is exactly where Q7 and the open-ended Q24 live, coming in Lecture 9.)
a) Binary sensors generate events only when the resident interacts with the environment: there is no fixed sampling rate, so with fixed time windows the distribution of events is inconsistent, and many windows contain no events during idle periods. Event-based windowing segments the stream into windows with a fixed number k of sensor events (a parameter to tune) covering a variable time interval: events are dense while activities are performed and sparse otherwise, avoiding unrepresentative windows. Most methods build one window per incoming event (the event plus the previous k−1 as context) and use as ground truth the activity performed on the last event — a single-event sliding window that also benefits real-time recognition.
b) Time dependency: in event-based windows there can be large time lags between consecutive events, so equal importance per event is wrong. The activation count becomes a discounted count C(i,j) = exp(−χ(tᵢ−tₖ)): each activation of sensor j is weighted by its temporal distance from the latest event, with decay rate χ; the feature is the sum of the contributions. χ too low → distant events still influence recognition; χ too high → even close events are deemed unimportant; tuned empirically. Sensor dependency: transition windows may contain temporally close but poorly related events. A Mutual Information matrix, computed on the training set as the chance of two event types occurring consecutively, weights the activation count w.r.t. the last event. Limit: the MI matrix may overfit the training set. The two weightings can be combined; additionally, the classifier's outputs on previous windows ("past activities") can be used as features, weighted by their confidence since they may be wrong.
a) An activity curve models an individual's generalized activity routine at a given time granularity (daily, weekly, monthly). It is built by segmenting an observation period into equal-size consecutive time windows and defining, for each window, a probability distribution over the activities (aggregated over the observation period). Significant routine changes — possible early indicators of cognitive decline — are detected by computing the distance between probability distributions: the KL divergence DKL(D₁∥D₂) = Σ d₁,ᵢ log(d₁,ᵢ/d₂,ᵢ). Since KL is not symmetric, the symmetric version SDKL = DKL(D₁∥D₂)+DKL(D₂∥D₁) is used, and two curves are compared by summing SDKL over all time intervals; a high distance may indicate a behavioral change. Because behavior depends on temporal context (weekday vs weekend), normal-behavior models are built at fine granularity and grouped hierarchically (working days, weekends…).
b) Wandering — a symptom of cognitive decline — is the tendency to move around aimlessly or without clear purpose, often in repetitive/looping patterns, characterized by loops in trajectories. Locomotion categories: direct (single efficient path), pacing (≥3 consecutive back-and-forth movements between two locations), lapping (circular movement across ≥3 distinct points repeated at least twice), random (aimless, non-direct movement across many locations). Trajectories come from environmental sensors mapped to fixed (x,y) coordinates (sequence of triggered events → approximated trajectory) or from indoor localization with a wearable tag. Loop detection: find sub-sequences whose algebraic sum of points is ≈ 0 within an approximation error ε → a loop, i.e., a wandering segment. Clinically relevant features — loop area, length, centroid coordinates, walking time — feed supervised classifiers of cognitive status (healthy, mild cognitive impairment, cognitively impaired).