The Multi-Inhabitant Setting
Most HAR literature assumes a single-inhabitant home. Real homes have multiple subjects, who may perform:
Same activity in cooperation
Bob and Alice are cooking together
Individual activities in parallel
Alice watches TV while Bob waters the plants
Group + individual in parallel
Alice & Bob cook, Carl & Dave watch TV, Erica reads
Applications beyond smart homes & healthcare: office automation (detecting meetings, presentations), building management, surveillance/security, work activities, smart retail (tracking multiple customers), sport.
The Data Association Problem
Example: Alice and Bob are in the kitchen. Alice relaxes at the dining table, Bob is cooking. A pressure sensor fires (someone sits), a magnetic sensor fires (fridge opened). Which event belongs to whom?
What the datasets look like
- Event-level annotation (e.g., MARBLE): each sensor event carries the
subject_idthat triggered it — ideal, but prohibitive to annotate. - Activity-level annotation (e.g., CASAS): only activity start/end per resident (R1_Work_In_Room begin/end), with parallel activities interleaved — the most likely annotation scenario.
Data-driven data association
Train a classifier that associates each event to the resident who triggered it — then run a classic single-subject HAR model per subject.
- A naive probabilistic approach (Naive Bayes): P(Residentn) = the resident's probability of triggering events; P(Eventi|Residentn) = probability of that event for that resident. Eventi is a complex feature vector encoding spatial and temporal relationships with past events + time features (time of day, day of week…).
- Extensions: e.g., LSTMs analyzing which events belong to which resident based on subject-specific patterns.
- Drawbacks: needs a significant amount of labeled data about subject–sensor relationships, and the trained model is specific to one environment and set of subjects — it can't easily generalize to different settings.
Implicit DA: Multi-Label Models
Many methods skip explicit data association: a single model takes windows of sensor events (possibly mixing residents) and outputs the activity performed by each resident — data association happens implicitly, via multi-label models.
Multi-label ≠ multiclass
Multiclass
- One label per input: y ∈ {c₁ … cL}
Multi-label
- More than one output per input: y ⊆ {c₁ … cL}
- Here the count is fixed: one activity per resident — and the labels are not independent!
The four approaches
| Method | Idea | Watch out |
|---|---|---|
| 1 · Factorized HMM | One independent probabilistic chain per resident (hidden = activity, observable = sensor event); the observable variables are shared between chains → introduces correlations between subjects' activities | Probabilistic graphical models are computationally prohibitive beyond 2 subjects |
| 2 · Combined labels | One single-label multiclass classifier with a label per combination (A1A2 = R1 does A1, R2 does A2) | Easy to implement, but with N residents and M labels the worst case is NM labels! |
| 3 · Binary relevance | One binary classifier per class; an activity is output if its classifier's confidence exceeds a threshold | Dependencies between residents' activities are ignored |
| 4 · Multi-output deep model | One softmax output layer per resident on top of shared layers (e.g., hybrid CNN-LSTM) capturing spatio-temporal relationships between the residents' activities | Still a multi-label model — see below |
Context-Aware Data Association
Assume each resident has a wearable device: it tracks the wearer's identity, location (with indoor localization — Lecture 5!), and low-level physical activities (Lecture 7b!). This context can be used to perform data association without training models.
- Available contexts: the user's posture, micro-localization, gestures, etcetera.
- A rule-based model defines relationships between sensor events and context based on common-sense knowledge (e.g., opening the fridge is necessarily done by a resident in the kitchen while standing) — often encoded in ontologies.
- When an event fires, compute its context-consistency with each user's context; the event is associated with user u only if context-consistent.
Challenges
- Often impractical to assume residents constantly wear wearables (especially elderly) — mitigated by miniaturized devices: smart clothes, smart rings, small BLE tags.
- Accuracy is constrained by the accuracy of the collected context: precision of micro-localization, detectable postures/gestures/physical activities.
Subject Separation
Method 1 — Supervised seq2seq
- Framed as a supervised seq2seq problem (Lecture 4!): the encoder maps sequences of sensor events into a latent space; the decoder separates them into per-subject sensor sequences, delimited by special tokens (SOS, EOS).
- Drawback: detailed annotation is still needed!
Method 2 — Unsupervised
- Assumption: in a first phase, unlabeled data can be collected when only a single resident is at home (detected automatically via PIR patterns or power consumption analysis) — not always realistic.
- The single-resident data is mined for statistics: e.g., the frequency of sensor-event pairs triggered consecutively or temporally close (within a threshold ε; frequency = occurrences of the pair / occurrences of the first event).
- Thresholds then decide if two events are likely triggered by the same resident — e.g., only pairs that occurred temporally close at least K times are attributed to the same resident.
Method 3 — Knowledge-based
- Uses common-sense spatio-temporal knowledge: the semantic and temporal similarity of consecutive events decides (via a threshold) whether they come from the same resident: sim(sei, sej) = (simT, simS).
- Temporal similarity: simT = max(0, 1 − |τ(sej) − τ(sei)| / Tmax), with Tmax the maximum time range considered (e.g., 24 h).
- Semantic similarity from an ontology (objects, locations, persons): the similarity of two concepts uses the Least Common Subsumer (LCS) — the most specific shared ancestor: simC(c₁,c₂) = 2N₃ / (N₁+N₂+2N₃). simS averages the concept similarity of the events' locations and of their associated objects.
- Similar consecutive events chain into the same parallel segment (fridge → stove: same subject; the computer event in the bedroom: another segment).
Challenges & what comes after
- Still hard when residents perform similar activities with similar sensors (Alice sets the table while Bob cooks — both in the kitchen); may not scale with more residents.
- After data association or subject separation: the output is one sensor stream per resident (identity attached only with data association) → apply any single-resident HAR method on each stream individually.
- Group activities remain an open challenge: "same activity ⇒ group activity" is true only sometimes (two people cooking → probably together, one kitchen; two people watching TV → maybe different rooms). Context post-processing decides if an activity is collaborative.
Wearable-Only HAR & People Counting
Collective behavior (mobile/wearables only)
- Inferring multiple subjects' behavior from their devices reveals coordinated movements: queues, unidirectional "lanes" in bi-directional pedestrian flows, intersections, groups gathered at a location.
- Applications: emergencies (optimizing evacuation at large events) and smart-city management (urban planning, pedestrian navigation).
Distributed (centralized)
- Each device recognizes its own individual activity (accelerometer) + contributes group cues (proximity with other subjects, direction)
- A central server aggregates everything to recognize individual and group activities
Decentralized
- No server: devices exchange information pairwise
- Each device keeps a "who is walking with me" set; the sets converge to the walking groups ({1,3,4} and {2,5})
Counting people in smart buildings
- Sometimes it's enough to count people, not recognize activities — e.g., occupancy patterns of office spaces for energy saving: HVAC, lighting and appliances adaptively controlled per space on short-term (hourly) usage changes (Lecture 6 energy!).
- Idea: an array of PIR sensors pointing in different directions → infer the number of subjects and their direction (entering/exiting a room).
- The raw analog voltage patterns feed a deep model; a multi-task supervised learner simultaneously predicts the number of people and their direction (with GAN-based data augmentation).
Final Quiz — Exam Style
10 MCQs + 2 open-ended. Question 2 is the actual simulation Q7 and open-ended 11 is the actual simulation Q24 — both reproduced verbatim.
How can wearable devices (e.g., a smartwatch) simplify data association in multi-inhabitant settings?
Open-ended
a) Data association is the problem of correctly associating each environmental sensor event (e.g., the stove turned on, the fridge opened) to the inhabitant that triggered it. It is a challenge in multi-inhabitant settings because environmental sensor events cannot identify who triggered them: with a single resident there is no ambiguity, but when two or more residents are at home (e.g., Alice relaxing at the dining table while Bob is cooking, with a chair pressure event and a fridge event firing) each event could belong to any of them. It is one of the most challenging problems limiting the deployment of real-world HAR systems in smart homes — also because annotating, for each sensor event, the resident who triggered it is usually prohibitive. Difference vs Subject Separation: subject separation is a weaker form of data association — it only determines which sensor events are likely triggered by different subjects, without associating them with the residents' identities. It is sufficient for applications where it only matters what activities are performed, not who performs them; after either process, single-resident HAR is applied to each resulting stream (with identity attached only in the data-association case).
b) One multi-label approach is the multi-output deep model: a single network (e.g., a hybrid CNN-LSTM) takes windows of sensor events as input and has one softmax output layer per resident, each predicting that resident's activity; the shared layers capture the spatio-temporal relationships between the activities performed (in parallel or jointly) by the different residents, so data association is performed implicitly. (Alternatives: Factorized HMM — one hidden chain per resident sharing the same observable sensor events; combined labels — one multiclass label per activity combination, up to NM labels in the worst case; binary relevance — one thresholded binary classifier per class, ignoring inter-resident dependencies.) General limits of multi-label approaches: they require a training set covering all possible combinations of activities performed by multiple residents, which is not practical; they cannot generalize to different homes with different residents and environments; and they have scalability problems — increasing the number of residents or activities significantly complicates training, with some methods (probabilistic graphical models) computationally prohibitive beyond 2 subjects.
a) Each resident wears a wearable/mobile device tracking their identity, their location (via the indoor positioning infrastructure), and additional context like posture, micro-localization and gestures. The pipeline: wearable + positioning data and environmental sensor data flow into a context aggregation module, then a context-aware data association module produces one sensor stream per resident. Association uses a rule-based model encoding common-sense relationships between sensor events and context (e.g., opening the fridge is necessarily done by a resident in the kitchen while standing) — typically encoded with ontologies. When an event fires, the system computes its context-consistency with each user's current context and associates the event with a user only if consistent (fridge opened → Bob, standing in the kitchen; chair pressure → Alice, sitting in the kitchen). Advantage: data association without requiring training. Challenges: assuming residents constantly wear devices is often impractical (especially for elderly subjects; mitigated by miniaturized devices such as smart clothes, smart rings, small BLE tags), and accuracy is bounded by the accuracy of the collected context (micro-localization precision, detectable postures/gestures).
b) Supervised (seq2seq): an encoder maps sequences of sensor events into a latent space and a decoder generates the separated per-subject sequences, delimited by special SOS/EOS tokens; limit — detailed annotation is still needed. Unsupervised: assumes an initial phase where unlabeled data is collected while a single resident is at home (detected via PIR patterns or power-consumption analysis; not always realistic); statistics are mined (e.g., frequency of event pairs triggered consecutively/temporally close within ε), and thresholds (pair observed close at least K times) decide whether two events belong to the same resident. Knowledge-based: uses semantic and temporal similarity between consecutive events against a threshold — simT = max(0, 1 − |τ(sej)−τ(sei)|/Tmax); simS averages the ontology-based concept similarity (via the Least Common Subsumer, simC = 2N₃/(N₁+N₂+2N₃)) of the events' locations and associated objects; similar events chain into the same parallel segment. General limits: separation stays hard when residents perform similar activities involving similar sensors in the same area (Alice setting the table while Bob cooks), and the methods may not scale well as residents increase.