AmI · Lecture 3 — Time Series Analysis

Time Series Basics

Definition

A time series is a collection of observations made sequentially in time — each observation is associated with the time at which it was obtained.

Univariate

Only one variable observed (e.g., outdoor temperature)

Multivariate

Multiple variables observed — a combination of univariate series (e.g., 3-axis accelerometer)
The most common situation in AmI

Discretization & sampling rate

The real process is continuous (e.g., acceleration), but the signal must be sampled to be stored digitally. The sampling rate (Hz) sets how many samples per second.

High rate → more energy + more expensive analysis
Low rate → information loss
Find a good utility ↔ efficiency trade-off

Time-domain vs Frequency-domain analysis

Time domain

Analysis with reference to time — intuitive & common
Good for linear trends & seasonalities (periodic events)
Hard to find complex/non-linear patterns or isolate noise

Frequency domain

Represent the series by its constituent frequencies
Via Fourier or Wavelet transform
Adds complexity; needs advanced signal processing

The four tasks we tackle in AmI: classification, clustering, anomaly detection, forecasting.

Pre-processing

Temporal alignment

Multivariate series may have different sampling rates, or be misaligned even at the same rate. Two strategies:

Time synchronization — devices aligned on the timestamp
Resampling — align all series to the same sampling frequency

Imputation (missing values)

Conventional imputation doesn't fit time series — the order of values matters. Approaches: backward/forward fill, mean/median imputation, and linear interpolation.

⭐ Denoising by smoothing (Simulation Q22b)

Apply filters to reduce noise. Small overhead, but a risk: changing the original data points may hurt the analysis.

Method	How it works
Simple Moving Avg (SMA)	Replace each value with the average of the last N points: SMA_t = (x_t+…+x_t-N)/N
Exponential Moving Avg (EMA)	EMA_t = α·x_t + (1-α)·EMA_t-1 — α weights the current value vs the past
Median Filter	Replace x_t with the median of its neighbors
Fourier denoising	FFT → frequency domain → remove noisy frequencies (low-pass/high-pass) → back to time domain

⭐ The key parameter — Simulation Q22b

The smoothing window size sets the trade-off between utility and outlier removal: too large → signal over-smoothed, risk of losing information; too small → little effect on noise removal. For EMA the analogous parameter is α.

Normalization / Scaling

In multivariate series, dimensions may have different ranges → high-value dimensions may dominate. Standardize/normalize each dimension separately (e.g., to [0,1]).

ThinkYou apply a moving-average filter to an accelerometer signal and the short, sharp peaks that distinguish "jumping" disappear. What went wrong, and which parameter do you adjust?

Answer: The smoothing window is too large — it over-smoothed the signal and erased the informative peaks (lost information). Reduce the window size (or increase α in EMA so the current sample matters more) to keep utility while still removing some noise.

⭐ Outliers (Simulation Q22a)

This is the exact subject of the open-ended Q22a. Nail the outlier-vs-anomaly distinction and the two outlier types.

Outlier detection ≠ Anomaly detection

Outlier detection finds unwanted data points → goal is data cleaning. Anomaly detection finds anomalous events of interest → goal is to analyze the outlier itself. The difference is sometimes subtle and depends on the application.

Two types of outliers

Point outlier

A single data sample that behaves unusually
vs the whole series → global outlier
vs neighboring points → local outlier

Subsequence outlier

Consecutive samples whose joint behavior is unusual
Each point alone is not necessarily a point outlier
Can also be global or local

Both types apply to univariate and multivariate data.

ThinkA heart-rate series reads 75, 76, 74, 210, 75, 76 (one spike) vs another where 30 consecutive samples sit at a flat, unusually-low plateau though no single value is extreme. Classify each.

Answer: The single 210 spike is a point outlier (likely global, since it's extreme vs the whole series). The 30-sample flat plateau is a subsequence outlier — no individual value is extreme, but the joint behavior of the consecutive samples is unusual.

Segmentation & Feature Extraction

Why segment?

Time series are potentially infinite. A single sample isn't representative — spatiotemporal aspects are crucial, so we reason over time intervals (segments).

Segmentation strategies

Fixed-size windows

Most common, simple
Time-based: each window = w seconds, overlap factor o
Event-based: each window = last k sensor events (for discrete smart-home data)

Dynamic segmentation

Boundaries set by significant changes in distribution (mean shift, std, slope, count)
Each segment captures an event of interest
Challenging & computationally demanding

Feature extraction

Handcrafted

Data scientist designs statistical features per window (time & frequency domain)
Fed to classic ML (Random Forest, SVM, MLP, K-Means)
Limit: designing good features is hard

Automatic

A DNN (CNN/RNN) extracts features from raw segments during training
The resulting feature vector = embedding (latent-space representation)
Proved more effective

Feature reduction (PCA, t-SNE) helps when features are many — and aids visualization.

CNNs for Time Series

CNN building blocks

Layer	Role
Convolutional	A filter/kernel (small matrix of learned weights) slides over the input; element-wise multiply + sum → feature map. Many filters per layer capture different aspects.
Pooling	Reduce feature-map dimensionality (fewer weights). Max pooling (max per patch) or average pooling.
Flattening	Converts CNN output into a vector for the dense layers.
Dense (fully-connected)	Transform high-level features into the final output (class probabilities or regression).

Activations & regularization

ReLU = max(0, z) — most popular, adds non-linearity, fast convergence. Problem: dying neuron (pre-activation always negative → gradient 0 → weights frozen).
Leaky ReLU — allows small negative values (e.g., 0.01x), so neurons don't saturate at 0; solves dying neuron, still non-linear.
Dropout — randomly drop a fraction p of neurons each iteration → regularization, reduces overfitting. Often after pooling.
Softmax output layer → probability distribution over classes (exp keeps positives, accentuates differences). Binary case: single sigmoid neuron.
Cross-entropy loss — pushes probability mass onto one class; risk of overconfidence (mitigated with regularization).

Applying CNNs to sequences

1D CNN — kernel moves only horizontally (along time). The standard for sequences.
Multi-head 1D CNN — each dimension processed by its own channel; more flexible.
2D CNN — frequency-transformed series (e.g., a scalogram) treated as an image.

Limit of CNNs for time series

CNNs capture strong spatial features but may miss temporal relationships between samples → motivates RNNs.

⭐ RNNs & LSTM (Simulation Q20)

Recurrent Neural Networks

Designed for sequential data — each input is a sequence, and sequences may have different lengths (unlike FCN/CNN with fixed input). Key difference: feedback loops.

The output for element x_t is used as extra info when processing x_t+1; the feedback connection has a learnable weight.
Unfolding/unrolling: visual way to show a cell processing each element over time — but it's really one cell with a feedback loop. Weights are shared, fixed regardless of sequence length.

The vanishing/exploding gradient problem

The feedback weight w is applied repeatedly. After a long sequence the output behaves like input × w^seqLength:

Exploding (w > 1)

e.g. w=2, length 50 → input × 2⁵⁰ (giant)
Too-large gradient → optimization oscillates, never settles

Vanishing (w < 1)

e.g. w=0.5, length 50 → input × 0.5⁵⁰ ≈ 0
Gradient ≈ 0 → training extremely slow

LSTM — the fix

Evolution of RNNs to mitigate vanishing/exploding gradients. Main idea: two separate paths instead of a single feedback loop — a long-term memory (cell state) and a short-term memory (hidden state). The unit is more complex.

⭐ The cell state — Simulation Q20

The cell state (long-term memory) carries information over long periods. It has no weights directly associated with it, so information flows without exploding/vanishing gradient. It can still be updated and/or cleared at each time step, and its role is to capture long-term dependencies. (The FALSE statement in Q20 is "its values do not depend on the short-term memory" — they do, via the gates.)

The three gates

Gate	What it does	Activation
Forget gate	Multiplies long-term memory by a sigmoid output → decides % of long-term memory to keep (small → forget)	σ (sigmoid)
Input gate	tanh = potential new long-term memory; sigmoid = % of new info to add	tanh + σ
Output gate	Updates short-term memory: tanh of cell state, sigmoid picks % to keep → this is the output (hidden state)	tanh + σ

Helpers: sigmoid (σ) maps any input to [0,1]; tanh maps to [-1,1] (a shifted/stretched sigmoid, keeps the cell state from blowing up).

Limit of LSTMs

With very long sequences, earlier info may still vanish/explode; data is processed sequentially so it can't be efficiently parallelized on GPUs → motivates Seq2Seq & Transformers (next lecture).

ThinkWhy can an LSTM carry information across a long sequence without the gradient vanishing, when a plain RNN can't?

Answer: In a plain RNN the single feedback weight w is applied at every step, so the signal scales like w^seqLength → vanishes (w<1) or explodes (w>1). The LSTM's cell state has no weights directly applied to it as it flows along the long-term path; it is only additively updated/cleared by the gates. So information (and gradient) can travel the sequence largely intact.

The Four Tasks

① Classification

Classify temporal segments (e.g., Human Activity Recognition, Emotion Recognition). Handcrafted features + classic ML, or automatic features (CNN/LSTM) with a softmax output + cross-entropy loss.

② Clustering

Find patterns in unlabeled data (e.g., discover human behaviors). The challenge is defining "similarity". Either cluster raw series with ad-hoc distances, or extract fixed-size features then use standard similarity.

LCSS

Longest Common Subsequence of similar points (threshold ε) that are temporally close (threshold σ)
Compares sequences of different lengths n, m
Not robust to noise

DTW (Dynamic Time Warping)

More robust to noise (but more complex)
Finds the pairing π minimizing distance between points
Conditions: match endpoints, every index matched, mapping monotonically increasing

Deep clustering: learn embeddings from raw segments (similar series close in space), then cluster the embeddings (PCA/t-SNE help).

⭐ ③ Anomaly Detection (Simulation Q13)

Anomalies are events of interest (not data cleaning). Three settings:

Approach	Training data	Idea
Supervised	normal + anomalous	Binary classification — but very unbalanced (anomalies rare & rarely labeled)
Semi-supervised	only normal	Model the expected behavior; flag samples that differ significantly
Unsupervised	unlabeled	Flag unrepresentative samples (assume normal = majority)

Method families: clustering-based (sample outside the normal subspace), density-based (low probability under the pdf), and reconstruction-based.

⭐ Autoencoder for anomaly detection — Simulation Q13

Train an autoencoder to reconstruct "normal" data. The model will fail to reconstruct abnormal sequences → a high reconstruction error on unseen data indicates an anomaly (captured with a threshold, e.g. 2σ or 3σ).

④ Forecasting

Use historical + current data to predict future values (e.g., load forecast in smart energy).

ARIMA — statistical: AR (previous p values) + MA (previous q errors). Good if the series is stationary (stats don't change over time); may need differencing to remove seasonality.
Deep forecasting — sensor data is usually not stationary → DL (recurrent nets) most effective; minimize prediction error (regression losses: MAE, MSE, RMSE…).

Linear forecasting

Input length n → output length m, both fixed

Autoregressive forecasting

Predict the following values with no fixed bound; feeds predictions back in (recurrent)

ThinkYou only have lots of "normal" ECG recordings and almost no labeled anomalies. Which anomaly-detection setting and method fit, and how is an anomaly flagged?

Answer: A semi-supervised setting (train only on normal data) using a reconstruction-based autoencoder. Train it to reconstruct normal ECG; at test time, a high reconstruction error (above a threshold like 2σ/3σ) signals an anomaly, because the model never learned to reproduce abnormal patterns.

✓

Final Quiz — Exam Style

9 MCQs + 3 open-ended. Several are modeled directly on the simulation exam (Q13, Q20, Q22).

⭐ Simulation exam Q13. How to use an autoencoder for anomaly detection in time-series?

AThe autoencoder is trained to reconstruct "anomalous" data. Low reconstruction errors on unseen data may indicate anomalous data points.

BThe autoencoder is trained to reconstruct "normal" data. High reconstruction errors on unseen data may indicate anomalous data points.

CThe autoencoder is trained to reconstruct both "normal" and "anomalous" data. High reconstruction errors on unseen data may indicate anomalous data points.

DThe autoencoder is trained to reconstruct "normal" data. Low reconstruction errors on unseen data may indicate anomalous data points.

B — slide: "learning a model for the accurate and full reconstruction of 'normal' time windows… the learned model will fail when reconstructing abnormal sequences" → anomalies = threshold on high reconstruction error. The four options are the 2×2 grid (trained-on × error-direction): decide each axis separately.

⭐ Simulation exam Q20. Which of the following sentences on the cell state of LSTM is FALSE?

AIt can be updated and/or cleared at each time step

BIts values do not depend on the short-term memory

CIts role is to capture long-term dependencies

DIt is not directly associated with weights, so it can mitigate exploding/vanishing gradients

B is false — the cell state IS influenced by the short-term memory (hidden state) through the gates. Verbatim from the simulation exam.

What is the difference between outlier detection and anomaly detection?

AThey are exactly the same task with different names

BOutlier detection finds events of interest; anomaly detection only cleans data

COutlier detection targets unwanted data for cleaning; anomaly detection targets anomalous events of interest to analyze

DOutlier detection only works on univariate data; anomaly detection only on multivariate

C — outliers → data cleaning (unwanted); anomalies → events of interest to analyze. The distinction is subtle and application-dependent.

A subsequence outlier is best described as:

AA single extreme value compared to the whole series

BA single value unusual only compared to its neighbors

CA missing value that must be imputed

DConsecutive samples whose joint behavior is unusual, though each alone may be normal

D — A and B describe global/local point outliers. A subsequence outlier is about the joint behavior of consecutive points.

In smoothing, what does making the window size too large cause?

ALittle to no effect on noise removal

BThe signal becomes over-smoothed, risking loss of useful information

CThe sampling rate automatically increases

DThe time series becomes stationary

B — large window → over-smoothing → information loss. Too small → little noise removal. The window size is the utility↔outlier-removal trade-off.

Why does the vanishing/exploding gradient problem occur in plain RNNs?

ABecause RNNs use the ReLU activation which always saturates

BBecause each time step uses a different randomly-chosen weight

CThe feedback weight is applied repeatedly, so the signal scales like w^(sequence length) — exploding if w>1, vanishing if w<1

DBecause RNNs cannot process sequences longer than 10 steps

C — repeated application of the same feedback weight makes the end-of-sequence output behave like input × w^seqLength.

What is the main advantage of DTW over LCSS for comparing time series?

ADTW requires the two series to have exactly the same length

BDTW is more robust to noisy data (at higher computational cost)

CDTW ignores temporal ordering entirely

DDTW does not require any distance metric

B — LCSS is not very robust to noise; DTW is more robust but more computationally complex.

Why is 1D convolution (rather than 2D) typically used for raw time series?

ATime series are sequential, so the kernel only needs to move horizontally along time

B1D convolution removes the need for any activation function

C2D convolution cannot be implemented on sequences at all

D1D convolution guarantees the model captures all temporal dependencies

A — in 1D CNNs the kernel slides only along the time axis. (2D CNNs are used when a series is turned into an image like a scalogram.)

When is the classic ARIMA model a good choice for forecasting?

AWhen the series is highly non-stationary and noisy

BOnly when using a softmax output layer

CWhen the time series is stationary (possibly after differencing)

DOnly for multivariate series with more than 3 dimensions

C — ARIMA suits stationary series; differencing can remove seasonality. Non-stationary sensor data favors deep (recurrent) forecasting.

Open-ended · Simulation Q22 (Time Series Analysis)

a) What is the difference between outlier and anomaly detection? Describe point and subsequence outliers.

b) Describe the idea behind smoothing to mitigate outliers. Which parameter determines the trade-off between utility and outlier removal, and why?

Model answer

a) Outlier detection aims to find unwanted data points so they can be removed — its goal is data cleaning. Anomaly detection aims to find anomalous events of interest that should be analyzed (the difference is subtle and depends on the application). A point outlier is a single sample that behaves unusually — vs the whole series (global) or vs its neighbors (local). A subsequence outlier is a set of consecutive samples whose joint behavior is unusual, even though each individual sample may not be a point outlier; both types apply to univariate and multivariate data.

b) Smoothing applies a filter (e.g., moving average, median filter) to reduce noise by replacing each point with an aggregate of its neighbors, at the risk of altering the original data. The window size controls the trade-off: a too-large window over-smooths the signal and risks losing useful information (utility), while a too-small window barely removes noise. (For EMA, the analogous parameter is α, weighting the current sample vs the past.)

Open-ended · RNNs & LSTM

a) Explain the vanishing/exploding gradient problem in RNNs and why it happens.

b) How does the LSTM cell state mitigate this problem? Mention the three gates.

Model answer

a) An RNN applies the same feedback weight w at every step of the unrolled sequence, so the end-of-sequence output (and the gradient during backprop) scales like input × w^(sequence length). If w > 1 this grows exponentially (exploding gradient → oscillating optimization that never converges); if w < 1 it shrinks toward 0 (vanishing gradient → extremely slow learning, earlier information lost).

b) The LSTM adds a separate cell state (long-term memory) with no weights directly applied as it flows along the sequence — it is only additively updated or cleared — so information and gradient travel largely intact. Three gates control it: the forget gate (sigmoid) decides the % of long-term memory to keep; the input gate (tanh for candidate value + sigmoid for %) adds new information; the output gate (tanh + sigmoid) produces the hidden state / output. The cell state captures long-term dependencies but can still be updated/cleared each step.

Open-ended · Anomaly Detection settings

a) Describe the three anomaly-detection settings (supervised, semi-supervised, unsupervised) and the data each uses.

b) Explain reconstruction-based anomaly detection with an autoencoder, including how the threshold is typically chosen.

Model answer

a) Supervised: trained on both normal and anomalous samples (binary classification) — but the task is very unbalanced since anomalies are rare and rarely labeled. Semi-supervised: trained only on normal data; it models expected behavior and flags samples that differ significantly. Unsupervised: trained on unlabeled data, assuming normal samples are the majority and anomalies are few/unrepresentative.

b) An autoencoder is trained to reconstruct normal time windows (encoder → latent embedding → decoder). The assumption is it will fail to reconstruct abnormal sequences, so a high reconstruction error (MSE between input and output) indicates an anomaly. The threshold is often set statistically on the reconstruction error of normal data — commonly 2σ or 3σ from the mean.

0/9