AmI · Lecture 4 — Seq2Seq & Transformers

Seq2Seq Models

Lecture 3 gave us CNNs and RNN/LSTM for time series. Those work well — but modern NLP introduced more sophisticated sequence-processing methods. Since time series are sequences too, we can borrow them.

Goal of Seq2Seq

Map a sequence into another sequence, where the input length may differ from the output length. Originally proposed for language translation; can be applied to time series (especially forecasting).

How? An encoder / decoder architecture

Input sequence→ Encoder (LSTMs)→ Context Vector→ Decoder (LSTMs)→ Output sequence

LSTMs are adopted to handle inputs/outputs of variable dimensions.

The Embedding Layer

In NLP, an embedding maps a discrete input (words) into a sequence of tokens — each token a real-valued vector. There are also special tokens (e.g. <EOS> to mark when a sequence ends).

Time series may also use an embedding layer, but no special tokens.

The Encoder & the Context Vector

The encoder uses several LSTM layers, processing one element at a time, to map the input into a latent representation.
This latent representation is the context vector — it summarizes the whole input sequence and is handed to the decoder.

The Decoder & Training

What the decoder does

Decodes the context vector into the output sequence, one element at a time, by "unfolding" the LSTM.
It begins from a starting token — <SOS> (Start Of Sequence) or <GO> — to predict the first output element.
Each next element is predicted from the previously predicted token + the long-term memory.
A softmax picks the most likely next token at each step.

When does generation stop?

Generation ends when the <EOS> (End Of Sequence) token is produced. During training, the process also stops when the ground-truth token is <EOS>, even if <EOS> wasn't generated.

Training: each generated token is matched against the ground truth

A specific loss compares each generated token with the ground truth and updates the weights with backpropagation. There are two ways to feed the decoder during training:

Free Running

Each decoder step sees the actual (predicted) output of the previous step
Realistic, but may converge slower — training is affected by error propagation

Teacher Forcing

Each decoder step uses the ground truth of the previous step
Faster convergence, but a higher risk of overfitting

ThinkDuring training your seq2seq model converges fast but generalizes poorly at inference — at test time it never sees ground-truth previous tokens. What training choice likely caused this, and what's the trade-off if you switch?

Answer: You almost certainly used teacher forcing — feeding the ground-truth previous token at every step gives fast convergence but overfits, and creates a mismatch with inference (where only the model's own predictions are available). Switching to free running makes training match inference and reduces that overfitting, but converges more slowly because early errors propagate through the sequence.

Seq2Seq for Time Series

What changes vs NLP

Time series are a stream of continuous values → don't necessarily need an embedding layer to obtain tokens.
But it's often good to use a linear projection layer to map raw sensor data into a latent space. (Tokenization is only needed for categorical time series — less common.)
The main application is usually forecasting, so the <EOS> token is not needed (e.g. "predict the next 100 values").
Instead of starting from <SOS>, the decoder starts from the last n samples of the input series.

In practice the recurrent unit can be a GRU or LSTM: encoder consumes the historical data → encoder state → decoder emits the predictions Y₁…Yₙ.

Forecasting with a sliding window

Split the series: an encoder sequence (history) feeds the model, which predicts the decoder sequence (future). A sliding window moves along the series to generate many training pairs.

A trained seq2seq model isn't only for forecasting

It can also do imputation (reconstruct missing portions), anomaly detection (when observed data differs too much from the forecast → likely anomaly — ties back to Lecture 3's reconstruction idea), feature representation (extract the encoder alone for embeddings), and classification (attach a classification head to the encoder and fine-tune).

ThinkIn Lecture 3 we used an autoencoder for anomaly detection. How does a forecasting seq2seq model detect anomalies, and how is the idea related?

Answer: The seq2seq model is trained to forecast normal behavior. At test time it compares the predicted values against the actual observed values — when the observed data differs too much from the forecast, it's flagged as an anomaly. It's the same reconstruction-error logic as the autoencoder (high error = anomaly), just using prediction error instead of reconstruction error.

⭐ The Attention Mechanism

The problem with plain Seq2Seq

Unfolding the encoder LSTMs produces a single context vector. Fine for short sequences, but a problem for long ones — LSTMs can still forget "old" elements, even with two memories (short- and long-term).

The idea of attention

Determine how much focus the decoder should place on different parts of the input when generating the current output. How? By adding many more paths between encoder and decoder — one for each input element — so each decoder step can access the output of every encoder step (similar to "skip connections").

Computing similarity

For each decoder step, compute the similarity of its output with the output of each encoder step (using the short-term memory). Usually with cosine similarity:

cos(θ) = (A · B) / (‖A‖ ‖B‖)

A note on similarity: dot product

The denominator just keeps cosine similarity in [-1, 1]. In practice it's not needed for attention — the numerator alone (the dot product) is sufficient and faster. So you'll usually see the dot product, not full cosine similarity, in attention.

Attention scores (weights)

For each decoder step you get many similarity values — one per encoder step.
These are normalized (usually with a softmax) into attention scores / attention weights that sum to 1.
Each score = the "importance" of an input element to producing the current output token.
The scores are combined (sometimes via a learnable MLP) and used to output the current decoder step.

ThinkWhy does attention specifically help with the long-sequence weakness of plain seq2seq?

Answer: Plain seq2seq squeezes the entire input into a single fixed context vector, so information from early/old elements of a long sequence gets diluted or forgotten. Attention adds a direct path from every encoder step to every decoder step, so the decoder can re-access any input element's representation directly — no longer bottlenecked through one vector.

Transformers

Limits of Seq2Seq + Attention

Even with attention, using LSTMs still causes long-range dependency problems, and slow training because sequences are processed one element at a time (deep models become inefficient). Transformers "get rid" of LSTMs.

The Transformer ("Attention Is All You Need")

An improved, advanced version of seq2seq — the most common model in NLP today (e.g. Large Language Models), and very powerful for time series too.
Still an encoder / decoder architecture, but the encoder processes the input in parallel (no recurrence).
The decoder takes the previous output tokens + the context vector from the encoder, and outputs one token at a time.

The blocks of each encoder / decoder

Encoder block

Self-Attention layer
→ Feed-Forward (fully-connected) layer
Applies self-attention to the whole input sequence

Decoder block

(Masked) Self-Attention
Encoder-Decoder Attention — the same attention we saw for seq2seq!
→ Feed-Forward layer

Stacked encoders & decoders

The encoder is actually a stack of encoders, and the decoder a stack of decoders — similar to how a CNN stacks many convolutional layers to learn a more representative latent representation.

Positional Encoding

Order matters in a sequence — but there's no recurrence to track it. Positional encoding injects the position of each element into its embedding, computed with periodic functions (sine & cosine) over the token's position and dimension. It is summed with the input embedding. Needed for time series too (combined with raw multivariate data or a linear projection's output).

⭐ Self-Attention & Q/K/V

Self-attention

An effective and efficient way to track long-range dependencies in a sequence. For each element we compute attention scores capturing its most important semantic relationships with every other element — and this can be done in parallel for all elements (the key efficiency win over LSTMs).

Query, Key, and Value

Each element of the sequence is transformed into three vectors, obtained by multiplying the input by three weight matrices learned during training:

Vector	Role
Query (Q)	Encodes the token when it is asking for information from other tokens
Key (K)	Encodes the token when it is being considered by another token (its potential to "respond" to a query)
Value (V)	Encodes the actual information used to update the output — weighted by the attention score

Q and K compute the attention scores; V is used for the weighted combination.

The output of self-attention

The attention score = similarity between Q and K, computed via the dot product (the cosine similarity without the denominator, as we saw). The scores are scaled, passed through softmax (weights summing to 1), then multiplied with V to produce the resulting embedding Z:

Z = softmax( (Q · Kᵀ) / √dₖ ) · V

Encoder vs Decoder self-attention

Encoder

Self-attention over the whole input sequence

Decoder — masked

Can not look into the future — only previously generated tokens
Future positions are masked → masked self-attention

Multi-head self-attention

There are actually multiple self-attention layers in parallel. They produce multiple representation subspaces of the same input, improving the model.

Beyond generating sequences

Decoder-only transformers — for autoregressive forecasting, you can use only the decoder. This is the approach of GPT models, and works for time series too. The decoder first sees the input, then generates one token at a time.
Classification head — attach a head at the end of the decoder so it outputs the most likely class instead of generating tokens. Many time-series classification approaches follow this.

The big picture (Seq2Seq → Attention → Transformers)

Seq2Seq gave variable-length sequence mapping with a context vector. Attention fixed the single-context-vector bottleneck. Transformers dropped recurrence entirely, using self-attention + positional encoding to process sequences in parallel — solving long-range dependencies and slow training at once.

ThinkTransformers have no recurrence, yet "order matters" in a sequence. How do they know which element came first, and why can the encoder run in parallel?

Answer: Order is injected by positional encoding — sine/cosine functions of each element's position are summed into its embedding, so position information lives in the vectors themselves rather than in a recurrent state. Because each element already carries its position and self-attention compares every element with every other directly (no step-by-step dependency), all positions can be processed in parallel — the source of the training-speed gain over LSTMs.

✓

Final Quiz — Exam Style

9 MCQs + 2 open-ended. Tuned to the simulation's style, with Q1 modeled on the simulation's Seq2Seq question (Q15).

What is the main goal of a Seq2Seq model?

ATo classify a single fixed-length input into one of several categories

BTo compress a sequence into a single scalar value

CTo map a sequence into another sequence, where input and output lengths may differ

DTo remove noise from a time series using a moving-average filter

C — Seq2Seq maps a sequence to another sequence (input length may differ from output). Built with an encoder/decoder of LSTMs; first proposed for translation, also used for forecasting.

In a Seq2Seq model, what is the context vector?

AThe set of attention weights produced by the decoder

BThe latent representation produced by the encoder that summarizes the whole input sequence

CThe positional encoding added to each input token

DThe ground-truth sequence used during teacher forcing

B — the encoder maps the input into a latent representation (the context vector), which is passed to the decoder.

What is the difference between teacher forcing and free running during training?

ATeacher forcing feeds the ground-truth previous token; free running feeds the model's own previous prediction

BTeacher forcing is used only at inference; free running only during training

CTeacher forcing disables backpropagation; free running enables it

DThey are two names for the same technique

A — teacher forcing uses the ground truth of the previous step (faster convergence, more overfitting); free running uses the actual predicted output (realistic, slower convergence).

When applying Seq2Seq to time series forecasting, which statement is correct?

AAn <EOS> token is always required to stop generation

BTime series must always be tokenized with an embedding layer like words

CThe decoder must start from a special <SOS> token

DThe <EOS> token is not needed, and the decoder starts from the last n samples of the input series

D — for forecasting (e.g. "predict the next 100 values") no <EOS> is needed; the decoder starts from the last n input samples. A linear projection layer is optional; tokenization is only for categorical series.

Why was the attention mechanism introduced for Seq2Seq models?

ATo remove the need for an encoder entirely

BA single context vector struggles with long sequences; attention lets each decoder step access every encoder step

CTo convert continuous time series into discrete tokens

DTo make teacher forcing unnecessary

B — unfolding LSTMs produces one context vector that forgets old elements in long sequences. Attention adds direct paths so each decoder step can focus on any encoder step.

In attention, why is the dot product often used instead of full cosine similarity?

AThe dot product guarantees the result is always exactly between 0 and 1

BCosine similarity cannot be computed on real-valued vectors

CThe numerator (dot product) alone is sufficient and faster; the denominator (normalization) isn't needed

DThe dot product automatically applies a softmax

C — the cosine denominator only keeps values in [-1,1]; in practice it's unnecessary for attention, and the dot product is sufficient and more efficient.

What are Query (Q), Key (K), and Value (V) in self-attention?

AThree fixed constants chosen by the data scientist before training

BThree vectors per element, obtained via three weight matrices learned during training; Q·K gives attention scores, V is the weighted information

CThe three gates of an LSTM cell

DThree separate loss functions combined during backpropagation

B — each element → Q, K, V via learned weight matrices. Q and K compute attention scores; V provides the information that gets weighted. Z = softmax(Q·Kᵀ/√dₖ)·V.

Why do Transformers need positional encoding?

AWithout recurrence there's no notion of order, so position info (via sine/cosine) is summed into the embeddings

BTo replace the softmax in the attention layer

CTo convert the cell state into a hidden state

DTo guarantee the input and output sequences have the same length

A — order matters but there's no recurrence to track it; positional encoding (periodic sine/cosine functions of position and dimension) is summed with the input embedding. Needed for time series too.

What is the key advantage of Transformer self-attention over LSTM-based encoders?

AIt removes the need for any training data

BIt processes one element at a time, making training slower but more accurate

CIt handles long-range dependencies and can be computed in parallel across the sequence

DIt requires no weights to be learned

C — LSTMs process sequentially (slow) and still struggle with long-range dependencies. Self-attention compares every element with every other directly and runs in parallel.

⭐ Simulation exam Q5. In the self-attention mechanism, the "Query", "Key", and "Value" vectors obtained from each element of the input sequence are used to compute the output of the self-attention layers. What is the main role of the "Value" vector?

AGenerating the latent representation of each sequence element, that will be weighted by the self-attention scores.

BComparing the similarity to the "Key" representations of the other elements.

CScaling the attention scores in the range [0,1]

DGiven an element of the sequence, determining the other ones most relevant through the "Query" vector.

A — the slide says it directly: "Value (V): encodes the token in a way that represents the actual information used to update the final output. It will be weighted by the attention scores." The distractors are the sibling roles from the same slide: B/D describe the Query–Key similarity matching, C describes the softmax. Z = softmax(Q·Kᵀ/√dₖ)·V — the V is the content being mixed.

⭐ Simulation exam Q15. When applying classic Seq2Seq models (without attention) for time series forecasting, which information is provided to the decoder?

Athe context vector and a special token to generate the first token

Bonly the context vector

Conly the last element of the input sequence

Dthe context vector and the last element(s) of the input sequence to generate the first token

D — this is the trap question. In NLP-style Seq2Seq the decoder starts from a special <SOS>/<GO> token (that's distractor A). But the stem says "for time series forecasting", and the slide is explicit: "Instead of starting from the <SOS> token, in time series the decoder starts from the last n samples of the input time series" — plus it always receives the encoder's context vector. When the stem adds a context qualifier, answer from the specialized slide, not the generic one.

Open-ended · Seq2Seq (Simulation-style)

a) Describe the encoder/decoder architecture of a Seq2Seq model and the role of the context vector.

b) How is a Seq2Seq model adapted for time series forecasting compared to NLP?

Model answer

a) A Seq2Seq model maps an input sequence to an output sequence of possibly different length using two parts. The encoder (several LSTM layers) processes the input one element at a time and compresses it into a context vector — a latent representation summarizing the whole input. The decoder (also LSTMs) takes the context vector and "unfolds" it, generating the output one element at a time: it starts from a start token (<SOS>/<GO>) and predicts each next element from the previously produced token plus the long-term memory, until it emits <EOS>. It can be trained with teacher forcing (ground-truth previous token, fast but overfits) or free running (own prediction, realistic but slower).

b) For time series the input is a stream of continuous values, so no embedding/tokenization is strictly needed (a linear projection into a latent space is common; tokenization only for categorical series). Since the main task is forecasting, the <EOS> token is not needed ("predict the next n values"), and instead of starting from <SOS> the decoder starts from the last n samples of the input. A sliding window generates training pairs (history → future). The same trained model can also do imputation, anomaly detection (forecast vs observed), feature extraction, and classification.

Open-ended · Attention & Transformers

a) Explain the attention mechanism: what problem it solves and how attention scores are computed.

b) What do Transformers add over Seq2Seq+attention? Mention self-attention, Q/K/V, and positional encoding.

Model answer

a) A plain Seq2Seq squeezes the whole input into a single context vector, which loses information from old elements of long sequences. Attention adds direct paths from every encoder step to every decoder step, so the decoder can decide how much to focus on each input element. For each decoder step we compute the similarity between its output and each encoder step's output — using the dot product (cosine similarity without the normalizing denominator, which is faster). The similarity values are normalized with a softmax into attention scores/weights (summing to 1), representing each input's importance, then combined to produce the current output.

b) Transformers drop the LSTMs, removing long-range-dependency issues and the slow one-element-at-a-time processing. They keep an encoder/decoder structure but use self-attention: each element is turned into a Query, Key, Value (via learned weight matrices); the attention score is Q·K (dot product), scaled and softmaxed, then multiplied by V → output Z. Self-attention captures relationships between all elements and runs in parallel; multi-head attention gives several representation subspaces; the decoder uses masked self-attention (can't see the future) plus encoder-decoder attention. Since there's no recurrence, positional encoding (sine/cosine of position) is summed into the embeddings to preserve order. Variants: decoder-only (GPT-style, autoregressive forecasting) and a classification head for time-series classification.

0/9