Think about the weather. Today's temperature is a pretty good guess for tomorrow's — warm days cluster with warm days, cold with cold. The past "echoes" into the present. A time series is just a list of measurements taken in order over time (daily temperatures, monthly sales, a stock's closing price), and the whole game is one question: how much does the past help you predict the next value?
Call today's value $X_t$ and yesterday's $X_{t-1}$. The simplest useful model says today is a fraction of yesterday plus a fresh random nudge $\varepsilon_t$ (the part nobody could have predicted):
$$X_t = \phi\,X_{t-1} + \varepsilon_t$$
This is an AR(1) model ("auto-regressive" — it regresses the series on its own past). The single number $\phi$ ("phi") is the strength of the echo. If $\phi=0.7$, then $70\%$ of yesterday carries into today: a value of $10$ today tugs tomorrow toward $7$ before the random nudge arrives. A second flavour, the MA(1) model $X_t=\varepsilon_t+\theta\varepsilon_{t-1}$, says today is shaped instead by today's and yesterday's random shocks.
Glue both ideas together and you get ARMA(1,1): $X_t=\phi X_{t-1}+\varepsilon_t+\theta\varepsilon_{t-1}$, with $\varepsilon_t$ independent noise. For the echo to fade rather than explode, AR models need $|\phi|<1$ — this is stationarity, meaning the series' mean and spread stay steady over time. Right at $\phi=1$ the model becomes a random walk: each step just adds a shock with no pull back toward the centre, so its variance grows without bound ($\sigma_X^2 \propto t$) and it wanders off — non-stationary. The fingerprints of these models live in the ACF (autocorrelation: how today correlates with $k$ steps ago) and the PACF. An AR($p$) shows a PACF that cuts off sharply after lag $p$; an MA($q$) shows an ACF that cuts off after lag $q$. The sliders map straight onto the math: phi is $\phi$, theta is $\theta$, phi2 is the second AR coefficient, and T sets how many time steps to simulate.
Set the model to AR(1) and slide $\phi$ from $0.2$ up toward $0.95$ — watch the line settle into longer, smoother runs as the echo strengthens, and the ACF bars decay more slowly. Switch to "Random Walk" and notice the series drift away and never return to zero. Finally pick MA(1) and open the ACF tab: only the first bar pokes past the dashed confidence band, then everything snaps to near zero — the tell-tale signature of an MA(1).
ARMA Models — Box & Jenkins, 1970
$\text{AR}(1):\; X_t=\phi X_{t-1}+\varepsilon_t,\quad |\phi|<1\text{ for stationarity}$
$\text{MA}(1):\; X_t=\varepsilon_t+\theta\varepsilon_{t-1}$
$$\boxed{\text{ARMA}(1,1):\; X_t=\phi X_{t-1}+\varepsilon_t+\theta\varepsilon_{t-1},\quad\varepsilon_t\overset{iid}{\sim}N(0,\sigma^2)}$$
| Model | ACF pattern | PACF pattern | Stationarity |
|---|---|---|---|
| AR(p) | Tails off (expo decay) | Cuts off after lag p | roots|>1 |
| MA(q) | Cuts off after lag q | Tails off | Always stationary |
| ARMA(p,q) | Tails off | Tails off | AR part: |roots|>1 |
| Random Walk | Tails off slowly | Spike at lag 1 | NOT stationary |
Mean: $E[X_t]=0$. Variance: $\sigma_X^2=\sigma^2/(1-\phi^2)$. ACF: $\rho(k)=\phi^k$ — exponential decay for $|\phi|<1$. For $\phi=1$ (random walk): variance grows as $t$, not stationary.
1. Check stationarity (plot, ADF test). 2. Difference until stationary (→ ARIMA). 3. Identify p,q from ACF/PACF. 4. Estimate parameters (MLE). 5. Diagnose residuals (should be white noise). 6. Forecast.
"A time series with an upward trend is stationary."
A trend means the mean is not constant over time — this violates stationarity. A stationary series has constant mean, constant variance, and ACF that depends only on lag, not time. Remove trend by differencing (first difference d=1) or detrending (subtract fitted trend). Random walks (phi=1) are NOT stationary — their variance grows with time.
📖 Box, Jenkins & Reinsel — Ch. 2
Using OLS regression for time series without checking autocorrelation
✅ OLS assumes independent errors. For time series with autocorrelated residuals, OLS estimates are unbiased but INEFFICIENT and standard errors are WRONG (usually too small → false significance). Check Durbin-Watson test and ACF of residuals. Use ARMA errors model or HAC (Newey-West) standard errors if autocorrelation is present.
🔍 Regressing GDP on time and concluding statistical significance without checking serial correlation.
Overdifferencing a time series
✅ Difference only until stationary (d=0 or 1, rarely 2). Over-differencing introduces artificial MA structure and makes forecasting harder. Apply ADF (Augmented Dickey-Fuller) or KPSS test to determine d. Rule: difference the minimum number of times needed to achieve stationarity.
🔍 Differencing three times when once was sufficient, creating complex MA error structure.