From a cloud of dots to one best line

Start — the everyday picture. Mark a dot for every student in your class: how many hours they studied (across the page) versus the score they earned (up the page). The dots never line up perfectly, yet you can still see a trend — more studying tends to go with a higher score. Linear regression is simply the rule for drawing the single straight line that best threads through that messy cloud, so you can say "each extra hour is worth about this much on the test."

Build — naming the pieces. Any straight line is fixed by two numbers. The slope $\hat\beta_1$ tells you how much $y$ rises for each one-unit step in $x$; the intercept $\hat\beta_0$ is where the line sits when $x=0$. Together they give the fitted line $\hat y = \hat\beta_0 + \hat\beta_1 x$. If one extra study-hour lifts the predicted score by 5 points, then $\hat\beta_1 = 5$. But which line is "best"? For every dot we measure the residual — the vertical gap between the real point and the line. We square each gap (so positives and negatives can't cancel) and add them all up. The best line is the one that makes this total, the $\text{RSS}$, as small as possible. That is exactly why the method is called least squares.

Deepen — the precise statement. Formally we minimize $\text{RSS}=\sum_i (y_i-\hat\beta_0-\hat\beta_1 x_i)^2$. Setting the two partial derivatives to zero gives a clean closed form: $\hat\beta_1 = S_{xy}/S_{xx}$ and $\hat\beta_0=\bar y-\hat\beta_1\bar x$, so the line always passes through the mean point $(\bar x,\bar y)$. The quality of the fit is summarized by $R^2\in[0,1]$, the fraction of $y$'s variation the line explains; for a single predictor it equals the squared correlation, $R^2=r_{xy}^2$. In this sim the True slope $\beta_1$ slider sets the hidden trend, Noise $\sigma$ controls how far the dots scatter from it, and n sets how many dots you draw.

Try this in the sim above. Slide Noise $\sigma$ down toward $0.05$ and watch $R^2$ climb to nearly $1$ as the dots snap onto the line; then push $\sigma$ up to $5$ and see $R^2$ collapse even though the true slope hasn't changed. Open the Slope Sweep tab and drag the manual slope — its RSS readout never drops below the OLS value, making "least squares" visible. Finally load the With outlier preset and watch a single stray point tug the whole fitted line toward it.

Simple Linear Regression — OLS — Gauss & Legendre ~1800

$$Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i,\quad \varepsilon_i \overset{iid}{\sim} \mathcal{N}(0,\sigma^2)$$

$$\boxed{\hat{\beta}_1 = \frac{S_{xy}}{S_{xx}} = \frac{\sum(x_i-\bar{x})(y_i-\bar{y})}{\sum(x_i-\bar{x})^2},\qquad \hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}}$$

Data: (1,2.1),(2,3.9),(3,6.2),(4,7.8),(5,10.1). $\bar{x}=3,\bar{y}=6.02$

$S_{xx}=10,\; S_{xy}=19.9$

$$\hat{\beta}_1=19.9/10=1.99,\quad\hat{\beta}_0=6.02-1.99\times3=0.05$$

$R^2=S_{xy}^2/(S_{xx}S_{yy})=19.9^2/(10\times 39.708)=0.997$ — almost perfect linear relationship.