Home / ML-finance / Linear regression

What does it mean
to train a model?

Linear regression from scratch.

By How to Train Your Model · Python + interactive experiments
What you need: a straight-line equation, basic Python, and a little curiosity.

When we say we are “training” a model, what does that mean? I think people know the basic idea that you feed a model data and it gives you a prediction. True. But what does it do? How does each model “learn” or “train” so that it gives a prediction? I mean, it is just code on a machine. How is it “learning”?

To understand all that, we will take a simple linear regression example and go through the entire process of preparing the data → feeding the data → outputting the prediction. And we will move the line ourselves before asking the code to do it.

The answer we are working towards is simple: training uses data to choose the model’s parameters according to an objective. Here, those parameters are just two numbers: the slope and the intercept. Let’s see what that actually means.

What is linear regression?

Linear regression is a supervised learning method used to model the relationship between an output and one or more inputs. It predicts continuous values. Let’s break it down instead of making it sound more complicated and intelligent.

So basically, supervised learning means we have examples where the input and the actual output are both known. These are called labelled observations. The input is also called a feature or independent variable, and the output we want to estimate is the target or dependent variable. “Independent” here does not mean that our inputs must be statistically independent.

For now, we have one input. We choose to represent its relationship with the output using a straight line:

ŷ_i = b + wx_iprediction = intercept + slope × input

If you have studied high-school level maths, you know that is just the equation of a straight line: y = mx + b. We are using w for the slope, and putting a hat on y to distinguish a prediction from the actual value.

x_i: the input for observation i.
y_i: its actual, observed output.
ŷ_i: the output predicted by our line.
w: the slope, or how much the prediction changes when the input increases by one unit.
b: the intercept, or the prediction when the input is zero.

This equation is also called the hypothesis function, often written as h(x). It is the rule used to make a prediction. We choose the straight-line structure; training chooses w and b. An arbitrary line is not already the best-fit line.

Keep these two parameters in mind. Once we have a way to measure error, you can try fitting the line yourself in the experiment below.

So what is the “magic” here?

It comes from minimising the error. For our example, we define the error as actual − predicted. On the data we are fitting, this difference is called a residual.

e_i = y_i − ŷ_i

But we have more than one observation. How do we turn all these errors into one number that tells us how well the line fits?

If the model has overestimated, the residual is negative. If it has underestimated, the residual is positive. So when we add both, the total can be lower, which would make us think there is less error. But that is not necessarily true.

Say one residual is +2 and another is −2. Their sum is zero. Both predictions are still wrong by two units. So we square the residuals first, and then we add them: 2² + (−2)² = 8.

This gives us the sum of squared errors (SSE). Fitting the line so that this number is minimum is called ordinary least squares (OLS). Divide that sum by the number of observations, and we get mean squared error (MSE).

MSE = sum of squared residuals / number of observationsSquare each residual, add the squares, then divide by n.

This is our cost function, also called the loss or objective here. It does exactly what it sounds like: it assigns a cost to the model’s predictions, and fitting tries to minimise it. SSE and MSE choose the same best-fit line for a fixed dataset, because dividing by n does not change which line has the lowest score.

You can choose a different cost function. Absolute values also stop positive and negative errors cancelling. Squaring does something extra: a residual of 4 contributes 16, while a residual of 2 contributes 4. Twice the error gets four times the penalty. This makes squared loss sensitive to large errors.

Before moving on: can a line have zero total residual and still be wrong?

Yes. Positive and negative residuals can cancel. Try +2 and −2: the total is zero, but both predictions are wrong. That is why the sum of signed residuals is not enough to judge a fit.

Now again, I just added one more formula for the model to solve. We still have not talked about how it solves it.

Gradient descent

This is an optimisation technique we can use to train the model. It is a method for minimising the cost function so that the model finds a best-fit line. It is not the only method: this regression problem also has a direct least-squares solution. But gradient descent lets us watch the learning happen, one update at a time.

So it starts from an initial point. That means initial values for the parameters. They can be random, but they do not have to be. Here we start with w = 0 and b = 0.

Then it calculates how the cost changes with each parameter. These derivatives together are the gradient. This is not the slope of the prediction line. It is the slope of the cost with respect to the parameters.

The gradient points towards the steepest local increase in cost. So we move in the opposite direction. The learning rate controls the size of the step.

new intercept = intercept − learning rate × intercept gradient
new slope = slope − learning rate × slope gradientCalculate both derivatives using the current parameters, then update both.

It keeps this process going until a stopping condition is met. For example, the gradient becomes very small, or we reach a maximum number of updates. It does not “know” the exact minimum. With a suitable learning rate, it approaches it.

Think of the cost as a bowl. A single-parameter slice can look like a parabola. With both slope and intercept free to change, we have a surface. The contour map below shows that surface from above: each ellipse joins parameter choices with the same MSE.

Try this before pressing Train. Move the slope and intercept until the line looks reasonable. Then let the model fit it. Did the lower-error line end up where you expected?

Watch the parameters learn

Here are six fixed example observations. Change the values, add a point, or try moving the line yourself. Then press Train and watch the line, the gradient descent path, and the loss curve change together.

This small teaching dataset is synthetic and always starts with the same values. The Python example below uses a real, built-in dataset.

Point	Input x	Actual y	Prediction	Residual

Slope w Intercept b

Learning rate

● Observations / current parameters · green line: current fit · + least-squares solution

Try a new input

The table uses the current line, so its predictions update during training. Learning rates are scaled to the inputs; the actual rate appears above. Edits reset training. The loss curve shows training error, not performance on unseen data.

Where do those derivatives come from?

Here J means the MSE cost, ∂ means a partial derivative, and ∑ means “add over all observations.” For one squared residual, applying the chain rule gives a factor of −2 times the residual. The slope derivative also gets a factor of x. Average over all observations:

∂J/∂b = −(2/n) ∑_i=1ⁿ (y_i − b − wx_i)
∂J/∂w = −(2/n) ∑_i=1ⁿ x_i(y_i − b − wx_i)

There is no trying both directions first. The derivatives determine the update direction; the learning rate scales the update. A sufficiently small positive step reduces this quadratic objective away from its minimum.

Can we just calculate the answer?

Yes. For simple linear regression with an intercept, ordinary least squares has a direct solution:

w* = ∑(x_i − x̄)(y_i − ȳ) / ∑(x_i − x̄)²
b* = ȳ − w*x̄x̄ and ȳ are the input and target means. The star marks an optimal parameter.

The slope depends on how the input and output vary together, relative to how much the input varies. The intercept makes the fitted line pass through their averages, (x̄, ȳ). If all input values are identical, the denominator is zero: the data cannot uniquely determine both a slope and an intercept.

So “Fit least squares” in the experiment calculates the answer directly. Gradient descent approaches the same solution on this dataset. Both count as training: both use the data to choose parameters that minimise the chosen objective. Training does not always mean repeated updates.

Now let’s put that into code

We will use NumPy for the arithmetic. No model-fitting function for our gradient descent implementation. This time, let’s use a real dataset that is already available in scikit-learn, instead of making up observations.

The diabetes dataset has 442 observations. We use its BMI feature as the input and its disease progression score as the target. This is a regression teaching example, not a financial dataset. We standardise the input so the learning rate is easier to tune.

Run the code. Watch the model learn.

Change the learning rate or number of updates, then run. Each chart update comes from the Python loop below; it is not a separate replay. Python and its bundled dataset load on the first run, so internet is needed.

Editable Python

import asyncio
import numpy as np
from sklearn.datasets import load_diabetes

# Real observations bundled with scikit-learn, not random data.
data = load_diabetes()
x = data.data[:, 2]  # BMI feature
y = data.target     # Disease progression score

# Express the input in standard-deviation units for training.
x = (x - x.mean()) / x.std()
b, w = 0.0, 0.0
learning_rate = 0.08
updates = 120

show_training(x, y, b, w, 0)
for step in range(1, updates + 1):
    predicted = b + w * x
    residual = y - predicted
    db = -2 * np.mean(residual)
    dw = -2 * np.mean(x * residual)
    b -= learning_rate * db
    w -= learning_rate * dw

# Send these actual Python parameters to the charts.
    show_training(x, y, b, w, step)
    await asyncio.sleep(0.08)
    if np.hypot(db, dw) < 1e-5:
        break

print(f"Learned intercept: {b:.4f}")
print(f"Learned slope: {w:.4f}")
print(f"Training MSE: {np.mean((y - b - w*x)**2):.4f}")

# Cross-check the fitted parameters with a direct solution.
xc = x - x.mean()
w_ols = np.sum(xc * (y - y.mean())) / np.sum(xc**2)
b_ols = y.mean() - w_ols * x.mean()
print(f"Least squares: b={b_ols:.4f}, w={w_ols:.4f}")
print("Agreement:", np.allclose([b, w], [b_ols, w_ols], atol=1e-4))

● Observations / current parameters · green line: current fit · + least-squares solution

Inspect the real data and current predictions

First 12 observations; all 442 are used for training. Predictions update from Python.

Observation	BMI (standardised)	Actual score	Prediction

Ready. Run Python to load the dataset and train.

show_training is this page’s display helper: it sends the current data and parameters to the charts. The gradient calculations above do the learning. asyncio.sleep spaces the updates so you can see them; it does not change the fitting method. With reduced motion enabled, the charts display the final state.

Look at what actually changes in the loop: b and w. The data does not change. The equation does not change. The cost function does not change. The parameter values change.

Once we have them, making a prediction is just putting an input into the equation. There is no extra learning step needed to evaluate that line. The final output also compares our parameters with the direct least-squares solution. Agreement checks the implementation; it does not prove the model is useful on new data.

What would this mean in finance?

A quick note before we apply it: linear regression is a very straightforward ML model. The simple version we have used works best when a straight line is a reasonable approximation of the relationship between the feature and the target. The relationship does not have to be perfectly linear, but if a straight line misses the important patterns, this model will struggle.

For finance, things are not usually that straightforward. Relationships can be noisy, change over time, and depend on more than one variable. So the example here is hypothetical: it explains how we would set up a regression, rather than showing a tested prediction or trading strategy. That does not mean linear regression is useless in finance. It can still help us study relationships, estimate exposures, and build a baseline to compare other models with.

However, we can use linear regression in real life. Here are some examples of questions it can help us investigate, provided the model is appropriate and we check its performance:

Real estate: estimating home prices from square footage, number of bedrooms, and location. Location needs to be represented as suitable input features.
Sales and marketing: forecasting sales revenue and studying how advertising spend relates to product demand. A relationship in the data does not, by itself, prove that the advertising caused the change.
Healthcare: analysing how dosage relates to a measured patient outcome, or modelling a continuous health measure using factors such as age and lifestyle. A yes-or-no outcome usually needs a model designed for classification.
Finance: estimating return relationships and exposure to risk factors. These estimates can inform portfolio decisions; fitting a regression alone does not optimise a portfolio or establish reliable stock-price forecasts.
Agriculture: modelling crop yields using inputs such as fertiliser usage and water supply, while checking whether a linear approximation makes sense.
Education: studying the relationship between study time and academic performance. Other factors may also explain the results, so we should not treat correlation as proof of cause and effect.

Let’s take a question: how does a stock’s daily return relate to the market’s daily return? The market return becomes our input, and the stock return becomes our target.

r̂_stock,t = b + w r_market,t

This is the same formula, just a different meaning for the inputs and outputs. If returns are measured in percentage points, a slope of 1.2 means that a one-percentage-point increase in the market return corresponds to a 1.2-percentage-point increase in the fitted stock return. The intercept is the fitted stock return when the market return is zero.

For actual data, we would document the source and date range, calculate returns consistently from appropriately adjusted prices, and pair observations on matching trading dates. A simple return is P_t/P_t−1 − 1. Both series must use the same units.

This is a same-day relationship, not an advance forecast. We do not know the full day’s market return before that day ends. A fitted intercept is not proof of trading alpha, and a low error is not proof of profitability. The interactive dataset in this article is synthetic; it is not an empirical stock-market result.

But there are assumptions here too

As with any model or formula in physics or maths, we should ask what we are assuming. Here we have chosen to approximate the relationship with a straight line. That does not mean the observations must sit perfectly on a line.

There is an important distinction: we can calculate a least-squares fit even when assumptions used for statistical inference do not hold. Statements about coefficient uncertainty need more care than simply drawing the fitted line.

Linearity: the conditional mean is represented by our chosen linear form. If we miss an important pattern, the residuals can reveal it.
Mean-zero errors conditional on the inputs: this supports interpreting OLS as estimating the intended mean relationship without systematic error related to the inputs. Zero average training residual alone does not establish this assumption.
Constant variance (homoscedasticity): the underlying errors have equal variance across input values. If their spread fans out, that is heteroscedasticity. It affects conventional uncertainty estimates; it does not stop the code fitting a line.
Error dependence: errors may be related across observations, particularly through time. Autocorrelation is one form of that dependence and matters for uncertainty and validation.
Normality of errors: normal errors support certain exact statistical tests and intervals. Normality is not needed just to compute the least-squares coefficients.

We can also have multiple linear regression, with more than one input:

ŷ = b + w₁x₁ + ⋯ + w_kx_k

This is the same idea, with more parameters. Perfectly redundant input columns prevent unique coefficient estimates; strong multicollinearity can make them unstable. And linear regression can include interaction terms such as x₁x₂ as features. “Linear” refers to how the coefficients enter the model, not a universal ban on interactions.

How well did it perform?

There are many evaluation metrics we can use. For this example, start with the MSE we already understand. RMSE is its square root, which puts the score back into the target’s units. MAE averages absolute errors instead of squared errors.

Another common metric is R², which compares squared error with the error from predicting the evaluation targets’ mean. It is not “percentage accuracy.” On training data, OLS with an intercept does at least as well as that mean-only baseline, up to numerical precision; on new data, R² can be negative.

Adjusted R², Ridge, Lasso, and Elastic Net are worth learning later. We do not need all of them to answer today’s question.

Quick check: what actually changes when we train?

The slope and intercept. We chose the equation and the cost function. The fitting method uses the observations to choose those two parameter values.

So what did the model learn?

Two numbers. That is it for this model: a slope and an intercept.

We gave it paired observations, chose a straight-line equation, and decided that squared error was our cost. The fitting method chose parameter values that reduced that cost. This is how this linear regression model “trains” or “learns”.

But the smallest error on these observations does not tell us how the model will perform on observations it has never seen. It does not establish causation either. We still have to test what happens outside the data used for training.

Pros

Simple to understand: we can see what each parameter does and follow every calculation. There is no need to treat training like a black box.
Fast to fit and use: ordinary least squares has a direct solution, and making a prediction only needs a little arithmetic.
A useful starting point: it gives us a baseline. Before choosing a more complicated model, we can ask whether that model actually performs better on unseen data.
Easy to inspect: coefficients and residuals help us examine the fitted relationship, as long as we keep the input units and model assumptions in mind.

Limitations

A straight line can miss the pattern: the simple model we used cannot capture bends or interactions by itself. Adding suitable features can extend linear regression, but choosing those features takes care.
Large errors can pull the fit around: squared loss gives outliers a lot of influence. One unusual observation can noticeably change the fitted line.
Missing information still matters: leaving out relevant inputs can distort how we interpret the coefficients. Strongly correlated inputs can also make individual coefficients unstable.
It does not prove causation: a fitted relationship does not tell us what would happen if we deliberately changed an input.
It can fail on new data: extrapolation and changing relationships can make predictions unreliable. A low training error is only the beginning of the evaluation.

And that is the next question: what happens when a model fits its training data well but performs poorly on new data? That takes us to overfitting.

Was this useful?

A quick signal helps me shape the next explanation.

What does it meanto train a model?

What is linear regression?

So what is the “magic” here?

Gradient descent

Can we just calculate the answer?

Now let’s put that into code

What would this mean in finance?

But there are assumptions here too

How well did it perform?

So what did the model learn?

Pros

Limitations

Was this useful?

What does it mean
to train a model?