Sample Data

First, let’s create a simple dataset. We’ll keep it in two dimensions to make it easier to visualize, but essentially everything I discuss in this article extends immediately to higher dimensions.

For this data, we’ll use the function as our data generating process. We’ll choose 100 values for , where each value will be a random number between zero and ten. will be the same as but with some noise added so that it will approximate a straight line (). Due to our choice of and , any models we fit to this data should tell us that the data is from the function .

Using NumPy to generate the data:

import numpy as np

1rng = np.random.default_rng(42)
2x = rng.random((100, 1)) * 10
3noise = rng.normal(loc=0, scale=0.5, size=x.shape)
4y = x + noise

If we plot these points and add the line , we can see that the data is approximately linear:

Ordinary Least Squares (OLS)

For this section, I assume a basic understanding of OLS regression.

As a reminder, a linear regression model with one independent variable () and one dependent variable () can be written as:

where is the intercept, is the coefficient for , and is the error term, or residual.

Two key features of OLS models are that they’re linear and that they minimize the sum of squared residuals. In other words, the model is trying to find the straight line that minimizes the sum of the squared differences between the observed values and the predicted values—the mean-squared error (MSE).

With OLS, we can easily solve this using a closed-form solution that is guaranteed to be the best possible solution:

where is the vector of fitted coefficients ( and in our example) and and are our data vectors.

Now, we’ll fit a standard OLS model to the data:

import statsmodels.api as sm

1x_with_const = sm.add_constant(x)
ols_model = sm.OLS(y, x_with_const).fit()

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.969
Model:                            OLS   Adj. R-squared:                  0.969
No. Observations:                 100   F-statistic:                     3053.
Covariance Type:            nonrobust   Prob (F-statistic):           1.14e-75
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const         -0.0257      0.101     -0.253      0.801      -0.227       0.175
x1             1.0038      0.018     55.257      0.000       0.968       1.040
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Note that the intercept coefficient is close to zero and the slope coefficient is close to one, which nearly matches our initial data generating process of .¹ The fitted line basically overlaps our line for , indicating the OLS model is fitting the data well:

If we used the same data to fit the model again, we’d get the same result because we’re using the closed-form solution, which is guaranteed to be the best possible solution.²

Stochastic Gradient Descent (SGD)

Rather than the typical OLS approach, we could instead fit the model numerically using an iterative method. This approach could look something like this:

1. Pick a random value for and .
2. Calculate the mean-squared error for the predicted values () and the true values ().
3. Pick another value for and and see if it’s better. If so, keep that one; otherwise, use the original one.
4. Repeat steps 1-3 N times or until the model converges.

This is a brute force approach. We could get clever and try to pick our values for more strategically depending on whether they seem to improve as we move in one direction or the other. There are a variety of algorithms for doing this—optimizing values numerically is an entire subfield of academic research.

One such approach is called Stochastic Gradient Descent (SGD). The gradient is the derivative of our loss function (MSE) with respect to the coefficient and intercept. It tells us the direction and magnitude of the greatest increase in the loss function. We can use the gradient to give us a more informed guess about how we should choose the new values to improve the model fit.³

Without getting into the precise details of SGD, we can use it to fit a linear model by numerically optimizing the coefficients and intercept:

from sklearn.linear_model import SGDRegressor

1sgd_model = SGDRegressor(loss="squared_error", penalty=None)
sgd_model.fit(x, y.ravel())
sgd_coefficent = sgd_model.coef_[0]
sgd_intercept = sgd_model.intercept_[0]

sgd_coefficient=0.9816
sgd_intercept=0.0730

We’re still using the MSE loss function, although we do have other options now. We can also regularize the coefficients, but we’ve turned that off here (penalty=None) to make it closer to OLS.⁴ In the end, we have a well-fitting linear model that serves the same purpose as OLS but gets us there in a different way.

One difference with OLS is that every time we fit the SGD linear model, we’ll get a different coefficient and intercept. The randomness in this case is due to the input data being randomized before fitting. The takeaway here is that where you begin the numerical optimization matters. It matters so much that you may end up in a local optimum, rather than the global one. Getting stuck like this isn’t possible with OLS.

Neural Network

So far we have used two techniques—OLS from classical statistics and SGD from machine learning—to fit a linear regression model. So how does this connect to deep learning and neural networks? A neural network is a series of linear models. Each linear model is called a layer and each layer is connected to the one before it and after it in the chain. These connections are called weights (what we’ve been calling coefficients) and biases (what we’ve been calling intercepts).

The first layer is called the input layer and the last layer is called the output layer. Any layers in between those two are called hidden layers. We must have an input layer and an output layer, but hidden layers are optional. If we have quite a few hidden layers, we call the neural network a “deep” neural network, thus the term deep learning.⁶

For our simple linear model, we have one input layer (), one output layer (), and no hidden layers. In general, each layer can take vectors and matrices as inputs and outputs, but our simple example just has a one-dimensional vector (a scalar) for both the input and output layers. If x were instead a vector (x), like we would see with multivariate regression, we would have the more general linear model that we frequently see when using OLS:

graph LR
    x["**x**"] -->|bias| y["y"]
    x -->|weight 1| y
    x -->|weight 2| y
    x -->|...| y
    x -->|weight n| y

With OLS, we have a closed-form, deterministic solution, although we saw earlier that stochastic gradient descent can do something similar numerically, albeit with some randomness and other potential drawbacks. Unfortunately we no longer have a closed form solution to solve the general representation used by neural networks. On the bright side, since we have to fit the model using optimization techniques anyway, we can make a vast number of changes to the model architecture with little additional penalty.

The optimal weights and biases for the neural network will be solved iteratively through a process called training. I’m not going to get into the specific details of how to train neural networks, but the general procedure is to:

1. Initialize the weights (coefficients) and biases (intercept) to some random values.
2. Pass the data through the network to get the predicted values using the current weights and biases (the forward pass).
3. Calculate the difference between the predicted values and the true values. OLS calls this the error or residual; it’s called the loss when training neural networks. We calculate it using a loss function, which is often the mean-squared error.
4. Use the loss to determine how we can adjust the weights and biases to hopefully improve the model fit. We can use the SGD method from Section 3 to do a lot of the heavy lifting for us by using the loss function as the input for the SGD optimizer (begins the backward pass).
5. Update the weights and biases in the direction that minimizes the loss.
6. Repeat steps 2-5 N times or until the model converges. Each pass through the data is called an epoch. We will typically set some maximum number of epochs to prevent the model from training forever, with the possibility of stopping early if the model fit is no longer improving.

This training loop is implicit with the SGDRegressor() function in the previous section; for neural networks, we’ll need to build it ourselves for maximum control and flexibility. In python code, the iterative training loop looks something like this for our sample data:

import torch
from torch.optim.sgd import SGD

nn_model = torch.nn.Linear(1, 1)  # 1 linear layer with 1 input and 1 output
1calculate_mse = torch.nn.MSELoss()
2sgd_optimizer = SGD(nn_model.parameters(), lr=0.01)

# convert numpy arrays to pytorch tensors
x_tensor = torch.from_numpy(x).float()
y_tensor = torch.from_numpy(y).float()

# training loop
NUM_EPOCHS = 1000
for _ in range(NUM_EPOCHS):
    # forward pass to calculate the loss
3    outputs = nn_model.forward(x_tensor)
    loss = calculate_mse(outputs, y_tensor)

    # backward pass to update weights/biases to try to improve next epoch
    sgd_optimizer.zero_grad()  # prepare to run backward pass
    loss.backward()  # compute gradient
    sgd_optimizer.step()  # use gradient to update weights

nn_weight = nn_model.weight.item()
nn_bias = nn_model.bias.item()

nn_weight=1.0026
nn_bias=-0.0179

Once again we see our weight/coefficient is close to one and the bias/intercept is close to zero, as expected. OLS, SGD, and neural networks all give us approximately the same result when fitting a linear model to our simple dataset:

Deep Learning

The next question is how to get from our simple neural network to practical deep learning models. At a basic level a deep neural network means a neural network with many hidden layers. But there are many considerations and options when building a usable deep learning model.