Page 11 of 14 · Neural networks & security

Multi-layer networks: stacking simple neurons

An artificial neuron is just logistic regression: weight the inputs, add a bias, pass the result through an activation function. One neuron can only draw a straight line. Connect layers of them, and the network can learn curved, complicated boundaries that no single line could ever separate.

Sinclair College - Dr. Mike Libassi

Inside one neuron

input x₁

input x₂

weight w₁

weight w₂

bias b

z=w₁x₁ + w₂x₂ + b=

output==

Why the activation function? Without it, stacking layers is pointless: a linear function of a linear function is still linear, so a 100-layer network would be no smarter than one neuron. The nonlinearity is what lets depth add power. ReLU (rectified linear unit) is the usual choice for hidden layers because it is cheap and trains well.

Train a network on a problem a line cannot solve

In the XOR pattern, the top-left and bottom-right corners are one class and the other two corners are the other. No straight line separates them. That limitation, pointed out in 1969, helped trigger the first AI winter. Start with no hidden layer and press Train, then add hidden neurons.

hidden neurons

accuracy

loss

epochs

How the network learns: backpropagation

Training a network is still gradient descent (page 04), just with many more weights. The trick is computing the gradient for every weight efficiently. Backpropagation does it in four repeating steps:

1. forwardFeed inputs through each layer to get a prediction.

2. lossCompare the prediction with the label, using log loss for classification.

3. backwardWork from the output back toward the input, using the chain rule from calculus to find how much each weight contributed to the error.

4. updateNudge every weight against its gradient, scaled by the learning rate.

∂L/∂w=∂L/∂a · ∂a/∂z · ∂z/∂w

The chain rule: the effect of a weight on the loss is the product of the effects along the path between them. Libraries such as PyTorch compute this automatically ("autograd"), so you never write it by hand.

Words you will hear

Deep network: one with many hidden layers.
Dense or fully connected layer: every neuron connects to every neuron in the previous layer.
Epoch: one pass through the training data. Batch size: examples per gradient step (page 05).
Dropout: switching off random neurons during training to reduce overfitting (page 07).

In Python

# scikit-learn: fine for small tabular problems
from sklearn.neural_network import MLPClassifier
net = MLPClassifier(hidden_layer_sizes=(16, 8), activation="relu", solver="adam", max_iter=500)
net.fit(X_train_s, y_train)

# PyTorch: the standard for real deep learning
import torch.nn as nn
model = nn.Sequential(
    nn.Linear(2, 8), nn.ReLU(),     # hidden layer: 2 inputs to 8 neurons
    nn.Linear(8, 1), nn.Sigmoid())  # output layer: 1 probability

A neuron computes a weighted sum plus bias, then applies a nonlinear activation.
Hidden layers let a network learn curved decision boundaries, like XOR, that a single neuron cannot.
Backpropagation uses the chain rule to get every weight's gradient; gradient descent does the rest.