5 Neural Networks

Neural network structure

A neural network is a machine-learning model made of connected computational units that transform inputs into predictions. The word “neural” is an analogy: artificial neurons are simple mathematical operations, not copies of biological brain cells. A network can learn patterns from examples by adjusting the strengths of connections between its units.

Neurons, layers, and forward computation

An artificial neuron receives values from the previous layer, combines them using learned weights, adds a learned bias, and applies an activation function. For inputs $x_1, x_2, \ldots, x_n$, it computes:

Here, $w_i$ are weights, $b$ is a bias, $\phi$ is the activation function, and $a$ is the neuron's output. Weights and biases are learned parameters. For example, with inputs $0.7$ and $0.2$, weights $0.5$ and $-1$, and bias $0.1$, the weighted sum is $0.5(0.7) - 1(0.2) + 0.1 = 0.25$. The activation function then determines the value passed onward.

Neurons are arranged in layers. The input layer receives features, such as measurements or pixel values; one or more hidden layers transform information; and the output layer produces a prediction, such as a numerical estimate or class scores. In a common design, each neuron in one layer connects to every neuron in the next.

During a forward pass, values flow layer by layer until the network produces an output. Adding a layer of only linear calculations does not, by itself, let the network represent nonlinear patterns: a composition of linear transformations remains linear. Activation functions provide the nonlinearity that makes layered networks more expressive.

Activation functions

An activation function transforms a neuron's weighted sum before passing it to the next layer. Common examples include:

• ReLU: $\phi(z) = \max(0,z)$. It returns zero for negative inputs and otherwise returns the input; it is widely used in hidden layers.

• Sigmoid: $\phi(z) = \frac{1}{1+e^{-z}}$. It maps values between $0$ and $1$.

• Tanh: maps values between $-1$ and $1$.

The activation function affects what a network can learn and how it trains. A sigmoid output can be useful for representing a probability in a binary prediction, while a hidden layer often uses ReLU. The appropriate choice depends on the model and task.

Training and parameter updates

Training adjusts a network's weights and biases so its predictions better match examples in the training data. In supervised learning, examples include inputs and their correct targets.

A common training process is:

1. Initialize the weights and biases, usually with small values.

2. Run a forward pass to calculate predictions.

3. Measure error with a loss function that compares predictions with target values.

4. Backpropagate the error by calculating how changes to parameters would affect the loss, working backward through the network.

5. Use an optimizer to update parameters, often using gradient descent to adjust them in a direction that reduces loss.

6. Repeat over many examples and training steps.

Backpropagation makes it practical to calculate gradients—the directions and rates of change of the loss—with respect to parameters in a multilayer network. The optimizer uses those gradients to update weights and biases. One full pass through the training examples is commonly called an epoch.

Training aims to learn useful patterns, not merely memorize examples. Performance on separate, unseen data helps assess whether the learned patterns generalize.

Deep learning and learned representations

A network with multiple hidden layers is often called a deep neural network, and training such networks is known as deep learning. Layers can build successively transformed representations of the input. In an image-recognition model, for instance, early layers may respond to simple visual features, while later layers combine them into patterns useful for identifying objects. This is a conceptual description of learned representations, not a guarantee that each layer has a neat human-readable meaning.

More layers and neurons can enable a model to represent more complex relationships, but they also increase the number of parameters and training demands. Deep networks therefore rely on suitable data, model design, and training procedures.

• weight