5 Neural Networks

Neural network structure

A neural network is a machine-learning model made of connected computational units arranged in layers. It learns patterns from examples by adjusting numerical parameters. Neural networks are especially useful when relationships in data are complex and difficult to describe with a simple formula or hand-written rules.

Layers and unit calculations

An example’s features enter through the input layer. They might be pixel values in an image or measurements in a table. Hidden layers transform those inputs: each unit combines values from the preceding layer using weights and a bias. The output layer produces a prediction, such as a category or a numerical estimate.

For one unit, the calculation is:

Here, $x_i$ are inputs, $w_i$ are weights, $b$ is a bias, and $\phi$ is the activation function. The network’s parameters are its learned weights and biases.

Activation functions apply a nonlinear transformation to a unit’s result. Without nonlinear activations, stacking layers would still produce only a linear relationship. ReLU, sigmoid, and tanh are common activation functions.

How neural networks learn

Training typically follows a repeated cycle:

1. Make a prediction. The input passes through the network layer by layer in a forward pass.

2. Measure the error. A loss function compares the prediction with the correct answer. For example, a classifier’s loss can be higher when it assigns a low probability to the correct class.

3. Adjust the parameters. Backpropagation calculates how changing each parameter would affect the loss. An optimization method such as gradient descent then nudges the parameters in directions expected to reduce it.

4. Repeat with training examples. After many updates, the model may learn patterns that help it predict answers for new examples, not just examples it trained on.

A separate validation set can help check generalization and reveal overfitting.

Example: recognizing cats

To train a network to recognize cats in photographs, provide labeled images. Early in training, its guesses may be poor. Repeated comparisons with the labels adjust the weights and biases; eventually, patterns in the images can help distinguish cats from other objects.

Applications

Neural networks are used for a range of tasks:

• Image and video analysis: recognizing objects or interpreting medical images.

• Language tasks: translating text, summarizing documents, and powering conversational systems.

• Speech: converting spoken language into text or recognizing voice commands.

• Prediction and pattern detection: estimating demand, identifying unusual transactions, or classifying documents.

Limitations and evaluation

Neural networks are not automatically the best choice for every problem. They can require substantial data and computing resources, may learn unwanted patterns in their training data, and can be difficult to interpret. Their performance should be evaluated on appropriate data before relying on their predictions.