A useful mental model

Think of a neural network as a stack of adjustable filters.

Each filter receives numbers from the previous layer. It gives each number a different amount of influence, adds an offset, and transforms the result. A layer sends its transformed numbers to the next layer. After several such transformations, the output layer produces a prediction, score, or generated value.

The usual diagram looks like a web because values flow between units:

input values          hidden layer             output

x₁ ───────────────┐   h₁ ───────────────┐
                  ├──>                  ├──> score
x₂ ───────────────┘   h₂ ───────────────┘

        weighted sums      weighted sum
        + activation

A unit is often called an artificial neuron. The name is historical. In practical terms, a neuron is a small numerical function. Its resemblance to a biological neuron is loose.

This simple picture shows a fully connected feedforward network, where information moves from input to output and each unit connects to every unit in the next layer. Not all neural networks use that exact wiring. Some reuse the same small filter across an image, carry state through a sequence, or let parts of the input interact through attention. They are still neural networks because they build a trainable computation from connected operations.

How a neural network works

For one typical neuron, the calculation is:

output = activation(w₁x₁ + w₂x₂ + ... + wₙxₙ + b)

The x values are the inputs. Each w is a weight that controls how strongly an input affects this neuron. The bias b shifts the result. The activation function then transforms that result before it continues through the network.

One common activation function is the rectified linear unit, or ReLU:

ReLU(x) = max(0, x)

ReLU returns zero for a negative input and leaves a positive input unchanged. Other activation functions transform values in other ways.

The activation is not a cosmetic extra. Suppose every layer performed only multiplication and addition. Two such layers could always be rewritten as one multiplication-and-addition step. Adding more layers would not let the network represent a nonlinear relationship. Nonlinear activations between layers prevent that collapse and let the network build more varied shapes from simple pieces.

When the network receives an input, it performs a forward pass:

  1. The input is represented as numbers.
  2. Each layer calculates new numbers from the previous layer.
  3. The output layer produces the network’s result.

During training, the result is compared with the desired result using a loss function, which measures error. Backpropagation efficiently calculates how a small change to each weight or bias would change that loss. An optimizer then adjusts those values, and the process repeats across many examples.

“Learning” means changing these adjustable numbers. The developer chooses the broad architecture and training process; the training algorithm finds the parameter values.

A forward pass by hand

Consider a toy network that receives two normalized signals for a card transaction:

  • x₁ = 0.8 for how unusual the amount is
  • x₂ = 0.3 for how unfamiliar the device is

These values are illustrative, not a real fraud model. The network has two hidden units with ReLU activations.

The first hidden unit calculates:

h₁ = ReLU(1.0 × 0.8 − 1.0 × 0.3 − 0.2)
   = ReLU(0.3)
   = 0.3

The second calculates:

h₂ = ReLU(−0.5 × 0.8 + 0.5 × 0.3 + 0.1)
   = ReLU(−0.15)
   = 0

The output unit combines the hidden values:

score = 1.5 × h₁ + 0.5 × h₂ − 0.4
      = 1.5 × 0.3 + 0.5 × 0 − 0.4
      = 0.05

That is one complete forward pass. A real classifier might apply another function to turn its raw score into a probability or compare the score with a decision threshold.

No programmer had to write the specific rule represented by these weights. Training would adjust them from data. Changing even one weight could change which hidden units are active and how strongly they affect the result.

It is tempting to label h₁ “amount risk” and h₂ “device risk.” The calculation does not justify those names. Hidden units often participate in patterns that only make sense in combination, and their meanings may not be cleanly interpretable by a person.

Why neural networks matter

Neural networks can learn useful intermediate representations instead of relying only on features that a person specifies in advance. In an image system, early computations might respond to local visual patterns while later computations combine them. In a language system, many layers transform representations of text based on context. The exact behavior is learned across the network rather than assigned to one named neuron at a time.

Their flexible function shape makes neural networks useful for images, language, audio, forecasting, control, and generation. It does not make them automatically best for every problem. A simpler model can be cheaper, easier to inspect, and better when data is limited or the relationship is straightforward.

Mathematical results show that certain neural networks can approximate broad classes of functions when given enough hidden units. That is a statement about what a network can represent. It does not promise that a particular dataset and training run will find a good function, avoid memorizing the examples, or behave reliably on unfamiliar inputs.

Common misconceptions

Neural networks work like brains

The terminology was inspired by biology, but artificial neurons omit nearly all of the structure and behavior of biological neurons. “Weighted numerical operation” is a more reliable mental model than “digital brain cell.”

Every neuron decides whether to fire

Some early models used thresholds, and ReLU does set negative values to zero. But modern units generally produce numerical activations, not simple yes-or-no decisions.

More layers always make a network better

Extra depth changes what a network can represent and how efficiently it may represent some patterns. It can also make the model harder or more expensive to train. Performance depends on the architecture, data, objective, optimization, and evaluation—not layer count alone.

Hidden layers automatically make a model nonlinear

A stack of linear layers is still linear. The nonlinear activation between layers is what prevents the stack from collapsing into a single linear transformation.

The network discovers understandable rules

It discovers parameter values that reduce its training objective. Those values can support useful behavior without corresponding to a short rule a person can read. Interpretation requires separate evidence.

Neural networks and deep learning mean the same thing

Deep learning usually refers to machine learning with neural networks that contain multiple processing layers. There is no universal layer-count cutoff for “deep,” and neural networks also include shallow models.

How neural networks fit into AI

Neural networks are one family within machine learning. A neural-network architecture specifies how computations are arranged. Training produces the learned model weights, which are part of the model’s parameters. Running the trained network on new input is inference.

Many systems described as an AI model use neural networks, but the terms are not interchangeable. “AI model” is broader. A transformer, for example, is a neural-network architecture built from attention, feedforward operations, normalization, and connections that route information around layers.

The key idea remains the same across these designs: arrange differentiable computations into a useful structure, then learn the adjustable values from data.

Where to go next

Read What Are Model Weights? to separate the network’s learned connection strengths from its architecture. Read What Are Model Parameters? for the broader set of adjustable values, then use Model Parameters vs. Model Weights to compare the terms directly. To see how neural networks generate language, continue to What Is an LLM?.