A useful mental model
Think of a model's architecture as a set of equations with empty numbered slots. The architecture decides where the slots are and how they connect. The parameters are the numbers placed in those slots.
That distinction matters. Two models can have the same architecture but behave differently because their parameter values differ. Before training, the slots contain initialized values. Training adjusts some or all of them using data. During inference, the model normally holds those values fixed and applies the resulting computation to new inputs.
A parameter is not a knob that a user turns for each request. It is part of the model itself.
How parameters work
A neural-network layer commonly performs a calculation of this form:
[
\text{output} = f(\text{input} \times \text{weight matrix} + \text{bias vector})
]
The weight matrix and bias vector contain parameters. The function (f) is part of the architecture and may contain no parameters of its own.
The architecture fixes the shapes of the parameter tensors. If a layer connects 2 input values to 3 output values, its weight matrix needs 6 scalar entries:
2 inputs ──> [2 × 3 weight matrix] ──> 3 outputs
+
[3-value bias vector]Each independently stored scalar entry counts as one parameter. The model uses the same entries for every example it processes. Applying a parameter a million times does not turn it into a million parameters.
During training, an optimizer receives information about how a small change to each trainable parameter would affect the model's error. It then updates the values. Repeating that process can produce a set of parameters that maps new inputs to useful outputs.
The word “trainable” needs care. A parameter can be frozen so that a particular training run does not update it. It is still part of the model's fitted state. Frameworks also store other state that affects computation but is not treated as a parameter. PyTorch calls such state a buffer, while Keras uses a broader “weights” collection and divides it into trainable and non-trainable weights.
A worked parameter count
Consider a small network with two input values, one hidden layer with three outputs, and one final output.
The first layer has:
- (2 \times 3 = 6) weights
- 3 biases
- 9 parameters in total
The final layer has:
- (3 \times 1 = 3) weights
- 1 bias
- 4 parameters in total
The complete network has (9 + 4 = 13) parameters.
The input values do not count as parameters because they change from one example to another. The intermediate outputs do not count either; they are temporary results. The activation functions in this example add no parameters. Only the independently stored values in the weight matrices and bias vectors contribute to the count.
This shape-based method scales to larger models: count the scalar entries in each parameter tensor, then add them. If parameters are shared, count the stored values once, not every place the model uses them.
Why parameter count matters
Parameter count describes the size of a model's learned numerical state. It affects several practical properties.
First, stored parameters take memory. The amount depends on how many parameters there are and how each value is represented. A 16-bit representation uses two bytes per stored scalar before accounting for non-parameter state and file overhead.
Second, parameters participate in computation. More parameters often mean more arithmetic, although architecture, parameter sharing, and which parts of a model are active can change that relationship.
Third, the number of parameters helps describe a model's capacity to fit patterns. It does not measure how well that capacity was used. Training data, objective, architecture, optimization, and evaluation all matter. A model with more parameters is not automatically more accurate, more knowledgeable, or more useful.
Parameter counts also need a label. A tool or paper may report total parameters, only parameters currently marked trainable, or a count that excludes embeddings. These numbers answer different questions and should not be compared as if they were identical.
Common misconceptions
Parameters are not hyperparameters
Parameters are values fitted as part of the model. Hyperparameters configure the model or training process. Learning rate and batch size are common hyperparameters: they influence how training proceeds, but they are not values learned inside the resulting model.
Parameters and weights are not exact synonyms
Weights are a major kind of parameter, so people often use the words interchangeably. But a bias is also a parameter, as are learned scale and offset values in some normalization layers. “Weights” can also have a broader, framework-specific meaning. The safest reading of a parameter count is the one defined by the tool or model publisher that produced it.
One parameter does not equal one fact
A trained model's behavior arises from many parameter values working together. A single value usually has no plain-language meaning such as “Paris is the capital of France.” Information can be distributed across many values, and one value can influence many outputs.
More parameters do not guarantee a better model
Parameter count measures quantity, not quality. Increasing capacity can help when the architecture, data, and training use it effectively. It can also waste memory and computation or fit unwanted patterns. Performance has to be measured on the task that matters.
Not every stored value is necessarily a parameter
Models may keep running statistics, caches, or other state alongside parameters. Frameworks classify this state differently. For an exact count, use the framework's parameter enumeration rather than the size of every tensor in a checkpoint.
How parameters fit into a model
The neural-network architecture determines the operations and the shapes of the parameter tensors. Initialization gives those tensors starting values. Training changes the trainable values. A checkpoint saves the resulting values, often with additional state. Inference loads them and uses them to compute outputs.
Parameters are therefore neither the whole model nor merely a model specification. A usable trained model combines an architecture with particular parameter values.
Where to go next
Read What Are Model Weights? for the role of the multipliers that make up most neural-network parameters. Then see Model Parameters vs. Model Weights for the exact boundary between the two terms.