Parameters and Hyperparameters

In the previous section, we established that machine learning models learn patterns from data. But what exactly does a model learn, and how do we control that learning process? This brings us to two fundamental types of settings associated with models: parameters and hyperparameters. Understanding the difference is essential for training and tuning your models effectively.

Parameters: What the Model Learns

Parameters are the internal variables of a model whose values are estimated or learned directly from the training data. Think of them as the knobs and dials that the model adjusts automatically during the training process to minimize errors and make better predictions. These are the variables that capture the patterns found in the data.

For example, consider the simple linear regression model we'll encounter later, which tries to fit a straight line to data points. The equation for a line is often written as $y = mx + b$. In this context:

• $m$ (the slope of the line) is a parameter.
• $b$ (the y-intercept of the line) is also a parameter.

During training, the linear regression algorithm will iteratively adjust the values of $m$ and $b$ to find the line that best represents the relationship between the input ($x$) and the output ($y$) in the training data. The final learned values of $m$ and $b$ define the specific model that resulted from the training process.

Other examples of parameters include:

• The weights and biases in neural networks.
• The coefficients in logistic regression.

The main takeaway is that parameters are learned from data by the algorithm itself. You don't set their values manually; the training process determines them.

Hyperparameters: How the Model Learns

Hyperparameters, on the other hand, are configuration settings that are external to the model and whose values cannot be estimated from data. They are set by the machine learning practitioner before the training process begins. Think of them as higher-level controls that define how the learning process itself should operate or constrain the model's complexity.

You can't know the optimal hyperparameter values just by looking at the data; they often require experimentation and tuning (a process we'll touch upon later).

Let's consider some examples related to algorithms mentioned in the course outline:

• Learning Rate (for Gradient Descent): In algorithms like linear regression or logistic regression that use gradient descent (an optimization technique we'll discuss), the learning rate controls how big the adjustment steps are when updating the parameters ($m$ and $b$, for instance). A learning rate that's too small might make training very slow, while one that's too large might prevent the model from finding the best parameter values. This learning rate is a hyperparameter you set beforehand.
• The Number of Neighbors (K in KNN): In the K-Nearest Neighbors algorithm (used for classification), 'K' represents the number of neighboring data points the algorithm considers when making a prediction for a new point. Should it look at the 3 closest neighbors? Or 5? Or 10? The value of 'K' is a hyperparameter you must choose before running the KNN algorithm.
• The Number of Clusters (K in K-Means): Similarly, in the K-Means clustering algorithm, 'K' represents the number of clusters you want the algorithm to find in the data. You decide this value before starting the clustering process.

Other examples include the number of layers or neurons in a neural network, the type of regularization to use (a technique to prevent overfitting), or the maximum depth of a decision tree.

Why Distinguish Them?

It's important to differentiate between parameters and hyperparameters:

Feature	Parameters	Hyperparameters
Origin	Learned from data during training	Set manually before training starts
Role	Define the specific model's prediction logic	Configure the training process, model structure
Determination	Estimated by the learning algorithm	Chosen by the practitioner (often via tuning)
Examples	Coefficients (Linear Reg), Weights (NNs)	Learning Rate, K (in KNN/K-Means), Tree Depth

Hyperparameters control the overall behavior and capacity of the learning algorithm. Finding good hyperparameters is often critical to achieving good model performance. You typically try different combinations of hyperparameters, train separate models for each combination, and evaluate them (often using a validation set, as discussed earlier) to find the settings that work best for your specific problem and data.

In summary, parameters are learned by the model from data, while hyperparameters are set by you to guide how the model learns. Both are integral parts of building and refining machine learning models.