Bias and Variance in ML: A Simple Guide for Understanding Your Machine Learning Model's Performance

 

Have you ever played a game with your friends and noticed that your performance can vary depending on the level or challenge? Machine learning models can also have a similar issue with performance variability, known as bias and variance.

Bias:

Bias is the overall difference between what the model predicts and what the correct answer should be, averaged across all examples. A model with high bias consistently makes the same types of errors, regardless of the specific input features or examples.

To illustrate this concept, let's consider a regression model that predicts the price of a house based on its square footage, number of bedrooms, and location. When evaluating this model on a test dataset, we calculate the mean squared error (MSE) between the predicted prices and the actual prices across all examples in the test dataset. If the MSE is high, this suggests that the model is biased and consistently predicts prices that are too high or too low compared to the true values.

To calculate bias, we need to compare the predicted values of a machine learning model to the actual values. We can then calculate the average difference between the predicted values and the actual values. If the average difference is close to zero, then the model has low bias, meaning it's making accurate predictions. On the other hand, if the average difference is large, then the model has high bias, meaning it's making inaccurate predictions.

Here's the mathematical formula to calculate bias: 

Bias = (1/n) * Σ(y_actual - y_predicted)

where, 

n = number of data points, 

y_actual = actual value of the target variable, 

y_predicted = predicted value of the target variable

Variance:

Variance is a measure of how much a machine learning model's performance varies when it's trained on different datasets. A model with low variance is more likely to perform well on new, unseen datasets, while a model with high variance may perform well on the training data but not generalize well to new datasets.

To understand variance, let's go back to the game you were playing with your friends. When you practice playing the game with your friends, it's like training a machine learning model on a dataset. Sometimes, the model gets really good at the game and can beat all the levels, and other times, it's not as good. This variability in how well the model performs on different datasets is called "variance". If the model performs well on many different datasets, then it has low variance. But if the model only performs well on one dataset, and not on others, then it has high variance.

In the context of machine learning, variance is typically measured as the variance of the model's predictions on different subsets of the training data. This is often done using cross-validation, where the dataset is split into multiple "folds", and the model is trained and evaluated on each fold.

To calculate the variance of the model's predictions, we can calculate the variance of the model's predicted values across the different folds. If the predicted values are similar across all the folds, then the model has low variance. If the predicted values are very different across the folds, then the model has high variance.

In summary, bias and variance are two important concepts in machine learning that can affect a model's performance. Understanding these concepts can help you diagnose and address issues with your models, leading to more accurate and reliable predictions.