# Bias-variance trade-off in model selection

Statistical model selection must seek a proper balance between overfitting and underfitting. It is the famous bias-variance trade-off. We need to balance simplicity against complexity. Simplicity here means fewer parameters to estimate, leading to lower variability, but associated with higher modeling bias. Complexity implies more parameters, which means a higher degree of variability but smaller modeling bias.

The bias-variance trade-off appears explicity on the formula of the widely used mean squared error (MSE) of an estimator ${\hat{\theta}}$ of a given unknown parameter ${\theta}$:

$\displaystyle MSE(\hat{\theta}) = Var(\hat{\theta}) + \left(Bias(\hat{\theta}, \theta)\right)^2$

Reference:

Claeskens, G., Hjort N. L. 2008. Model Selection and Model Averaging. Cambridge university press. (Chapter 1)