Statistical model selection must seek a proper balance between overfitting and underfitting. It is the famous bias-variance trade-off. We need to balance simplicity against complexity. Simplicity here means fewer parameters to estimate, leading to lower variability, but associated with higher modeling bias. Complexity implies more parameters, which means a higher degree of variability but smaller modeling bias.
The bias-variance trade-off appears explicity on the formula of the widely used mean squared error (MSE) of an estimator of a given unknown parameter :
Claeskens, G., Hjort N. L. 2008. Model Selection and Model Averaging. Cambridge university press. (Chapter 1)