Bias-variance trade-off in model selection

Statistical model selection must seek a proper balance between overfitting and underfitting. It is the famous bias-variance trade-off. We need to balance simplicity against complexity. Simplicity here means fewer parameters to estimate, leading to lower variability, but associated with higher modeling bias. Complexity implies more parameters, which means a higher degree of variability but smaller modeling bias.

The bias-variance trade-off appears explicity on the formula of the widely used mean squared error (MSE) of an estimator {\hat{\theta}} of a given unknown parameter {\theta}:

\displaystyle MSE(\hat{\theta}) = Var(\hat{\theta}) + \left(Bias(\hat{\theta}, \theta)\right)^2


Claeskens, G., Hjort N. L. 2008. Model Selection and Model Averaging. Cambridge university press. (Chapter 1)


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s