Appendix 3B: Maximum Likelihood
Maximizing the likelihood and numerical methods
- Algebraic maximization of ln L(β|x,y) (i.e., the log-likelihood) is rarely possible
- Numerical methods search for the maximum using the slope and change in slope of the likelihood equation (i.e., first and second derivatives)
- β estimates correspond to the peak of the log-likelihood curve—when the first derivative = 0
- Standard error (SE) estimates are largely a function of the rate of change around the β estimate (the top of the curve). This is the second derivative. More peaked distributions will produce smaller SEs (more certainty about β estimate). Flatter distributions will produce larger SEs (less certainty about β estimate)
- Estimates of parameters are usually very close in different software, with perhaps small differences in standard errors
*NOTE: It is more complicated when we have multiple IVs—but this basic intuition about how MLE works with one IV should suffice for the purposes of this class
Possible problems with ML
- A flat likelihood function makes convergence difficult
- Which curve pictured below would produce smaller SEs?
- Little variation in a DV or for a given IV can cause problems
- Some models (but not those we consider in this class) can have a local maxima
Maximum Likelihood Estimates
- Under general conditions, the ML estimates are:
- Consistent: The mean of the sampling distribution approaches the true value
- Asymptotically efficient: Data are used as well as possible
- Asymptotically normal: The sampling distribution becomes normal with large samples
MLE and sample size
- Consistency, normality, and efficiency are asymptotic properties
- OLS tends to work well with very small samples
- MLE often requires larger samples (but, some bespoke solutions)
- Should consider both overall size of sample as well as sparseness of the data for what you are modeling
- ML estimators are not necessarily bad in small samples
- But small sample behavior is largely unknown
ML estimators
- Maximum likelihood is an extremely flexible tool for estimating models. As you learn more complex models (e.g., hierarchical models, panel models, latent variable models, structural equation models, etc.) there is a high likelihood (pun!) that the standard practice will be to estimate the model via maximum likelihood
