Heteroskedasticity: The variance of errors depends on \(X\) \[\text{Var}(\varepsilon_i | X_i) = \sigma_i^2 \neq \sigma^2\]

Right panel: The spread increases as X increases

Good news: OLS estimator \(\hat{\beta}\) is still unbiased and consistent

Bad news: The usual standard error formula is wrong!

Standard OLS variance formula assumes: \[\text{Var}(\hat{\beta}) = \sigma^2(X^TX)^{-1}\]

Problem: This formula assumes constant \(\sigma^2\), but we have varying \(\sigma_i^2\)

Using standard SEs with heteroskedasticity leads to:

1. Invalid hypothesis tests (wrong p-values)
2. Invalid confidence intervals (wrong coverage)
3. Misleading inference (false positives or negatives)

The coefficient is fine, but we can’t trust our uncertainty estimates!

With heteroskedasticity, the true variance of \(\hat{\beta}\) is: \[\text{Var}(\hat{\beta}) = (X^TX)^{-1} \left(\sum_{i=1}^n X_i X_i^T \sigma_i^2\right) (X^TX)^{-1}\]

Problem: We don’t know the true \(\sigma_i^2\)

Solution: Estimate each \(\sigma_i^2\) with the squared residual \(\hat{\varepsilon}_i^2\)

Robust variance estimator (HC0): \[\widehat{\text{Var}}(\hat{\beta}) = (X^TX)^{-1} \left(\sum_{i=1}^n X_i X_i^T \hat{\varepsilon}_i^2\right) (X^TX)^{-1}\]

Called the “sandwich” because of its structure: \[\underbrace{(X^TX)^{-1}}_{\text{bread}} \underbrace{\left(\sum X_i X_i^T \hat{\varepsilon}_i^2\right)}_{\text{meat}} \underbrace{(X^TX)^{-1}}_{\text{bread}}\]

This estimator is robust to heteroskedasticity:

• If errors are homoskedastic: Robust SE ≈ Standard SE (asymptotically)
• If errors are heteroskedastic: Robust SE is correct, Standard SE is wrong

Big picture: Robust SEs work whether or not you have heteroskedasticity

Several versions exist, with different finite-sample adjustments:

HC0 (Original): \(\widehat{\text{Var}}(\hat{\beta}) = (X^TX)^{-1} \sum X_i X_i^T \hat{\varepsilon}_i^2 (X^TX)^{-1}\)

HC1: Multiply by \(\frac{n}{n-k}\) for degrees of freedom

HC2: Weight by \((1-h_{ii})^{-1}\) where \(h_{ii}\) is leverage

HC3: Weight by \((1-h_{ii})^{-2}\) (more conservative)

In practice: Use HC1 or HC3; differences usually small with large \(n\)

In this example: Robust SE is larger (standard SE underestimates uncertainty)

Idea: If we know the variance structure, we can use it to get more efficient estimates

Suppose we know \(\text{Var}(\varepsilon_i | X_i) = \sigma^2 w_i\) for known weights \(w_i\)

WLS: Transform the model by \(1/\sqrt{w_i}\): \[\frac{y_i}{\sqrt{w_i}} = \frac{X_i}{\sqrt{w_i}}\beta + \frac{\varepsilon_i}{\sqrt{w_i}}\]

Result: Transformed errors have constant variance!

WLS minimizes weighted sum of squared residuals: \[\hat{\beta}_{WLS} = \arg\min_\beta \sum_{i=1}^n \frac{1}{w_i}(y_i - X_i\beta)^2\]

Solution: \[\hat{\beta}_{WLS} = (X^TW^{-1}X)^{-1}X^TW^{-1}y\] where \(W = \text{diag}(w_1, \ldots, w_n)\)

When weights are correct (\(w_i = \sigma_i^2/\sigma^2\)):

1. \(\hat{\beta}_{WLS}\) is unbiased
2. \(\hat{\beta}_{WLS}\) is BLUE (Best Linear Unbiased Estimator)
3. \(\hat{\beta}_{WLS}\) is more efficient than OLS
4. Standard SEs are valid (no need for robust SEs)

WLS has smaller standard error (more efficient)

Setup: Use WLS with correct weights \(w_i = \sigma_i^2/\sigma^2\)

Advantages:

• Most efficient estimator (smallest variance among unbiased estimators).
• Standard inference procedures are valid.
• No need for robust SEs

Setup: Use WLS with correct weights \(w_i = \sigma_i^2/\sigma^2\)

Requirements:

• Must know the correct weights
  
• Weights must be correctly specified

Setup: Use OLS and compute robust standard errors

Advantages:

• Don’t need to know variance structure
  
• More flexible (works with any heteroskedasticity pattern).
• Robust to weight misspecification

Setup: Use OLS and compute robust standard errors

Disadvantage:

• Less efficient than WLS (larger variance)

Which estimator has smaller variance?

WLS with correct weights has smaller variance than OLS

WLS distribution is tighter (has smaller variance)

WLS inference is valid when:

1. Weights are correctly specified: \(w_i = \sigma_i^2/\sigma^2\) exactly
2. Other assumptions hold: Linearity, independence, etc.

Problem: If weights are wrong, WLS inference is invalid

Example: Using \(w_i = x_i\) when true variance is \(\sigma_i^2 = \exp(x_i)\)

• WLS estimator is still consistent but not efficient
  
• Standard errors are wrong
  
• Hypothesis tests maybe wrong

These standard errors are not reliable because weights are misspecified

Robust SE inference is valid when:

1. Large sample size: Robust SEs are asymptotically valid
2. Correct model specification: Need correct \(E[Y|X]\)
3. Independence: Observations are independent
4. No severe leverage points: Extreme \(X\) values can cause problems

Key advantage: Don’t need to know variance structure!

Important caveat: Robust SEs are asymptotically valid

• Need “large enough” sample size
• Can underestimate uncertainty in small samples
• Various corrections (HC2, HC3) help with small samples

In practice, we rarely know the true weights!

Problem with WLS:

• Need to model \(\sigma_i^2\) as function of \(X\)
  
• If model is wrong, inference is invalid
  
• Hard to know if model is correct

Robust SEs:

• No need to model variance structure
  
• Valid for any heteroskedasticity pattern
  

Suppose we think variance depends on \(x\), so we try: \[\sigma_i^2 = \exp(\alpha_0 + \alpha_1 x_i)\]

Steps to implement WLS:

1. Fit OLS and get residuals \(\hat{\varepsilon}_i\)
   
2. Regress \(\log(\hat{\varepsilon}_i^2)\) on \(x\) to estimate \(\alpha_0, \alpha_1\)
   
3. Compute weights \(\hat{w}_i = \exp(\hat{\alpha}_0 + \hat{\alpha}_1 x_i)\)
   
4. Fit WLS using these weights

Robust SEs protect against model misspecification

WLS with wrong weights:

• Estimator still consistent
  
• But SEs are wrong
  
• Tests have wrong size.
• CIs have wrong coverage

OLS with robust SEs:

• If variance model is wrong, no problem!
  
• Inference still valid (asymptotically)

WLS with wrong weights: Coverage far from 95%! Robust SEs: Coverage close to nominal level

Robust SEs are simpler:

• No need to specify variance model
  
• Fewer modeling choices

WLS requires:

• Choosing functional form for variance
  
• Estimating variance model
  
• Potential for specification error
  

Researcher degrees of freedom:

With WLS, researchers might:

• Try different variance models.
• Pick the one that gives desired results.
• “Optimize” standard errors

Robust SEs prevent this:

• One standard approach
  
• Less flexibility for p-hacking
  
• More credible inference

Despite the advantages of robust SEs, use WLS when:

1. You truly know the weights (e.g., aggregated data with known group sizes)
2. Efficiency is critical (small sample, need every bit of precision)
3. Variance structure is simple and obvious (e.g., repeated measurements)

Example: Survey data with known sampling weights