Study List for Linear Models Exam

This list may not be exhaustive but should be a good start, I encourage you to go through our slides, exercises, and notes from the class to best prepare for the upcoming midterm.

Key Formulas

OLS Estimator:

  • \(\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}\)

Normal Equations:

  • \(\mathbf{X}^T\mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}^T\mathbf{y}\)

Hat Matrix:

  • \(\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\)
  • \(\hat{\mathbf{y}} = \mathbf{H}\mathbf{y}\)
  • Properties: \(\mathbf{H}^T = \mathbf{H}\) (symmetric), \(\mathbf{H}^2 = \mathbf{H}\) (idempotent)

Variance of OLS:

  • \(\text{Var}(\hat{\boldsymbol{\beta}}) = \sigma^2(\mathbf{X}^T\mathbf{X})^{-1}\)
  • \(\text{Var}(\hat{\beta}_j) = \sigma^2[(\mathbf{X}^T\mathbf{X})^{-1}]_{jj}\)

Sums of Squares:

  • TSS = \(\mathbf{y}^T\mathbf{y} - n\bar{y}^2\) = \(\sum_{i=1}^n(y_i - \bar{y})^2\)
  • SS\(_{\text{Reg}}\) = \(\mathbf{y}^T\mathbf{H}\mathbf{y} - n\bar{y}^2\)
  • SSE = \(\mathbf{y}^T(\mathbf{I}-\mathbf{H})\mathbf{y}\)
  • Fundamental Identity: TSS = SS\(_{\text{Reg}}\) + SSE

Coefficient of Determination:

  • \(R^2 = \frac{\text{SS}_{\text{Reg}}}{\text{TSS}} = 1 - \frac{\text{SSE}}{\text{TSS}}\)

Variance Estimation:

  • \(\hat{\sigma}^2 = \frac{\text{SSE}}{n-p}\) (MSE)

Test Statistics:

  • t-statistic: \(t = \frac{\hat{\beta}_j}{\text{se}(\hat{\beta}_j)}\) where \(\text{se}(\hat{\beta}_j) = \sqrt{\hat{\sigma}^2[(\mathbf{X}^T\mathbf{X})^{-1}]_{jj}}\)
  • F-statistic: \(F = \frac{\text{SS}_{\text{Reg}}/(p-1)}{\text{SSE}/(n-p)} = \frac{\text{MS}_{\text{Reg}}}{\text{MSE}}\)

Random Vector Properties

  • \(E[\mathbf{A}\mathbf{Y} + \mathbf{b}] = \mathbf{A}E[\mathbf{Y}] + \mathbf{b}\)
  • \(\text{Var}(\mathbf{A}\mathbf{Y} + \mathbf{b}) = \mathbf{A}\text{Var}(\mathbf{Y})\mathbf{A}^T\)

Gauss-Markov Assumptions

  1. Linearity: \(\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}\)
  2. Zero mean errors: \(E[\boldsymbol{\varepsilon}] = \mathbf{0}\)
  3. Constant variance & independence: \(\text{Var}(\boldsymbol{\varepsilon}) = \sigma^2\mathbf{I}\)
  4. Full rank: \(\mathbf{X}\) has full column rank

Degrees of Freedom

  • Error: \(n - p\)
  • Regression: \(p - 1\)
  • Total: \(n - 1\)

Key Conceptual Points

Orthogonality Condition:

  • \(\mathbf{X}^T\hat{\boldsymbol{\varepsilon}} = \mathbf{0}\) (residuals perpendicular to column space)

QR Decomposition:

  • \(\mathbf{R}\boldsymbol{\hat\beta} = \mathbf{Q}^T\mathbf{y}\)
  • How to backsolve to get \(\hat\beta\)

BLUE:

  • Best = minimum variance
  • Linear = estimator is linear in \(\mathbf{y}\)
  • Unbiased = \(E[\hat{\boldsymbol{\beta}}] = \boldsymbol{\beta}\)
  • Estimator

For Unbiased Estimator \(\mathbf{C}\mathbf{y}\):

  • Must satisfy: \(\mathbf{C}\mathbf{X} = \mathbf{I}\)

Positive Semi-Definite:

  • Matrix \(\mathbf{A}\) is PSD if \(\mathbf{x}^T\mathbf{A}\mathbf{x} \geq 0\) for all vectors \(\mathbf{x}\)
  • \(\mathbf{D}\mathbf{D}^T\) is always PSD

Matrix Algebra Rules You’ll Need

  • \((\mathbf{AB})^T = \mathbf{B}^T\mathbf{A}^T\)
  • \((\mathbf{AB})^{-1} = \mathbf{B}^{-1}\mathbf{A}^{-1}\)
  • \((\mathbf{A}^T)^{-1} = (\mathbf{A}^{-1})^T\)