Study List for Linear Models Exam 2

This list may not be exhaustive but should be a good start, I encourage you to go through our slides, exercises, and notes from the class to best prepare for the upcoming midterm.

OLS

\(\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}\)
\(\hat{\mathbf{y}} = \mathbf{H}\mathbf{y}, \quad \mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\)
\(\hat{\boldsymbol{\beta}} \sim N(\boldsymbol{\beta},\; \sigma^2(\mathbf{X}^T\mathbf{X})^{-1})\)
\(\hat{\sigma}^2 = \text{SSE}/(n-p)\)
\(\text{se}(\hat{\beta}_j) = \hat{\sigma}\sqrt{[(\mathbf{X}^T\mathbf{X})^{-1}]_{jj}}\)

Inference

\(t = \hat{\beta}_j / \text{se}(\hat{\beta}_j) \sim t_{n-p}\)
\(\hat{\beta}_j \pm t_{\alpha/2,\,n-p} \cdot \text{se}(\hat{\beta}_j)\)
Confidence interval for \(E[y_\text{new}]\): \(\hat{y}_\text{new} \pm t_{\alpha/2,\,n-p}\cdot\hat{\sigma}\sqrt{\mathbf{x}_\text{new}^T(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{x}_\text{new}}\)
Prediction interval for \(y_\text{new}\): \(\hat{y}_\text{new} \pm t_{\alpha/2,\,n-p}\cdot\hat{\sigma}\sqrt{1 + \mathbf{x}_\text{new}^T(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{x}_\text{new}}\)

Model Fit

\(R^2 = 1 - \text{SSE}/\text{SST}\)
\(R^2_\text{adj} = 1 - \dfrac{\text{SSE}/(n-p)}{\text{SST}/(n-1)}\)
\(\text{AIC} = n\log(\text{MSE}) + 2p + C\)
\(C_p = \text{SSE}/\hat{\sigma}^2 - n + 2p\)
\(\text{LOOCV} = \dfrac{1}{n}\displaystyle\sum_{i=1}^n \left(\dfrac{y_i - \hat{y}_i}{1 - h_i}\right)^2\)

Leverage & Diagnostics

\(h_i = H_{ii}, \quad \sum h_i = p, \quad\) flag if \(h_i > 2p/n\)
\(\text{Var}(\hat{\varepsilon}_i) = \sigma^2(1-h_i)\)
Standardized: \(r_i = \hat{\varepsilon}_i \;/\; \hat{\sigma}\sqrt{1-h_i}\)
Studentized: \(t_i = \hat{\varepsilon}_i \;/\; \hat{\sigma}_{(i)}\sqrt{1-h_i} \sim t_{n-p-1}\)
Cook’s \(D\): \(D_i = \dfrac{r_i^2}{p}\cdot\dfrac{h_i}{1-h_i}\)

Ridge Regression

\(\min_{\boldsymbol{\beta}}\;(\mathbf{y}-\mathbf{X}\boldsymbol{\beta})^T(\mathbf{y}-\mathbf{X}\boldsymbol{\beta}) + \lambda\boldsymbol{\beta}^T\boldsymbol{\beta}\)
\(\hat{\boldsymbol{\beta}}_\text{ridge} = (\mathbf{X}^T\mathbf{X}+\lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}\)
\(E[\hat{\boldsymbol{\beta}}_\text{ridge}] = (\mathbf{X}^T\mathbf{X}+\lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{X}\boldsymbol{\beta}\)
\(\text{Var}(\hat{\boldsymbol{\beta}}_\text{ridge}) = \sigma^2(\mathbf{X}^T\mathbf{X}+\lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{X}(\mathbf{X}^T\mathbf{X}+\lambda\mathbf{I})^{-1}\)
\(\text{df}(\lambda) = \text{tr}(\mathbf{H}_\lambda), \quad \mathbf{H}_\lambda = \mathbf{X}(\mathbf{X}^T\mathbf{X}+\lambda\mathbf{I})^{-1}\mathbf{X}^T\)

Splines

\(\hat{y} = \hat\beta_0 + \hat\beta_1 x + \hat\beta_2 x^2 + \hat\beta_3 x^3 + \displaystyle\sum_{k=1}^{K}\hat\beta_{3+k}(x-\xi_k)_+^3\)
\((x-\xi_k)_+^3 = (x-\xi_k)^3\) if \(x > \xi_k\), else \(0\)