Random Vectors and Gauss-Markov Theorem Problem Set

Instructions

This problem set covers random vector properties, variance-covariance matrices, and the Gauss-Markov theorem. Show all work and provide clear explanations for your reasoning. For computational problems, you may verify your answers using R, but show the mathematical work first.

Problem 1

Consider the random vector \(\mathbf{Y} = \begin{bmatrix} Y_1 \\ Y_2 \\ Y_3 \end{bmatrix}\) where \(E[\mathbf{Y}] = \begin{bmatrix} 3 \\ -1 \\ 2 \end{bmatrix}\).

a) Let \(\mathbf{A} = \begin{bmatrix} 2 & 0 & 1 \\ 1 & -1 & 3 \end{bmatrix}\). Calculate \(E[\mathbf{A}\mathbf{Y}]\) using the linearity property.

b) What would \(E[\mathbf{A}\mathbf{Y} + \mathbf{c}]\) be if \(\mathbf{c} = \begin{bmatrix} 5 \\ -2 \end{bmatrix}\)?

Problem 2

Given the random vector \(\mathbf{Z} = \begin{bmatrix} Z_1 \\ Z_2 \end{bmatrix}\) with variance-covariance matrix:

\[\text{Var}(\mathbf{Z}) = \begin{bmatrix} 9 & 2 \\ 2 & 4 \end{bmatrix}\]

a) What are \(\text{Var}(Z_1)\), \(\text{Var}(Z_2)\), and \(\text{Cov}(Z_1, Z_2)\)?

b) Calculate \(\text{Var}(3Z_1 - 2Z_2)\) using the matrix formula \(\text{Var}(\mathbf{A}\mathbf{Z}) = \mathbf{A}\text{Var}(\mathbf{Z})\mathbf{A}^T\).

c) Find \(\text{Var}\begin{bmatrix} Z_1 + Z_2 \\ 2Z_1 - Z_2 \end{bmatrix}\). Show your matrix multiplication steps.

Problem 3

Consider the linear regression model \(\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}\) where:

• \(\mathbf{y}\) is \(n \times 1\)
  
• \(\mathbf{X}\) is \(n \times p\) with full column rank
  
• \(\boldsymbol{\beta}\) is \(p \times 1\)
  
• \(\boldsymbol{\varepsilon}\) is \(n \times 1\)

a) State the Gauss-Markov assumptions clearly.

b) Explain what \(\text{Var}(\boldsymbol{\varepsilon}) = \sigma^2\mathbf{I}\) means in plain English. What two conditions does this impose on the error terms?

c) If \(\text{Var}(\varepsilon_1) = 4\), \(\text{Var}(\varepsilon_2) = 9\), and \(\text{Cov}(\varepsilon_1, \varepsilon_2) = 1\), do the errors satisfy the Gauss-Markov assumptions? Explain why or why not.

Problem 4

For the OLS estimator \(\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}\):

a) Starting from \(E[\hat{\boldsymbol{\beta}}] = E[(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}]\), show step-by-step that OLS is unbiased. Clearly indicate where you use each Gauss-Markov assumption.

b) In the proof, we treat \(\boldsymbol{\beta}\) as a constant rather than a random variable. Explain why this is appropriate in the classical regression framework.

c) What would happen to your proof if \(E[\boldsymbol{\varepsilon}] = \mathbf{c}\) for some non-zero constant vector \(\mathbf{c}\) instead of \(E[\boldsymbol{\varepsilon}] = \mathbf{0}\)?

Problem 5

a) Derive \(\text{Var}(\hat{\boldsymbol{\beta}})\) starting from the result that \(\hat{\boldsymbol{\beta}} = \boldsymbol{\beta} + (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\boldsymbol{\varepsilon}\). Show each step clearly.

b) For the simple linear regression case with \(\mathbf{X} = \begin{bmatrix} 1 & 1 \\ 2 & 4 \end{bmatrix}\), calculate \((\mathbf{X}^T\mathbf{X})^{-1}\) by hand.

c) Using your result from part (b), what is \(\text{Var}(\hat{\boldsymbol{\beta}})\) in terms of \(\sigma^2\)? What is the variance of \(\hat\beta_0\)? What is the variance of \(\hat\beta_1\)?

Problem 6

The key step in proving OLS is BLUE involves writing any linear unbiased estimator as: \[\mathbf{C} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T + \mathbf{D}\]

a) Explain in your own words why this decomposition is “clever” and what intuition it provides about comparing estimators.

b) Show that if \(\mathbf{C}\mathbf{X} = \mathbf{I}\) (the unbiasedness constraint), then \(\mathbf{D}\mathbf{X} = \mathbf{0}\).

c) In the final step of the proof, we use the fact that \(\mathbf{D}\mathbf{D}^T\) is positive semi-definite.

• Define what “positive semi-definite” means
  
• Explain why any matrix of the form \(\mathbf{D}\mathbf{D}^T\) must be positive semi-definite
  
• How does this property ensure that OLS has minimum variance?

d) If \(\mathbf{D} = \begin{bmatrix} 1 & 2 \\ 0 & -1 \end{bmatrix}\), calculate \(\mathbf{D}\mathbf{D}^T\) and verify that all eigenvalues are non-negative.

Bonus

Read this post on Is OLS BLUE or BUE?