Homework 02: properties of linear regression

Due Friday September 26 at 5:00pm

Conceptual exercises

The conceptual exercises are focused on explaining concepts and showing results mathematically. Show your work for each question.

You may write the answers and associated work for conceptual exercises by hand or type them in a Quarto document. Note: there is no GitHub repository for this assignment since there are no coding exercises.

In all exercises below, you may assume \(E[\boldsymbol{\varepsilon}| \boldsymbol{X}] = \boldsymbol{0}\) and \(\text{cov}(\boldsymbol{\varepsilon}| \boldsymbol{X}) = \sigma^2 \boldsymbol{I}\).

Exercise 1

In lecture, we defined the hat matrix \(\boldsymbol{H}\) as a projection matrix that projects \(\boldsymbol{y}\) onto \(Col(\boldsymbol{X})\) and discussed the properties of a projection matrix. In class we showed that \(\boldsymbol{H}\) is symmetric and idempotent. Now we will focus on two other properties.

  1. Show that for any vector \(\boldsymbol{v}\) in \(Col(\boldsymbol{X})\). \(\boldsymbol{H}\boldsymbol{v} = \boldsymbol{v}\).

  2. Show that any vector \(\boldsymbol{v}\) orthogonal to \(Col(\boldsymbol{X})\), \(\boldsymbol{H}\boldsymbol{v} = \boldsymbol{0}\)

Exercise 2

Derive the expected value and covariance matrix of the residual vector, \(\hat{\boldsymbol{\varepsilon}}\). In other words, derive the \(E[\hat{\boldsymbol{\varepsilon}}| \boldsymbol{X}]\) and \(\text{var}(\hat{\boldsymbol{\varepsilon}}| \boldsymbol{X})\).

Hint: consider the hat matrix.

Exercise 3

Show that all eigenvalues of the hat matrix \(\boldsymbol{H}\) are 0 or 1.

Exercise 4

Let \(\boldsymbol{y}= \boldsymbol{X}\beta + \epsilon\) where \(\beta \in \mathbb{R}^p\) and \(p = 2\).

  1. Show that \((I-\boldsymbol{H})\) is orthogonal to \(Col(\boldsymbol{X})\).

  2. Show \(E\left[\frac{\text{RSS}}{n-2}| \boldsymbol{X}\right]\) is an unbiased estimator of \(\sigma^2\).

Hint: note the fact that \(a = tr(a)\) for all scalar numbers \(a\). Also note the cyclic property of trace: \(tr(ABC) = tr(CAB) = tr(BCA)\).

Exercise 5

Show that if \(\bar{x} > 0\), \(\text{cov}(\hat{\beta}_0, \hat{\beta}_1) < 0\) in simple linear regression.

Exercise 6

Because the least squares estimator \(\hat{\beta} = \boldsymbol{A}\boldsymbol{y}\), for some matrix \(\boldsymbol{A}\), we call \(\hat{\beta}\) a linear estimator.

If \(E[\boldsymbol{\varepsilon}| \boldsymbol{X}] = \boldsymbol{0}\) and \(\text{var}(\boldsymbol{\varepsilon}| \boldsymbol{X}) = \sigma^2 \boldsymbol{I}\), then show (a) that \(E[\hat{\beta}] = \beta\) and (b) that \(\hat{\beta}\) has the smallest variance among all linear, unbiased estimators.

Hint for part (b): consider an alternative linear estimator \(\tilde{\beta} = \boldsymbol{B} \boldsymbol{y}\) and show that its variance is equal to the variance of \(\hat{\beta}\) plus some positive semi-definite matrix.

Exercise 7

Describe, in your own words, the difference between \(\hat{\beta}\) and \(\beta\) as well as the difference between \(\hat{y}\) and \(y\). In your explanation, specifically identify whether each is random or fixed, known or unknown both before and after collecting the data.

Submission

Warning

Remember – you must turn in a PDF file to the Gradescope page before the submission deadline for full credit.

If you write your responses to conceptual exercises by hand, you will need to combine your written work to the completed PDF for the applied exercises before submitting on Gradescope.

Instructions to combine PDFs:

To submit your assignment:

  • Access Gradescope through the menu on the Canvas website.

  • Click on the assignment, and you’ll be prompted to submit it.

  • Mark the pages associated with each exercise. All of the pages of your lab should be associated with at least one question (i.e., should be “checked”).