Computational Statistics

64 REGRESSION

2.2.2 Least Squares Estimation

A first idea of estimation in a linear model can be gained from the following relation: given X,

we have E(Y ) = Xβ. As X is a matrix, we cannot simply solve this relation for β using a

division by X. But we can expand the relation to X

>

E(Y ) = X

>

Xβ. X

>

X is a positive semi-

definite symmetric matrix. If X has rank p, the full rank, this matrix is invertibe. In general

at least a pseudo-inverse exists and we can calculate (X

>

X)

−

X

>

E(Y ) = β.

2

This equation

motivates the following estimator:

b

β = (X

>

X)

−

X

>

Y. (2.9)

Using the model relation Y = Xβ + ε 2.2 and E(ε) = 0, in the full rank case we get

E(

b

β) = E

“

`

X

>

X

´

−