Search for probability and statistics terms on Statlect
StatLect
Index > Fundamentals of statistics

Linear regression - Hypothesis testing

by , PhD

This lecture discusses how to perform tests of hypotheses about the coefficients of a linear regression model estimated by ordinary least squares (OLS).

Table of Contents

Normal vs non-normal model

The lecture is divided in two parts:

How to choose which test to carry out after estimating a linear regression model.

The linear regression model

The regression model is[eq1]where:

There are $N$ observations in the sample, so that $i=1,ldots ,N$.

Matrix notation

We also denote:

Using this notation, we can write[eq5]

Moreover, the OLS estimator of $eta $ is [eq6]

We assume that the design matrix X has full-rank, so that the matrix $X^{ op }X$ is invertible.

Tests of hypothesis in the normal linear regression model

We now explain how to derive tests about the coefficients of the normal linear regression model.

The vector of errors epsilon is assumed to have a multivariate normal distribution conditional on X, with mean equal to 0 and covariance matrix equal to[eq7]where I is the $N imes N$ identity matrix and sigma^2 is a positive constant.

It can be proved (see the lecture about the normal linear regression model) that the assumption of conditional normality implies that:

Test of a restriction on a single coefficient (t test)

In a test of a restriction on a single coefficient, we test the null hypothesis[eq11]where $eta _{k}$ is the k-th entry of the vector of coefficients $eta $ and $qin U{211d} $.

In other words, our null hypothesis is that the k-th coefficient is equal to a specific value.

This hypothesis is tested with the test statistic[eq12]where $S_{kk}$ is the k-th diagonal entry of the matrix [eq13].

The test statistic $t$ has a standard Student's t distribution with $N-K$ degrees of freedom. For this reason, it is called a t statistic and the test is called a t test.

Proof

Under the null hypothesis [eq14] has a normal distribution with mean $q$ and variance $sigma ^{2}S_{kk}$. As a consequence, the ratio[eq15]has a standard normal distribution (mean 0 and variance 1). We can write[eq16]Since [eq17] has a Gamma distribution with parameters $N-K$ and sigma^2, the ratio[eq18]has a Gamma distribution with parameters $N-K$ and 1. It is also independent of $z$ because [eq19] is independent of [eq20]. Therefore, the ratio[eq21]has a standard Student's t distribution with $N-K$ degrees of freedom (see the lecture on the Student's t distribution).

The null hypothesis is rejected if $t$ falls outside the acceptance region.

How the acceptance region is determined depends not only on the desired size of the test, but also on whether the test is:

For more details on how to determine the acceptance region, see the glossary entry on critical values.

Test of a set of linear restrictions (F test)

When testing a set of linear restrictions, we test the null hypothesis[eq22]where $R$ is a $L imes K$ matrix and $q$ is a $L imes 1$ vector. $L$ is the number of restrictions.

Example Suppose that $eta $ is $2 imes 1$ and that we want to test the hypothesis [eq23]. We can write it in the form $Reta =q$ by setting[eq24]

Example Suppose that $eta $ is $3 imes 1$ and that we want to test whether the two restrictions [eq25] and $eta _{3}=0$ hold simultaneously. The first restriction can be written as[eq26]So we have[eq27]

To test the null hypothesis, we use the test statistic[eq28]which has an F distribution with $L$ and $N-K$ degrees of freedom. For this reason, it is called an F statistic and the test is called an F test.

Proof

Under the null and conditional on X, the vector $Rwidehat{eta }$, being a linear transformation of the normal random vector $widehat{eta }$, has a multivariate normal distribution with mean[eq29]and covariance matrix[eq30]Thus, we can write[eq31]Since $Rwidehat{eta }$ is multivariate normal, the quadratic form $Q$ has a Chi-square distribution with $L$ degrees of freedom (see the lecture on quadratic forms involving normal vectors). Furthermore, since [eq17] has a Gamma distribution with parameters $N-K$ and sigma^2 the statistic[eq33]has a Chi-square distribution with $N-K$ degrees of freedom (see the lecture on the Gamma distribution). Thus, we can write[eq34]Thus F is a ratio between two Chi-square variables, each divided by its degrees of freedom. The two variables are independent because $Q$ depends only on $widehat{eta }$ and $P$ depends only on [eq17], and $widehat{eta }$ and [eq17] are independent. As a consequence, F has an F distribution with $L$ and $N-K$ degrees of freedom (see the lecture on the F distribution).

The F test is one-tailed.

A critical value in the right tail of the F distribution is chosen so as to achieve the desired size of the test.

Then, the null hypothesis is rejected if the F statistics is larger than the critical value.

When you use a statistical package to run a linear regression, you often get a regression output that includes the value of an F statistic. Usually this is obtained by performing an F test of the null hypothesis that all the regression coefficients are equal to 0 (except the coefficient on the intercept).

Tests based on maximum likelihood procedures (Wald, Lagrange multiplier, likelihood ratio)

As we explained in the lecture on maximum likelihood estimation of regression models, the maximum likelihood estimator of the vector of coefficients of a normal linear regression model is equal to the OLS estimator $widehat{eta }$.

As a consequence, all the usual tests based on maximum likelihood procedures (e.g., Wald, Lagrange multiplier, likelihood ratio) can be employed to conduct tests of hypothesis about $eta $.

Tests of hypothesis when the OLS estimator is asymptotically normal

In this section we explain how to perform hypothesis tests about the coefficients of a linear regression model when the OLS estimator is asymptotically normal.

As we have shown in the lecture on the properties of the OLS estimator, in several cases (i.e., under different sets of assumptions) it can be proved that:

  1. the OLS estimator $widehat{eta }$ is asymptotically normal, that is,[eq37]where [eq38] denotes convergence in distribution (as the sample size $N$ tends to infinity), and $zeta $ is a multivariate normal random vector with mean 0 and covariance matrix V; the value of the $K imes K$ matrix V depends on the set of assumptions made about the regression model;

  2. it is possible to derive a consistent estimator $widehat{V}$ of V, that is,[eq39]where [eq40] denotes convergence in probability (again as $N$ tends to infinity). The estimator $widehat{V}$ is an easily computable function of the observed inputs $x_{i ext{ }}$and outputs $y_{i}$.

These two properties are used to derive the asymptotic distribution of the test statistics used in hypothesis testing.

Test of a restriction on a single coefficient (z test)

In a z test the null hypothesis is a restriction on a single coefficient:[eq11]where $eta _{k}$ is the k-th entry of the vector of coefficients $eta $ and $qin U{211d} $.

The test statistic is[eq42]where $widehat{V}_{kk}$ is the k-th diagonal entry of the estimator $widehat{V}$ of the asymptotic covariance matrix.

The test statistic $z_{N}$ converges in distribution to a standard normal distribution as the sample size $N$ increases. For this reason, it is called a z statistic (because the letter z is often used to denote a standard normal distribution) and the test is called a z test.

Proof

We can write the z statistic as[eq43]By assumption, the numerator [eq44] converges in distribution to a normal random variable $zeta $ with mean 0 and variance $V_{kk}$. The estimated variance $widehat{V}_{kk}$ converges in probability to $V_{kk}$, so that, by the Continuous Mapping theorem, the denominator [eq45] converges in probability to $sqrt{V_{kk}}$. Thus, by Slutsky's theorem, we have that $z_{N}$ converges in distribution to the random variable[eq46]which is normal with mean[eq47]and variance[eq48]Therefore, the test statistic $z_{N}$ converges in distribution to $z$, which is a standard normal random variable.

When $N$ is large, we approximate the actual distribution of $z_{N}$ with its asymptotic one (standard normal).

We then employ the test statistic $z_{N}$ in the usual manner: based on the desired size of the test and on the distribution of $z_{N}$, we determine the critical value(s) and the acceptance region.

The test can be either one-tailed or two-tailed. The same comments made for the t-test apply here.

The null hypothesis is rejected if $z_{N}$ falls outside the acceptance region.

Test of a set of linear restrictions (Chi-square test)

In a Chi-square test, the null hypothesis is a set of $L$ linear restrictions[eq49]where $R$ is a $L imes K$ matrix and $q$ is a $L imes 1$ vector.

The test statistic is[eq50]which converges to a Chi-square distribution with $L$ degrees of freedom. For this reason, it is called a Chi-square statistic and the test is called a Chi-square test.

Proof

We can write the test statistic as[eq51]By the assumptions on the convergence of $widehat{eta }$ and $widehat{V}$, and by the Continuous Mapping theorem, we have that[eq52]By Slutsky's theorem, we have[eq53]But $Rzeta $ is multivariate normal with mean[eq54]and variance[eq55]Thus,[eq56]but, by standard results on normal quadratic forms, the quadratic form on the right hand side has a Chi-square distribution with $L$ degrees of freedom ($L$ is the dimension of the vector $Rzeta $)

When setting up the test, the actual distribution of $chi _{N}^{2}$ is approximated by the asymptotic one (Chi-square).

Like the F test, also the Chi-square test is usually one-tailed.

The desired size of the test is achieved by appropriately choosing a critical value in the right tail of the Chi-square distribution.

The null is rejected if the Chi-square statistics is larger than the critical value.

Learn more about regression analysis

Want to learn more about regression analysis? Here are some suggestions:

How to cite

Please cite as:

Taboga, Marco (2021). "Linear regression - Hypothesis testing", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix. https://www.statlect.com/fundamentals-of-statistics/linear-regression-hypothesis-testing.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.

Featured pages
Explore
Main sections
About
Glossary entries
Share
  • To enhance your privacy,
  • we removed the social buttons,
  • but don't forget to share.