Regression analysis aims at constructing relationships between a single dependent or response variable and one or more independent or predictor variables, and is one of the more widely used methods in data analysis. Although the computations and analysis that underlie regression analysis appear more complicated than those for other procedures, simple analyses are quite straightforward.

The general model that underlies regression analysis is

data = predictable component + unpredictable component

"Data" in this case are the observed values of the dependent variable, the predictable component consists of the predictions generated by the regression equation, and the unpredictable component consists of the "residuals" or unpredictable parts of the data.  The general idea in regression analysis is to move information into the predictable component, leaving the unpredictable component with no information or pattern.

Regression basics

• the regression model and alternative representations
• other quantities in regression analysis (fitted values, residuals, sums of squares)
• "fitting" the regression equation

Example

• example of the fitting and diagnostic analysis of a regression model

Examining the regression equation

This examination leads to an assessment of the "strength" of the relationship between the response variable and the predictor variable(s).  Two hypothesis tests are examined

• a test of the null hypothesis that the proportion of the variance of the response variable "explained" by the predictor variable(s) is not significant (an F-test, analogous to the one in ANOVA of the hypothesis that the group means are not different)
• individual tests of the null hypotheses that the estimated regression coefficients (b₀, b₁) are equal to zero.

The two hypotheses are tested by examining the regression equation and related statistics

• decomposition of individual deviations
• how strong is the relationship? (F-test, R²)
• are the coefficients significant? (t-tests and standard errors of the predictions)

The fit of the regression model can also be displayed by plotting confidence intervals (which allow variability in the regression line to be visually assessed) and prediction intervals (which allow variability in the data to be assessed).

Readings: 

Kuhnert & Venebles (An Introduction...):  p. 109-120;  Maindonald (Using R...):  ch. 5