Regression
-
Upload
buddy-krishna -
Category
Technology
-
view
2.428 -
download
9
Transcript of Regression
Multiple regression analysis And
Stepwise regression
History:The earliest form of regression was the method of least squares, which was published by Legendre in 1805, and by Gauss in 1809.
The term "regression" was used by British biometrician sir Francis Galton in the (1822-1911), to describe a biological phenomenon.
Sir Galton's work on inherited characteristics of sweet peas led to the initial conception of linear regression.
Introduction: Regression is a statistical technique for investigating and modeling the relationship between variables.
Applications of regression are numerous and occur in almost every field, including engineering, the physical and the social sciences, and the biological sciences.
Usually, the investigator seeks to ascertain the causal effect of one variable upon another—the effect of a price increase upon demand, for example, or the effect of changes in the money supply upon the inflation rate.
Definition: Regression is the measure of the average relationship between two or more variables in terms of the original units of the data. It is unquestionably the most widely used statistical technique in social sciences. It is also widely used in biological and physical science. Regression equation is (y) =a + b xSlope (b) = (NΣXY-(ΣX)( ΣY)) / (NΣX2 – (ΣX)2)Intercept (a) = (ΣY-b(ΣX)) / N
Review of Simple linear regression. A simple linear regression is carried out to estimate the relationship between a dependent variable, Y and a single explanatory variable, x given a set of data that includes observations for both of these variables for a particular population.
•For ex: A real estate agent wishes to examine the relationship between the selling price of a home and its size (measured in square feet)•A random sample of 10 houses is selected
Dependent variable (Y) = house price Independent variable (X) = square feet
Simple Linear Regression Model
ii10i εXββY Linear component
Population
Y intercept
Population SlopeCoefficient
Random Error term
Dependent Variable
Independent Variable
Random Error component
i10i XbbY
The simple linear regression equation provides an estimate of the population regression line
Estimate of the regression intercept
Estimate of the regression slope
Estimated (or predicted) Y value for observation i Value of X
for observation i
The individual random error terms ei have a mean of zero
Prediction equation is given by:
21
xx
yyxx
SS
SS
i
ii
xx
xy
xy 10ˆˆ
Estimation of coefficients:
Where
Measures of Variation
Total variation is made up of two parts:
SSE SSR SST Total Sum of Squares
Regression Sum of Squares
Error Sum of Squares
2i )YY(SST 2
ii )YY(SSE 2i )YY(SSR
where:
= Average value of the dependent
variable
Yi = Observed values of the dependent
variable
i = Predicted value of Y for the given Xi
value
Y
Y
Measures of Variation
Xi
Y
X
Yi
SST = (Yi - Y)2
SSE = (Yi - Yi )2
SSR = (Yi - Y)2
__
_
Y
Y
Y_Y
Coefficient of Determination, r2
• The coefficient of determination is the portion of the total variation in the dependent variable that is explained by variation in the independent variable
• The coefficient of determination is also called r-squared and is denoted as r2
1r0 2 note:
squares of sum
squares of regression2
total
sum
SST
SSRr
Multiple linear regression
Introduction: The general purpose of multiple regression (the term was first used by Pearson, 1908) is to learn more about the relationship between several independent or predictor variables and a dependent or criterion variables.
Definition: A regression model that involves the relationship between two or more explanatory variables and a response variable by fitting a linear equation to observed data (more than one regressor variable) is called a multiple regression model. Every value of the independent variable x is associated with a value of the dependent variable y. Suppose that the yield in the pounds of conversation in a chemical process depends on temperature and the catalyst concentration. A multiple regression model that might describe the relationship is
y=β0+β1x1+β2x2+ε
where y denotes the yield,x1denotes the temperature,x2
denotes the catalyst concentration. This is multiple linear regression model with two regressor variables. The term linear is used because equation is a linear function of the known parameters β0,β1& β2 and ε is error term.
The parameter β1 indicates that the expected
change in response (y) per unit change in x1 when x2 is held
constant. Similarly β2 measures the expected change in (y)
per unit change in x2 when x1 held constant.
In general, the response y may be related to k regressor (or) predictor variables. The model y= β0+β1x1+β2x2+……………+ βkxk+ε
is a multiple linear regression with k regressors. The parameters βj, j=0,1,…….k. are called regression coefficients.
The parameter βj represents the expected change in the response
(y) per unit change in xj when all of the remaining regressor variables xi
(i≠j) are held constant. For this reason the parameters βj, j=1,…….k are
often called partial regression coefficients.
Assumptions of Regression
• For any given set of values of x1, x2, … , xk, the random error has a probability distribution with the following properties:
• 1. Mean equal to 0
• 2. Variance equal to 2
• 3. Normal distribution
• 4. Random errors are independent
Regression Analysis: Model Building
• General Linear Model
• Determining When to Add or Delete Variables
• Analysis of a Larger Problem
• Multiple Regression Approach
to Analysis of Variance
General Linear Model
Models in which the parameters (β0, β1, . . . , βp) all have exponents of one are called linear models.
• First-Order Model with One Predictor Variable
y x 0 1 1y x 0 1 1
Variable Selection Procedures
• Stepwise Regression
• Forward Selection
• Backward Elimination
Iterative; one independent
variable at a time is added or
deleted Based on
the F statistic
Variable Selection Procedures• F Test
• To test whether the addition of x2 to a model involving x1 (or the deletion of x2 from a model involving x1and x2) is statistically significant
F0=MSR/MSRes
(MSR=SSR/K)
The p-value corresponding to the F statistic is the criterion used to determine if a variable should be added or deleted
(SSE(reduced)-SSE(full))/ number of extra termsMSE(full)
F
Forward Selection
• This procedure is similar to stepwise-regression, but does not permit a variable to be deleted.
• This forward-selection procedure starts with no independent variables.
• It adds variables one at a time as long as a significant reduction in the error sum of squares (SSE) can be achieved.
Backward Elimination
• This procedure begins with a model that includes all the independent variables the modeler wants considered.
• It then attempts to delete one variable at a time by determining whether the least significant variable currently in the model can be removed because its p-value is less than the user-specified or default value.
• Once a variable has been removed from the model it cannot re enter at a subsequent step.
Stepwise regression:
Procedure of simultaneous forward and backward selection also available
In a stepwise regression, predictor variables are entered into the regression equation one at a time
based upon statistical criteria. At each step in the analysis the predictor variable that
contributes the most to the prediction equation in terms of increasing the multiple correlation, R, is
entered first. This process is continued only if additional variables add anything statistically to the
regression equation.
The choosing is done according to following manner
i.e.) delete x.i if ^i2 E(2 )(Z1
lZl)-1 ii<Fout=F1,n-r-1 pout
enter x.j if (n-r-2)cjq
2 (cjjcqq-cjq2) > Fin=F1,n-r-2 pin
here either pin or pout are specify the stepwise procedure is terminated when either of the two
following points happens
We can’t enter or delete the variables according to the above criteria i.e.) this includes the case where enter all regressor & can’t delete any.
The processor dictates that the same regressor be enter and deleted in successive operations the stepwise selection procedure is an attempt to
achieve to insert variables in terms until the regression equation is satisfactory.
When additional predictor variables add anything statistically meaningful to the regression equation, the analysis stops. Thus, not all
predictor variables may enter the equation in stepwise regression.
There are a number of multiple regression variants. Stepwise is usually a good choice though one can enter all variables simultaneously as an alternative. Similarly, one can enter all of the variables simultaneously and gradually eliminate predictors one by one if elimination does little
to change the overall prediction.Stepwise regression procedure is the best procedure
when compared to the all procedures we have see earlier.
Uses of Regression Analysis:1.Regression analysis helps in establishing a functional
Relationship between two or more variables.
2. Since most of the problems of economic analysis are based on cause and effect relationships, the regression analysis is a highly valuable tool in economic and business research.
3. Regression analysis predicts the values of dependent variables from the values of independent variables.
4. We can calculate coefficient of correlation (r) and coefficient of determination (R2) with the help of regression coefficients.
ANOVA TABLE:Source Degrees of freedom Sum of squares Mean Square F Regression 2 5550.8166 2775.4083 4.7*10-16 Residual 22 233.7260 10.6239
Total 24 5784.5426
R2 = 0.9596 Adjusted R2 = 0.9559
Scatter plot for cases and distance