Can we use effectively use predictors of success to allocate funding?

A study on the correlation between GRE scores and other factors of success in engineering

Presented by:Melissa Lee

Department of Civil Engineering

Presentation outline

• Introduction• Literature review• Methodology, results and discussion• Conclusions

Introduction

• Limited funding opportunities for students• Funding in the form of assistantships and

fellowships helps attract the best potential students

• How do we recruit and identify students?• How do we identify those most likely to succeed

through their graduate school applications?

Introduction

• Helps us better effectively manage enrolment and offers of funding

• How do we know what we know about graduate school admission?

• What do we know about how we admit students, our admission profile, and funding profile?

Existing LiteratureLittle to No correlation

Very accurate predictor

Valid but with other factors

• Cooksey and Stenning (1981)

• Morrison and Morrison (1995)

• Powers (2004) found that there is some validity, though it is clearly accompanied by other corresponding factors

• Holt, Bleckmann et al. (2006), found that GRE scores were a fairly accurate predictor of graduate school success

• Nathan Kuncel

Cooksey, L., & Stenning, W. F. (1981). The Empirical Impact of the Graduate Record Examination and Grade Point Average on Entry and Success in Graduate School at Texas A&M University. ERIC. Holt, D. T., Bleckmann, C. A., & Zitzmann, C. C. (2006). The Graduate Record Examination and Success in an Engineering Management Program: A Case Study. Engineering Management Journal, 18(1), 10-16. Morrison, T., & Morrison, M. (1995). A Meta-Analytic Assessment of the Predictive Validity of the Quantitative and Verbal Components of the Graduate Record Examination with Graduate Grade Point Average Representing the Criterion of Graduate Success. Educational and Psychological Measurement, 55(2), 309-316. doi: 10.1177/0013164495055002015Powers, D. E. (2004). Validity of Graduate Record Examinations (GRE) General Test Scores for Admissions to Colleges of Veterinary Medicine. Journal of Applied Psychology, 89(2), 208-219.

Existing literature

• Orlando (2005) – Reviewed a range of studies between 1977 and 2001– Different findings regarding validity– Some correlation, though there is disagreement to the extent of

correlation– Difference in disciplines, skills demanded, type of admission,

age

Orlando, J. (2005). The reliability of GRE scores in predicting graduate school success: a meta-analytic, cross-functional, regressive, unilateral, post-kantian, hyper-empirical, quadruple blind, verbiage-intensive and hemorrhoid-inducing study. Ubiquity, 2005, 1-1. doi: 10.1145/1071916.1071921

Existing Literature

• Nathan Kuncel – extensive research on GRE scores as predictor of success– Meta-analysis of different factors influencing graduate school success

(2001)– Synthesized studies done on all standardized tests (2007)– Another meta-analytic study on GRE scores for Master’s and Doctoral

programs (2010)

• Existing studies discussed the validity of GRE scores, but what can we do with this knowledge?

Kuncel, N. R., & Hezlett, S. A. (2007). Standardized Tests Predict Graduate Students' Success. Science, 315(5815), 1080-1081. doi: 10.1126/science.1136618Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127(1), 162-181. doi: http://dx.doi.org/10.1037/0033-2909.127.1.162Kuncel, N. R., Wee, S., Serafin, L., & Hezlett, S. A. (2010). The Validity of the Graduate Record Examination for Master’s and Doctoral Programs: A Meta-Analytic Investigation. Educational and Psychological Measurement, 70(2), 340-352. doi: 10.1177/0013164409344508

Questions• Is there a relationship between GRE scores and GPA?

Are GRE scores an accurate predictor of success?• If GRE scores could predict success, how do we use this

knowledge to allocate funding?

Can GRE scores be used as an accurate determinant of success in graduate school?

Profile

• Civil engineering graduate students at a large, research university

• Enrolled each Fall from 2003-2013• Enrolments at three different degree

programs, Master of Engineering, Master of Science, and Doctoral program

Citizenship status

71

28

Citizenship status (% of total enrolment)

InternationalDomestic

Degree program

24

31

43 MENMSPhD

Percentage of students enrolled in each degree program

Is there a correlation between GRE scores and graduate school GPA?

• GRE scores and graduate school GPA were taken from the Fall enrolments between 2003-2013

• Total of 1,083 students sampled over this period• Multiple regression was applied across

enrollments during the 10 year period• GRE scores as the independent variable• Graduate school GPA as dependent variable

Correlation between GRE scores and GPA

  2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Correlation 0.379 0.108 0.114 0.246 0.403 0.182 0.196 0.190 0.262 0.221 0.209

Coefficient 0.143 0.012 0.013 0.061 0.162 0.033 0.038 0.036 0.069 0.049 0.044

Significance -F 0.0065 0.732 0.634 0.099 0.0002 0.131 0.102 0.129 0.033 0.069 0.043

Sample size 68 56 72 77 100 123 120 114 99 110 144

Correlation between GRE scores and GPA (international students)

  2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Correlation 0.144 0.216 0.257 0.611 0.117 0.156 0.147 0.270 0.243 0.288

Coefficient 0.021 0.047 0.066 0.374 0.013 0.024 0.022 0.073 0.059 0.083

Significance -F 0.632 0.376 0.201 0.003 0.562 0.540 0.485 0.130 0.177 0.051

Sample size 47 44 50 28 86 53 69 57 60 72

Correlation between GRE scores and GPA (domestic)

  2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Correlation 0.174 0.083 0.347 0.611 0.343 0.156 0.174 0.317 0.256 0.166

Coefficient 0.030 0.007 0.120 0.374 0.118 0.024 0.030 0.100 0.065 0.028

Significance -F 0.912 0.918 0.215 0.003 0.119 0.540 0.522 0.127 0.204 0.495

Sample size 9 28 27 28 37 53 45 42 50 53

Is there a correlation between GRE scores and graduate school GPA?

• Positive correlation, but fairly weak• Corresponds to earlier studies between

GRE scores and graduate school GPA• Kuncel’s studies support that GRE scores

have some validity with CGPA

Kuncel, N. R., & Hezlett, S. A. (2007). Standardized Tests Predict Graduate Students' Success. Science, 315(5815), 1080-1081. doi: 10.1126/science.1136618Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127(1), 162-181. doi: http://dx.doi.org/10.1037/0033-2909.127.1.162Kuncel, N. R., Wee, S., Serafin, L., & Hezlett, S. A. (2010). The Validity of the Graduate Record Examination for Master’s and Doctoral Programs: A Meta-Analytic Investigation. Educational and Psychological Measurement, 70(2), 340-352. doi: 10.1177/0013164409344508

Can GRE scores therefore be used to help allocate funding?

1.Can GRE scores be used as an accurate determinant for success in graduate school?

2.Can we identify and allocate funding to those most likely to succeed?

Questions

1. Are we using GRE scores in the selection of our graduate assistants?

2. How does our graduate assistant scores match up against those who don’t have assistantships?

3. Are we directing funding to the most successful students? (how does their GPA look against others?)

4. If we are not awarding assistantships to those who are not most likely to succeed based off GPA, then is the assistantship helping them or hurting them?

Does having an assistantship make a difference?

• Assistantship data from 2009 to 2013 is used for this determination

• Students offered either a teaching assistantship or graduate assistantship in their first year of enrolment in the program• GRE scores and Graduate school GPA• Undergraduate GPA and Graduate school

GPA

Assistantships - Correlation between GRE scores and GPA

• Total of 96 students were sampled between 2009-2013

• Only students who were offered assistantships in their first semester

• Correlation of 0.141. Positive, but a weak correlation between GRE scores and the GPAs of those given assistantships

Relationship between U-GPA and G-GPA

• Correlation of 0.04• Extremely weak correlation between

undergraduate GPA and graduate GPA for those who are on assistantships

• How else would we determine who is best suited to be teaching and research assistants?

Questions

1. Are we using GRE scores in the selection of our graduate assistants?

2. How does our graduate assistant scores match up against those who don’t have assistantships?

3. Are we directing funding to the most successful students? (how does their GPA look against others?)

4. If we are not awarding assistantships to those who are not most likely to succeed based off GPA, then is the assistantship helping them or hurting them?

Average GRE Verbal scores for those on Assistantships

• For each of the years analyzed, it was found that:

Year Median GRE - Verbal

Total number of GAs

Above Below

2009 470 32 13 17

2010 480 20 12 6

2011 490 24 12 11

2012 530 10 4 5

2013 490 12 6 4

Average GRE Quantitative scores for those on Assistantships

• For each of the years analyzed, it was found that:

Year Median GRE - Quantitative

Total number of GAs

Above Below

2009 750 32 17 11

2010 760 20 8 8

2011 760 24 12 11

2012 780 10 4 5

2013 770 12 6 4

Questions

1. Are we using GRE scores in the selection of our graduate assistants?

2. How does our graduate assistant scores match up against those who don’t have assistantships?

3. Are we directing funding to the most successful students? (how does their GPA look against others?)

4. If we are not awarding assistantships to those who are not most likely to succeed based off GPA, then is the assistantship helping them or hurting them?

Average GPA

Average GPA With assistantships No assistantships

MEN 3.69 3.58

MS 3.7 3.7

PhD 3.75 3.68

Questions

1. Are we using GRE scores in the selection of our graduate assistants?

2. How does our graduate assistant scores match up against those who don’t have assistantships?

3. Are we directing funding to the most successful students? (how does their GPA look against others?)

4. If we are not awarding assistantships to those who are not most likely to succeed based off GPA, then is the assistantship helping them or hurting them?

Average time to degree (semesters)

Average time to degree With Assistantships No Assistantships

MEN 4.5 4.37

MS 6.58 6.05

PhD 11.36 11.82

Conclusions

• We cannot rely on GRE alone in predicting future success in graduate school (determined by graduate school GPA and time to degree)

• Students on assistantships tend to do better in terms of overall grades and time taken to complete their degrees

Further research

• More study is required on how we admit students and distribute funding to students

• Bigger sample size will help give a more accurate picture of how and why students are admitted

• Indicators of success for graduate students• Can we create a matrix of sorts to help better allocate funding?

• Best practices in admission and funding• Eliman (1991) and Suhayda, Hicks and Fogg (2008) have also explored this

possibility in other disciplines

• Is graduate school admission an art or a science?• Considering the different aspects of evaluating applicants, particularly the

non-measurable factors, can we reduce each and every applicant in graduate school to a data point?

Eliman, A. A. (1991). A decision support system for Univeristy admission policies. European Journal of Operational Research, 50(2), 140-156. doi: http://dx.doi.org/10.1016/0377-2217(91)90237-PSuhayda, R., Hicks, F., & Fogg, L. (2008). A Decision Algorithm for Admitting Students to Advanced Practice Programs in Nursing. Journal of Professional Nursing, 24(5), 281-284. doi: http://dx.doi.org/10.1016/j.profnurs.2007.10.002

ReferencesAttiyeh, G., & Attiyeh, R. (1997). Testing for Bias in Graduate School Admissions. Journal of Human Resources, 32(3), 524-548.

Cieboter, F. J. (1969). Factors Related to the Performance of Foreign Graduate Students. The Journal of Educational Research, 62(8), 360-365. doi: 10.2307/27532232

Cooksey, L., & Stenning, W. F. (1981). The Empirical Impact of the Graduate Record Examination and Grade Point Average on Entry and Success in Graduate School at Texas A&M University. ERIC.

Eliman, A. A. (1991). A decision support system for Univeristy admission policies. European Journal of Operational Research, 50(2), 140-156. doi: http://dx.doi.org/10.1016/0377-2217(91)90237-P

Feeley, T. H., Williams, V., & Wise, T. (2005). Testing the Predictive Validity of the GRE Exam on Communication Graduate Student Success: A Case Study at University at Buffalo. Communication Quarterly, 53(2), 229-245. doi: 10.1080/01463370500090209

Freeman, R. B. (2005). Fellowship Stipend Support and the Supply of Science and Engineering Students: NSF Graduate Research Fellowships. American Economic Review, 95(2), 61-65.

Holt, D. T., Bleckmann, C. A., & Zitzmann, C. C. (2006). The Graduate Record Examination and Success in an Engineering Management Program: A Case Study. Engineering Management Journal, 18(1), 10-16.

House, J. D., & Johnson, J. J. (1993). Graduate Record Examination Scores and Academic Background Variables as Predictors of Graduate Degree Completion. Educational and Psychological Measurement, 53(2), 551-556. doi: 10.1177/0013164493053002025

Ingram, R. E. (1983). The GRE in the graduate admissions process: Is how it is used justified by the evidence of its validity? Professional Psychology: Research and Practice, 14(6), 711-714. doi: http://dx.doi.org/10.1037/0735-7028.14.6.711

Kuncel, N. R., & Hezlett, S. A. (2007). Standardized Tests Predict Graduate Students' Success. Science, 315(5815), 1080-1081. doi: 10.1126/science.1136618

Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127(1), 162-181. doi: http://dx.doi.org/10.1037/0033-2909.127.1.162

Marston, A. R. (1971). It is time to reconsider the Graduate Record Examination. American Psychologist, 26(7), 653-655. doi: http://dx.doi.org/10.1037/h0032124

Morrison, T., & Morrison, M. (1995). A Meta-Analytic Assessment of the Predictive Validity of the Quantitative and Verbal Components of the Graduate Record Examination with Graduate Grade Point Average Representing the Criterion of Graduate Success. Educational and Psychological Measurement, 55(2), 309-316. doi: 10.1177/0013164495055002015

ReferencesOrlando, J. (2005). The reliability of GRE scores in predicting graduate school success: a meta-analytic, cross-functional, regressive, unilateral, post-kantian, hyper-empirical, quadruple blind, verbiage-intensive and hemorrhoid-inducing study. Ubiquity, 2005, 1-1. doi: 10.1145/1071916.1071921

Powers, D. E. (2004). Validity of Graduate Record Examinations (GRE) General Test Scores for Admissions to Colleges of Veterinary Medicine. Journal of Applied Psychology, 89(2), 208-219.

Robertson, M., & Nielsen, W. (1961). The graduate record examination and selection of graduate students. American Psychologist, 16(10), 648-650. doi: http://dx.doi.org/10.1037/h0043209

Rosser, M. (1970). Criteria used for admission by graduate departments. American Psychologist, 25(6), 558-560. doi: http://dx.doi.org/10.1037/h0037862

Rubio, D. M., Rubin, R. S., & Brennan, D. G. (2003). How Well Does the GRE Work for Your University? An Empirical Institutional Case Study of the Graduate Record Examination across Multiple Disciplines. College and University, 78(4), 11-17.

Saaty, T. L., France, J. W., & Valentine, K. R. (1991). Modeling the graduate business school admissions process. Socio-Economic Planning Sciences, 25(2), 155-162. doi: http://dx.doi.org/10.1016/0038-0121(91)90013-H

Sternberg, R. J., & Williams, W. M. (1997). Does the Graduate Record Examination predict meaningful success in the graduate training of psychology? A case study. American Psychologist, 52(6), 630-641. doi: http://dx.doi.org/10.1037/0003-066X.52.6.630

Suhayda, R., Hicks, F., & Fogg, L. (2008). A Decision Algorithm for Admitting Students to Advanced Practice Programs in Nursing. Journal of Professional Nursing, 24(5), 281-284. doi: http://dx.doi.org/10.1016/j.profnurs.2007.10.002

Thornell, J. G., & Mccoy, A. (1985). The Predictive Validity of the Graduate Record Examinations for Subgroups of Students in Different Academic Disciplines. Educational and Psychological Measurement, 45(2), 415-419. doi: 10.1177/001316448504500229

Thank you!