Advancing Assessment of Quantitative and Scientific Reasoning: A Progress Report on an NSF Project...
-
Upload
lee-webster -
Category
Documents
-
view
214 -
download
0
Transcript of Advancing Assessment of Quantitative and Scientific Reasoning: A Progress Report on an NSF Project...
Advancing Assessment of Quantitative and Scientific
Reasoning: A Progress Report on an NSF Project
Donna L. Sundre, Douglas Hall, Karen Smith,
Advancing Assessment of Quantitative and
Scientific Reasoning
Donna L. Sundre
Center for Assessment and Research Studies (CARS)
James Madison Universitywww.jmu.edu/assessment/
Overview of talk
Current NSF Research project
History of the test instrument
Phase I: Generalizability of the instrument
Phase II: Assessment Practice and Validity
Results from some of our partners:
James Madison University
Truman State University
St. Mary’s University
Current NSF Project
3-year grant funded by National Science Foundation: “Advancing assessment of scientific and quantitative reasoning”
Hersh & Benjamin (2002) listed four barriers to assessing general education learning outcomes: confusion; definitional drift; lack of adequate measures, and misconception that general education cannot be
measured
This project addresses all of these concerns with special emphasis on the dearth of adequate measures
Partner Institutions
Michigan State University: State-supported; Research institution
Truman State University: State-supported; Midwestern liberal arts institution
St. Mary’s University (Texas): Independent; Roman-Catholic; Hispanic Serving institution
Virginia State University: State-supported; Historically Black institution
Objectives of NSF project
Explore psychometric quality and generalizability of the QR and SR instruments
Build scientifically based assessment plans Build assessment capacity at partner institutions Develop new assessment models for adoption and adaptation Document potential barriers to assessment practice and explore
solutions Create scholarly communities of assessment practitioners to
sustain work
History of the instrument
Tests have been under development since 1997 at JMU Quantitative Reasoning (QR- 26 items) and Scientific Reasoning (SR- 49 items)
Designed to measure 8 general education learning objectives
Test information and manuals available at
http://www.jmu.edu/assessment/resources/prodserv/cbts.htm
Project phases
Phase I: First Faculty institute (conducted July 2007 at JMU); followed by data collection, identification of barriers, and reporting of results
Phase II: Assessment practice and validity studies; research questions developed at July 2008 Faculty Institute; dissemination of findings and institutional reports
Early content validity evidence
Results strongly support generalizability of test items Truman State: 100% of items mapped Michigan State: 98% (1 item not mapped) Virginia State: 97% (2 items unmapped) St. Mary’s: 92% (5 items not mapped)
Mapping of items alone is not sufficient
Balance across objectives must be obtained
Teams then created additional items to cover identified gaps in content coverage 14 for MSU; 11 for St. Mary’s; 10 for Truman State; 4 for VSU
Research at JMU
Highlights of our Findings: Grades in relevant courses are positively correlated with QR
and SR scores Student QR and SR scores improve with additional course work
AP and JMU credits show greater improvement Transfer credits do not show as marked gains
Students completing their requirements perform better than those who have not
Sophomores and juniors score higher than entering first year students
Research at JMU
Highlights of our Findings: Sophomore students who have completed 3 or 4 courses score
higher than sophomores who have not. We have established faculty ‘standards’ for performance
Many of our students are not meeting those high expectations
Of those completing requirements: QR: 70% SR: 73% These percentages are much higher than those observed for
entering students or students who have not completed their requirements
We ‘filter’ our data using motivation Effort scores This removes about 30-35 scores out of 1,100
Research Plan
Administration of QRSR to incoming freshman classes (Fall 2007 & 2008)
Administration to students with junior standing Spring 2008 and 2009
Link results to various student groups and other academic data
QRSR Freshman Results
Fall 2007
Mean QRSR (%)
Fall 2008
Mean QRSR (%)
All Freshmen 61.2 58.1
Schools
Business 60.9 57.3 b (-0.33)
Human & Soc Sci 59.1 a (-0.39) c 57.9 b (-0.27)
Sci Eng & Tech 63.8 61.5
a – p<0.05 vs. SET; b – p<0.01; c – Effect size
QRSR Score Correlations
Fall 2007 Fall 2008
SAT- Total 0.64 0.60
ACT Composite 0.69 0.63
CCTST a 0.49 0.50
a – California Critical Thinking Skills Test
QRSR Junior Results
Spring 2008 Mean QRSR Score (%)
Spring 2009 Mean
QRSR Score (%)
All Juniors 57.2 59.7
Schools Business 51.7 a (-0.63) b 53.6 a (-0.63) b
(-0.53 ) c
Human & Soc Sci 55.2 a (-0.50) 61.3
Sci Eng & Tech 62.7 63.2
a – p<0.01 vs. SET; b – vs. SET; c – vs. HSS
Test Scores and Student Motivation Level
• Student Opinion Survey (SOS) developed by JMU
• 10 items – 1-5 scale
• Score Range 10-50
• 3 scores• Effort• Importance• Total Motivation
Freshman QRSR And Motivation Levels
Motivation Level Fall 2007
Mean QRSR (%)
Fall 2008
Mean QRSR (%)
No Response 57.8 (N=93) 58.2 (N=103)
10-19 62.3 (N=7) 50.4 (N=12)
20-29 58.4 (N=85) 54.6 (N=80)
30-39 62.7 (N=196) 60.3 (N=204)
40 or higher 66.9 (N=45) 64.6 (N=54)
Spring 2009 Junior QRSR And Motivation Levels
Motivation Level Mean
QRSR (%)
Mean
Cum GPA
No Response (N=40) 62.3 2.90
10-19 (N=11) 45.6 3.05
20-29 (N=85) 57.5 3.24
30-39 (N=98) 60.6 3.11
40 or higher (N=17) 67.6 3.27
Junior QRSR And Motivation Levels
Motivation Level Mean QRSR Score (%)
Mean Total Motivation
Business 53.6 28.86 a (-0.48)
Human & Soc Sci 61.3 29.93
Sci Eng & Tech 63.2 32.08
a – p<0.05 vs. SET
Truman State University
QRSR results
Institution Characteristics
• Public liberal arts
• Highly selective
• High economic diversity
• Low ethnic diversity – predominately white
• Long history of assessment
• Good infrastructure for data collection
Questions
1. Reliability of QRSR v CAAP?
2. Correlations with number of science and quantitative classes?
3. Correlations with ACT?
4. Comparison of majors v nonmajors
5. Correlation with STAT 190 performance?
6. Comparison of Juniors’ scores to first-year students’?
QRSR Administration• Juniors
– Part of normal junior testing – Spanned two academic years: Fall 07- Spring 09– All Jr.’s participate – roughly 50/50 between JMU and
CAAP science and math– Paper-pencil administration– 2283 total
• Smaller scale study of First-year students– Invitations to instructors of first-year experience– Online administration– 135 total
• Both versions include 10 additional items for coverage of outcomes
RQ 1: How does the reliability of QRSR compare to the CAAP?
Overall reliability is comparable
CAAP: .84 - .86*
QRSR Juniors
2007-2008: .80 (calculated) 2008-2009: .81 (calculated)First-year students
Fall 2008 .86 (calculated)(*http://www.act.org/caap/pdf/handbook/Chapter4.pdf ) (No item data available)
– Reliability of outcome-based SUBSCALES (Juniors 07-08)
• Physical science outcome 1 – 38 items: .639• Physical science outcome 2 – 26 items: .579• Physical science outcome 3 – 9 items: .460• Physical science outcome 4 – 8 items: .189
• Life science outcome 1 – 38 items: .639• Life science outcome 2 – 26 items: .579• Life science outcome 3 –19 items: .563• Life science outcome 4 – 5 items: .174• Life science outcome 5 – 16 items: .518
• Math outcome 1 – 27 items: .666• Math outcome 2 –5 items: .274• Math outcome 3 – 22 items: .609• Math outcome 4 – 5 items: .439• Math outcome 5 – 5 items: .396
RQ 2: How do QRSR scores and CAAP scores correlate with number of classes taken (at
Truman) in science and quantitative areas? CAAPMath CAAPSci QRSR
#AGSC -0.053 -0.030** -0.008
#BIOL 0.156** 0.215** 0.118**
#CHEM 0.270** 0.277** 0.146**
#CS 0.203** 0.091* 0.068*
#ECON 0.127** -0.103* 0.061
#MATH 0.190** -0.011 0.054
#PHYS 0.316** 0.279** 0.148**
#POL -0.032 0.01 0.109**
#PSYC -0.068 -0.063 -0.023
#SOAN -0.151** -0.032 -0.021
#STAT 0.125** -0.053 0.039
RQ 3: How do ACT science and math subscores correlate with science and math
subscores on the two assessment instruments?
CAAP Math CAAP Sci. JMU
ACT Math .685 .543 .517
ACT Sci. .516 .635 .518
ACT Comp. .568 .655 .618
CORRELATIONS
RQ 4: Does QRSR discriminate science/math majors from non science/math majors?
Yes.• Science and Math majors : 85.3%
• Other majors average 79.7%
These differences are statistically significant for the overall score (t(584) = 5.85, p < .001) and for each of the outcome subscores.
RQ5: Does STAT 190 predict student performance of the QRSR?
Too few students without STAT 190 credit to test those with the course v those without.
Correlation with Truman STAT 190 course grades
QRSR: .318
CAAP MATH: .374
CAAP SCI: .282
RQ6: How do scores of Juniors compare to those of first-year students?
First-year scores and junior scores are significantly different, p < .01, effect size .215 (for junior 07-08)
0
10
20
30
40
50
60
70
80
First Year Juniors 07-08 Juniors 08-09
Raw
sco
re m
ean
Challenges
• Collecting data from first-year students
• Estimates of student motivation
• Sharing the model outside quantitative and scientific disciplines
• Using the data in a changing curriculum
Uses & Future directions
• Considered as part of gen ed curriculum reform
• Data analysis from 08-09 juniors continues
Thank you for coming!
Questions??
All slides will be made available from the NASPA website in a week or two.