Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd
description
Transcript of Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd
![Page 1: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/1.jpg)
COPYRIGHT WESTED, 2011
Integrating Simulation-Based Science Assessments into Balanced State Science
Assessment Systems:Findings and Implications
Richard VineyardNevada Department of Education
Edys Quellmalz and Matt SilberglittWestEd
Presented to the Council of State Science Supervisors, San Francisco, CA, March 9, 2011
Enhanced Assessment Grant
![Page 2: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/2.jpg)
COPYRIGHT WESTED, 2011
Enhanced Assessment Grant State Partners
• Nevada (Lead State) • Utah • North Carolina • Massachusetts • Connecticut • Vermont
![Page 3: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/3.jpg)
COPYRIGHT WESTED, 2011
Advisors
• Rebecca Kopriva, Wisconsin Center for Education Research–Assessments for English Learners
• Martha Thurlow, National Center on Education Outcomes–Assessments for Students with
Disabilities
![Page 4: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/4.jpg)
COPYRIGHT WESTED, 2011
Subcontractors
• WestEd STEM (SimScientists)– Project Management– Content Development
• Redhill Studios– Interactive (Flash) Programming
• UCLA/CRESST– External Evaluation
![Page 5: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/5.jpg)
COPYRIGHT WESTED, 2011
EAG Goals
• Establish the technical quality of simulation-based assessments – for two topics, ecosystems and force and motion
• Study the feasibility and utility of simulation-based assessments in classrooms from 3 states
![Page 6: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/6.jpg)
COPYRIGHT WESTED, 2011
EAG Goals
• Study the measurement of science knowledge and inquiry by simulation-based curriculum-embedded assessments and unit benchmark assessments in comparison to traditional items for:– All students– ELLs– Students with disabilities
• Propose alternative models for integrating simulation-based assessments into state assessment systems
![Page 7: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/7.jpg)
COPYRIGHT WESTED, 2011
Relevance to Current Assessment Programs
• Need innovative, technology-enhanced assessments that align with new frameworks that focus on – fewer, deeper, more integrated core knowledge
targets • E.g., the new Framework for Science Education and
next generation national science standards • Models as structures for understanding and studying
science systems (model-based learning)• Science practices for using knowledge and inquiry in
significant, recurring authentic tasks
![Page 8: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/8.jpg)
COPYRIGHT WESTED, 2011
Relevance to Current Assessment Programs
Need innovative, technology-enhanced assessments that• Target 21st century skills within STEM• Use technology to engage students in use of 21st
century “tools of the trade” • Provide evidence supporting one type of technology-
enhanced performance assessment– For summative and formative purposes – Called for in the RTT Consortia
![Page 9: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/9.jpg)
COPYRIGHT WESTED, 2011
EAG Design
• Align to national science standards and to science standards of the pilot states– Ecosystems and force & motion– At middle school level– Nevada, North Carolina, Utah
• Accommodations– Extended time– Audio– Zoom
![Page 10: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/10.jpg)
COPYRIGHT WESTED, 2011
Ecosystem Target Model
System Model Levels
Model LevelDescriptions
Content Targetsby Model Level
Inquiry Targets
Components What are the components of the system and their rules of behavior?
Every ecosystem has a similar pattern of organization with respect to the roles (producers, consumers, and decomposers) that organisms play in the movement of energy and matter through the system.
Use principles to identify role of organisms.
Interactions How do the the individual components interact?
Matter and energy flow through the ecosystem as individual organisms participate in feeding relationships within an ecosystem.
Observe interactions among organisms.
Emergent Behaviors What is the overall behavior or property of the system that results from many interactions following specific rules?
Interactions among organisms and among organisms and the ecosystem’s nonliving features cause the populations of the different organisms to change over time.
predict observe explain investigate
![Page 11: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/11.jpg)
COPYRIGHT WESTED, 2011
Multiple Modes of RepresentationActive Inquiry
Organism box shows icons that appear and disappear
Graph shows size of population over time.
Data inspector shows population for one point on graph.
Table shows population at start, end, and point selected by data inspector.
![Page 12: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/12.jpg)
COPYRIGHT WESTED, 2011
EMBEDDED IN CLASSROOM INSTRUCTION
Assessment Module Components for Each Topic/System
Online assessment without feedback
Teacher scores constructed responses
Bayes Net
Proficiency report
Benchmark Summative Unit Assessments
Online assessment with feedback and coaching
Follow up Classroom Reflection ActivityProgress report
Embedded Assessments for Formative Purposes (2 to 4)
![Page 13: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/13.jpg)
COPYRIGHT WESTED, 2011
Ecosystem Embedded Assessment: Feedback,
Coaching, and Self-AssessmentDemo
![Page 14: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/14.jpg)
COPYRIGHT WESTED, 2011
Progress Reports to Students
![Page 15: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/15.jpg)
COPYRIGHT WESTED, 2011
Progress Reports to Teachers
NH = needs help P = making progress OT = on track
![Page 16: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/16.jpg)
COPYRIGHT WESTED, 2011
Grouping Recommendations for Classroom Reflection Activity
![Page 17: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/17.jpg)
COPYRIGHT WESTED, 2011
Classroom Reflection Activity
• Formative use of assessment results– Students assigned to teams based on embedded results
• Transfer to different, more complex system• Jigsaw structure
– Allows differentiated instruction via tasks of varying difficulty– Promotes small and large group discourse and collaboration
• Guidance for teacher– Teacher review of key points in simulation– What to look for during group work and questions to pose in response– Presentations– Evaluation of presentations
![Page 18: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/18.jpg)
COPYRIGHT WESTED, 2011
Benchmark Assessment:Assess Transfer to New Context
Demo
![Page 19: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/19.jpg)
COPYRIGHT WESTED, 2011
Benchmark Report: Class Summary
![Page 20: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/20.jpg)
COPYRIGHT WESTED, 2011
Benchmark Report: Student Details
![Page 21: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/21.jpg)
COPYRIGHT WESTED, 2011
Samples
• 28 think aloud sessions• Pilot testing in 4 states
• 60 participating teachers• ~5,800 students• 40 schools• 28 districts– Response data on the assessments, post tests and
classroom observations
![Page 22: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/22.jpg)
COPYRIGHT WESTED, 2011
Research Evidence• Quality of the assessments (external review)
– Alignment with standards and curriculum– Coherence– Science content and inquiry– Item design
• Implementation– Feasibility in schools and classrooms– Utility for students and teachers
![Page 23: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/23.jpg)
COPYRIGHT WESTED, 2011
Research Evidence
• Technical Quality (Validity & Reliability)– Alignment with science standards– Cognitive labs– Analyses of responses
• Comparison of student responses on simulation-based and traditional items– Cognitive labs �– IRT model fits�
![Page 24: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/24.jpg)
COPYRIGHT WESTED, 2011
Reliability of Benchmark and Post Tests
• Ecosystems– Benchmark (n=3844)
• Mean score was 28.14 (max 39)• Standard Deviation 6.64• Coefficient Alpha: 0.85
– Post Test (n=3422)• Mean score: 16.52 (max 30) • Standard deviation: 6.68• Coefficient Alpha (KR 20): 0.88
![Page 25: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/25.jpg)
COPYRIGHT WESTED, 2011
Reliability of Benchmark and Post Tests
Force and Motion– Benchmark (n=1504)
• Mean score: 28.15 (max 40) • Standard Deviation: 5.51• Coefficient Alpha: 0.79
– Post Test (n=2166)• Mean score: 14.87 (max 28) • Standard Deviation: 5.73• Coefficient Alpha (KR 20): 0.82
![Page 26: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/26.jpg)
COPYRIGHT WESTED, 2011
Correlation of Benchmark to Posttest
Ecosystems(n=2924)
Force and Motion(n=1496)
ContentAbility
estimate
InquiryAbility
estimate
ContentAbility
estimate
InquiryAbility
estimateCorrelation with ability estimates on posttest
.51** .24** .40** .45**
** Correlation is significant at the 0.01 level (2-tailed)
![Page 27: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/27.jpg)
COPYRIGHT WESTED, 2011
CRESST Evaluation Questions
• How effective was SimScientists implementation?
• Was SimScientists feasible and useful in middle school classrooms?
• How effective were SimScientists’ accommodations?
![Page 28: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/28.jpg)
COPYRIGHT WESTED, 2011
CRESST Evaluation Methods
• Convenience sample of teachers from all three pilot states, both topics– Eight teachers with 3-4 classes– Five schools
• Observation and interview protocols– Assessments– Reflection Activities
• Teacher survey
![Page 29: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/29.jpg)
COPYRIGHT WESTED, 2011
Teacher’s Role (Assessment)
![Page 30: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/30.jpg)
COPYRIGHT WESTED, 2011
Class Organization (RA)
![Page 31: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/31.jpg)
COPYRIGHT WESTED, 2011
CRESST Interviews
“I think that [students] are way more engaged … using this kind of interactive assessment … it was more engaging them simply, “Here is a pencil, here is a paper, answer questions” .”
![Page 32: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/32.jpg)
COPYRIGHT WESTED, 2011
CRESST Interviews• “…I like that kind of feedback when it doesn’t just go
to the next page and they don’t know whether they did it right or not.”
• “…Yes, the science content is really being tested. Students are asked to conduct experiments, investigate, and draw conclusions and to use scientific skills. Students are not able to guess on the multiple choice questions because it probes them until they choose the right answer. Students are also taught about food webs in one biome and they are tested on another biome.”
![Page 33: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/33.jpg)
COPYRIGHT WESTED, 2011
CRESST Evaluation Conclusions
• Observations showed that students were actively engaged most of the time during assessments.
• Both teachers and students generally believed that the SimScientists program was beneficial to learning.
• Teachers found the automatically scored, immediate feedback—especially the reports generated by the questions—helpful to students. The instant reports allowed teachers to easily see which questions students had the most difficulty with so that they could tailor their lessons accordingly.
![Page 34: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/34.jpg)
COPYRIGHT WESTED, 2011
CRESST Evaluation Conclusions
• Teachers collectively agreed that the simulation assessments had greater benefits than traditional paper-and-pencil tests because of the simulation’s instant feedback, interaction, and visuals.
• Teachers agreed that the assessments would be useful in measuring their individual state standards.
![Page 35: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/35.jpg)
COPYRIGHT WESTED, 2011
Multilevel Balanced Assessment System Models
• Reporting benchmark results alongside district and state data
• Matrix sampling of short “signature” tasks from different topics
![Page 36: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/36.jpg)
COPYRIGHT WESTED, 2011
![Page 37: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/37.jpg)
COPYRIGHT WESTED, 2011
Matrix Sampling Short “Signature Tasks” Across Topics
![Page 38: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/38.jpg)
COPYRIGHT WESTED, 2011
Policy Brief
• Summary of alternative models for integrating simulation-based assessments into balanced state assessment reporting systems
• Summary of study findings on feasibility, utility, and technical quality
• Implications for further study
![Page 39: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/39.jpg)
COPYRIGHT WESTED, 2011
Related Activities• Increased use of technology in classroom instruction and
assessment• Continued (and growing) frustration around measuring
student’s knowledge and skills above conceptual level• Growth in costs associated with hand scoring of CR items on
science assessments at grades 5 and 8• Development of electronic and on-line resources for
instruction, assessment, and professional development for math and science – MINES, Science Builder, Pathways to Nevada’s Future
![Page 40: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/40.jpg)
COPYRIGHT WESTED, 2011
Other Projects• Race to the Top• Assessment consortium related to CCSS in
math and reading• Multilevel Assessments of Science Standards• Science Assessments with UDL • EAG proposal for 2011?
– Study series of assessment suites within a grade
![Page 41: Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd](https://reader038.fdocuments.net/reader038/viewer/2022110214/56815ac8550346895dc897a8/html5/thumbnails/41.jpg)
COPYRIGHT WESTED, 2011
Contact information
Email: [email protected]@[email protected]
Website: www.simscientists.org