What are Best Practices in Large Scale Testing? A New Book: 2013 Edition Outlines Those Practices...
31
What are Best Practices in Large Scale Testing? A New Book: 2013 Edition Outlines Those Practices Operational Best Practices for Statewide Assessment Programs Update – 2013 Edition 1
-
Upload
dorcas-sharp -
Category
Documents
-
view
213 -
download
0
Transcript of What are Best Practices in Large Scale Testing? A New Book: 2013 Edition Outlines Those Practices...
What are Best Practices in Large Scale Testing?
A New Book: 2013 Edition Outlines Those Practices
Operational Best Practices for Statewide Assessment Programs
Update – 2013 Edition
1
Introductions
2
Presenters
Christopher Hanczrik – State of WA, Office of Superintendent of Public Instruction (Co-Chair)
Pat Roschewski – Data Recognition Corporation
Sarah Susbury – Virginia Department of Education
Lisa R. Ehrlich – Measured Progress (Co-Chair)
Outline
3
Background
Chapter 0 - Assessment Program Design (Pat)
Chapter 9 - Test Administration (Sarah)
Chapter 15 - Technology-based Assessments and Technical Support (Sarah)
Chapter 22 - System and Program Interoperability (Lisa)
Chapter 8 – Security (Christopher)
Background – 2010 Edition
4
Began in June of 2006 - USDOE, ATP, CCSSO, AAP
A collaborative effort between CCSSO and ATP - Testing Directors and Service Providers together at the same table
Focus was predominately to define and describe Best Practices in operational fulfillment of paper-based large-scale, high-stakes assessment
- Includes examples and implementation suggestions
- Articulate what to do but not how to do it
- These are not standards but practices
- Voluntary, inclusive and nonproprietary
- Released July 2010 and available for purchase on ATP/CCSSO websites (thousands of copies sold)
2013 Edition
5
Convened new working group in January 2011 Focus was review existing document and expand or
augment technology-based and operational assessment topics
Author new and/or significantly expand existing chapters to address technology specific topics
- Interoperability
- Assessment Program Design
- Security
- Technology-Based Assessments
- Accessibility focus, especially for technology-based assessments
- Online Reporting
2013 Edition - Status
6
Completion of review draft – September 2012
Stakeholders review October – November 2012
- Internal review by ATP “Heads of Houses” and staffs, as well as CCSSO, State DOEs, EIMAC, Assessment Directors.
Public review February – April 2013
Release of 2013 Edition – expected August
How do vendors use the OBP?
7
2010 Edition of the Operational Best Practices is typically used in a number of different ways.
TrainingOnboarding of incoming team membersRecurring “recertification” training for existing team members
Program Design
- Test Design - Pilot Testing
- Delivery Model - Distribution Requirements
- Item Bank Development - Reporting Plan
8
Process DevelopmentInternal QA procedures across each departmentSecurity protocols throughout the program cycleProgram reporting requirements both internally and externally
Client/Service Provider DialogueProvides foundation and continued reference source
How do we use the OBP?
Pat RoschewskiSenior Director of Education Solutions
Data Recognition Corporation
Chapter “Zero” – Assessment Design
9
Chapter “Zero” – Assessment Design
10
The chapter’s name – defines its purpose
Had I only known...
Designed for state directors – new and veteran
Chapter “Zero” – Assessment Design
11
Checklist of topics, considerations – pre-RFP
Each “topic” forecasts a chapter
The chapter order...assessment cycle
New name of Chapter “0”
Pre-Chapter: State Considerations for Assessment
Program Design
Sarah SusburyDirector of Test Administration, Scoring, and Reporting
Virginia Department of Education
Chapter 9. Test Administration
Chapter 15 - Technology-based Assessments and Technical Support
12
Chapter 9 - Test Administration
13
Paper-based, Technology-based, and Mixed Mode Testing
Calendars and Schedules
Defined Roles at the State, District, and School Levels. Test Administration Technology
Test Administration Directions for Students
Test Environment Protocols
Materials and Procedures for Training and Test Administration
Chapter 15 - Technology-based Assessments and Technical Support
14
Previously Chapter 15 was named “Online Assessment and Technical Support”
Content of the chapter transitioned from aspirational to best-practices
Infrastructure
Devices and Software
Assistive Technology
Chapter 15 - Technology-based Assessments and Technical Support
15
Security
Technical Support
Configuration of Testing Environments
Training
System Performance and Monitoring
Change Management
What to do, not how to do it.
Lisa EhrlichCOO
Measured Progress Inc.
Chapter 22.
System and Program Interoperability
16
Interoperability
17
System
- Hardware
- Operating Systems
- Application software
Assessment System
- Data
- Reports
Networks and Data transport
Industry Standards
18
Content
Data/Reporting
Integration Components- Seamless- Content – TEI- Delivery- Online reporting's
Student Registration- Student related data- Accessibility- SIS
Standardized Processes for Assessment Systems
19
System Components
- Hardware – Servers, interface devices
- Software – operating systems, applications, SIS, delivery , APIs
- Network – WAN, LAN, Wireless, Cloud
Migration Paths for legacy components
Data/Content Interoperability Considerations
20
Item Development
Item Banking
Delivery
SIS
Accessibility
Scoring
Results
Interoperability Standards
21
SIF – School Interoperability Framework
- XML data formant
IMS Global
- QTI [Question and Test Interoperability Specification]
- APIP [Accessibility Portable Item Protocol]
CEDS – Common Education Data Standards
AIF – Assessment Interoperability Framework
Christopher HanczrikState of WA
Office of Superintendent of Public Instruction
Chapter 8. Security
22
Chapter 8. Security
23
So what is so new about test security?
Paper and technology-based assessments addressed together and separately
Chapter 8. Security
24
Define and design a comprehensive security plan
Internal processes
Computer systems
Electronic transfers
Test administration
Chapter 8. Security
25
How to reduce the likelihood of a security breach
How to deal with a security breach when it occurs
Chapter 8. Security
26
Protect your assessment program, your organization –client and service provider
Chapter 8. Security
27
Develop the Plan
Implement it!!
How does this book benefit us?
28
All states are in transition- Assessment Consortium- New ESEA legislation- Enhancements to system
Service provider and state perspectives – learning from each other - Partnership
“Hot button issues” – security, technology-based testing
How else does this book benefit us?
29
Additional players joining the large scale, high-stakes assessment environment:
- Data systems & network specialists
- Hardware & software developers
- Technology leaders
- New assessment leaders
Additional need for current and up-to-date information:- “examples and implementation suggestions”
- “non-proprietary views”
- “what to do but not how to do it”
- operational best practices
Questions?
30
Thank YouIf you should have any questions, please contact
[email protected] or [email protected]
Please complete the session evaluation that has been distributed to you.
31
