Under Lock and Key: Conducting a Physical Security Audit John Fremer, Ph.D – President, Caveon
Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment -...
-
Upload
caveon-test-security -
Category
Education
-
view
40 -
download
0
Transcript of Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment -...
![Page 1: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/1.jpg)
www.caveon.com 1
Caveon Webinar Series
July 22, 2015
Walt Drane, MS DOEJamie Mulkey, Caveon Test Security
Steve Addicott, Caveon Test Security
Lessons Learned at the National Conference on Student Assessment
![Page 2: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/2.jpg)
www.caveon.com 2
Agenda for Today
• Important themes • Keynote• Security sessions
– Walt– Jamie
• New Resources• Q&A
![Page 3: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/3.jpg)
www.caveon.com 3
State Assessment Director Breakfast
• Wow!• State Assessments =
Politics• Common Core or No
Common Sense?• Exodus from
Consortia
![Page 4: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/4.jpg)
www.caveon.com 4
Impact of Technology on Security• Devices• Technology
Enhanced Items• Automated
Scoring• Computer
Adaptive Testing• Others?
![Page 5: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/5.jpg)
www.caveon.com 5
Keynote – Dr. Andreas SchleicherHow the U.S. and Other Countries Compare:
Lessons Learned from International Education Data
What is the OECD?
Thirty-four democracies work together promoting economic growth, prosperity, & sustainable development
What is PISA?
Test that evaluates 15-year-old students in reading, mathematics, and science, worldwide
![Page 6: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/6.jpg)
www.caveon.com 6
Keynote Key Messages
• “Math skills” vs math skills
• Problem solving skills = Career success
• Great teachers require development
![Page 7: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/7.jpg)
Developing Teaching as a profession
Recruit top candidates into the profession
Support teachers in continued
development of practice
Retain and recognise effective teachers – path for growth
Improve the societal view of teaching as a profession
Mean mathematics performance, by school location, after accounting for socio-economic status77 Capacity at the point of delivery
![Page 8: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/8.jpg)
www.caveon.com 8
Test Security Sessions…
We noticed….there weren’t many!!
![Page 9: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/9.jpg)
www.caveon.com 9
Essential Tips for Leveraging Your
Vendor’s Expertise to Strengthen the Security
of Statewide Assessments
![Page 10: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/10.jpg)
www.caveon.com 10
State Assessment Needs
• Valid and reliable score results
• If results aren’t valid, then actions may be taken – or not taken – that actually harm rather than help
![Page 11: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/11.jpg)
www.caveon.com 11
Holistic Test Security FrameworkDet
er
Detect
Decide (now and for the future)
![Page 12: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/12.jpg)
www.caveon.com 12
Vendor’s Role in DETERRING Negative Activities?Security Audits• Use the Vendor to “Audit” the Security Audit
Process• Thinking Outside the Box
Support/Consulting on policies and procedures• Use vendor to “audit” current policies and
procedures• Student Assessment Handbook, Appendix F, and
Legislation
Invalidations as a deterrent• Use the Vendor during the appeals process
![Page 13: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/13.jpg)
www.caveon.com 13
Vendor’s Role in DETECTING?
Data Forensics
• Systems that utilize sophisticated statistical analyses of test-response data to identify patterns indicative of test fraud (e.g. cheating and piracy).
• Erasure Analysis for paper and pencil assessments
• Analysis of test responses of on-line Assessments
![Page 14: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/14.jpg)
www.caveon.com 14
State’s Expectation of Vendor When DECIDING?To inform our decisions, we expect from our vendors
• Statistical analyses for invalidationsoConsistently use the data you haveoPlan for future challenges (more than
data)
• Information to support go/no go decisions for investigationsoPlay to win - don't pick a fight you are
not reasonably sure you can win
• Deep-dive data analysis and interpretation
![Page 15: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/15.jpg)
www.caveon.com 15
Test Security’s Changing Landscape in Computerized
Testing Environments
![Page 16: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/16.jpg)
www.caveon.com 16
Security in Computerized AssessmentsWhy should we still be concerned about test security with the move to computerized assessments?
![Page 17: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/17.jpg)
When Threats Occur
Before During After
Unauthorized access Computer hacking Keystroke logging Storing outside delivery system Testing interruptions Testing window Memorable items Student workstations Electronic devices Social media
![Page 18: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/18.jpg)
www.caveon.com 18
Policies and Procedures
– How do state assessment programs implement computerized assessments? What policies, procedures, and practices have they put in place?
– Focus on Exam Administration:• Assessment monitoring• Proctor training• PITA Hotline• Online training modules (25% of which
deal with test security)• Resources for
staff/teachers/coordinators
![Page 19: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/19.jpg)
www.caveon.com 19
Test Security Investigations
How can you conduct legally defensible investigations that deliver appropriate consequences?
• Having the right documentation in place
• Collecting all related information • Conducting interviews as needed• What are some of the consequences
![Page 20: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/20.jpg)
www.caveon.com 20
New Resources Made Available at NCSA
![Page 21: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/21.jpg)
www.caveon.com 21
New Resources Available
http://bit.ly/1CHUvsi
![Page 22: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/22.jpg)
www.caveon.com 22
New Resources Available
http://bit.ly/1J71f8q
![Page 23: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/23.jpg)
www.caveon.com 23
New Resources Available
http://bit.ly/1VlyARX
![Page 24: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/24.jpg)
www.caveon.com 24
New Resources Available
http://bit.ly/1RBCKWm
![Page 25: Caveon Webinar Series - Lessons Learned at the 2015 National Conference on Student Assessment - July 2015](https://reader038.fdocuments.net/reader038/viewer/2022103118/55c9c192bb61eb7f458b4616/html5/thumbnails/25.jpg)
www.caveon.com 25
Thank You!
Walt DraneState Assessment Director, Mississippi [email protected]
Steve AddicottVice President, [email protected]
Follow Caveon on twitter @caveonCheck out our blog www.caveon.com/blogLinkedIn Group “Caveon Test Security”
Jamie MulkeyVice President, [email protected]