1 Examination Accreditation Program Exam Security: Preventing Exam Espionage Saturday, October 1...
-
Upload
ashlynn-webb -
Category
Documents
-
view
218 -
download
2
Transcript of 1 Examination Accreditation Program Exam Security: Preventing Exam Espionage Saturday, October 1...
1
Examination Accreditation Program
Exam Security:Preventing Exam Espionage
Saturday, October 1Baltimore – Maryland - USA
2
Exam Security: Introducing the Presenters
During Test Development
Lawrence J. Fabrey, Ph.D.
During Test AdministrationDave Moore
After the TestChris Beer
Tadas Dabsys
3
Benefits of ARELLO-Accreditation
Independent Evaluation Confidence and Protection Defensible Standards Reasonable Consistency in Quality Facilitates License Recognition
www.ARELLO.org (ARELLO Guidelines for Accreditation)
4
ARELLO Additional Resources
About Exam Development An Accreditation Overview ARELLO® Matters to Consider in Exam RFP ARELLO® Top Ten Questions and Complaints on
Licensing Exams ARELLO® Supreme Court Case - Exam Validity ARELLO® Examination Accreditation Program
- White Paper on Determining Minimum Passing Scores
5
Security During Examination Development
• Physical Environment • Electronic Access • Staffing • Inventory Storage• Subject Matter Experts• Other Functions
• Translation, Printing, Artwork, etc.
6
Development – Physical Access
• Secure Room/Area • Vault – Key Access • Limited Authorized Access• Chain-of-Custody• Shredding – including notes/drafts
7
Development – Electronic Access
• Passwords & Screensavers• Selective Access Permission • Specific Time Period• Read-only, non-print, non-export• Webinars
8
Development - Staffing
• Staff • Confidentiality Agreements • Conflict of Interest • Audits
9
Development - SME’s and Other Functions
• Subject Matter Experts • Translators• Artwork• Supplies for Practical Exams• Printers• Couriers
10
Development – Inventory and Distribution
• Printing (Serial #’s, Shrink wrap)• Shipping (Traceable, Armed Courier)• Electronic Transmission• Web Access
11
Security During Exam Administration
• Paper & Pencil • Computer-based Testing• Practical Exams• Open-book Exams• Special Accommodations • Facilities & Staff• Candidates
12
Administration – Paper & Pencil
• Printing • Shipping Receipt and Inventory• Storage• Contingency Plan• Return Shipment & Inventory
13
Administration - CBT
• CBT Centers • Restricted Login Access• Restricted Time Periods• Restricted Printing• Cameras, DVR’s
14
Administration – Special Administrations
• Practical Exams • Raters (SME) – Conflict of Interest• Supplies
15
Administration – Open-book
• Open-book Exams • Notes• Highlight/Underline• Post-its/Dividers• Screening
16
Administration - Facilities
• Limited Access• Spacing Candidates• Monitor Restroom Breaks• Secure Storage - Supplements• Special Accommodations
17
Administration - Staffing
• Conflict of Interest• Self-regulating• Restricted Access (Email, Internet)• Audits• Training – Active Proctoring• Termination
18
Administration - Special
• Special Accommodations• Interpreters
19
Administration - Candidates
• Candidates• Identification• Roster Signature• Fingerprint Login• Clothing/Purses• Activity/Mannerisms
20
Available through SkyMall and Web
21
Caught on Camera
Smart phone
Notes
22
I know I will be Caught
23
24
Show Me the Money!
25
Security After the Test
• Data Forensics• Monitoring Websites• Item Updating• Facility Screening• Audits• Incident Reporting
26
After – Data Forensics - Items
p-value: Percentage of candidates who answer an item correctly
pbis: Correlation between performance on an item and performance on the overall exam
option stats: Percentage of candidates who answer each option correctly
option pbis: Correlation between performance on the options and the overall exam
27
After – Data Forensics - Items
p-value: Too high, too low
pbis: Too low
options:Distractors with no responses
Distractors with more responses than key
O pbis: Distractors higher than key
Omits Above expectations
28
After – Data Forensics - Candidates
Suspicious Candidate Activity
Test for significant differences in pass rates Better performance on harder items High scores in low test times Suspicious improvement in scores
29
After – Monitoring Websites
• Cram Schools• Sample Exams• False Advertising• Social Networking
30
After – Item Updating
• Monitor Item Statistics• Replace Items• Add New Items
31
After – Facility Screening
• Look for Notes in Testing Station• Screen Restrooms and Hallways
32
After - Audits
• Test Center Audits• “Secret Shopper” Programs• Inventory• Continuing Education for Proctors
33
After – Incident Reporting
• Documentation• Evidence• Reporting to Client
34
Examples
• Cheating Incidents
35
ARELLO Accredited Providers
Applied Measurement Professionalshttp://www.goamp.com
Pearson VUEhttp://www.pearsonvue.com
PSI Services LLChttp://psionline.com
36
QUESTIONS?COMMENTS!
Fabulous Prize Giveaway