Caveon Webinar - Weathering the Perfect Test Security Storm May 2015 edited
© Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and...
-
Upload
barnard-andrews -
Category
Documents
-
view
217 -
download
0
Transcript of © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and...
![Page 1: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/1.jpg)
© Caveon, LLC
Legal Issues in TestingLegal Issues in Testing(Part One)(Part One)
A legal framework for the appropriate and defensible use of Caveon Data Forensics™
June 21, 2006
Bob Hunt, J.D. Ph.D.Vice President of Legal Services and General Counsel
Caveon Test Security
![Page 2: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/2.jpg)
© Caveon, LLC
Overview …Overview …
What is “data forensics” The need for competitive fairness in
testing Adequacy of proctoring Legal rights to use statistics to:
monitor testing behavior cancel scores and take other actions
Examples of Concepts
![Page 3: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/3.jpg)
© Caveon, LLC
Fairness in TestingFairness in Testing
Does the test objectively evaluate relevant knowledge and skills?
Does the test provide consistent
results?
Is the testing process equally fair?
![Page 4: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/4.jpg)
© Caveon, LLC
Why is Competitive Fairness Why is Competitive Fairness Important?Important?
Without competitive fairness, tests are simply biased rituals
The validity and reliability of test results hinge on fair competition
Consistent results
(reliability)
Test Relevance (validity)
Relational Fairness(level playing field)
![Page 5: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/5.jpg)
© Caveon, LLC
The (in)Adequacy of ProctoringThe (in)Adequacy of Proctoring
Cheating behaviors are increasingly “invisible” to proctors Theft of test material Wider use of stolen test content
obtained from the Internet Wireless communications
Human proctoring is hampered by: Scope: one to many Errors of observation, memory and bias
![Page 6: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/6.jpg)
© Caveon, LLC
Data Forensics Promotes Data Forensics Promotes Competitive FairnessCompetitive Fairness
More comprehensive (no limits on effectiveness due to volume of cheating)
Fewer errors of observation, memory and bias (need to elaborate why scientific error is more trustworthy)
Flags “trusted” test results Reinforces other detection efforts
Confirm or clear other suspicions
![Page 7: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/7.jpg)
© Caveon, LLC
Caveon Data Forensics Caveon Data Forensics
“Low-resolution” analyses1.Gain-score2.Response aberrance3.Latency aberrance
“High-resolution” analyses1.Copying indices2.Erasures
![Page 8: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/8.jpg)
© Caveon, LLC
Questions?
![Page 9: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/9.jpg)
© Caveon, LLC
The $1,000,000 Legal QuestionThe $1,000,000 Legal Question
Can statistics improve test security if they don’t provide “first-hand” evidence of cheating?
![Page 10: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/10.jpg)
© Caveon, LLC
Answer: Cheating is OverratedAnswer: Cheating is Overrated Testing programs have broad authority to
manage the validity of test results Courts only require a general “lack of
confidence” in a test result Lots of (other) reasons to lack confidence in
test results: Disruption Illness Technology issues Mishandling, etc.
![Page 11: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/11.jpg)
© Caveon, LLC
“We are satisfied that the relevant public and private interests are fairly accommodated by a procedure which permits ETS to cancel scores upon an adequate showing of substantial question as to their validity, without any necessity for a showing of actual cheating or other misconduct.” Scott v. ETS (N.J. Sup.1991)
Validity vs. CheatingValidity vs. Cheating
![Page 12: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/12.jpg)
© Caveon, LLC
Courts Understand the Need for a Courts Understand the Need for a Broader SolutionBroader Solution
“To demand that ACT prove by eyewitness testimony that an individual cheated before invalidating a score would undermine ACT's primary function of providing colleges with scores that are highly reliable.”
“ACT could not possibly catch every student who cheats on its exams if it had to produce an eyewitness to confirm every instance of misconduct.”
Langston v. ACT, (11th Cir 1989)
![Page 13: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/13.jpg)
© Caveon, LLC
Question: Does that Mean that Test Question: Does that Mean that Test Scores can be Cancelled for ANY Scores can be Cancelled for ANY
Reason? Reason?
Answer: Nearly Testing programs enjoy broad
discretion in evaluating the validity of test scores
Score cancellations based on statistical evidence have been upheld
Courts that have addressed the issue directly have stated that it is not necessary to prove wrongdoing
![Page 14: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/14.jpg)
© Caveon, LLC
Implement Uncertainty!Implement Uncertainty!
“Your scores may be classified as indeterminate if the scores are at or above the passing level and the USMLE program cannot certify that they represent a valid measure of your knowledge or competence as sampled by the examination.”
U.S. Medical Licensing Board
Test retakes replace committee reviews, hearings, etc. as the chief due process opportunity
![Page 15: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/15.jpg)
© Caveon, LLC
Questions?
Case Studies: Scott v. ETS: Private/Contract law
Murray v. ETS: Public/Constitutional law
![Page 16: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/16.jpg)
© Caveon, LLC
Scott v. ETS Scott v. ETS (N.J. Sup. 1991)(N.J. Sup. 1991)
National Teachers Examination administered by ETS for the State
of New Jersey Decided on constitutional grounds
![Page 17: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/17.jpg)
© Caveon, LLC
Scott v. ETS (cont.)Scott v. ETS (cont.)
ETS detected a score gain of 42 points Scott’s second and third attempts
ETS then detected similarity between the Scott’s test-responses and those of another examinee (not seated near her).
ETS cancelled Scott’s score but offered her an opportunity to retest.
![Page 18: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/18.jpg)
© Caveon, LLC
Scott v. ETS (cont.)Scott v. ETS (cont.)
The issue: Whether Scott had been denied due
process by ETS’s failure to prove that she had actually cheated.
![Page 19: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/19.jpg)
© Caveon, LLC
Scott v. ETS (cont.)Scott v. ETS (cont.)
The Decision: “We are satisfied that the relevant public and
private interests are fairly accommodated by a procedure which permits ETS to cancel scores upon an adequate showing of substantial question as to their validity, without any necessity for a showing of actual cheating or other misconduct.”
![Page 20: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/20.jpg)
© Caveon, LLC
Scott v. ETS (cont.)Scott v. ETS (cont.) The “relevant public and private interests”
ETS: accuracy of its test results and predictions
Test-takers: no unfair advantage
School officials: reliability Public: reserving teacher certification for
those who fulfill its requirements
![Page 21: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/21.jpg)
© Caveon, LLC
Scott v. ETS (cont.)Scott v. ETS (cont.)
Meaning of “Substantial Question of validity”:
“Here ETS questioned plaintiff's scores on the basis of a statistical analysis showing hardly more than a 4 in 10 million chance that they were fairly earned … That gave ETS, and would give any other observer, substantial grounds for doubting the reliability of the scores.”
![Page 22: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/22.jpg)
© Caveon, LLC
Case Study: Murray v. ETS Case Study: Murray v. ETS (5(5thth Cir. 1999) Cir. 1999)
U. Texas El-Paso basketball recruit Needed an SAT score of 820 (of 1600) to
receive a scholarship Scored 700 on his first attempt in March Enrolled in “Testbusters” Scored 1300 in June
![Page 23: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/23.jpg)
© Caveon, LLC
Murray v. ETS (cont.)Murray v. ETS (cont.)
Test-use agreement:
“ETS reserves the right to cancel any test score if … the student engages in misconduct, if there is a testing irregularity, or if ETS believes there is a reason to question the score's validity.”
SAT I Registration Bulletin
![Page 24: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/24.jpg)
© Caveon, LLC
Murray v. ETS (cont.)Murray v. ETS (cont.)
The Decision: “The only contractual duty ETS owed to Murray
was to investigate the validity of Murray's scores in good faith.”
Good-faith means: Having a reason to lack confidence in a test
result (from statistics or some other source) Providing an opportunity to retake the test and
to present other evidence supporting scores Allowing independent review
![Page 25: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/25.jpg)
© Caveon, LLC
Murray v. ETS (cont.)Murray v. ETS (cont.)
“ETS dutifully fulfilled its contract [ ..] by following established procedures for determining the validity of questionable scores.”
![Page 26: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/26.jpg)
© Caveon, LLC
ConclusionsConclusions
Statistical analyses of test data like Caveon Data Forensics provide powerful test security information
Managing cheating with test-use agreements, statistics and retakes is within the legitimate power of testing programs to manage the “validity” of their tests
![Page 28: © Caveon, LLC Legal Issues in Testing (Part One) A legal framework for the appropriate and defensible use of Caveon Data Forensics™ June 21, 2006 Bob Hunt,](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d955503460f94a7cfeb/html5/thumbnails/28.jpg)
© Caveon, LLC