Post on 28-Jun-2015
1
Liz Burns, Juniper NetworksJennifer Semko, Baker & McKenzie
Jack Terry, NBEORachel Schoenig, ACT
Dennis Maynes, Caveon Test Security
Lessons Learned from Using Statistics to Invalidate Scores
June 19, 2014
Caveon Webinar Series presents:
2
Agenda for Today
• Question/Answer Format
• Please use the chat window to ask questions
• Conclusions
3
To Kick Off the Discussion…
Let’s chat about why an organization would use statistics to invalidate scores.
5
Q1
If a candidate hires legal counsel and contests a score invalidation, what are the three most important things you should do?
6
Q1 - Answer
1. Don’t second guess yourself. Remain confident.
2. Be prepared to provide legal counsel with a high level summary.
3. Bring your counsel on board in a timely manner.
If a candidate hires legal counsel and contests a score invalidation, what are the three most important things you
should do?
7
Q2
In your opinion, which statistics provide the most credible information concerning potential test fraud?
8
Q2 - Answer
• Old/New or EVT* Items• Trojan Horse Items• Value• Consistent measurement• Identify candidates who benefit• Can be automated• Easy to explain/understand
In your opinion, which statistics provide the most credible information concerning potential test fraud?
*EVT = Embedded Verification Test
9
Q3
When and how do you believe one should recommend score invalidations to your board or governing body?
10
Q3 - Answer
When? When you have a good faith basis for questioning the validity of a score
How? Provide a full and frank summary; give your Board the tools they need
When and how do you believe one should recommend score invalidations to your board or governing body?
11
Q4
Describe at least three things that should be avoided when using statistics to invalidate scores?
12
Q4 - Answer
1. Using complicated statistical explanations as rationale
2. Using complex charts or graphs to illustrate your point
3. Using the word ‘cheat’, ‘cheater’, or other emotionally-charged, accusatory language
Describe at least three things that should be avoided when using statistics to invalidate scores?
13
Q5
Based on your experience, what are pros and cons to using statistics to invalidate scores?
14
Q5 - AnswerBased on your experience, what are pros and cons to
using statistics to invalidate scores?
Pros• Statistics are powerful and
illustrative.• Statistics can be applied
consistently.• Statistics can provide you
with a “targeted action” approach.
15
Q5 - AnswerBased on your experience, what are pro’s and con’s
to using statistics to invalidate scores?
Cons• Staying ahead of breaches to
your program is challenging.• Statistics are not a “silver
bullet.”• Exam security requires a
comprehensive plan that is constantly evolving.
16
Q6
Describe a time when you were successful in using statistics to invalidate a test score (or scores). What factors contributed to yoursuccess? Have you ever used only statistics to successfully invalidate a score (i.e., there was no physical or other corroborating evidence of cheating besides the statistical results)?
17
Q6 - AnswerDescribe a time when you were successful in using
statistics to invalidate a test score.
• Studied similarity data, performed analysis
• Reviewed seating charts• Identified index values >7.5• Created a report and referred
to board
18
Q7
Describe a time when you had statistical evidence of aberration, but were unsuccessful in using that evidence to invalidate the score (e.g., you lost a legal dispute or you decided not to pursue the case further for some reason). Explain why you were not successful.
19
Q7 - AnswerDescribe a time when you had statistical evidence of
aberration, but were unsuccessful in using that evidence to invalidate the score. What happened?
• Lack of corroborating evidence
• Unable to convey the compelling nature of the statistical evidence
20
Q8
If a program has a solid testing agreement in place with proctors and candidates, how much time, effort, and money do you estimate is required to invalidate one test score based on statistics?
21
Q8 - AnswerIf a program has a solid testing agreement in place with proctors
and candidates, how much time, effort, and money do you estimate is required to invalidate one test score based on statistics?
This varies based on your organization and processes followed.
• Two examples from our panelists• Example 1: Three primary areas of
expense• Example 2: Internal, automated
process
22
Q9
What internal challenges and obstacles did you confront in instituting your invalidation program? Were they operational? Legal? Political? Please describe.
23
Q9 - AnswerWhat internal challenges and obstacles did you confront in instituting your invalidation program? Were they operational? Legal? Political? Please describe.
• Political – Invalidated scores belonged to customers and partners. Stakeholders had to know what to expect and why this was beneficial.
• Legal – Is process legally defensible? Do we have all the right documentation in place?
• Operational – Continually evolving and staying on top of technologies is key.
26
Q10
Have you instituted any KPIs or other measurements around your use of statistical analyses and invalidations to measure program impact and success? What are they?
27
Q10 - AnswerHave you instituted any KPIs or other measurements around your use of statistical analyses and invalidations to measure
program impact and success? • Metrics re: cases opened, closed,
cancel rates• Metrics re: resolution options
selected by examinee• KPIs to measure effectiveness of
program
28
Q11
What legal agreements or legal foundation need to be in place in order to consider invalidating based on statistics?
29
Q11 - AnswerWhat legal agreements or legal foundation need to be in place in order to consider invalidating based on statistics?
• Clear, comprehensive, signed candidate agreements
• Ethics policies in place• Have a plan, be
comprehensive• Court will look to your agreements
30
Audience Questions
31
Audience Questions
“We see a lot of exam takers who score very high in very short amounts of time. We know that it would be impossible to even read the questions in the short time spans, but it’s hard to prove cheating.
Is there a guideline on the amount of time taken per question that we can use to take these cheaters down?”
32
Audience Question - Answer
• Some reading rates exceed 1,000 wpm (trained speed reading)
• Average adult reading rates are 250-300 wpm on non-technical content
• Reading rates should be verified• High-speed, erratic reading
strongly indicates braindump usage
Is there a guideline on the amount of time taken per question that we can use to take these cheaters down?
33
Conclusions
“I would urge us to reframe our concerns about test data integrity not as cheating concerns, but as a validity issue.”
“I would first strongly caution everyone NOT to do any analyses at all until first developing and adopting policies and procedures about what to DO with the results of any analyses.”
– Gregory Cizek, NCSA, 2013
http://www.caveon.com/tilsa-test-security-guidebook-next-steps/
34
Conclusions“Perhaps the worst situation one could be in would be a situation where analyses have been conducted and it must be admitted publically that nothing has been done with the results.”
“It is important to treat each similarly situated case the same way, and a coherent, comprehensive set of policies and procedures, uniformly applied, is essential.”
– Gregory Cizek, NCSA, 2013
http://www.caveon.com/tilsa-test-security-guidebook-next-steps/
35
Thank you!
- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”
Liz Burns, Juniper NetworksRachel Schoenig, ACT
Jennifer Semko, Baker & McKenzieDr. Jack Terry, NBEO
Dennis Maynes, Caveon Test Security