Francesco Gratton 2013 Testing in the time of crisis BILC PROFESSIONAL SEMINAR Stockholm, October 14...
-
Upload
marion-marshall -
Category
Documents
-
view
213 -
download
1
Transcript of Francesco Gratton 2013 Testing in the time of crisis BILC PROFESSIONAL SEMINAR Stockholm, October 14...
Francesco Gratton
2013
Testing in the time Testing in the time of crisisof crisis
BILC PROFESSIONAL SEMINARBILC PROFESSIONAL SEMINAR
Stockholm , October 14 - 17, 2013Stockholm , October 14 - 17, 2013
INNOVATIVE TEST DESIGNS AND FORMATSINNOVATIVE TEST DESIGNS AND FORMATS
Lt.Col. F. Lt.Col. F. GrattonGratton
Francesco Gratton
2013
Summary:Summary:
• Past situation (up to Sept 2013)Past situation (up to Sept 2013)
• New Course Of Actions adoptedNew Course Of Actions adopted
• A do-it-all softwareA do-it-all software
• ProposalsProposals
Francesco Gratton
2013
1: Multilevel test (level 1 to 4)
2: Multiple choice questions (60 for L & R)
3: No penalties for wrong answers
4: Duration: R 105’ / L 90’
5: Separate Sections (& levels)
6: # of correct answers multiplied for a coefficient
7: Potential use of “F” factor
RECEPTIVE SKILLS (Listening & Reading)RECEPTIVE SKILLS (Listening & Reading)
Stanag Proficiency Test 1.0 Stanag Proficiency Test 1.0
Francesco Gratton
2013
How STANAG levels were awarded
(Stanag Proficiency TestStanag Proficiency Test 1.0)
# of correct answers
fixed coefficient (1,66)
Multiplied by
RECEPTIVE SKILLSRECEPTIVE SKILLS
Francesco Gratton
2013
CORRECT % CORRECT % % % %
1 1,67 11 18,37 26 43,42 41 68,47 56 93,52
2 3,34 12 20,04 27 45,09 42 70,14 57 95,19
3 5,01 13 21,71 28 46,76 43 71,81 58 96,86
4 6,68 14 23,38 29 48,43 44 73,48 59 98,53
5 8,35 15 25,05 30 50,1 45 75,15 60 100,2
6 10,02 16 26,72 31 51,77 46 76,82
7 11,69 17 28,39 32 53,44 47 78,49
8 13,36 18 30,06 33 55,11 48 80,16
9 15,03 19 31,73 34 56,78 49 81,83
10 16,7 20 33,4 35 58,45 50 83,5
21 35,07 36 60,12 51 85,17
22 36,74 37 61,79 52 86,84
23 38,41 38 63,46 53 88,51
24 40,08 39 65,13 54 90,18
25 41,75 40 66,8 55 91,85
LEVEL 0 LEVEL 1
CORRECT % CORRECT % % % %
1 1,67 11 18,37 26 43,42 41 68,47 56 93,52
2 3,34 12 20,04 27 45,09 42 70,14 57 95,19
3 5,01 13 21,71 28 46,76 43 71,81 58 96,86
4 6,68 14 23,38 29 48,43 44 73,48 59 98,53
5 8,35 15 25,05 30 50,1 45 75,15 60 100,2
6 10,02 16 26,72 31 51,77 46 76,82
7 11,69 17 28,39 32 53,44 47 78,49
8 13,36 18 30,06 33 55,11 48 80,16
9 15,03 19 31,73 34 56,78 49 81,83
10 16,7 20 33,4 35 58,45 50 83,5
21 35,07 36 60,12 51 85,17
22 36,74 37 61,79 52 86,84
23 38,41 38 63,46 53 88,51
24 40,08 39 65,13 54 90,18
25 41,75 40 66,8 55 91,85
LEVEL 1
CORRECT % % CORRECT % % %
1 1,67 11 18,37 26 43,42 41 68,47 56 93,52
2 3,34 12 20,04 27 45,09 42 70,14 57 95,19
3 5,01 13 21,71 28 46,76 43 71,81 58 96,86
4 6,68 14 23,38 29 48,43 44 73,48 59 98,53
5 8,35 15 25,05 30 50,1 45 75,15 60 100,2
6 10,02 16 26,72 31 51,77 46 76,82
7 11,69 17 28,39 32 53,44 47 78,49
8 13,36 18 30,06 33 55,11 48 80,16
9 15,03 19 31,73 34 56,78 49 81,83
10 16,7 20 33,4 35 58,45 50 83,5
21 35,07 36 60,12 51 85,17
22 36,74 37 61,79 52 86,84
23 38,41 38 63,46 53 88,51
24 40,08 39 65,13 54 90,18
25 41,75 40 66,8 55 91,85
LEVEL 2
CORRECT % % % CORRECT % %
1 1,67 11 18,37 26 43,42 41 68,47 56 93,52
2 3,34 12 20,04 27 45,09 42 70,14 57 95,19
3 5,01 13 21,71 28 46,76 43 71,81 58 96,86
4 6,68 14 23,38 29 48,43 44 73,48 59 98,53
5 8,35 15 25,05 30 50,1 45 75,15 60 100,2
6 10,02 16 26,72 31 51,77 46 76,82
7 11,69 17 28,39 32 53,44 47 78,49
8 13,36 18 30,06 33 55,11 48 80,16
9 15,03 19 31,73 34 56,78 49 81,83
10 16,7 20 33,4 35 58,45 50 83,5
21 35,07 36 60,12 51 85,17
22 36,74 37 61,79 52 86,84
23 38,41 38 63,46 53 88,51
24 40,08 39 65,13 54 90,18
25 41,75 40 66,8 55 91,85
LEVEL 2 LEVEL 3
CORRECT % % % % CORRECT %
1 1,67 11 18,37 26 43,42 41 68,47 56 93,52
2 3,34 12 20,04 27 45,09 42 70,14 57 95,19
3 5,01 13 21,71 28 46,76 43 71,81 58 96,86
4 6,68 14 23,38 29 48,43 44 73,48 59 98,53
5 8,35 15 25,05 30 50,1 45 75,15 60 100,2
6 10,02 16 26,72 31 51,77 46 76,82
7 11,69 17 28,39 32 53,44 47 78,49
8 13,36 18 30,06 33 55,11 48 80,16
9 15,03 19 31,73 34 56,78 49 81,83
10 16,7 20 33,4 35 58,45 50 83,5
21 35,07 36 60,12 51 85,17
22 36,74 37 61,79 52 86,84
23 38,41 38 63,46 53 88,51
24 40,08 39 65,13 54 90,18
25 41,75 40 66,8 55 91,85
LEVEL 3 LEVEL 4
Francesco Gratton
2013
1: Functional language assessed in a global manner
2: Structured interview
3: Tailored to the candidate
4: Checks & probes
5: 1 to 2 role plays
SPEAKING (SPEAKING (holistically assessed holistically assessed ))
PRODUCTIVE SKILLSPRODUCTIVE SKILLSPRODUCTIVE SKILLSPRODUCTIVE SKILLSProficiency Test 1.0Proficiency Test 1.0Proficiency Test 1.0Proficiency Test 1.0
Francesco Gratton
2013
Three tasks (one for each level)
PRODUCTIVE SKILLSPRODUCTIVE SKILLSPRODUCTIVE SKILLSPRODUCTIVE SKILLSJFLT 1.0JFLT 1.0JFLT 1.0JFLT 1.0
WRITING (WRITING (holistically assessed holistically assessed ))
Francesco Gratton
2013
Summary:Summary:
• Past situationPast situation
• New COAs adoptedNew COAs adopted
• A do-it-all softwareA do-it-all software
• ProposalsProposals
Francesco Gratton
2013
•Specifications Specifications
•Cut-off score Cut-off score
• Joint DatabaseJoint Database
New COAs adoptedNew COAs adopted
Francesco Gratton
2013
Francesco Gratton
2013
• PURPOSEPURPOSE• PURPOSEPURPOSE
• ADMINISTRATION PROCEDURESADMINISTRATION PROCEDURES• ADMINISTRATION PROCEDURESADMINISTRATION PROCEDURES
• VALIDATION PROCEDURESVALIDATION PROCEDURES• VALIDATION PROCEDURESVALIDATION PROCEDURES
• TEST FORMATTEST FORMAT• TEST FORMATTEST FORMAT
• LEVELS OF LINGUISTIC KNOWLEDGELEVELS OF LINGUISTIC KNOWLEDGE• LEVELS OF LINGUISTIC KNOWLEDGELEVELS OF LINGUISTIC KNOWLEDGE
• TEST CONTENTTEST CONTENT• TEST CONTENTTEST CONTENT
Francesco Gratton
2013
1: Multilevel test (level 1 to 4)
2: Multiple choice questions (60 for L & R)
3: No penalties for wrong answers
4: Duration: R 105’ / L 90’
5: Separate Sections (& levels)
6: # of correct answers multiplied by a coefficient
7: Potential use of “F” factor
RECEPTIVE SKILLS (Listening & Reading)RECEPTIVE SKILLS (Listening & Reading)
Stanag Proficiency Test 1.0 Stanag Proficiency Test 1.0 TWAS HIGH
TIME!!!
Francesco Gratton
2013
Stanag Proficiency Test 2.0Stanag Proficiency Test 2.0New Key-factors New Key-factors
• Each section is a mini-test (L & R)
• Plus levels
• Percentages
• Each section is a mini-test (L & R)
• Plus levels
• Percentages
Francesco Gratton
2013
•Specifications Specifications
•Cut-off score Cut-off score
• Joint DatabaseJoint Database
New COAs adoptedNew COAs adopted
Francesco Gratton
2013
Section 1 –Stanag level 1 (questions from 1 to 15)
No.Correct answ.
Level awarded
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0 0+ 1Section 2 –Stanag level 2 (questions from 16 to 30)
No.Correct answ.
Level awarded
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 1+ 2Section 3 –Stanag level 3 (questions from 31 to 45)
No.Correct answ.
Level awarded
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2 2+ 3
Section 4 –Stanag level 4 (questions from 46 to 60)
No.Correct answ.
Level awarded
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
3 3+ 4
JFLT 2.0: RDS (Listening & Reading)
Francesco Gratton
2013
EXAMPLE Level 1 (15 questions) Level 2 (15 questions) Level 3 (15 questions)
Max score: 45
Level 1 (15 questions) Level 2 (15 questions) Level 3 (15 questions)
Max score: 45
All candidates answer correctly to 30 questionsAll candidates answer correctly to 30 questionsAll candidates answer correctly to 30 questionsAll candidates answer correctly to 30 questions
Old test Old test Old test Old test WithWithWithWith they would all get thethey would all get thethey would all get thethey would all get the same scoresame scoresame scoresame score
Francesco Gratton
2013
CANDIDATO CORRECT ANSWERS
LEVEL 1(15 Questions)
LEVEL 2(15 Questions)
LEVEL 3(15 Questions) FINAL LEVEL
BIANCHI 30 10 10 10 3ROSSI 30 11 10 9 2+VERDI 30 13 12 5 2GIALLI 30 15 8 7 1+ARANCIONI 30 15 6 9 1
Francesco Gratton
2013
•Specifications Specifications
•Cut-off score Cut-off score
• Joint DatabaseJoint Database
New COAs adoptedNew COAs adopted
Francesco Gratton
2013
NEW LISTENING & READING ITEMSNEW LISTENING & READING ITEMS
JDB (JOINT DATA BASE)JDB (JOINT DATA BASE) JOINT EFFORTJOINT EFFORT
HOW NOT TONOT TO MAKE JOINT EFFORTSHOW NOT TONOT TO MAKE JOINT EFFORTS
Francesco Gratton
2013
Test-writers involvedTest-writers involved
• Accustomed to military environmentAccustomed to military environment
• Language Testing Seminar Language Testing Seminar
• QualifiedQualified
• Norming SessionsNorming Sessions
Francesco Gratton
2013
THE JDB FLOW
THE JDB FLOW
CARABINIERICARABINIERIARMYARMY
AIR FORCE
AIR FORCE
NAVYNAVY
Francesco Gratton
2013
SEP OCT NOV DEC JAN FEB MAR APR MAJ JUN JUL2012 2013
1st
Group
• PREP
ARATION O
F
FIRST
BATCH O
F
ITEM
S
•SENT T
O OTH
ER
GROUP
Phase1
TEST
ERS M
EET
FOR 1°
REVISI
ON
Phase2
PRE-
TEST
ING
REVISI
ON TRIA
LLIN
#
50 IT
EMS
(-30
%)
Phase3
FINAL
MODIFI
CATION
APPROVAZIO
NE
Phase4
AUGWG GdL GdL GdL
2nd
Group
FEBRUARY 201320 NEW ITEMS.
INTO JDB
•PREP
ARATION O
F
FIRST
BATCH O
F
ITEM
S
•SENT T
O OTH
ER
GROUP
Phase1
TEST
ERS M
EET
FOR 1°
REVISI
ON
Phase2
Phase3
PRE-
TEST
ING
REVISI
ON TRIA
LLIN
#
50 IT
EMS
(-30
%)
Phase4
FINAL
MODIFI
CATION
APPROVAZIO
NE
•PREP
ARATION O
F
FIRST
BATCH O
F
ITEM
S
•SENT T
O OTH
ER
GROUP
Phase1
TEST
ERS M
EET
FOR 1°
REVISI
ON
Phase2
Phase3
PRE-
TEST
ING
REVISI
ON TRIA
LLIN
#
50 IT
EMS
(-30
%)
3rd
Group
4th
Group •REP
ERIM
ENTO
• PREP
ARAZIONE
# 12
0 TTIV
AZIONI
•INVIO
AD A
LTRO
GRUPPO
Phase1
MAJ 201340 NEW ITEMS.
INTO JDB
NOVEMBER 201380 NEW ITEMS.
INTO JDB
TIMINGSTIMINGS
AUGUST 201360 NEW ITEMS.
INTO JDB
Francesco Gratton
2013
Summary:Summary:
• Past situationPast situation
• COA (specs, JDB, cut-off score)COA (specs, JDB, cut-off score)
• A do-it-all softwareA do-it-all software
• ProposalsProposals
Francesco Gratton
2013
WHAT’S THE DIFFERENCE ?WHAT’S THE DIFFERENCE ?
Francesco Gratton
2013
PC-assessment-related terminologyPC-assessment-related terminologyTerm Definition Stakes
Assessment Any systematic method of obtaining evidence (through questions) for a purpose.
Quiz … measures for the purpose of providing feedback to the student. Low
Survey … to determine needs required to fulfill a defined purpose. Low
Test … measures knowledge for the purpose of informing the student on their current level
Medium
Exam … measures knowledge for the purpose of documenting the current level of knowledge High
Francesco Gratton
2013
the software is used for:the software is used for:
• Needs analysis:(surveys) • Placement test• Any training activity• Assessment:
– First level survey – Post-course– Pre-certification – CertificationCertification
Francesco Gratton
2013
Software’s system Software’s system Schema Schema
Software’s system Software’s system Schema Schema
Create questions
And organize them in
tests using a windows
based PC
Francesco Gratton
2013
Assessment ……Assessment ……
via Browser
QuestionManager
AssessmentManager
AssessmentDefinitions
Questions
… allows to choose:
• Time limits
• Feedback to test-taker
• Styles (Template)
• Jumps
• Question shuffling
• Instructions to test-
takers
… allows to choose:
• Time limits
• Feedback to test-taker
• Styles (Template)
• Jumps
• Question shuffling
• Instructions to test-
takers
On Windows PC
QuestionManager
AssessmentManager
Francesco Gratton
2013
AssessmentsAssessments
• … also created with
authoring manager by
selecting Qs previously
created
• Any question can be
chosen from the
database
• … also created with
authoring manager by
selecting Qs previously
created
• Any question can be
chosen from the
database
AssessmentDefinitions
Questions
su PC Windows
QuestionManager
AssessmentManager
via Browser
QuestionManager
AssessmentManager
Francesco Gratton
2013
Software’s system Software’s system Schema Schema
Software’s system Software’s system Schema Schema
Create questions
And organize them in
tests using a windows
based PC
Set security parameters,
schedule assessment and link to
other (Learning
Management Systems
assessment published using any browser, secure browsers, or a PC/MAC
Result reports, CIA, graphs, gimmicks, you name it …..
Francesco Gratton
2013
Types of Questions: Multiple ChoiceTypes of Questions: Multiple ChoiceTypes of Questions: Multiple ChoiceTypes of Questions: Multiple Choice
Francesco Gratton
2013
Likert Scale (for questionnaires) Likert Scale (for questionnaires)
Francesco Gratton
2013
Essay QuestionEssay Question
1. Candidate can write free text in the space provided
2. Testers will evaluate later
1. Candidate can write free text in the space provided
2. Testers will evaluate later
Francesco Gratton
2013
Francesco Gratton
2013
Summary:Summary:
• Past situationPast situation
• COA (specs, JDB, cut-off score)COA (specs, JDB, cut-off score)
• A do-it-all softwareA do-it-all software
• Proposals Proposals
Francesco Gratton
2013
• Wide projectWide project
• Sharing experience & Sharing experience &
capabilitiescapabilities
• Optimization of resourcesOptimization of resources
• No alternative to B.A.T.No alternative to B.A.T.
Testing in the time Testing in the time of crisisof crisis
Francesco Gratton
2013
Francesco Gratton
2013
A Bilateral-based CDB
A multilateral-based CDB
COMBINED DATABASE
or …
or …
Francesco Gratton
2013
Francesco Gratton
2013
FLOWFLOW
TIME SCHEDULETIME SCHEDULE
SPECSSPECS1 2
3
Francesco Gratton
2013
Francesco Gratton
2013
Thank youThank you
[email protected]@gmail.com
[email protected]@sclingue.esercito
.difesa.it.difesa.it