1st NRC Meeting, 16-19 October 2006, Amsterdam 1 ICCS 2009 Field Operations.
-
Upload
matthew-johns -
Category
Documents
-
view
220 -
download
0
Transcript of 1st NRC Meeting, 16-19 October 2006, Amsterdam 1 ICCS 2009 Field Operations.
1 1st NRC Meeting, 16-19 October 2006, Amsterdam
ICCS 2009 Field Operations
2 1st NRC Meeting, 16-19 October 2006, Amsterdam
Field Operations Overview
National Context Questionnaire
Item Pilot
Field Trial Operations Overview
Online Data Collection
Support Materials
Field Trial Timeline
Next Steps
Contact Information
3 1st NRC Meeting, 16-19 October 2006, Amsterdam
National Context Questionnaire
Language will be English only
NRCs will have to gather information from several sources
Data for National Context Survey will be collected online
NRCs can login as often as required
Questionnaire will be online end of 2006/beginning of 2007
– NRCs will receive email with individual access information
4 1st NRC Meeting, 16-19 October 2006, Amsterdam
Item Pilot
Submit Items to ACER via email at [email protected]
Review Items and commenting on study website http://iccs.acer.edu.au
End of reviewing process mid January 2007
5 1st NRC Meeting, 16-19 October 2006, Amsterdam
Field Trial Operations Overview
6 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
7 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
8 1st NRC Meeting, 16-19 October 2006, Amsterdam
Adapt Questionnaires & DocumentAdaptations
TranslationVerification
Layout Verification
TranslateQuestionnaires& Verifiy within
Country
Correct &Assemble
Questionnaires
CorrectQuestionnaires(if necessary)
Send Out
9 1st NRC Meeting, 16-19 October 2006, Amsterdam
NRCIEA Sec./ISC/DPCtime
Time planning before administration
Layout Verification
Implement changes &assemble Instruments
Translation Verification
Translate Instruments
2 weeks
4 weeks
3 weeks
2 weeks
4 weeks
Start data collection15
weeks
Instruments Send Out
Printing
10 1st NRC Meeting, 16-19 October 2006, Amsterdam
due dateNRC/ISC/IEAtime
Time planning before administration
Layout Verification
Implement Changes &Assemble Instruments
Translation Verification
Translate Instruments
4 weeks
3 weeks
2 weeks
4 weeks
Instruments Send Out 2. July 2007
30. July 2007
27. August 2007
17. September 2007
1. October 2007
15. October 2007Start data collection
Print Instruments2 weeks 15. October 2007
11 1st NRC Meeting, 16-19 October 2006, Amsterdam
Meanwhile…
12 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
13 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
14 1st NRC Meeting, 16-19 October 2006, Amsterdam
Contact Schoolsand get
within SchoolInformation
Perform Within-School
Sampling
Prepare List of Schools
and send it to DPC
Recruit and Train Test Administrators
DPC provides list of sampled
Schools
Recruit and Train Scorers
Translate School Coordinator Manual
Translate Test Administrator Manual
15 1st NRC Meeting, 16-19 October 2006, Amsterdam
School Contact
Send sampling frame (list of schools) to DPC
Sampling of schools will be done by DPC preferably
List of sampled schools will be provided by DPC
Contact schools, ensure participation
Identify liaison person (School Coordinator)
Provide School Coordinator Manual
Record status for each sampled school in WinW3S
Adaptation and translation of School Coordinator and Test Administrator Manual (no verification on international level, modifications to some extend possible)
16 1st NRC Meeting, 16-19 October 2006, Amsterdam
Within-School Sampling (Students)
Two levels: students and teachers
Identification of classes on Class Listing Forms
– Class name
– Number of students
Random within-school sample of classes
Identification of students within sampled classes on Student Listing Forms
Send list of sampled students to School Coordinator on Student Tracking Form
WinW3S serves as survey administration tool
17 1st NRC Meeting, 16-19 October 2006, Amsterdam
Within-School Sampling (Teachers)
Identification of eligible teachers on Teacher Listing Forms
Within-school random sample of teachers
Send Teacher Tracking Form to School Coordinator
18 1st NRC Meeting, 16-19 October 2006, Amsterdam
Within-School Sampling
WinW3S will support NRCs with
– Listing of classes / teachers / students
– Sampling of classes / teachers
– Assignment of forms
– Survey administration
– Tracking of participation
Will be subject to Data Management Seminar
19 1st NRC Meeting, 16-19 October 2006, Amsterdam
Assigning Questionnaires
School Questionnaire
Teacher
– Every Teacher will receive one questionnaire
Student
– Booklet
– Questionnaire (perception and background part in one form)
– Regional Modules instrument(s)
WinW3S will assign forms and provide labels for questionnaires
20 1st NRC Meeting, 16-19 October 2006, Amsterdam
NRCSchool
Time Planning before Administration
time
Contact Schools
Decision to participate
Send Listing Forms
List Students/Teachers
Sample Teachers
Send Tracking Forms
Send Questionnaires
Start data collection
Name School Coordinator
Sample Classes
List Classes
21 1st NRC Meeting, 16-19 October 2006, Amsterdam
School ContactInstrumentstime
Time Planning before Administration
Contact Schools
Send Listing Forms
Sample Teachers
Send Tracking Forms
Send Questionnaires
Implement changes & assemble Quest.
Translate Instruments
Printing
Data collection
Sample Classes
22 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
23 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
24 1st NRC Meeting, 16-19 October 2006, Amsterdam
Adapt Codebooks To National Questionnaires
Document Adaptations inNational Adaptaion Forms
25 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
26 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
27 1st NRC Meeting, 16-19 October 2006, Amsterdam
Monitor Test Administrators
Ad hoc Support
Arrange for Material Receipt
Train Scorers
Train Data Entry Staff
28 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
29 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
30 1st NRC Meeting, 16-19 October 2006, Amsterdam
Organize Material for (direct) Data Entry
Organize Material for Scoring and Reliability Scoring
Check if Information and Material is complete
Get back to School
if necessary
Enter Participation Information
31 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
32 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
33 1st NRC Meeting, 16-19 October 2006, Amsterdam
Conduct and Monitor Scoring
Conduct and Monitor
Reliability Scoring
Organize Material forData Entry
34 1st NRC Meeting, 16-19 October 2006, Amsterdam
Scoring
All open ended items have to be scored according to scoring guide
Reliability scoring for open ended items for a certain amount of booklets
Subject to Scoring Training Seminar in 2007
35 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
36 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
37 1st NRC Meeting, 16-19 October 2006, Amsterdam
Complete Participation Information
Verify Data
Enter Data
38 1st NRC Meeting, 16-19 October 2006, Amsterdam
Data Entry
Data entry software (WinDEM) and international codebooks for the FT are provided by the DPC in time
National staff should be trained on data entry, even if all or parts are being outsourced
Computer workstations need to be set up
National Centers enter and verify data from paper questionnaires
Adapt codebooks to reflect national conditions (e.g. ISCED levels)
39 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
40 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
41 1st NRC Meeting, 16-19 October 2006, Amsterdam
Set of Final Instruments
National Codebooksand Documentation (NAF)
VerifiedData sets
TrackingForms
Reliability Coding Data
42 1st NRC Meeting, 16-19 October 2006, Amsterdam
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
43 1st NRC Meeting, 16-19 October 2006, Amsterdam
Online Data Collection
44 1st NRC Meeting, 16-19 October 2006, Amsterdam
Online Data Collection (ODC)
Used for all questionnaires that National Centers should answer
– National Context Questionnaire
– Survey Activities Questionnaire
International Option for
– School Questionnaire
– Teacher Questionnaire
Countries that decide to use ODC in Main Study are obliged to conduct Field Trial in a split half design
Ensure fall-back to paper questionnaire
True mixed mode design
Goal: reduce response burden
– Ask schools for preferred mode (in Main Study)
– Mode assignment for all questionnaires of one type possible
45 1st NRC Meeting, 16-19 October 2006, Amsterdam
Codebook Adaptation
InstrumentPreparation
Sampling and PreparingAdministration
Data Entry and
Verification
Ad
min
istr
ati
on
Rece
ivin
g M
ate
rial
Data
Sen
doutCodebook Adaptation
Prepare ODC
Monit
or
OD
C
46 1st NRC Meeting, 16-19 October 2006, Amsterdam
School contactInstrumentstime
Time planning before administration
Contact Schools
Sample Students / Teachers
Send Tracking Forms
Send Questionnaires
Implement changes & assemble Quest.
Translate Instruments
Printing
Data collection
PrintingPrepareODC
Send Quest./Cover Letters
Send Listing Forms
Sample Classes
47 1st NRC Meeting, 16-19 October 2006, Amsterdam
Online Data Collection (ODC) - Facts
Based on concept of cultures
All languages supported (character sets, text directions,…)
Participants receive “cover letter”
– Label with login information provided by WinW3S
– Culture specific URL
– Personal username and password
– Sent to school coordinator for distribution
Participants can log in and out as often as desired
Same administration period as paper questionnaires
48 1st NRC Meeting, 16-19 October 2006, Amsterdam
Support Material
49 1st NRC Meeting, 16-19 October 2006, Amsterdam
Support Material
Survey Operations Manual (FT)– Part 1: Survey Operations Overview, Dates and Activities– Part 2: Survey Administration
School Coordinator ManualTest Administrator ManualWithin School Sampling Software (WinW3S) + ManualIEA SurveySystem Designer + ManualNational Quality Control Monitor Manual
– Part 3: Data EntryScoring GuideCodebooksWindows Data Entry Manager (WinDEM) + Manual
Distribution of all materials and software packagesthrough study website only
50 1st NRC Meeting, 16-19 October 2006, Amsterdam
FT Timeline
51 1st NRC Meeting, 16-19 October 2006, Amsterdam
Field Trial Timeline
Receive instruments & NAF by 2 July 2007
Submit translated questionnaires with NAF v1 for international translation verification by 30 July 2007
Receive verified instruments by 27 August 2007
Submit for layout verification by 17 September 2007
Start printing and distribution of questionnaires by 1 October 2007
Proposed FT administration period (subject to country conditions):15 October – 16 November 2007
Submit data to DPC by end of 2007
52 1st NRC Meeting, 16-19 October 2006, Amsterdam
Next Steps
53 1st NRC Meeting, 16-19 October 2006, Amsterdam
Next Steps
Send items to ISC via email to: [email protected]
Review and comment on items at http://iccs.acer.edu.auuntil Mid January 2007
Complete National Context Questionnaire beginning of 2007
Set up team at national center and plan for– Staff– Translation– School Contact– Test Administrators– Scorers– Data Management (Data Entry and Verification)
NRC meeting in June 2007
Data Management Seminar in July 2007
Be prepared to receive Field Trial instruments in July 2007
54 1st NRC Meeting, 16-19 October 2006, Amsterdam
Contact Information
55 1st NRC Meeting, 16-19 October 2006, Amsterdam
Contact Information
Field Operations / Data Management:
IEA Data Processing Center
Falk Brese Michael Jung
56 1st NRC Meeting, 16-19 October 2006, Amsterdam
Thanks for your attention!