1 3rd NRC Meeting, 9-12 June 2008, Windsor

Data Management Procedures

Michael Jung, IEA Data Processing and Research Center

2 3rd NRC Meeting, 9-12 June 2008, Windsor

Contents

Receipt of FT Data and Documentation

FT Data Processing at the IEA DPC

Evaluation of FT Data Quality

WinW3S - Within-School Sampling Software

WinDEM – Data Entry Manager

SurveySystem – On-line Data Collection Software

Material Submission after Test Administration

MS Data Processing at the IEA DPC

3 3rd NRC Meeting, 9-12 June 2008, Windsor

Receipt of FT Data and Documentation

4 3rd NRC Meeting, 9-12 June 2008, Windsor

Data from 32 countries altogether– 16 data sets arrived on time

– 16 data sets arrived after January, 25th

Receipt of FT Data and Documentation

5 3rd NRC Meeting, 9-12 June 2008, Windsor

25. Nov.

5. Dez.

15. Dez.

25. Dez.

4. Jan.

14. Jan.

24. Jan.

3. Feb.

13. Feb.

23. Feb.

4. Mrz.

14. Mrz.

24. Mrz.

3. Apr.

13. Apr.

23. Apr.

3. Mai.

13. Mai.

23. Mai.

0 5 10 15 20 25 30

Receipt of FT Data

25-Jan

6 3rd NRC Meeting, 9-12 June 2008, Windsor

Data from 32 countries altogether

– 16 data sets arrived on time

– 16 data sets arrived after January, 25th

Documentation from 32 countries altogether

– 10 countries sent on time and complete

– 22 countries sent after January, 25th and complete

Receipt of FT Data and Documentation

7 3rd NRC Meeting, 9-12 June 2008, Windsor

18. Dez

28. Dez

07. Jan

17. Jan

27. Jan

06. Feb

16. Feb

26. Feb

07. Mrz

17. Mrz

27. Mrz

06. Apr

16. Apr

26. Apr

06. Mai

16. Mai

0 5 10 15 20 25 30

25-Jan

Receipt of FT Documentation

8 3rd NRC Meeting, 9-12 June 2008, Windsor

FT Data Processing at the IEA DPC

9 3rd NRC Meeting, 9-12 June 2008, Windsor

Data Processing at the IEA DPCData Processing at the IEA DPC

Data & Codebook

sInstrument

s & Forms

INP

UT

IEA DPC

Country

1. Import to SAS and adapting the

file structure

10 3rd NRC Meeting, 9-12 June 2008, Windsor

Adapting the File StructureAdapting the File Structure

WinW3S database and WinDEM data files have been merged, renamed and converted to SAS data files

Unique system numbers (CSYSTEM, TSYSTEM, SSYSTEM) have been added

National Scores and derived variables (e.g. final participation indicator) have been added

National variables have been removed (will be sent to countries in a separate data file on demand)

11 3rd NRC Meeting, 9-12 June 2008, Windsor

Structure of the Merged Data Structure of the Merged Data SetSet

Variable ValueIDPUNCH 1

...,ITPART1..2,

IDSCHOOL,

IDCLASS,...

3...

8001

800101

IDSTUD 80010101

IDCHECK 373

TOKEN01 3

CS2GSEX 9

SSYSTEM,... 000001

Information:

TrackingForms Information:

StudentQuestionnair

e

Addedduring

cleaningprocess

Example: Student Questionnaire Data File

WinDEM(ISGxxxF2.DBF)

12 3rd NRC Meeting, 9-12 June 2008, Windsor

Data Processing at the IEA Data Processing at the IEA DPCDPC

Country Databas

eNew Data

Structure

Reports, Statistic

s & Docu.

INP

UT

OU

TP

UTStructur

e Check

Background

Cleaning

IEA DPC

Country

ID Cleaning

Linkage Cleanin

g

Data & Codebook

sInstrument

s & Forms2. Cleaning

13 3rd NRC Meeting, 9-12 June 2008, Windsor

Evaluation of FT Data Quality

14 3rd NRC Meeting, 9-12 June 2008, Windsor

Evaluation of FT Data QualityEvaluation of FT Data Quality

General data quality in almost all countries was VERY GOOD

Most common problems– Not all changes in the instruments reflected in the

codebooks

– Not administered variables deleted in codebooks instead of hidden

– WinDEM quality control checks not performed or not all problems have been solved

– Inconsistencies between Student Questionnaire files and Occupation data files

– Information in WinW3S not completed and thus database not exported

15 3rd NRC Meeting, 9-12 June 2008, Windsor

WinW3S - Within-School Sampling Software

16 3rd NRC Meeting, 9-12 June 2008, Windsor

WinW3S

Management tool for all ICCS within-school sampling operations

Identifies the schools, classes, teachers and students that will participate in ICCS

Creates a hierarchical ID system + passwords for ODC

Samples classes + teachers

Assigns booklets to the students

Produces and prints all forms and labels

17 3rd NRC Meeting, 9-12 June 2008, Windsor

WinW3S – What's NewNew option for participation in Asian Module added

Import of students and teachers from MS Excel files

New participation code for students added: „participated, but excluded”

New participation code for teachers added: „teacher on temporary leave”

Assignment of reliability booklets changed: random assignment of ~100 R booklets per booklet type

Assignment of different class IDs for ICCS Target Grade and Additional Grade students

Listing of teachers birth information and gender optional on Teacher Listing Forms

Possible to list more than 99 teachers per school

Teacher and school principal questionnaire mode can be changed on individual level– ODC optional ONLY for countries that have trialed ODC in FT

Participation status for on-line participants can be imported from SurveySystem Monitor

18 3rd NRC Meeting, 9-12 June 2008, Windsor

WinDEM – Data Entry Manager

19 3rd NRC Meeting, 9-12 June 2008, Windsor

WinDEM

Provides mask for data entry from booklets and paper questionnaires

Based on codebooks, provided by the IEA DPC

Allows adaptation of codebooks

Validates data during data entry

Provides various data verification checks after data entry

Allows for comparison of double-entered data

20 3rd NRC Meeting, 9-12 June 2008, Windsor

WinDEM – What's New

New coding module for coding of occupations integrated in WinDEM– Entry of occupation data directly from WinDEM– Occupation data will be stored in an MS Access database– Text answers and occupation codes also available in

table mode

21 3rd NRC Meeting, 9-12 June 2008, Windsor

IEA SurveySystem

On-line Data Collection Software

22 3rd NRC Meeting, 9-12 June 2008, Windsor

ODC – What's New

Export tool in SurveySystem Monitor application for participation status of teachers and school principals– data can be imported into WinW3S later on

23 3rd NRC Meeting, 9-12 June 2008, Windsor

Material Submission after Test Administration

24 3rd NRC Meeting, 9-12 June 2008, Windsor

Material to be submitted

WinW3S database exported for IEA DPC Sampling

(MICS<country>1.dbf)(MICS<country>2.dbf)

Send Material to IEA DPC Sampling

National Adaptation Forms (IV) (*.xls)

Set of final National Instruments (*.pdf)

Send Material to ISC at ACER

Send Material toIEA DPC

25 3rd NRC Meeting, 9-12 June 2008, Windsor

Material to be submitted

Send Material to IEA DPC Sampling

Send Material to ISC at ACER

Send Material toIEA DPC

National Adaptation Forms (IV) (*.xls)

Set of final National Instruments (*.pdf)

WinW3S database exported for IEA DPC

(MICS<country>1.mdb)(MICS<country>2.mdb)

Student and Teacher Tracking Forms (paper copies or scanned)

26 3rd NRC Meeting, 9-12 June 2008, Windsor

Material to be submitted

Send Material to IEA DPC Sampling

Send Material to ISC at ACER

Send Material toIEA DPC

WinDEM data files (*.dbf) + WinDEM codebooks (*.sdb)

(ICGICSM2.DBF; *SDB)

(ITGICSM2.DBF; *SDB)

(ISGICSM2.DBF; *SDB)

(ISAICSM2.DBF; *SDB)

(ISRICSM2.DBF; *SDB)

(ISEICSM2.DBF; *SDB)

(ISLICSM2.DBF; *SDB)

(ISSICSM2.DBF; *SDB)

Occupation data file(ISOICSM2.mdb)

27 3rd NRC Meeting, 9-12 June 2008, Windsor

Material to be submitted

Data and Documents have to be submitted

by 27-Mar-2009 latest for SH countries and

by 29-Aug-2009 latest for NH countries

IEA DPC Sampling

ISC at ACER

IEA DPC

28 3rd NRC Meeting, 9-12 June 2008, Windsor

File-Exchange using the IEA DPC WebFTP-Server

The IEA DPC has setup secure FTP sites to exchange materials between countries and the - ISC at ACER and/ or- IEA Secretariat and/ or - IEA DPC

Can easily be accessed using a browser (e.g. Internet Explorer)

Individual country logins from FT are still valid

FT folders and documents have been moved to a folder “0_Old_Material_from_Field_Trial “ please do not upload any documents into that folder any more

29 3rd NRC Meeting, 9-12 June 2008, Windsor

File-Exchange using the IEA DPC WebFTP-Server

MS Materials should be uploaded using a predefined folder structure1_Instruments_and_NAF_for_Adaptation_Verification_ACER

2_Instruments_and_NAF_for_Translation_Verification_IEA_Secretariat

3_Instruments_and_NAF_for_Layout_Verification_ACER

4_Final_Instruments_and_NAF_for_ACER _and_IEA_DPC

5_SurveySystem_Definition_Files_for_IEA_DPC

6_Codebooks_and_Data_Files_for_IEA_DPC

7_Exported_WinW3S_Database_for_IEA_DPC

8_Other_Materials

9_To_Country

30 3rd NRC Meeting, 9-12 June 2008, Windsor

File-Exchange using the IEA DPC WebFTP-Server

Please inform the recipients via email as soon you have uploaded any materials

Updated WebFTP-Server Manual will be sent to countries

31 3rd NRC Meeting, 9-12 June 2008, Windsor

MS Data Processing at the IEA DPC

32 3rd NRC Meeting, 9-12 June 2008, Windsor

Data Processing at the IEA Data Processing at the IEA DPCDPC

Data Processing of the MS data at the IEA DPC basically follows the same prcedures as for the FT

But: Any problems and/or inconsitencies will be reported back to countries and need correction and/or approval

Countries need to be available after data submission

33 3rd NRC Meeting, 9-12 June 2008, Windsor

Data Processing at the IEA Data Processing at the IEA DPCDPC

Country Databas

eNew Data

Structure

Reports, Statistic

s & Docu.

INP

UT

OU

TP

UTStructur

e Check

Background

Cleaning

IEA DPC

Country

ID Cleaning

Linkage Cleanin

g

Data & Codebook

sInstrument

s & Forms Cleaning

34 3rd NRC Meeting, 9-12 June 2008, Windsor

Thank you very much for your attention.