Data Collection

Data is your friend

Agenda

• Action research check-up

• Measures (aka, ways to collect data)

• Midterms

Public Service Announcement

APA style guide websites• http://owl.english.purdue.edu/owl/resource/560/01/

• http://citationmachine.net/index.php?reqstyleid=2

Action Research Projects

• Where you should be

• Finding research related to, but not necessarily identical to, your topic

• Research questions may shift

Action Research Projects

• Organizing Sources for literature reviewliterature review

• Example• Topic A (Source A, C, G, H, K)

• Subtopic 1 (Source G, K)

• Subtopic 2 (Source A, G, H)

• Subtopic 3 (Source A, C, H, K)

• Topic B (Source B, D, E, F, G, H)• Subtopic 1 (Source B, F, G, H)

• Subtopic 2 (Source D, E, F, G)

Action Research Projects

• Collecting Data• What types of data should I collect to answer

my research question?• What types of data should I collect to help

answer the why questions in my study?

Measures of Data Collection

• Interviews

• Questionnaires

• Observations

• Tests

Measures (Means of Data Collection)

You must match the instrument to the research question!

Examples to critique

• Measures• Questionnaire – Psychological School

Membership Survey used with middle school students

• Interview protocol – for teachers & counselors regarding professional development issues

• Observation instrument – PDE 430 for student teachers

• What are 2 benefits and 2 limitations of this measure?

Interviews

• What are some important things happening in this video related to interviewing?

http://www.youtube.com/watch?v=eAfTHwVTOrY

Interviews

• Advantages• Establish rapport & enhance motivation• Clarify responses through additional questioning• Capture the depth and richness of responses• Allow for flexibility• Reduce “no response” and/or “neutral” responses

• Disadvantages• Time consuming• Expensive• Small samples• Subjective – interviewer characteristics,

contamination, bias

Questionnaires

• Used to obtain a subject’s perceptions, attitudes, beliefs, values, opinions, or other non-cognitive traits

• Example of psychology questionnaires measuring aspects of happiness, “signature strengths,” well-being

http://www.authentichappiness.sas.upenn.edu/

Questionnaires

• Scales - a continuum that describes subject’s responses to a statement • Likert• Checklists• Ranked items

Questionnaires

• Likert scales• Response options require the subject to

determine the extent to which they agree with a statement

• Debate over odd v. even number responses• Statements must reflect extreme positive or

extreme negative positions

Questionnaires

• Checklists• Choose options

• Ranked items • Sequential order• Avoids marking everything high or low

Questionnaires

• Problems with measuring non-cognitive traits• Difficulty clearly defining what is being measured

• Self-concept or self-esteem

• Response set• Responding same way (Ex - all 4’s on CATS)

• Social desirability /faking• “PC filter”

• Agreeing with statements because of the negative consequences associated with disagreeing

Questionnaires

• Controlling problems• Equal numbers of positively and negatively

worded statements• Alternating positive and negative statements • Providing confidentiality or anonymity to

respondents

Designing Questionnaires

• Online resources• www.surveymonkey.com• http://pareonline.net/getvn.asp?v=5&n=3• http://www.peecworks.org/PEEC/PEEC_Inst/

I0004E536• http://www.statpac.com/surveys/

Observations

• Observations - direct observations of behaviors• Provide first hand account (ameliorates issues

of self-reporting in questionnaires)• Natural or controlled settings

• Ex – classroom vs. lab (child attachment studies)

• Structured or unstructured observations• Ex – frequency counts vs. narrative record

• Detached or involved observers

Observations

• Inference• Low inference - involves little if any inference

on the observers’ part• Student participation

• High inference - involves high levels of inference on the observers’ part

• Teacher effectiveness – PDE form 430

Observations

• Controlling observer effects• Observer bias

• Training• Inter-rater reliability (Cronbach’s alpha)• Multiple observers

• Contamination - knowledge of the study influences the observation

• Training• Targeting specific behaviors• Observers do not know of the expected outcomes• Observers are “blind” to which group is which

Observations

• Observer effects• Halo effect - initial ratings influence subsequent

ratings

• Hawthorne effect - increased performance results from awareness of being part of study

• Leniency - wanting everyone to do well

• Central Tendency - measuring in the middle

• Observer Drift - failing to record pertinent information

Tests

• PSSA Writing Assessment• Purpose• Domains• Scoring

• Questions to Consider

Score Interpretation of Tests

• Know the norming group on a norm-referenced test

• Self-report information is not very reliable• Inferences must be limited to what is being

tested• Grade equivalency scores should not be

interpreted to indicate grade assignment• Comparisons on a norm-referenced test can not

be made to populations outside the given norming group.

Tests

• Standardized tests• Uniform procedures for administration, scoring,

and interpreting test scores• Benefits and limitations?• Validity and reliability?

Benefits

• Forces teachers to get students on track with specific standards; quality control; assures students getting skills

• Red flag for a poor teacher who is negligent

• Consistency across school districts; use scores as consistent measure

• Consistency of subject matter

• Accountability of teachers

• Find areas where students are lacking and teach to their deficits

• Force students to be accountable

Limitations

• “If test scores improve (as I believe they will) and parent and policy makers are pleased with the results, are children receiving a better education?” (Cuban, 1983, p. 696).

• Once we know the expectation we limit what we teach to just what is covered on the test; restricts content taught

• Teaching to the test• Alignment between test and standard; is test valid?• Which came first—the standard or the assessment?• How to provide opportunities for remediation for those not meeting the

standard without cheating the “smart” kids.• Doesn’t measure creativity• Leads to labels of students• Bad test takers penalized

Validity & Reliability

• What are issues of validity and reliability that must be addressed in relation to standardized testing?

• PSSA issues

Standardized Testing Resources

• AERA statement about standardized tests• http://www.aera.net/policyandprograms/?id=378

• AERA “Research Points”• http://www.aera.net/uploadedFiles/Journals_and_Publications/

Research_Points/RP_Spring03.pdf

• Article by James Popham• http://school.familyeducation.com/educational-testing/educational-

philosophy/38778.html

• Dept of Education guide for policy makers and educators

• http://www.ed.gov/offices/OCR/archives/pdf/TestingResource.pdf