Evidence - Based Research: Continuing on after your PT3 is gone

download Evidence - Based Research: Continuing on after your PT3 is gone

of 27

  • date post

    14-Jan-2016
  • Category

    Documents

  • view

    15
  • download

    0

Embed Size (px)

description

Evidence - Based Research: Continuing on after your PT3 is gone. By Gerald Knezek Professor of Technology & Cognition University of North Texas SITE Annual Meeting Atlanta, Georgia March 2, 2004. Dedicated to PT3 Program Pioneers:. Kelly Green Susana Bonis Tom Carroll All of US - PowerPoint PPT Presentation

Transcript of Evidence - Based Research: Continuing on after your PT3 is gone

  • Evidence - Based Research:Continuing on after your PT3 is goneByGerald KnezekProfessor of Technology & CognitionUniversity of North TexasSITE Annual MeetingAtlanta, Georgia

    March 2, 2004

  • Dedicated to PT3 Program Pioneers:Kelly GreenSusana BonisTom CarrollAll of USPT3 Capacity 99-00PT3 Implementation 00-03PT3 Core Evaluation Group 01-03Challenge Grant Evaluator 99-05AERA TACTL SIG 02 - 04+SITE V.P. for Research 04-07

  • Major TopicsAs a project concludes, what are research questions that need to be addressed? How can studies be conducted related to these questions?

    What is necessary to conduct Scientifically Based Research? (Intro. to afternoon symposium)

  • General Guidelines1. Build on successesWhat were/are you good at?2. Use data already gathered3. Publish, Publish, PublishSubmit to SITE, AERA TACTL SIG ($5;)Accept book chapter offers, assemble panelsWrite journal articles (AACE, ISTE etc.)

  • Pay attention to which way the wind is blowing

  • Current Winds:Quantitative, Randomized, Replicated

  • Keep an Eye to the Future(APA Guidelines, 2001)The Publication Manual of the American Psychological Association (APA, 2001) strongly suggests that effect size statistics be reported in addition to the usual statistical tests. To quote from this venerable guide, "For the reader to fully understand the importance of your findings, it is almost always necessary to include Some index of effect size or strength of relationship in your Results section" (APA, 2001, p. 25). This certainly sounds like reasonable advice, but authors have been reluctant to follow this advice and include the suggested effect sizes in their submissions. So, following the lead of several other journals, effect size statistics are now required for the primary findings presented in a manuscript.

  • Chart1

    0.050.440.22

    ScienceScienceScience

    -0.12-0.060.02

    Social StudiesSocial StudiesSocial Studies

    0.030.210.22

    MathMathMath

    0.060.110.22

    VPAVPAVPA

    2000-2001

    2001-2002

    2002-2003

    MLTI 9 Project School Scores vs. 200 Other Maine Middle Schools, in Standard Deviation Units

    Effect of Maine Learning Technology Initiative 2000 - 2003

    Sheet1

    Trends in MEA Scores for Demonstration Schools vs. Other Maine Middle Schools Over Three Years

    MEA 2000-2001: Group 1 = 9 Demonstation Schools, Group 2 = All Others

    Group Statistics (Order Revised 11-12-03)Effect Size (Cohen's D)

    GROUPNMeanStd. DeviationStd. Error MeanPooled SD

    SCISS19529.11113.951091.317030.054.23

    2204528.89714.254720.29789

    SSSS19531.33334.387481.46249-0.124.52

    2204531.88734.540140.31787

    MTHSS19527.77783.86581.28860.034.98

    2204527.61275.031140.35225

    VPASS195315.590171.863390.065.51

    2204530.64715.524060.38676

    MEA 2001-2002: Group 1 = 9 Demonstation Schools, Group 2 = All Others

    Group Statistics

    GROUPNMeanStd. DeviationStd. Error MeanEffect Size (Cohen's D)

    SCISS19529.55563.844191.28140.444.27

    2214527.66824.274310.29219

    SSSS19529.44444.362081.45403-0.065.16

    2214529.7575.196320.35521

    MTHSS19527.77786.610182.203390.215.75

    2214526.59355.721070.39108

    VPASS19530.33334.716991.572330.116.04

    2213529.67146.096320.41771

    MEA 2002-2003: Group 1 = 9 Demonstation Schools, Group 2 = All Others

    Group Statistics

    GROUPNMeanStd. DeviationStd. Error MeanEffect Size (Cohen's D)

    SCISS195293.4281.1430.224.48

    2211528.034.5170.311

    SSSS19531.443.3211.1070.025.33

    2211531.355.4050.372

    MTHSS19528.443.8771.2920.224.89

    2211527.374.940.34

    VPASS19531.674.51.50.226.02

    2211530.376.0770.418

    2000-20012001-20022002-2003

    0.050.440.22

    Science

    -0.12-0.060.02

    Social Studies

    0.030.210.22

    Math

    0.060.110.22

    VPA

    Figure 1. Trends for Maine Learning Technology Initiative (MLTI) eighth grade standard assessment

    scores across 9 Demonstration school sites (vs. 200 other Maine middle schools representing origin = 0.0).

    Note: Total n = approx. 17,367 8th grade students per year.

    Note2: Standard assessment = Maine Educational Assessment (MEA) test scores.

    Sheet1

    2000-2001

    2001-2002

    2002-2003

    MLTI 9 Project School Scores vs. 200 Other Maine Middle Schools, in Standard Deviation Units

    Effect of Maine Learning Technology Initiative 2000 - 2003

    Sheet2

    Sheet3

  • Maine 2003

  • And to the PastCampbell, D. T. & Stanley, J. C. (1966). Experimental and Quasi-Experimental Designs for Research on Teaching. From Gage, N. L. (Ed.) Handbook of Research on Teaching. Boston: Rand McNally, 1963.

    Frequently references:McCall, W. A. (1923). How to Experiment in Education.

  • Examine Longitudinal Trends

    Chart1

    4.1

    4.86

    0.62

    4.29

    4.88

    0.53

    4.45

    4.81

    0.38

    4.37

    5.2

    0.78

    Stages

    Stages of Adoption: CECS 4100 (Computers in Education) Univ. of North Texas

    Sheet1

    Fall 2001Spring 2002Fall 2002Spring 2003

    PrePostESPrePostESPrePostESPrePostES

    Stages4.14.860.624.294.880.534.454.810.384.375.20.78

    Sheet1

    Stages

    Stages of Adoption: CECS 4100 (Computers in Education) Univ. of North Texas

    Sheet2

    Sheet3

  • Texas Attitudes Toward School by Grade Level: 2001 (6 Items)

    Chart1

    3.513.483.573.653.213.193.253.293.66

    3.333.363.453.683.173.023.213.233.58

    3.213.43.413.653.152.873.223.193.53

    3.133.283.363.543.032.513.083.073.45

    3.193.293.383.522.962.463.13.043.45

    3.13.263.323.492.912.213.0333.41

    2.91253.1153.193.2952.79252.34252.9052.8453.25

    2.91253.18253.163.2652.70252.25252.79252.76253.22

    2.7852.98753.04753.12.5752.1552.68752.64253.0775

    2.923.27253.072.92752.58252.0952.68752.65752.98

    2.773.28752.982.76252.52251.92252.61252.6052.8375

    3.00252.98753.01752.95752.6352.19252.68752.682.995

    Creativity

    Empathy

    Computer Importance

    Computer Enjoyment

    Motivation

    Att. School

    Study Habits

    Motivation to Study

    Att. Comp.

    Sheet1

    Allen ISD Fall 1999 Original Data: Grades 1-6 on 4-Point Scale, Grades 7-12 on 5-Point Scale

    Measurement ScaleGrade 1Grade 2Grade 3Grade 4Grade 5Grade 6Grade 7Grade 8Grade 9Grade 10Grade 11Grade 12

    Creativity3.513.333.213.133.193.103.553.553.383.563.363.67

    Empathy3.483.363.403.283.293.263.823.913.654.034.053.65

    Computer Importance3.573.453.413.363.383.323.923.883.733.763.643.69

    Computer Enjoyment3.653.683.653.543.523.494.064.023.803.573.353.61

    Motivation3.213.173.153.032.962.913.393.273.103.113.033.18

    Attitudes Toward School3.193.022.872.512.462.212.792.672.542.462.232.59

    Study Habits3.253.213.223.083.103.033.543.393.253.253.153.25

    Motivation to Study3.293.233.193.073.043.003.463.353.193.213.143.24

    Attitude toward Computer3.663.583.533.453.453.414.003.963.773.643.453.66

    Grades 7-12 Transformed from 5-Point to 4-Point Scale

    Measurement ScaleGrade 1Grade 2Grade 3Grade 4Grade 5Grade 6Grade 7Grade 8Grade 9Grade 10Grade 11Grade 12

    Creativity3.513.333.213.133.193.102.912.912.792.922.773.00

    Empathy3.483.363.403.283.293.263.123.182.993.273.292.99

    Computer Importance3.573.453.413.363.383.323.193.163.053.072.983.02

    Computer Enjoyment3.653.683.653.543.523.493.303.273.102.932.762.96

    Motivation3.213.173.153.032.962.912.792.702.582.582.522.64

    Att. School3.193.022.872.512.462.212.342.252.162.101.922.19

    Study Habits3.253.213.223.083.103.032.912.792.692.692.612.69

    Motivation to Study3.293.233.193.073.043.002.852.762.642.662.612.68

    Att. Comp.3.663.583.533.453.453.413.253.223.082.982.843.00

    Sheet1

    Creativity

    Empathy

    Computer Importance

    Computer Enjoyment

    Motivation

    Att. School

    Study Habits

    Motivation to Study

    Att. Comp.

    Sheet2

    Sheet3

  • Hawaii Attitudes Toward School by Grade Level: 1971 (20 Items)

  • Address Issues of MethodologyQuantitative Currently in favor, heavy on analysis methodologyQualitativeRicher, takes longerMixed MethodsSeeing process in operation often necessary to find out why in educationTheory Building vs. Theory TestingExploratory/Data Mining vs. Hypothesis Testing

  • Seek RandomizationRandom assignment (currently emphasized)For internal validity (fidelity of experiment)Start with large groupRandomly assign 1/2 treatment, 1/2 control(Versus)Random samplingDrawing from larger populationFor generalizability to larger populationExternal validity (Trust that this would work elsewhere)Also very important

  • Always Focus on InstrumentationMuch emphasis on standardized outcome measures as ultimate (valid) criteriaLess attention to reliability/accuracy of legislated tests and measuresLittle attention to how/where/when (or numerous other holes in) the data gatheredMistrust of teacher self appraisal/reflection

  • Instruments Book (http://iittl.unt.edu)

    Instruments

    for

    Assessing

    Educator Progress

    in

    Technology Integration

    By

    Gerald Knezek

    Rhonda Christensen

    Keiko Miyashita

    Margaret Ropp

    UNIVERSITYof

    NORTH TEXAS

  • Instruments SourcebookTechnology Evaluation Sourcebook Now Available

    Assessing the Impact of Technology in Teaching and Learning: A Sourcebook for Evaluators (edited by Jerome Johnston, University of Michigan, and Linda Toms Barker, Berkeley Policy Associates). The Sourcebook provides an overview of measurement issues in seven areas, from learner outcomes to technology integration. A co