Reliability and Validity of Data collection
-
Upload
venus-pugh -
Category
Documents
-
view
61 -
download
12
description
Transcript of Reliability and Validity of Data collection
![Page 1: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/1.jpg)
RELIABILITY AND VALIDITY OF DATA COLLECTION
![Page 2: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/2.jpg)
RELIABILITY OF MEASUREMENT • Measurement is reliable when it yields the same values
across repeated measures of the same event• Relates to repeatability• Not the same as accuracy
• Low reliability signals suspect data
![Page 3: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/3.jpg)
THREATS TO RELIABILITY
1. Human error
• Miss recording a data point
• Usually result from poorly designed measurement systems
• Cumbersome or difficult to use
• To complex
• Can reduce by using technology – Cameras
![Page 4: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/4.jpg)
2. INADEQUATE OBSERVER TRAINING
• Training must be explicit and systematic
• Careful selection of observers
• Must clearly define the target behavior
• Train to competency standard
• Have on-going training to minimize observer drift
• Have back up observers observe the primary observers
![Page 5: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/5.jpg)
3. UNINTENDED INFLUENCES ON OBSERVERS
• Causes all sorts of problems
• Expectations of what the data should look like
• Observer reactivity when she/he is aware that others are evaluating the data
• Measurement bias
• Feedback to observers about how their data relates to the goals of intervention
![Page 6: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/6.jpg)
SOLUTIONS TO RELIABILITY ISSUES
1. Design a good measurement system
• Take your time on the front end
2. Train observers carefully
3. Evaluate extent to which data are accurate and reliable
4. Measure the measurement system
![Page 7: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/7.jpg)
ACCURACY OF MEASUREMENT
• Observed values match the true values of an event
• Issue: Do not want to base research conclusions or treatment decisions on faulty data
![Page 8: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/8.jpg)
PURPOSES OF ACCURACY ASSESSMENT:
• Determine if data are good enough to make decisions
• Discover and correct measurement errors
• Reveal consistent patterns of measurement error
• Assure consumers that data are accurate
![Page 9: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/9.jpg)
OBSERVED VALUES MUST MATCH TRUE VALUES
• Determined by calculating correspondence of each data point with its true value
• Accuracy assessment should be reported in research
![Page 10: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/10.jpg)
INTER- OBSERVER AGREEMENT (IOA) OR RELIABILITY (IOR)
• Is the degree to which two or more independent observers report the same values for the same events
• Used to:
• Determine competency of observers
• Detect observer drift
• Judge clarity of definitions and system
• Increase validity of the data
![Page 11: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/11.jpg)
REQUIREMENTS FOR IOA / IOR
• Observers must:
• Use the same observation code and measurement system
• Observe and measure the same participants and events
• Observe and record independently of one another
![Page 12: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/12.jpg)
METHODS TO CALCULATE IOA / IOR
• (Smaller Freq. / Larger Freq.) * 100 = percentage
• Can be done with intervals as well
• Agreements / Agreements + Disagreements X 100
• Methods can compare:
• Total count recorded by each observer
• Mean count-per-interval
• Exact count-per-interval
• Trial-by-trial
![Page 13: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/13.jpg)
TIMING RECORDING METHODS:
• Total duration IOA
• Mean duration-per-occurrence IOA
• Latency-per-response
• Mean IRT-per-response
![Page 14: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/14.jpg)
INTERVAL RECORDING AND TIME SAMPLING:
• Interval-by-interval IOA (Point by point)
• Scored-interval IOA
• Unscored-interval IOA
![Page 15: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/15.jpg)
CONSIDERATIONS IN IOA
• During each condition and phase of a study
• Distributed across days of the week, time of day, settings, observers
• Minimum of 20% of sessions, preferably 25-30%
• More frequent with complex systems
![Page 16: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/16.jpg)
CONSIDERATIONS IN IOA
• Obtain and report IOA at the same levels at which researchers will report and discuss it within the results
• For each behavior
• For each participant
• In each phase of intervention or baseline
![Page 17: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/17.jpg)
OTHER CONSIDERATIONS
• More conservative methods should be used• Methods that will overestimate actual agreement should
be avoided• If in doubt, report more than one calculation• 80% agreement usually the benchmark
• Higher the better• Depends upon the complexity of the measurement
system
![Page 18: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/18.jpg)
REPORTING IOA
• Can use
• Narrative
• Tables
• Graphs
• Report how, when, and how often IOA was assessed
![Page 19: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/19.jpg)
VALIDITY• Many types
• Are you measuring what you believe you are measuring
• Ensures the data are representative
• In ABA, usually measure:
• a socially significant behavior
• dimension of the behavior relevant to the question
![Page 20: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/20.jpg)
THREATS TO VALIDITY
• Measuring a behavior other than the behavior of interest• Measuring a dimension that is irrelevant or ill suited to
the reason for measuring behavior• Measurement artifacts
• Must provide evidence that the behavior measured is directly related to behavior of interest
![Page 21: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/21.jpg)
EXAMPLES
• Discontinuous measurement
• Poorly scheduled observations
• Insensitive or limiting measurement scales
![Page 22: Reliability and Validity of Data collection](https://reader036.fdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/22.jpg)
CONCLUSIONS
• Reliabiltiy and validity of data collection are important
• Impacts the client,
• Impacts your reputation for good work