Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the...
Transcript of Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the...
![Page 1: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/1.jpg)
Presented byEndia J. Lindo, Ph.D.Michael Faggella-Luby, Ph.D.Amy Elleman, Ph.D.
Conducting Systematic and Meta-Analytic Reviews: Basics for Early Career Researchers
![Page 2: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/2.jpg)
Professional Reading & Writing, is…
• a way for the field to catalog where it is, where it as been, and where it might go!
![Page 3: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/3.jpg)
Professional Reading & Writing, is…
• a way for the field to disseminate information from research to practice (and in reciprocal form from practice to research!)
![Page 4: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/4.jpg)
Professional Reading & Writing, is…
• organized around a pre-agreed and consistent form of written language, rules, and grammar
![Page 5: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/5.jpg)
Professional Reading & Writing, is…
• an important step to master along your professional path
![Page 6: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/6.jpg)
![Page 7: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/7.jpg)
Systematic Reviews of the Literature
• Critical evaluations of material already published• Synthesizing existing ideas into one research review or meta-analysis• Reviews:
• Define and Clarify a Problem• Summarize Previous Investigations• Identify Relations and Inconsistencies in Literature• Suggest Novel Approaches
• May include IES practice guides• Stand alone as publications, more than just an introduction
![Page 8: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/8.jpg)
What is the Systematic Review Process?
1. Select a Topic2. Search the Literature3. Develop the Argument4. Survey the Literature5. Critique the Literature6. Write the Review
![Page 9: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/9.jpg)
Step 1: Select a Topic• Interest in a practical or theoretical problem
• Desire to learn more for teaching your students or solve a challenge in your school
• Examine textbooks, articles, future goals
• **Stated as a question accessible to an academic discipline**▫ Use specific language▫ Refine the focus of interest▫ Select the academic vantage point
![Page 10: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/10.jpg)
Step 2: Search the Literature
• Go find the information to support your thesis!
• Preview the data
• Select the data
• Organize the data
• Catalog the data
• Requires skimming, scanning, and mapping!
![Page 11: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/11.jpg)
Step 3: Develop the Argument
• Form your case from the data▫ Arrange your claims logically
• Present your case from the data▫ Organize the relevant data into a body of evidence that explains what
is known about the topic
![Page 12: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/12.jpg)
Step 4: Survey the Literature
• Assemble, synthesize and analyze the data
• Goal: Form an argument about the current knowledge of the academic discipline on the topic
▫ Data is used to establish evidence that creates logical and defensible conclusions (or claims)
▫ Claims provide basis for answering research question
![Page 13: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/13.jpg)
Step 5: Critique the Literature
• Interprets the current understanding of the topic• Analyzes how the previous knowledge answers the research question• Note quality indicators for different types of research
![Page 14: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/14.jpg)
Step 6: Write the Review
• Transform the research project into a document for others• Requires writing, auditing and editing
![Page 15: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/15.jpg)
The Systematic Review Model
Machi & McEvoy, 2009, Figure I.3, p. 5
![Page 16: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/16.jpg)
• Provide a conceptual context for the research and state clear research questions and objectives.
• Develop and report eligibility criteria
• Report search procedures used to identify potentially eligible research studies
• Report procedures for retrieving studies and document the proportion of potentially eligible studies retrieved
• Describe methods used to screen studies for inclusion and document reasons particular studies were not included
What are quality indicators of Systematic Reviews?
See Talbott, Maggin, Van Acker & Kumm (2017)
![Page 17: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/17.jpg)
• Develop a coding system that addresses critical aspects of the research.
• Describe adherence to key indicators of study quality.
• Describe the research participants and contexts included across the studies.
• Describe the variables under study to evaluate the consistency of operational descriptions and data collection methods used.
• Develop a data analysis plan to address the research questions.
• Interpret the results considering the research findings, study quality, and conditions to which the research might generalize.
Quality indicators of Systematic Reviews (continued)
See Talbott, Maggin, Van Acker & Kumm (2017)
![Page 18: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/18.jpg)
• Adolescent Literacy▫ Faggella-Luby, M., Drew, S., & Schumaker, J. (2015). Not such a simple story: Contradictory evidence from a review of story
structure research for students at risk. Learning Disabilities Research & Practice. 30(2), 61-75.
• Postsecondary Education for Students with Disabilities▫ Faggella-Luby, M., Dukes, L., Madaus, J. Gelbar, N., Lombardi, A., & Lalor, A. (in press) Universal design and college students
with disabilities: Does the data equal the zeal? Currents in Teaching and Learning. ▫ Dukes, L. L. III., Madaus, J. W., Faggella-Luby, M., Lombardi, A., & Gelbar, N. (in press). PASSing College: A taxonomy for
students with disabilities in postsecondary education. Journal of Postsecondary Education and Disability▫ Lombardi, A., Gelbar, N., Dukes, L. L., Kowitt, J., Wei, Y., Madaus, J. W., Lalor, A., & Faggella-Luby, M. (in press). Higher
education and disability: A systematic review of assessment instruments designed for students, faculty, and staff. Journal of Diversity in Higher Education
▫ Madaus, J. Gelbar, N., Dukes, L., Lalor, A., Lombardi, A., Kowitt, J. & Faggella-Luby, M. (2016). Literature on postsecondary disability services: A call for research guidelines. Journal of Diversity in Higher Education.
▫ Gelbar, N., Madaus, J., Lombardi, A., Faggella-Luby, M., & Dukes, L. (2015). College students with physical disabilities: Common on campus, uncommon in the literature. Physical Disabilities: Education and Related Services. 34(2), 14-31.
▫ Faggella-Luby, M., Lombardi, A., Lalor, A., & Dukes III, L. (2014). Methodological trends in disability and higher education research: Historical analysis of the journal of postsecondary education and disability.
▫ Madaus, J., Gelbar, N., Dukes, L., Faggella-Luby, M., Lalor, A., & Kowitt, J. (2013). Thirty-five years of transition: A review of CDTEI issues from 1978-2012. Career Development and Transition for Exceptional Individuals.
Example Systematic Review Publications
![Page 19: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/19.jpg)
• Lindo, E., Park, N., & Nix, S.(In progress). Parents as Trainers: A synthesis of the literature examining the impact do parent-implemented academic interventions with students in grades 3-12.
• Lindo, E., Al-Otaiba, S., Allor, J., Rouse, A., &. Mellado-De La Cruz, V.* (In progress). A meta-analysis of volunteer tutoring programs for beginning readers.
• Elleman, A. M., Lindo, E., J., Talbert, S., & Cooper, J. (in progress). Examining contextual and participant characteristics that impact the effectiveness of comprehension instruction: A meta-analytic review.
• Elleman, A. M., Lindo, E. J., Talbert, S. M., Cooper, J., Kim, K., Barnes, Z., & Fields, S. (in progress). Examining the sensitivity of standardized and proximal measures in comprehension interventions: A meta-analytic review.
• Parrish, J. L., Talbert, S., Elleman, A. M. (2016, July). Integrated science and literacy K-12 instruction: A meta-analysis. Poster presented at the Twenty-Third Annual Meeting of the Society for the Scientific Study of Reading, Porto, Portugal.
Example Meta-Analytic Reviews• Elleman, A. M. (2017). Examining the Impact of Inference Instruction on the literal and inferential comprehension of skilled and less skilled
readers: A meta-analytic review. Journal of Educational Psychology, 109, 761-782. http://dx.doi.org/10.1037/edu0000180
• Lindo, E. J., Kleimann, K., Combes, B. H., & Franks, J. (2016). Managing stress levels of parents of children with developmental disabilities: A meta-analytic review. Family Relations, 65(1), 207-224. doi: 10.1111/fare.12185
• Elleman, A. M., Lindo, E. J., Morphy, P., & Compton, D. L. (2009). The impact of vocabulary instruction on passage-level comprehension of school-age children: A meta-analysis. Journal of Research on Educational Effectiveness, 2, 1-41. http://dx.doi.org/10.1080/19345740802539200
![Page 20: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/20.jpg)
• What is it?• Why use it?• How to do it?• Challenges and benefits?• Current trends?
Meta-Analysis
![Page 21: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/21.jpg)
• Meta-analysis is a statistical technique for analyzing the results of a collection of independent studies, on a related topic, in order to determine an overall estimate of treatment effect.
What is meta-analysis?
![Page 22: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/22.jpg)
Cumulative number of publications about meta-analysis over time, until 17 December 2009 (results from Medline search using text "meta-analysis")Source: Hippokratia. Dec 2010; 14(Suppl 1): 29–37.
Hierarchy of evidence
![Page 23: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/23.jpg)
Why use meta-analytic technique?
• Single studies are not sufficient to determine the utility of an intervention or a hypothesis’ validity.
• Ex. ‘The Great Debate’ on Psychotherapy’s effectiveness (1952-1978)
• Applies structure and affords less subjectivity than traditional narrative reviews.
• Can integrate a much larger sample of studies.
• Used as original publications, but also serve as introductions of topics or justification for grant proposals.
![Page 24: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/24.jpg)
Source: Practical Meta-Analysis -- D. B. Wilson
24
When Can You Do Meta-analysis? Meta-analysis is applicable to collections of research that
Are empirical, rather than theoretical
Produce quantitative results, rather than qualitative findings
Examine the same constructs and relationships
Have findings that can be configured in a comparable statistical form (e.g., as effect sizes, correlation coefficients, odds-ratios, proportions)
Are “comparable” given the question at hand
![Page 25: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/25.jpg)
• Analyze and integrate outcomes▫ Aggregating effect sizes▫ Assessing heterogeneity▫ Test moderators
• Interpret evidence▫ Address strengths and limitations▫ Assess for publication bias
• Present results/Publish
• Specifying the problem
• Establish inclusion and exclusion criteria
• Identify and retrieve research reports
• Develop coding protocol and code reports
• Determine and compute appropriate effect sizes
How do you conduct a meta-analysis?
![Page 26: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/26.jpg)
Specifying the problem (One example)
•Need: to determine what constitutes evidence-based practice in regards to approaches to address the stress experienced by parents of individuals with moderate to severe developmental disabilities.
Research Questions:What interventions have been effective in reducing the stress levels of parental caregivers of children with developmental disabilities?
Are these effects moderated by treatment, parent or child characteristics?
![Page 27: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/27.jpg)
Search far & wide (Not just peer reviewed)
• Search in electronic reference databases. • Search reference list of previous reviews
and pertinent primary studies on your topic.
• Hand search journals that frequently publish on the topic.
• Contact experts who have published on your topic.
• Check reference list of each study included in your review.
Identification of Reports PRISMA 2009 Flow DiagramPRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses
![Page 28: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/28.jpg)
Terms: Parent Stress AND intervention AND Disab*
• Consider other potential terms:▫ Parent Stress: OR Parental stress OR caregiver stress▫ Intervention: OR Support OR therapy▫ Disab*: OR Disabilities OR autism OR delay OR intellectual disability
• Search references of key reports and other literature reviews▫ Hastings & Beck, 2004▫ Singer, et al., 2007
Finding Reports of Parental Stress Management Interventions…
![Page 29: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/29.jpg)
Parental Stress Management Interventions…
Inclusionary criteria
Present an intervention targeted at relieving the stress of parents of individuals with moderate to profound cognitive/developmental disabilities.
Employ a outcome measuring parental stress; NOT depression
Use an experimental or quasi-experimental design reporting data that permits calculation of a numeric effect size for at least one eligible outcome. (Single Subject Exluded)
Reported in English and conducted in North America.
Electronic searches of ERIC, PsychInfo, Professional Development Collection
• (support OR intervention) AND (disab*) AND (parent stress)
Electronic searches of Academic Search Complete, AgeLine, AltHealth Watch, Educational Source, Family and Society Studies Worldwide, Health source, Professional Development Collection, PsycArtilces, PsychInfo and SocIndex
• (parental OR caregiver stress) AND (disabilitiesOR autism OR delay OR intellectual disability) AND (intervention OR therapy)
![Page 30: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/30.jpg)
• Develop coding protocol ▫ Establish inclusionary and exclusionary
criteria
▫ Clearly define variables
▫ Use previous reviews and key studies as guides for other predictor variables (potential moderators) to code.
• Establish procedures for managing data.
• Assess inter-coder agreement and check data entry.
Coding of Reports
![Page 31: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/31.jpg)
APA CitationParticipant CharacteristicsAge/GradeSESReading AbilityEthnicityRural/UrbanGender
DesignExperimental, quasi-experimental, pre-post only w/no control groupOther
Intervention Characteristics (Independent Variable) DescriptionLength of study (in days/weeks) # of hours# of sessions
Who administeredTraining of teacher/researcherFidelity reported
Dependent Variables (Outcome Measures)NameDescriptionStandard Norm-Referenced or CustomReliability reported
Data Analysis Method
ResultsSignificanceESAuthors’ conclusionsAlignment of methods, results, and authors’ conclusions
Notes
Sample Coding
![Page 32: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/32.jpg)
Source: Wilson
• Not understanding or planning the analysis prior to coding
• Underestimating time, effort, and technical/statistical demands
• Using a spreadsheet for managing a large review
• Variable names not on coding forms• Not breaking apart difficult judgments• Over-coding—Trying to extract more detail
than routinely reported**
Common Mistakes
• Regular meetings (develops normative understandings)
• Annotate coding manual
Training Coders
Coding
![Page 33: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/33.jpg)
Comments on Managing the Bibliography• Major activity• Information you need to track▫ source of reference (e.g., PsychLit, Dissertation Abs.)▫ retrieval status retrieved, requested from ILL, etc.
▫ eligibility status eligible not eligible relevant review article
▫ coded status• Word processor not up to the task• Spreadsheets are cumbersome• Use a database of some form
Source: Wilson
![Page 34: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/34.jpg)
Approaches to coding
For analysis
• Gender:(0) All Females (1) More Females (2) Equal (3) More Males (4) All Males (999) can’t tell
For descriptive report
• Gender:▫ Female: n = 100▫ Male: n = 75
Same for variable like ethnicity, risk status, SES, etc.
• Percentage of males in sample:(0) 0-24%(1) 25-49%(2) 50-74%(3) 75-100%(4) Unspecified(5) Not reported
![Page 35: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/35.jpg)
Approaches to coding (con’t)
Unless item occurs in consistent units avoid creating a category for each
• Intervention1= 2 weeks2 = 6 weeks3 = 12 weeks (and so)
Better• Intervention
1= <6 weeks2 = 6-12 weeks3 = >12 weeks (etc.)
Do not code as if only for the studies you found. Typically coding planned first. Also, you may find other studies that go beyond your current range.
![Page 36: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/36.jpg)
Approaches to coding (con’t)
One option
School Prep 0= not discussed 1= visual system (cueing) 2= social skills training3 = vocational training4 = strength-based planning- including academic 5 = goal setting 6 = self-advocacy 7 = counseling & guidance 8 = mentoring
Another option
• School prep: 0= not discussed, 1= discussed
• Visual system (cueing)▫ 0=no, 1= yes
• Social skill training▫ 0=no, 1= yes
etc. . .
![Page 37: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/37.jpg)
![Page 38: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/38.jpg)
Synthesizing Research
Qualitative
• Recurring themes across reports.
Quantitative
• Vote counting▫ Emphasis on direction of findings▫ 0= not significant,▫ 1= significant in unexpected direction. ▫ 2= significant in expected directionThis approach has issues. Significance tests highly influenced by sample size.
• Measuring direction and strength of relationship (Effect sizes: d, r, Odds Ratio)▫ Reporting range found▫ Computing average
![Page 39: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/39.jpg)
Forest plot of vocabulary intervention ESs by measure and reading status
Source: Elleman, A. M., Lindo, E. J., Morphy, P., & Compton, D. (2009). The impact of vocabulary instruction on passage-level comprehension of school-age children: A meta-analysis. Journal of Research on Educational Effectiveness, 2, 1-43.
Calculating Effect Sizes• An effect size (ES) is a measure of the magnitude of a
relationship between two variables or a difference between groups.
• Main types of effect sizes are based on:• means (Cohen’s d, Hedges’ g, raw unstandardized
difference)
• binary data (risk ratio, odds ratio, risk difference)
• correlations (Pearson’s correlations, Fisher’s Z)
• survival data (hazard ratio)
• Compute ES and its variance, standard error, and confidence interval to provide an estimate of the precision of an ES.
• Only 1 ES per study in an analysis (select 1 or aggregate).
![Page 40: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/40.jpg)
Interpreting Effects
Cohen’s “Rules-of-Thumb”
standardized mean difference effect size
small = 0.20medium = 0.50
large = 0.80
correlation coefficientsmall = 0.10
medium = 0.25large = 0.40
odds-ratiosmall = 1.50
medium = 2.50large = 4.30
Practically important effectsWWC = 0.25Hattie = 0.4
Annual Reading and Math Growth
Lipsey (2006) Effect sizes found in meta-analyses of educational interventions. (Note: Based on 126 distinct meta-analysis, 777 mean ESs reported.)
![Page 41: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/41.jpg)
Programs for Analysis
• Excel ()
• Effect Size calculator https://campbellcollaboration.org/effect-size-calculato.html
• SPSS, STATA, or SAS Macros▫ http://mason.gmu.edu/~dwilsonb/
ma.html
• Comprehensive Meta-Analysis http://www.meta-analysis.com
Fixed v. Random Effects
Fixed Effect assumes studies identified are the entire population of relevant studies
Random Effect Assumes variability reflects both population effects and sampling error.
Homogeneity Analyses▫ Is the variance due only to sampling error?▫ Q statistic: Interpreted like a chi square, establishes if there is
significant heterogeneity across studies
Moderator AnalysisAnalyze excess between study (ES) variability
categorical variables with the analog to the one-way ANOVA continuous variables and/or multiple variables with weighted
multiple regression
Aggregating and Analyzing the Data
![Page 42: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/42.jpg)
• funnel plot
• Egger’s linear regression method
• Begg and Mazumdar’srank correlation method
• Duval and Tweedie’s Trim and Fill method
• Rosenthal’s Fail-safe N
Assessing Publication Bias
![Page 43: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/43.jpg)
• Meta-Analysis Reporting Standards (MARS). ▫ included in APA Manual, 6th ed. and Cooper (2010, 2015)
• Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)▫ http://prisma.thetacollaborative.ca/
• Campbell Collaboration ▫ https://campbellcollaboration.org/research-resources/writing-a-campbell-systematic-review.html
• Meta-analysis of Observational Studies in Epidemiology (MOOSE)▫ Stroup et al., JAMA 2000▫ www.consort-statement.org/Media/Default/Downloads/Other Instruments
Reporting Standards
![Page 44: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/44.jpg)
Source: Practical Meta-Analysis -- D. B. Wilson
Benefits of Meta-analysis
Imposes a discipline on the process of summing up research findings
Represents findings in a more differentiated and sophisticated manner than conventional reviews
Capable of finding relationships across studies that are obscured in other approaches
Protects against over-interpreting differences across studies Can handle a large numbers of studies (this would
overwhelm traditional approaches to review)
![Page 45: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/45.jpg)
Source: Practical Meta-Analysis -- D. B. Wilson
Challenges of Meta-analysis
Takes significant effort
Not all studies provide adequate data for inclusion and analysis
Mechanical aspects don’t lend themselves to capturing more qualitative distinctions between studies
“Apples and oranges” criticism
Heterogeneity of study populations
Most meta-analyses include “blemished” studies to one degree or another (e.g., a randomized design with attrition)
Publication bias poses a continual threat
Requires advanced statistical techniques
![Page 46: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/46.jpg)
• Continued increases in number of meta-analyses conducted.
• Quickly becoming the norm for published literature syntheses.
• Multi-level meta-analyses• Ex. Marsh et al. (2009)
• Meta-analysis of meta-analyses▫ Ex. Hattie (2009). Visible Learning
Current Trends and Extensions of the Approach
Source: Hattie (2009)
![Page 47: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/47.jpg)
• Cooper, H. (2015). Research synthesis and meta-analysis: A step-by-step approach (5th Ed.). Thousand Oaks: Sage.
• Haidich, A.B. (2010). Meta-analysis in medical research. Hippokratia, 14(Suppl 1): 29-47.
• Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York: Routledge.
• Marsh, H.W., Bornmann, L. Mutz, R., Daniel, H., & O’mara, A. (2009). Comprehensive Meta-Analysis Comparing Traditional and Multilevel Gender Effects in the Peer Reviews of Grant Proposals. Review of Educational Research, 79, 1290.
• Moher D, Liberati A, Tetzlaff J., Altman D.G., The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(6): e1000097. doi:10.1371/journal.pmed1000097
Select References
![Page 48: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/48.jpg)
Training Videos
• David Wilson, The Campbell Collaboration • Effect size calculation and basic meta-analysis, https://www.youtube.com/watch?v=nkcZFAmmKeE
• Michael Borenstein, Comprehensive Meta-Analysis• Meta-Analysis 001: https://www.youtube.com/watch?v=xWiXeKR3dB4• Basic Level 3: https://www.youtube.com/watch?v=O6qDlov5-Is• Meta-Analysis | Common mistakes and how to avoid them | Fixed effect vs. random effects
https://www.youtube.com/watch?v=3_sbitgMeUg• Meta regression using Comprehensive Meta-Analysis
https://www.youtube.com/watch?v=39PKfU4cRBc
Select References
![Page 49: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/49.jpg)
• Learning Disability Quarterly (LDQ)• Learning Disabilities Research and Practice (LDR&P)• Journal of Learning Disabilities (JLD)• Review of Educational Research (RER)• Journal of Educational Psychology (JEP)
Suggested Publication Outlets
![Page 50: Conducting Systematic and Meta-Analytic Reviews: Basics ... · • Preview the data • Select the data • Organize the data • Catalog the data • Requires skimming, scanning,](https://reader036.fdocuments.net/reader036/viewer/2022070904/5f728520d80f964df269626a/html5/thumbnails/50.jpg)
• Endia Lindo [email protected]• Michael Faggella-Luby [email protected]• Amy Elleman [email protected]
Thanks for joining us today!
Contact Information