Data Extraction Workshop

Lindsay Shepard30 March 2011

PhD 7113 Systematic ReviewsCollege of Social Work, University of Utah

OverviewO The PurposeO The ProcessO The ContentsO Data Extraction FormsO Example Data Extraction FormsO Example Data ExtractionO Class Workshop

The Purpose of Data Extraction

O “to describe the study in general, to extract the findings from each study in a consistent manner to enable later synthesis, and to extract information to enable quality appraisal so that the findings can be interpreted. Ideally this should be undertaken in such a way as to require minimal reference to the original papers at data synthesis stage.” (Social Care Institute for Excellence, 2006)

The Rationale for Data Extraction Processes

O “The findings of a systematic review depend critically on decisions relating to which studies are included, and on decisions relating to which data from these studies are presented and analysed. Methods used for these decisions must be transparent, and they should be chosen to minimize biases and human error” (Cochrane Handbook, Section 7.1, emphasis added)

The Process of Data Extraction

O Plan data collection in advanceO Data – “any information about (or deriving from) a

study, including details of methods, participants, setting, context, interventions, outcomes, results, publications and investigators” (Cochrane Handbook, Section 7.3.1)

O Develop an extraction strategyO ReplicableO Transparent

O Use resourcesO Checklists and Review Group guidelinesO Topical systematic reviews and primary studiesO Data extraction forms

O Pilot test methodO Duplicate data extraction

Source•Study ID (created by review author).•Report ID (created by review author).•Review author ID (created by review author).•Citation and contact details.Eligibility•Confirm eligibility for review.•Reason for exclusion.Methods•Study design.•Total study duration.•Sequence generation*.•Allocation sequence concealment*.•Blinding*.•Other concerns about bias*.Participants•Total number.•Setting.•Diagnostic criteria.•Age.•Sex.•Country.•[Co-morbidity].•[Socio-demographics].•[Ethnicity].•[Date of study].Interventions•Total number of intervention groups.For each intervention and comparison group of interest:•Specific intervention.• Intervention details (sufficient for replication, if feasible).• [Integrity of intervention].

Outcomes•Outcomes and time points (i) collected; (ii) reported*.For each outcome of interest:•Outcome definition (with diagnostic criteria if relevant).•Unit of measurement (if relevant).•For scales: upper and lower limits, and whether high or low score is good.Results•Number of participants allocated to each intervention group.For each outcome of interest:•Sample size.•Missing participants*.•Summary data for each intervention group (e.g. 2×2 table for dichotomous data; means and SDs for continuous data).•[Estimate of effect with confidence interval; P value].•[Subgroup analyses].Miscellaneous•Funding source.•Key conclusions of the study authors.•Miscellaneous comments from the study authors.•References to other relevant studies.•Correspondence required.•Miscellaneous comments by the review authors.

• *Full description required for standard items in the ‘Risk of bias’ tool (see Chapter 8, Section 8.5).

Data Collection Checklist, Cochrane Handbook, Table 7.3.a

Items without parentheses should normally be collected in all reviews

Items in square brackets may be relevant to somereviews and not others

Method CollectionO “Different research methods can influence study

outcomes by introducing different biases into results. Basic study design characteristics should be collected” (Cochrane Handbook, Section 7.3.2)O Randomized?O Cluster or cross-over design?O Duration of the study?O Non-randomized studies included? (see Cochrane

Handbook, Section 13.4)O “Information should also be collected to facilitate

assessments of the risk of bias in each included study using the tool described in Chapter 8 (Section 8.5)” (Cochrane Handbook, Section 7.3.2)O Sequence generation?O Allocation sequence concealment?O Blinding?O Incomplete outcome data?O Selective outcome reporting?, etc.

Participant and Setting Collection

O “Typically, aspects that should be collected are those that could (or are believed to) affect presence or magnitude of an intervention effect and those that could help users assess applicability” (Cochrane Handbook, Section 7.3.3)O Participant characteristics?

O Total number?O Sub-group analyses? (Age, Sex,

Ethnicity, Country, Co-morbidity, etc.)O Study setting?O Diagnostic criteria?

Intervention CollectionO “Again, details are required for aspects that could

affect presence or magnitude of effect, or that could help users assess applicability. Where feasible, information should be sought (and presented in the review) that is sufficient for replication of the interventions under study” (Cochrane Handbook, Section 7.3.4)O Number of intervention groups?O Routes of delivery? (Oral or intravenous, Surgical

technique, etc.)O Dosage? (Amount, Intensity, Frequency, etc.)O Timing? (Between diagnosis and treatment, etc.)O Length of treatment?O Contents of the intervention?O Who delivered or administered?O Format and timing of delivery?O Intervention integrity/fidelity/compliance?

Outcome Measure CollectionO “Review authors should decide in advance whether they

will collect information about all outcomes measured in a study, or about only those outcomes of (pre-specified) interest in the review. Because we recommend in Section 7.3.6 that results should only be collected for pre-specified outcomes, we also suggest that only the outcomes listed in the protocol be described in detail” (Cochrane Handbook, Section 7.3.5)O Definition? (Diagnostic method, scale, threshold, or behavior)O Timing?O Unit of measurement?O Scale interpretation? (Upper/Lower limits; Favorable outcomes,

etc.)O Economic outcomes? (See Cochrane Handbook, Section

15.4.2)O Patient-reported outcomes? (See Cochrane Handbook, Chapter

17)O Adverse outcomes? (See Cochrane Handbook, Chapter 14)O Method of collection? O Number of outcomes reported?

Results CollectionO “Other outcomes should not be extracted unless the

protocol is modified to add them, and this modification should be reported in the review…Review authors should be alert to the possibility of important, unexpected findings, particularly serious adverse effects” (Cochrane Handbook, Section 7.3.6)O Number of participants allocated to each intervention?O Variation in outcome measure/time/summary statistic reporting?O Content for meta-analysis? (see Cochrane Handbook, Section

7.7)O Unit of analysis?O Type of outcome? (Dichotomous, Continuous, etc.)O Summary data for each intervention group? (Mean, Standard

Deviation, Standard Error, Confidence Interval, Test Statistic, etc.)

O Missing participants? (CONSORT Flow diagram)O Estimate of effect? (Effect size, confidence interval, and p-value)O Sub-group analyses?O Number of outcomes reported?

Additional Information for Collection

O “Other information will be required from each report of a study, including the citation, contact details for the authors of the study and any other details of sources of additional information about it” (Cochrane Handbook, Section 7.3.7)O Eligibility for inclusion?O Page numbers of content/extraction?O Funding source or potential conflicts of interest?O Key conclusions? (Direction of effect)O References to other relevant studies?O Study quality notes? (Ethical approval, sample size

calculation, etc.)O Correspondence required?O Comments/Notes (Date of extraction, dates of

revisions, name of extractor, etc.)

Rationale for Data Extraction Forms

O “The data collection form serves several important functions (Meade 1997)…1. the form is linked directly to the review question

and criteria for assessing eligibility of studies, and provides a clear summary of these that can be applied to identified study reports…

2. the data collection form is the historical record of the multitude of decisions (and changes to decisions) that occur throughout the review process…

3. the form is the source of data for inclusion in an analysis” (Cochrane Handbook, Section 7.5.1)

O “Given the important functions of data collection forms, ample time and thought should be invested in their design” (Cochrane Handbook, Section 7.5.1, emphasis added)

Data Extraction Form TypesPaper Advantages

Electronic Advantages

O convenience or preference;O data extraction can be

undertaken almost anywhere;O easier to create and

implement (no need for computer programming or specialist software);

O provides a permanent record of all manipulations and modifications (providing these manipulations and modifications are not erased); and

O simple comparison of forms completed by different review authors

O convenience or preference;O combines data extraction and

data entry into one step;O forms may be programmed (e.g.

using Microsoft Access) to ‘lead’ the author through the data collection process, for example, by posing questions that depend on answers to previous questions;

O data from reviews involving large numbers of studies are more easily stored, sorted and retrieved;

O allows simple conversions at the time of data extraction;

O rapid comparison of forms completed by different review authors; and

O environmental considerationsCochrane Handbook, Section 7.5.2

Data Extraction Form Examples

O Sample Data Extraction Forms:O Berkeley Systematic Reviews Group:

http://www.medepi.net/meta/forms.html O Cochrane Cystic Fibrosis and Genetic Disorders

Group: http://cfgd.cochrane.org/Files/Study%20selection%20and%20%20extraction%20form.doc

O Petticrew and Roberts Systematic Reviews in the Social Sciences: A Practical Guide: http://onlinelibrary.wiley.com/doi/10.1002/9780470754887.app4/pdf

O Cochrane Schizophrenia Group: http://www.cochrane-net.org/openlearning/Other/Example_from_a_review_in_schizophrenia.pdf

Data Extraction Form Development

O “If review authors plan to develop their own electronic forms using spreadsheet or database programs, we recommend that O (i) a paper form is designed first, and piloted

using more than one author and several study reports;

O (ii) the data entry is structured in a logical manner with coding of responses as consistent and straightforward as possible;

O (iii) compatibility of output with RevMan is checked; and

O (iv) mechanisms are considered for recording, assessing and correcting data entry errors” (Cochrane Handbook, Section 7.5.2)

Example Data Extraction

O Systematic Review Research Question: Amongst women in polygamous marriages, as compared to women in monogamous marriages in the same population, what is the prevalence of mental health issues?O See Paper Data Extraction Exercise

ExampleO See Electronic Data Extraction Form

Example

Study Setting Sample

Subgroups

Measurement Tools

Analyses

Results

Al-Krenawi & Slonim-Nevo, 2008

The Negev Desert Region, Israel

Quasi-Random Sample; Bedouin Arab Women Recruited by Phone in Conjunction with a Randomized Trial Involving Their Children (N=315)

159 Monogamous and 156 Senior Wives of Two Wife Families

A Socio-demographic Questionnaire, the Self-Esteem scale (SE), the Brief Symptom Inventory (BSI), the McMaster Family Assessment Device (FAD), the ENRICH questionnaire, and the Index of Parental Attitudes

Independent Samples t-test; Multiple Linear Regressions: Full Model and Best Predictive Model

“The findings show that the wives in polygamous marriages suffered from more psychological difficulties than those in monogamous marriages” with higher levels of somatization, depression, anxiety, hostility, and paranoid ideation (p<0.05); greater general severity (GSI) and total number of symptoms (p<0.05); lower self-esteem, less marital satisfaction, and more problematic family functioning (p<0.001). Family functioning was the strongest and most consistent predictor of mental distress; when family functioning was held as a dependent variable, polygamy was only significantly predictive of self-esteem (p<0.001)

Workshop TaskO For your Systematic Review Research

Question:O Begin to develop a data extraction

formO For one paper in your review, begin to

pilot your data extraction O Consider how to revise your form or

process to make the most of your timeO How to balance detail with usefulness?

(i.e., too little versus too much information)

Writing the ProtocolO “The methods section of both the protocol

and the review should detail:O the data categories that are to be collected;O how verification of extracted data from each

report will be verified (e.g. extraction by two review authors, independently);

O whether data extraction is undertaken by content area experts, methodologists, or both;

O piloting, training and existence of coding instructions for the data collection form;

O how data are extracted from multiple reports of the same study; and

O how disagreements are handled if more than one author extracts data from each report” (Cochrane Handbook, Section 7.6.6)

ReferencesO Higgins JPT, Green S (editors).

Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org

O Social Care Institute for Excellence (2006). The conduct of systematic research reviews for SCIE knowledge reviews. Available from http://www.scie.org.uk/publications/researchresources/rr01.pdf