National Immunization Survey: Data Quality and Public-Use Data Files Meena Khare, National Center...
-
Upload
lucas-jordan -
Category
Documents
-
view
231 -
download
0
Transcript of National Immunization Survey: Data Quality and Public-Use Data Files Meena Khare, National Center...
National Immunization Survey:
Data Quality and Public-Use Data Files
Meena Khare, National Center for Health Statistics
Michael P. Battaglia, Abt Associates Inc.
July 16, 2002
Outline
National Immunization Survey
Quality of the data
Confidentiality issues
Public Use Data Files (PUF)
Summary
2
National Immunization Survey (NIS) Large ongoing RDD survey, conducted by CDC since 1994
Measures vaccination coverage among children aged 19-35 months at National, State, and Urban area levels (78 IAP areas)
Monitors Healthy People 2000 and 2010 goals of immunization coverage >>90% Coverage: 4DTP, 3Polio, 1MMR, 3HepB, 90% Coverage: 4DTP, 3Polio, 1MMR, 3HepB,
3Hib, and 4:3:1:3 series3Hib, and 4:3:1:3 series
Monitors introduction of new vaccines
<4% of households in the U.S. contain child of age 19-35 months
3
NIS Operations
N o Co n se n t o r n o p rov id er in fo
4 Q T R E s tim a tes
In te rn a l A n a lytic F ileC h ild re n w ith ad e qu a te P ro vid e r D a ta o n ly
F in a l P UF
D R B /co nfid e n tia lity R e v iew
C h ild re n w ith com p le te d ho u seh o ld in te rv ie wsa n d a de qu ate p rov ide r da ta
Q C Ch e ck/ D a ta p roce ss in g / M a tch in g S h e e t re v iew
IH Q R e tu rn e d w ith d a ta IH Q N o n re sp o n sem iss in g p ro v id er d a ta
P ro v ide r R eco rd Ch e ck S tu dyIH Q M a iled
W ith C o nse n t a n d w ith p rov id er in fo
H o u se h o ld In te rv iewC o m ple te d in te rv ie ws
C A T I S c re e n er
N ISR D D S a m p le
4
NIS Data Collection
Household CATI Screener and Interview Household CATI Screener and Interview Parent/Guardian Parent/Guardian (most knowledgeable person)(most knowledgeable person) Socio-demographic information: Socio-demographic information: mother and mother and
childchild Shot card use or memory recall Shot card use or memory recall Vaccination dates (shot card only)Vaccination dates (shot card only) Provider’s contact information with consentProvider’s contact information with consent
Provider Record Check Study (mailed Provider Record Check Study (mailed IHQ)IHQ) Provider’s office (e.g., staff, nurse, manager)Provider’s office (e.g., staff, nurse, manager) Mail, Fax, Telephone Mail, Fax, Telephone Completed IHQ or copy of medical recordsCompleted IHQ or copy of medical records Matched on DOB, Gender, NameMatched on DOB, Gender, Name Vaccination historyVaccination history Provider’s informationProvider’s information
5
Goals of Quality Control in the NIS
Evaluate and improve quality of Evaluate and improve quality of the information collectedthe information collected
Reduce/eliminate discrepanciesReduce/eliminate discrepancies
Improve quality of the estimatesImprove quality of the estimates
Reduce potential bias in official Reduce potential bias in official estimates of vaccination estimates of vaccination coverage ratescoverage rates
7
Quality Control Procedures used in the NIS
Extensive monitoring and QC review of Extensive monitoring and QC review of the data collected from CATI and the data collected from CATI and mailed IHQsmailed IHQs
Automated edit software for processing Automated edit software for processing combined household and provider data combined household and provider data
Manual ‘Matching Sheet’ review of Manual ‘Matching Sheet’ review of discrepant cases discrepant cases
Resolution by using guidelines Resolution by using guidelines developed by expert immunization developed by expert immunization program managers program managers
8
Review of Data Errors in the IHQ
Data Entry ValidationData Entry Validation Double data entryDouble data entry Approximately 400 forms Approximately 400 forms
reviewed/quarterreviewed/quarter Error rates ranged from 0.23% - 0.67% Error rates ranged from 0.23% - 0.67%
(~30,000 fields with data, NIS Q4/1999-(~30,000 fields with data, NIS Q4/1999-Q4/2001)Q4/2001)
~350-1000 manual Matching Sheet ~350-1000 manual Matching Sheet reviews/quarter for discrepant datareviews/quarter for discrepant data
Most errors in dates are found in the Most errors in dates are found in the original provider-reported recordsoriginal provider-reported records
<1% children had data errors <1% children had data errors 9
Observed Discrepancies in Dates
DOBDOB Between household (HH) and Between household (HH) and
provider(s)provider(s) Between multiple providers for a Between multiple providers for a
childchild
Shot datesShot dates Between HH and provider(s)Between HH and provider(s) Within a single providerWithin a single provider Between multiple providers for a Between multiple providers for a
childchild
10
DOB and Shot datesDOB and Shot dates 122 children classified as age-ineligible122 children classified as age-ineligible 69 IHQs filled out for the ‘wrong’ child69 IHQs filled out for the ‘wrong’ child 21 children had changes in best DOB21 children had changes in best DOB 2163 children had shot dates before DOB 2163 children had shot dates before DOB
or vaccination interval < 1 month, or other or vaccination interval < 1 month, or other shot date editsshot date edits
Hepatitis B ‘birth’ doseHepatitis B ‘birth’ dose 361 children had missing provider ‘birth 361 children had missing provider ‘birth
dose' of Hepatitis B and providers had dose' of Hepatitis B and providers had checked the ‘given at birth’ box on IHQchecked the ‘given at birth’ box on IHQ
34 children had imputed dates from the household 34 children had imputed dates from the household shot cardshot card
327 children had imputed dates from the distribution 327 children had imputed dates from the distribution of provider-reported dates for the birth doseof provider-reported dates for the birth dose
Matching Sheet Review, 2000 NIS
11
Results of Quality Control Evaluation
No single data source is 100% complete and No single data source is 100% complete and accurateaccurate
Some discrepancies in DOB and vaccination Some discrepancies in DOB and vaccination dates/number of doses could not be verified and dates/number of doses could not be verified and remained after edits remained after edits
Number of discrepancies increased if more than Number of discrepancies increased if more than one immunization provider per child (1.35 one immunization provider per child (1.35 providers per child in 1999 NIS ) was identified providers per child in 1999 NIS ) was identified
Errors in Errors in Original Immunization records Original Immunization records Locating records for the correct child Locating records for the correct child Transcribing shot dates to the IHQTranscribing shot dates to the IHQ Illegible dates, transposed dates, missing day/month, Illegible dates, transposed dates, missing day/month,
incorrect dates (especially year)incorrect dates (especially year) Shot date interval< 1 month, or shot dates<DOBShot date interval< 1 month, or shot dates<DOB Missing ‘first’ Hepatitis B dose given at birthMissing ‘first’ Hepatitis B dose given at birth 12
‘Best’ Vaccination Value Research
Currently, coverage estimates from the Currently, coverage estimates from the providerprovider data are used as data are used as official official estimates estimates
The goal of ‘best’ value research is to The goal of ‘best’ value research is to obtain the most complete and accurate obtain the most complete and accurate immunization data possible from immunization data possible from twotwo sourcessources Supplement incomplete provider data with Supplement incomplete provider data with
household data for children who are UTD from household data for children who are UTD from ‘shot card’ ‘shot card’
Substitute completely missing provider data Substitute completely missing provider data with household data for children who are with household data for children who are 4:3:1:3 UTD from ‘shot card’ 4:3:1:3 UTD from ‘shot card’
13
NIS 2000 Sample
35,960 households with eligible 35,960 households with eligible children 19-35 monthschildren 19-35 months
33,477 (93.1%)of eligible households 33,477 (93.1%)of eligible households with completed interviewswith completed interviews
34,087 age-eligible children with 34,087 age-eligible children with completed interviewscompleted interviews
22,958 (67.4%) children with 22,958 (67.4%) children with completed interviews and ‘adequate’ completed interviews and ‘adequate’ provider dataprovider data
14
Sources of Immunization Histories, 2000 NIS
SourceSource Household data from Household data from Shot Card Shot Card
‘‘Adequate’ Adequate’ Provider Provider data data
YesYes NONO TotalTotal
nn %% nn %% nn %%
YesYes 11,911,96363
73.873.8 10,910,99595
61.561.5 22,922,95858
67.467.4
NoNo 4,244,2499
26.226.2 6,886,8800
38.538.5 11,111,12929
32.632.6
TotalTotal 16,216,21212
47.647.6**
17,817,87575
52.452.4**
34,034,08787
100.100.00
* row percents15
Assessment of Provider-Reported 4:3:1:3* Coverage Estimates among Children with
Household Data,2000 NIS
Source of HH reportsSource of HH reportsand measures of qualityand measures of quality
Shot CardShot Card ‘‘Memory’ Memory’ Recall**Recall**
Sample Size Sample Size 11,96311,963 5,5265,526
SensitivitySensitivity 67.167.1 44.844.8
SpecificitySpecificity 61.861.8 65.365.3
Misclassification (prov)Misclassification (prov)(HH UTD)/ Prov NUTD)(HH UTD)/ Prov NUTD)
7.47.4 8.78.7
Overall agreementOverall agreement 66.166.1 49.949.9
Overall disagreementOverall disagreement 33.933.9 50.150.1
Net Difference (HH Cov - Prov Net Difference (HH Cov - Prov cov)cov)
-19.1-19.1 -32.7-32.7
*4DTP/3Polio/1MCV/3Hib** Other 5,469 non-shot card children have missing 4:3:1:3 UTD status
16
Comparison of Vaccine-specific Coverage Rates by Source of
Immunization Histories, 2000 NIS
17
0
20
40
60
80
100
4+ DTP 3+ POLIO 1+ MCV 3+ HIB 3+ Hep B 1+ VRC 4:3:1 4:3:1:3
Type of Vaccine
Co
vera
ge
rate
(%
)
Household Provider Best Value Revised Best Value
Best Value: Combined Provider and household ‘shot card’ data
Revised Best Value: After adjusting Best Value for missing provider data among non-shot card children
NIS Data Files and Estimates
In-house analytic filesIn-house analytic filesPublic-use data files (PUF )Public-use data files (PUF )National, State, and Urban area National, State, and Urban area
level estimates of vaccination level estimates of vaccination coveragecoverage Internet: Internet: www.cdc.gov/nip/coveragewww.cdc.gov/nip/coverage MMWR MMWR Journal ArticlesJournal Articles Articles in ProceedingsArticles in Proceedings Presentations at National and State Presentations at National and State
ConferencesConferences Publications are listed on NIS Web sitePublications are listed on NIS Web site
18
Confidentiality Issues Legal obligation to protect confidentiality of Legal obligation to protect confidentiality of
respondents and reduce risk of disclosurerespondents and reduce risk of disclosure Sample size (4 quarters)Sample size (4 quarters)
~ 440 children per IAP with interview data~ 440 children per IAP with interview data ~ 290 children per IAP with provider data ~ 290 children per IAP with provider data
Only telephone households (90% HHs with Only telephone households (90% HHs with telephone)telephone)
Rare population (children 19-35 months)Rare population (children 19-35 months) Geographic identifiersGeographic identifiers
State, Urban AreasState, Urban Areas Census RegionCensus Region Smaller geographic areas (e.g., Zip code, county)Smaller geographic areas (e.g., Zip code, county)
Socio-demographic detailsSocio-demographic details Analytic cells and cell sizes (within IAP area)Analytic cells and cell sizes (within IAP area)
Age, gender, race/ethnicityAge, gender, race/ethnicity WIC participation, Poverty statusWIC participation, Poverty status Minimum population sizeMinimum population size>5 in unique cells5 in unique cells
Date of birth, shot dates, interview datesDate of birth, shot dates, interview dates Clearance by the NCHS Disclosure Review Clearance by the NCHS Disclosure Review
Board (DRB)Board (DRB) 19
The NIS PUF
Public-use data files (1995-2000): Child-Public-use data files (1995-2000): Child-level records with 78 IAP area (state and level records with 78 IAP area (state and urban) identifiers urban) identifiers (2001 NIS PUF by Fall 2002)(2001 NIS PUF by Fall 2002)
Approximately 35,000 age-eligible children Approximately 35,000 age-eligible children with household interview datawith household interview data
Approximately 23,000 children with Approximately 23,000 children with household interview household interview andand ‘adequate’ ‘adequate’ provider data; on average 295 children per provider data; on average 295 children per IAP area IAP area
PUFs released on the Internet and CD-PUFs released on the Internet and CD-ROMsROMs Internet : Internet : www.www.cdccdc..govgov//nisnis (1995-2000) (1995-2000) CD_ROMs No 1-5CD_ROMs No 1-5
20
Contents of The NIS PUF
Household Interview data itemsHousehold Interview data items Unique sequence ID, age at interview Unique sequence ID, age at interview
(months), gender, race/ethnicity, …(months), gender, race/ethnicity, … Mother’s age group, education, marital status, Mother’s age group, education, marital status,
…… WIC participation, income, poverty status,…WIC participation, income, poverty status,… Vaccine specific immunization history Vaccine specific immunization history
(#doses, up-to-date status) (#doses, up-to-date status) IAP area and State identifiers, Census Region IAP area and State identifiers, Census Region
Provider data itemsProvider data items Vaccine-specific Immunization History: Vaccine-specific Immunization History: age at vaccination(days/months), number of age at vaccination(days/months), number of
doses, up-to-date status (UTD)doses, up-to-date status (UTD) Providers’ information (e.g., facility type, VFC Providers’ information (e.g., facility type, VFC
participation)participation)
21
Contents of NIS PUF (cont.)
Composite vaccine-specific and vaccine Composite vaccine-specific and vaccine series flags with up-to-date statusseries flags with up-to-date status
Sampling weights and key analytic Sampling weights and key analytic variablesvariables Sampling weights: Sampling weights: Children with household interview (HY_WGT)Children with household interview (HY_WGT) Children with Provider data(W0)Children with Provider data(W0) Stratum + PSU identification variablesStratum + PSU identification variables
22
SummaryComplicated Childhood Immunization Complicated Childhood Immunization
Schedule Schedule
No single source is 100% complete or accurateNo single source is 100% complete or accurate
<1% children were observed with <1% children were observed with discrepancies in household and/or provider discrepancies in household and/or provider reported datareported data
Data reported from written ‘shot card’ are of Data reported from written ‘shot card’ are of high quality; Shot card use varies by State/IAP high quality; Shot card use varies by State/IAP areaarea
High vaccine-specific agreement in the UTD High vaccine-specific agreement in the UTD status between HH ‘shot card’ and provider status between HH ‘shot card’ and provider report(s) whenreport(s) when HH reported UTD HH reported UTD
23
Substantial underreporting in vaccination Substantial underreporting in vaccination coverage rates from the household coverage rates from the household reportsreports
Strongly supports the use of provider-Strongly supports the use of provider-reported immunization histories in reported immunization histories in computing vaccination coverage computing vaccination coverage estimatesestimates
Official estimates from provider reports Official estimates from provider reports may slightly underestimate coverage may slightly underestimate coverage rates due to missing provider data; rates due to missing provider data; however, provider data produce however, provider data produce comparable estimates across IAP areascomparable estimates across IAP areas
24
Summary
Visit the Web Sites
For information on the NIS For information on the NIS PUF and coverage estimates PUF and coverage estimates http://www.cdc.gov/nishttp://www.cdc.gov/nishttp://www.cdc.gov/nip/coverahttp://www.cdc.gov/nip/covera
gege