Steps in assessing the transferability of
foreign CEA studies
Lebanese Society for Pharmacoeconomics and Outcomes Research (LSPOR)
Silvia Evers/Professor of Public
Health Technology Assessment
Transferability, a lot of terms
• applicability
• exchangeability
• extrapolation
• external validity
• generalisability
• portability
• relevance
• transportability
Definition
“Generalisability is defined as the extent to
which the results of a study can be
generalised to the population from which the
sample was drawn.
Whereas, transferability is conceptualised as
the extent to which the results of a study, as it
applies to a particular setting, hold true for a
different population or setting”
Knies et al, European Journal of Hospital Pharmacy-Pharmacoeconomics, 2008; 14(4): 51-54
Definition (2)
“The Task Force’s working definitions were that economic evaluations were generalizable if they applied, without adjustment, to other settings. On the other hand, data were transferable if they could be adapted to apply to other settings. Also, the terms ‘country’, ‘jurisdiction’ and ‘location’ are used interchangeably to mean any setting where there is a need for local estimates of cost-effectiveness.
ISPOR Task Force
• Applying economic evaluations across
jurisdictions (setting needing local cost-
effectiveness estimate):
– generalizable: without adjustment
– transferable: adaptation needed and
possible
– impossible: own study necessary
Drummond M et al. Transferability of economic evaluations across jurisdictions:
ISPOR good research practices task force report
Value in Health 2009; 12(4): 409-418
Transferability issues
Content
• Transferability limiting factors
• How to diagnose transferability?
• Questions
Limiting factors transferability (1)
• Basic demography and epidemiology of
disease (age structure of the population,
incidence of various disease, etc)
• Availability of health resources and variation
in clinical practice (range of treatments,
clinical guidelines)
• Incentives to health care professionals and
institutions (physicians fee-for-service or
salary; hospitals input or output based) OECD. A disease-based comparison of health systems, 2003
Limiting factors transferability (2)
• Relative prices or costs (prices do not
reflect costs, difference in relative prices
affect CEA)
• Population values (health state
preferences valued may vary by
demographical area)
OECD. A disease-based comparison of health systems, 2003
Limiting Factors; Goeree (1)
• Patient Characteristics – age, gender, race, education, socio-economic status,
population density, immigration and emigration patterns, risk factors, comorbidities, lifestyle, genetic factors, environmental factors, mortality
– attitude toward health care and treatment, compliance and adherence rates, utility valuation of health states
– patient incentives such as the impact of user fees, co-payments and deductibles on the use of health care
• Disease Characteristics (incidence and prevalence of disease, disease severity, and progression/prognosis of disease)
Goeree et al, Current Medical Research and Opinion, 2007; 24(4): 671-682
Limiting Factors; Goeree (2)
• Provider Characteristics (clinical practice, conventions, guidelines, or norms across countries, rate hospitalizations, rates of diagnostic testing, lengths of stay in hospital, the frequency of physician follow-up, and drug dosing regimens, experience, education, training, skills, efficiency)
• Health Care System Characteristics (absolute and relative unit costs, types and magnitude of resources available, ‘usual care’, inputs used in health care delivery, the organization and structure of the health care system, the level of technological innovation used, level of technical efficiency)
Goeree et al, Current Medical Research and Opinion, 2007; 24(4): 671-682
Limiting Factors; Goeree (3)
• Methodological Characteristics (study
perspective, the timing of the economic
evaluation itself, and clinical endpoints
or outcome measures used in the study)
Goeree et al, Current Medical Research and Opinion, 2007; 24(4): 671-682
How to diagnose transferability?
Eligibility to transferability:
• Heyland
• Späth
• general knock-out criteria Welte
Instruments to diagnose
• Welte
• Boulenger/EURONHEED
• Urdahl
Heyland’s generalizability criteria
Clinical generalizability
1. Are the patients described in the analysis similar to those
patients you see in your own setting?
Systems generalizability
1. Is the viewpoint of the analysis relevant to your clinical setting/
situation?
2. Is the intervention under study generalizable to your setting?
3. Are the costing methods applicable to the health care system in
which you work?
4. Are the outcomes measured appropriate to your setting?
5. Is the discount rate applicable to your setting?
Heyland et al. Critical Care Medicine. 1996; 24(9):1591-1598
Welte
• User friendly list based on knock-out criteria
• Steps:
Diagnosing the transferability
- general knock-out criteria
- specific knock-out criteria
- estimate relevance of a specific knock-out criteria
- estimate correspondence between 2 countries relating to these specific knock-out criteria
- estimate effect on the ICER of decision country
Welte et al, Pharmacoeconomics, 2004; 22(13): 857-876
General knock-out criteria
1. Evaluated technology not comparable
to the one used in decision country
2. Comparator is not comparable to
decision country (drug not licensed)
3. Study does not possess and
acceptable quality
If any yes, no transferability possible
Welte et al, Pharmacoeconomics, 2004; 22(13): 857-876
Specific knock-out criteria (1)
Methodological criteria:
- Perspective (NICE NHS perspective, other guidelines societal perspective) € & E
- Discount rate € & E
- Medical cost approach (how are costs valued (charges, fees, market prices/inclusion of overhead approach) €
- Productivity cost approach (HCA, FCM, QALY) €
Welte et al, Pharmacoeconomics, 2004; 22(13): 857-876
Specific knock-out criteria (2)
Health care system characteristics:
• Absolute and relative prices in healthcare €
• Practice variation € & E
• Technology availability €
Above factors influenced by
(i) Market structure and regulation
(ii) Staff characteristics & effects of learning
(iii) Incentives to health care providers
(iv) Place of technology
Welte et al, Pharmacoeconomics, 2004; 22(13): 857-876
Specific knock-out criteria (3)
Population characteristics:
• Disease incidence/prevalence €
• Case-mix € & E
• Life expectancy € & E
• Health-status preferences E
• Acceptance, compliance, incentives to patients € & E
• Productivity and work-loss time €
• Disease spread € & E
Welte et al, Pharmacoeconomics, 2004; 22(13): 857-876
Welte and transferability factors
Transferability
factors
Factors
Methodological
characteristics
perspective; discount rate; medical cost
approach; productivity cost approach
Health care
characteristics
absolute and relative prices; practice variation;
technology availability
Population
characteristics
incidence/prevalence; case-mix; life
expectancy; health-status preference;
acceptance, compliance and incentives to
patients; productivity and work-loss time1;
disease spread
Specific knock-out criteria (4)
Following steps:
• estimate relevance of a specific knock-out criteria (not relevant, very low to very high)
• estimate correspondence between 2 countries (very low to very high)
• estimate effect on the ICER of decision country (unbiased, to low, to high)
Welte et al, Pharmacoeconomics, 2004; 22(13): 857-876
Analysis specific knock-out
criteria • Determine relevance (item relevant)
• Assessment of correspondence (high low)
• Assessment of transferability
Relevance Correspondence Judgement
transferability
High High No problem
High Low Problem
Low High No problem
Low Low Could be problem
Level of adaptation
Approach
Source of data by transferability
factor
Clinical
efficacy
Resource
utilization
Unit cost
data
1 Study country
only
Study country
only
Mixture
2 Study country
only
Study country
only
Target country
only
3 Study country
only
Mixture Target country
only
4 Study country
only
Target country
only
Target country
only
5 Target country
only
Target country
only
Target country
only
Least to most
country specific
analysis
Goeree et al, Current Medical Research and Opinion. 2007; 24(4): 671-682
Simple adaptation
1. Exchange rates
2. GDP and PPP
• Gross Domestic Product (GDP) = price component conversion of local currency into 2000 constant local currency
• Purchasing Power Parities (PPP) = convert 2000 constant local currency into international dollars by taking into account the difference in price level and purchasing power between countries (Mc Donalds factor) – The PPP theory uses the long-term equilibrium exchange rate of
two currencies to equalize their purchasing power
– This purchasing power exchange rate equalizes the purchasing power of different currencies in their home countries for a given basket of goods
Evers et al, European Journal of Clinical Microbiology & Infectious Diseases, 2007, :531-40
Statistical analysis
1. Studies based on the analysis of patient-
level data - regression analysis
advocated as a means of looking at
variability in economic results across
locations.
2. Multi-level approaches
Beyond (before?) diagnosing
transferability?
3 opportunities to increase transferability:
• Design stage
• Analysis stage
• Reporting stage
Recommendation for increasing
transferability; design stage Study element Selection
Study sites (centers) Representative jurisdiction
Patient enrollment Reflect the normal clinical case load; variation in
centres and patients
Treatment alternatives Selection relevant for the jurisdiction; comparator
should reflect current practice
Perspective(s) Differences in requirement; government, insurance
and societal; preferred societal
Resource use and costs Collect quantities separately from prices/unit costs.
Health state preference
values
Generic instruments (e.g. EQ-5D; HUI
Drummond, et al International Journal of Technology Assessment in Health Care, 2005, 165-171
Recommendation for increasing
transferability; analysis stage
• Data that are used to populate a decision model should be justified given the stated target decision-maker(s) or jurisdiction(s). This justification will apply not just to unit costs but to resource use, effectiveness, and preference value data.
• Meta-analysis or multi-parameter analysis
• Data from outside; exchangeable?
• Analysis should assess variability
Drummond, et al International Journal of Technology Assessment in Health Care, 2005, 165-171
Recommendation for increasing
transferability; reporting stage Study element Reporting recommendations
Study sites (centers) Describe the characteristics of the centers participating in the trial. If these are from
different countries also report the relevant features of the various health-care
systems.
Patient enrollment Report the types of patients excluded from the trials and the percentage of the normal
caseload that these represent. Comparison with the relevant patient population
outside the trial centers.
Treatment alternatives Describe the alternatives in detail, so that study users can assess the relevance to their
own setting.
Perspective(s) Report costs and benefits by each relevant perspective.
Resource use and
costs
Report quantities separately from prices/unit costs.
Health state preference
values
Report the source of the values and any instrument used.
Analysis of variability Provide details of quantitative analysis of variability by location. Ideally, this will be
based on statistical analysis (such as multilevel modeling), but should at least
incorporate standard sensitivity analysis.
Other analytical issues Provide details on the extent of incomplete observations (i.e., missing and censored
data).
Detail the characteristics of patients with incomplete data.
Describe the methods used to address the problem.
ISPOR Task Force
ISPOR Good Research Practices Task Force on Economic Data Transferability
(i) to review what national guidelines for economic evaluation say about transferability;
(ii) to discuss which elements of data could potentially vary from setting to setting;
(iii) to recommend good research practices for dealing with aspects of transferability (including analytic strategies and guidance for considering the appropriateness of evidence from other countries).
Drummond et al, Value in Health. 2009; 12(4): 409–418
Questions !!!
Welte, Boulenger, Urdahl,
transferability, generalisability #
Top Related