Reliability and Validity

Introduction to Study Skills & Research Methods (HL10040)

Dr James Betts

Lecture Outline:•Definition of Terms

•Types of Validity

•Threats to Validity

•Types of Reliability

•Threats to Reliability

•Introduction to Measurement Error.

Some definitions…• Validity

• Reliability

• Objectivity

Types of Experimental Validity• Internal

• External

Logical Statistical

Validity

Logical Validity

Statistical Validity

Logical/Statistical Validity

Interesting Example: Breast Cancer• Incidence: ~1 % (0.8 %)

(i.e. a positive result should be detected for approximately 1 in every 100 women tested)

• Sensitivity: ~90 % (87 %)(the mammogram is sensitive enough that approximately 90 in every 100 breast cancer patients will receive a positive result)

• Specificity: ~90 % (93 %)(the mammogram is specific enough that approximately 90 in every 100 healthy patients will receive a negative result).

Data from Kerlikowske et al. (1996)

Quick Test

• What is the probability that a patient receiving a positive result actually has breast

cancer?

Threats to Internal Validity

e.g.

O1 T O2

PO3

RO4

Threats to Internal/External Validity

Threats to Internal/External Validity

Threats to External Validity

Reliability

Subject 1 60 ml.kg-1.min-1 60 ml.kg-1.min-1 60 ml.kg-1.min-1

Subject 2 55 ml.kg-1.min-1 55 ml.kg-1.min-1 55 ml.kg-1.min-1

Subject 3 70 ml.kg-1.min-1 70 ml.kg-1.min-1 70 ml.kg-1.min-1

Reliability

Subject 1 60 ml.kg-1.min-1 65 ml.kg-1.min-1 65 ml.kg-1.min-1

Subject 2 55 ml.kg-1.min-1 60 ml.kg-1.min-1 60 ml.kg-1.min-1

Subject 3 70 ml.kg-1.min-1 75 ml.kg-1.min-1 75 ml.kg-1.min-1

Reliability

Subject 1 60 ml.kg-1.min-1 72 ml.kg-1.min-1 57 ml.kg-1.min-1

Subject 2 55 ml.kg-1.min-1 61 ml.kg-1.min-1 52 ml.kg-1.min-1

Subject 3 70 ml.kg-1.min-1 40 ml.kg-1.min-1 84 ml.kg-1.min-1

Types of Reliability

Subject 1 60 ml.kg-1.min-1 63 ml.kg-1.min-1 57 ml.kg-1.min-1

Subject 2 55 ml.kg-1.min-1 56 ml.kg-1.min-1 48 ml.kg-1.min-1

Subject 3 70 ml.kg-1.min-1 65 ml.kg-1.min-1 66 ml.kg-1.min-1

Rater Reliability (Objectivity)• Intrarater reliability

• Interrater reliability

Threats to Reliability

Measurement Errors• Ultimately, reliability is dependent on the

degree of measurement error in a given study

• The overall error in any measurement is comprised of both systematic and random error

• We will address measurement error further next week…

Literature Search Assignment

• The handout lists 8 questions which can be answered through retrieving the corresponding source articles

• Answer as many as possible and hand them in at next week’s lecture

• Does not contribute to final assessment but will be marked

• DO NOT contact author or order articles.

Selected Reading• Atkinson, G. and A. M. Nevill. Statistical methods for

assessing measurement error (Reliability) in variables relevant to sports medicine. Sports Medicine. 26:217-238, 1998.

• Holmes, T. H. Ten categories of statistical errors: a guide for research in endocrinology and metabolism. American Journal of Physiology. 286: E495-501.

• Thomas J. R. & Nelson J. K. (2001) Research Methods in Physical Activity, 4th edition. Champaign, Illinois: Human Kinetics

J.Betts@bath.ac.uk