Is better measurement the solution?The case of ‘SES’ and ‘age’

Federica RussoCenter Leo Apostel, Vrije Universiteit Brussel

Centre for Reasoning, University of Kent

2

Overview

Measurement in social scienceSome classic and more recent discussionsA common theme: better measurement is better

Measuring ‘SES’ and ‘Age’Challenges in measurement and interpretationChallenges to the theme: better measurement is not always the

solution

Integrating qualitative and quantitative methodsObserve on small scale before you measureMeasure on large scale based on observation

3

MEASUREMENT IN SOCIAL SCIENCE

4

A theory of measurement

Suppes 1998

Two problems for measurement:

The problem of representationAttach a number to an ‘object’

Look at the structure of the theory, and yet …

The problem of determining the procedureThe choice of the scale also depends on theory

5

The focus on procedural aspects

Zeller and Carmines 1980

Follow Blalock: measurement is the process of linking abstract concepts to

empirical indicators

The possibility to answer research questions depends on robustness of our measurement procedures

Measurement procedures above theorising

6

Measurement and realism Cartwright and Chang 2008

Practitioner’s problem: whether measurements are correctPhilosopher’s problem: whether we measure what we want to measure

Nominalism conventionalism OR operationalismNaïve realism problem of justification and nomic measurement

In social science Suppes’ measurement theory solves representation problem and leaves

open procedure problemhow to measure a concept within a theory

Variability and contingency of concepts to measureNon value-free measurements

7

Realism and indicators

Bohrnstedt

In social science there are some clear and tangible measuresE.g. age, birth, number of children, marital status …

For more blurred conceptsObserve the covariation between indicators, and infer their reality

8

Establishing a trend

The worryWhat do we measure? Is it real?

The solutionIt must be real, somehow The better we measure the better we represent ‘real’ objects

Is it always the case?

Is realism the problem?And is better measurement the solution?

10

Measurement itself, especially if carried out using sophisticated instruments or analysed using complex methodology, is seen to have the attributes of ‘science’, and often taken effectively as a justification for believing the results that

are presented as if they have a meaningful relation to whatever social process they are claimed to measure.

Harvey Goldstein 2012

11

MEASURING SOCIO-ECONOMIC STATUS AND AGE

12

At the extremes of measurement, a common problem

Age• Very easy to measure

• What does it represent?• Does it have any

explanatory import?

SES• Very controversial how we

should measure it

• What does it represent?• What is its import in

explanation of social or social / health outcomes?

13

Measuring ‘Ballung’ conceptsCartwright and Bradburn

Measurement requirements:Characterisation; Representation; Procedure

ConceptsRefer to a single quantityHave unclear boundaries and relations (Ballung)

They hinder a development of social science into ‘proper’ science

How to represent Ballung concepts“One is to represent them with a table or vector of features laying out the dimensions along which the family resemblances in question lie […] The other is to shed much of the original meaning and zero in on some more precisely definable feature from the congestion that constitutes the concept.”

Then, go ahead with chosen procedure

14

Measuring SES

Theoretical approachesWeberian, Marxist, Colemanian

Identification of different indicators, different types of variables

Class stratificationGoldthorpe Class Schema

Grouping of types of workers

15

What do we need SES for?

Consider social epidemiology

SES is highly correlated with health outcomesAsbestos related deaths in Barking Cancer related deaths in Eternit workersCancer incidence in Taranto…

16

What does SES do?

Categorise?A classificatory variableWhat part of the populations are more exposed, have higher

prevalence …

Explain?Active part in the explanation of diseases

Mixed aetiology!

What are the active causal pathways from exposure to outcome?

Social practices / norms / habits to explain (and to prevent) exposure

17

Which one to choose?

Measurement – categorisation – explanation

Measurement, alone, does not explain

Measurement, alone, only categorises

Include SES to explain a phenomenon

18

Measuring age

Easy to measureAccessibility of data, straightforward question, …

Choose to measureCategoricallyContinuously

Easy data to get – use it!

19

Typical uses of ‘age’

ControlAdjust results of statistical analyses (control for age)

PredictAge structure helps predict results

Categorisegrouping and collapsing multiple categories into fewer categories

Care with loss of information, residual confounding

20

What age stands for

Biological ageA typical health status, for that age

Social ageSocial practices that are typical of that age

…

Any explanatory import?

21

SOURCES OF INFORMATION

22

Where do we get the information from?

Quantitative studies

Large samples, large data sets

Correlations to be validated

The bigger the better, the more precise the better

23

Where do we get the information from?

Qualitative studies

Small samples, small numbers

Detailed description of practices

Small does not allow generalisation

24

Establishing a trend

Sample:The bigger the better

MeasurementThe more precise the better

Should we always follow this trend?

26

The ‘extra’ information that statistics does not give us

Description of Practices

Interactions

Influences

Background

Norms

…

GO small FIRST!

27

The information that statistics does give us

Categorise the ‘practices, interactions, backgrounds, …’ into measurable variables

Is it generalisable?

An empirical question!

Now go BIG!

28

TO SUM UP

29

Traditional problem of measurement in social scienceThe trend: justify naïve realism by better measurement

Question the trend through two examplesSES and Age

One step backWhere do we get information

Focus on explanation rather than realism

We may need to describe before measuring

30

TO CONCLUDE

31

Better measurement is not necessarily panacea

To measure better we need to describe better

Difficulty: not just a social science trend

Oppose the trend in requests from policymakers

What is evidence

What information we can trust

What methods we can trust

32

What / why do we measure?In the area of data collection and presentation at the present time, likewise, there seems little ground for optimism. Even in those societies, such as parts of Australia, where crude league tables used to be eschewed, increasing political and commercial pressures seem to be gaining the upper hand. New technologies such as powerful dynamic computer graphics do have the potential to convey findings and patterns

in powerful ways, but whether they are used to inform rather than merely impress, remains an open question.

Perhaps the most that one can hope for is that we could reflect more on Galton and his legacy. In particular, a better understanding is needed of

the difference between data that ‘confirms’ a theory by

providing a good model fit, and data that allows us to explain observed data patterns using as much potentially falsifiable information as possible.

Harvey Goldstein 2012

33

REFERENCES

34

George W. Bohrnstedt, An Overview of Measurement in the Social Sciences. http://www7.nationalacademies.org/dbasse/Measurement_in_Social_Sciences.pdf

Burt R. 1991 Measuring age as a structural concept. Social Networks 13

Cartwright N. and Chang H. 2008 Measurement, in The Routledge Companion to Philosophy of Science, pp. 367-375.

Cartwright N. and Bradburn N., A theory of measurement.http://www7.nationalacademies.org/dbasse/Common%20Metrics_Measurement_for_Science_and_Policy.pdf

Goldstein H. 2012. Francis Galton, measurement, psychometrics and social progress. Assessment in Education: Principles, Policy & PracticeVol. 19, No. 2

Marks G. The measurment of socioeconomic status and social class in the LSAY project. Technical Paper http://www.acer.edu.au/documents/LSAY_techrep14.pdf

Reijneveld S A 1998 Age in epidemiological analysis, J Epidemiol Community Health 2003;57

Suppes P. 1998 Theory of Measurement. E. Craig (Ed.), Routledge Encyclopedia of Philosophy. pp. 243-249.

Zeller and Carmines 1980. Measurement in the social sciences. The link between theory and practice. CUP