European Social Survey THE EUROPEAN 3rd International ......ESS Conference Lausanne 15 July 2016 2...

European Social Survey

ESS is a European Research Infrastructure Consortium (ESS ERIC)

THE EUROPEAN SOCIAL SURVEY INFRASTRUCTURE: PAST, PRESENT AND FUTURE Dr. Rory Fitzgerald Director, ESS ERIC City University London

3rd International ESS Conference Lausanne

15 July 2016

2

European Social Survey infrastructure

Looking back: the past

Where we currently are: the present

Looking forward: the future

Today’s session:

THE PAST

3

4

European Social Survey: the past

Why we needed ESS

“There is a wealth of data on individuals and households that are regularly collected by statistical offices and other administrative agencies. These data, however, are frequently not comparable across nations and are often not accessible to researchers for reasons of administrative self-interest or data protection. Absolutely crucially, however, they do not deal with the whole range of individual orientations (attitudes, beliefs and behaviours) that are central in understanding modern societies, and that can be assessed by survey research. “Very many individual surveys have been and will be conducted in particular European countries. Inevitably, however, these are tailored to individual projects and interests. The social sciences, if they are to make progress, require regular cross-national surveys that are conceptually well anchored, conducted according to rigorous methodological standards and are available at little cost to the entire research and policy community.” ESF Blueprint, March 1998

5

European Social Survey: the past

Key milestones for ESS: ESS commission funding (2001) 22 countries took part in Round 1 (2002) 36 European countries involved (2002-16) Received Descartes Prize (2005) Infrastructure funding (from 2006) Groves Review (2008) ESFRI Roadmap ERIC (2013) ESFRI landmark (March 2016)

“The panel unanimously finds that the importance of ESS, its demonstrated success in initial launch, and its clear signals of impact justify fully continuous funding at levels necessary to achieve its vision and maintain its quality.” Groves Review, 2008

THE PRESENT

6

7

European Social Survey: the present

Use of ESS infrastructure

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

100000

janv..05 janv..06 janv..07 janv..08 janv..09 janv..10 janv..11 janv..12 janv..13 janv..14 janv..15 janv..16

Registered ESS Users (January 2005 - January 2016)

8


Data users and data sessions per month

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Dec 2003

Dec 2004

Dec 2005

Dec 2006

Dec 2007

Dec 2008

Dec 2009

Dec 2010

Dec 2011

Dec 2012

Dec 2013

Dec 2014

Dec 2015

Data users

Data sessions

9


Data users and data sessions per year

0

5000

10000

15000

20000

25000

30000

35000

40000

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

Data users

Data sessions

10


Data downloaders per year

0

2000

4000

6000

8000

10000

12000

14000

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

Downloaders

Trends in ESS academic publishing (Google Scholar, 2003-2014, N=2705)

1131

89

133

193210

263292

313

375 379

416

0

50

100

150

200

250

300

350

400

450

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Overall Journal articles Books & chapters Conference & working papers

2705 publications overall:▪ 1281 journal articles▪ 472 books & chapters▪ 470 working papers▪ 330 conference papers

Journals with most ESS publications (2003-2014, N=1281)

11.9

2.9

5

6.3

8.1

11.5

19.2

35.1

0 5 10 15 20 25 30 35 40 45 50

Other

Media, Public opinion

Psyshology

Health, Medicine

Methods, Statistics

Economy

Political science

Sociology

▪ Social Indicators Research 60 ▪ European Sociological Review 55 ▪ European Societies 32 ▪ Journal of European Social Policy 19 ▪ International Journal of Public Opinion Research 18 ▪ Survey Research Methods 17 ▪ International Journal of Sociology 14 ▪ International Journal of Comparative Sociology 14 ▪ Comparative Political Studies 14 ▪ West European Politics 14 ▪ Journal of Cross-Cultural Psychology 13

Topics addressed in ESS publications (2003-2014, N=2705)

4053333

4795

111114116120

136139139

159185

209222227233234

282289

338600

0 100 200 300 400 500 600 700

MethodsTransition

EnvironmentEuropeNation

Media-ICTCrime

ReligionEducation

HealthGender

CitizenshipAge groups

Culture, valuesInequalities

EconomySocial capital

FamilySWB-QOL

WorkPublic policies, welfare

ImmigrationPolitics, democracy

67

71

73

83

102

199

0 100 200 300

Democratic system

Political attitudes, culture

Political protest

Political parties

Political trust

Political participation

Distribution of 1st author's affiliation across countries (2003-2014, N=2705)

3132363738

52555660616364

7172

106137

150164

245272

319363

0 100 200 300 400

RomaniaHungary

AustriaGreece

IsraelEstoniaIreland

PortugalPoland

DenmarkFinlandFrance

SwitzerlandNorway

ItalySpain

SwedenBelgium

NetherlandsUSA, Canada

GermanyUnited Kingdom

49.2

13.4

18

8.9

10.5 Europe-WestEurope- NorthEurope-SouthEurope-EastNorth America

% share of publications where a country was included (1)

(sorted according to the number of rounds fielded, 2003-2014, N=1520)

38.139.7

4852.9

55.755.856.1

6060.3

63.965.666.6

68.970.570.971.472.372.473.573.7

0 20 40 60 80 100

SlovakiaEstonia

Czech RAustria

SwitzerlandHungarySlovenia

PolandIreland

NorwayPortugal

FranceFinland

NetherlandsBelgium

SpainDenmarkSweden

United KingdomGermany 7 rounds

6 rounds

Trend of multiple round usage % share of publications where more that one ESS round was used, (2003-2014,

N=1580)

06.3

1523 25.4 26.1

34.2 33.742.4

49.8

0

10

20

30

40

50

60

70

80

90

100

<2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

more than 1 round 3 rounds 4 rounds 5 rounds

18


Non-academic impact of the ESS Setting new standards in cross-national research Open data Methods adopted Policymakers and Government Press coverage

19


Non-academic impact of the ESS

HAS PRESENT MET CHALLENGES OF PAST

20

MEASUREMENT EQUIVALENCE ACHIEVED?

In each round two complex concepts in the ESS are tested for measurement equivalence, i.e. whether the same construct is being measured across countries or linguistic groups.

Testing in the ESS is done in three steps:

Configural invariance: testing evaluates whether a theoretical model holds cross-nationally, i.e. whether the factor structure is the same.

Metric invariance: testing restricts the factor loadings to be equal across groups. Its interpretation is that respondents across groups attribute the same meaning to the latent concept under study. →Only if metric invariance is established the relationship with other variables can be studied.

Scalar invariance: testing restricts (in addition) the intercepts to be equal across groups.

→ If established groups’ latent means can be compared.

Concept tested Number of indicators

Across countries / groups (combination of language and country)

Interest in Political Issues presented in the Media

3 29/30 countries (R1-R4)

Depression 8 6/27 countries (R3 and R6)

Trust in police procedural fairness

3 18/27 countries (R5)

Political satisfaction 2 10/25 (R1- R5)

Quality of state services 2 7/25 (R1 – R5)

Trust in political institutions 2 21/33 groups (R2-R5)

Trust in political authorities 2 33/33 groups (R2-R5)

Internal political efficacy 3 14/18 groups (R7 first data release)

External political efficacy 3 15/ 18 groups (R7 first data release)

Scalar invariance - Allowing for comparison of relationships and means across groups in each Round

Concept tested Number of indicators

Within country longitudinal results

Interest in Political Issues presented in the Media

3 29/30 countries (R1-R4)

Depression 8 22/22 countries (R3 and R6)

Political satisfaction 2 For most countries that participated in 4 or 5 rounds, within-country scalar equivalence established over 4-5 rounds.

Quality of state services 2 For most countries that participated in 4 or 5 rounds, within-country scalar equivalence established over 4-5 rounds.

Scalar invariance - Allowing for comparison of relationships and means across rounds within each group

Equivalence achieved? • Scalar invariance test is a strict test requiring loadings and intercepts to

be equal across groups, even though it could be established for most of the groups tested.

• Sometimes substantive researchers are more interested in relationships with other variables than in the comparison of latent means. For that metric invariance is a sufficient requirement. ESS concepts exhibit high metric invariance.

• When invariance is established the measurement of a concept is not dependent on group membership, revealing differences across countries as substantive.

• Invariance testing provides valuable information for analysts, for instance:

Common understanding of political affairs across Europe:

- Concepts asked in the ESS about political attitudes exhibit high measurement equivalence.

- ‘Interest in political issues presented in the media’ was scalar invariant across 4 rounds tested and 29 out of 30 countries

- ‘Trust in political authorities’ was scalar invariant in 4 rounds tested and 33 out of 33 groups

Interpretation of the ‘depression scale’ is not the same across European countries although it is consistent over time within countries.

THE FUTURE

25

26

European Social Survey: the future

ANALYSIS CHALLENGES Weighting

Not always applied (not clear in journals) Most common user query Complex sampling designs not accounted for Use of a harmonized methodology for post stratification weights needs to be evaluated

Interviewer effects not accounted for Differences in question quality rarely accounted for Equivalence testing not always performed The curse of the ESS total column in tables More on-line training and guidance required in future Combining data across rounds to increase sample size of sub groups – issues?

27


Reasons for rejecting mixed mode interviewing Synthesis report (Villar and Fitzgerald, 2016) ESS has always used face-to-face interviews Six studies into measurement, representativeness and cost efficiency Same sampling frame as the standard ESS Response rates remained lower in all mixed-mode designs compared to face-to-face design Telephone interviews had poor response rates Mixed-mode online performed better – results still disappointing Nonresponse bias may be increased by moving to mixed-mode Mode effects would be introduced but no easy way to correct for these Estonia – hope for future?

A collaboration between:

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 654221.

www.seriss.eu @SERISS_EU

• European Social Survey (ESS ERIC)

• Survey of Health, Ageing and Retirement in Europe (SHARE ERIC)

• Consortium of European Social Science Data Archives (CESSDA AS)

• Generations and Gender Programme (GGP)

• European Values Study

• WageIndicator Survey

• Strengthen social science research across Europe and beyond by overcoming fragmentation and fostering interoperability, harmonisation and innovation

• Equip social science infrastructures to play a major role in addressing

key societal challenges and ensure that national and European policy making is built on a firm socio-economic evidence base

• Promote the value of the social sciences to the wider research

community, and policymakers



Designed to:

Achieved by:

• Addressing key challenges for cross-national data collection e.g. accurately representing the population, achieving equivalence through translation

• Breaking down barriers between social science infrastructures

via training and networking events and the development of shared online tools to facilitate harmonised data collection and documentation.

• Embracing the future of the social sciences by examining the legal

and ethical challenges associated with new forms of data, developing a cross-national probability-based web survey and exploring automated coding for socio-economic variables



32


Methodology: CRONOS web panel Assessing viability of obtaining accurate data online CRONOS will use ESS to recruit respondents from three countries Recruits will take part in 12-month web panel Respondents will be asked to complete a 20-minute web survey every other month at any point during a two-month period Surveys will be programmed and administered using Questback-EFS Questions will be taken from European Value Study, ESS and European Quality of Life Survey Respondents will be given a small token of appreciation Some data will be freely accessible If positive outcome, could be adopted by ESS in future

Commitment to the ESS

16 countries ESS ERIC Members (6 Guests)

Austria Portugal

Belgium Slovenia

Czech republic Sweden

Estonia Switzerland* France UK Germany (Hungary) Ireland

Lithuania (Finland) Netherlands (Iceland) Norway (Israel) Poland (Russia)**

36 Countries have taken part in at least one round of the survey:

27 EU states (not Malta)

+ 9 others:

Norway, Switzerland, Israel, Turkey, Russia, Ukraine, Iceland, Kosovo, Albania

34


ESS SUSTAIN Membership Development activities

Membership Development Advisor appointed Roaming Ambassadors appointed Events / seminars in target countries

Funding and Governance Working Group (including BREXIT) Impact case study Enhanced communications

Increasing media presence Launched social media presence Updating News section on ESS website Working with national teams to promote ESS in media across Europe

Funded by Commission (H2020)

35

European Social Survey: conclusions

Clear success of the ESS has been demonstrated and the current model is sustainable Have to keep making the case for rigour Need to further increase visibility of the project Use of data and comparability findings suggest ESS is a leader in the field Mixed mode would undermine quality but FtoF fieldwork costs rising: web additions in the future? ESS core questionnaire will be reviewed Rotating modules continue to allow new topics to be included SERISS project offers opportunities for joint methodological development Especially CRONOS panel SUSTAIN project helping to secure the future but challenging landscape BREXIT is a new challenge

CONTACT

www.europeansocialsurvey.org [email protected]

@ESS_Survey european-social-survey

European Social Survey THE EUROPEAN 3rd International ......ESS Conference Lausanne 15 July 2016 2...

Documents

Transcript of European Social Survey THE EUROPEAN 3rd International ......ESS Conference Lausanne 15 July 2016 2...