June 5 2006Bled panel Surveying e-values1 Surveying eValues: Experiences and Challenges Panel at the...

June 5 2006 Bled panel Surveying e-values 1

Surveying eValues:Experiences and Challenges

Panel at the Bled ConferenceMonday June 5, 2006

Ronald Batenburg

Institute of Informationand Computing Sciences

Utrecht UniversityThe Netherlands


Agenda for this panel

14:00-14:20 Ronald Batenburg, Utrecht University Georgios Karageorgos, European Commission, DG

Enterprise & Industry (not present)14:20-14:40 Fernando Reis, European Commission, DG Eurostat14:40-15:00 Hannes Selhofer, Empirica Gesellschaft für

Kommunikations- und Technologieforschung15:00-15:20 Vasja Vehovar, University of Ljubljana15:20-15:30 Closing remarks an discussion


Introduction

Why this panel? My recent survey experience The STILE project and beyond My three main problems to tackle

in the next 5 years


Example: evaluating the Dutch‘Go Digital’ program

Goal of this policy program: improving the ICT/e-business maturity of Dutch SMEs to meet the ‘European’ level

Sponsored/funded by the Dutch Ministry of Economic Affairs

Executed by intermediary organizations, providing (free) IT advise, consultancy, workshops, etc

Budget: € 40 Mio over 2001-2005, app. 20,000 Dutch SMEs participated


Evaluation questions

Was the program executed efficiently and effectively?

Did the program made a significant difference to the ICT-maturity of Dutch SMEs?

(how) to continue with Go Digital program for Dutch SMEs in the near future?


Research questions

How to measure ICT/e-business maturity and effectiveness in SMEs?

How to conclude if the program was successful, i.e.:– How to make a ‘ fair’ comparison

between large, small and non-participants of the Go Digital program?

– How to control for other determinants of ICT maturity and developments over time?


Result 1: Go Digital participantsare very ICT mature

0

20

40

60

80

100

have an intranet have a corporatew ebsite

supportcustomers and e-

sales

IT is externallyconnected for

sales

IT is externallyconnected for

purchasing

Dutch average Go Digital participants non Go Digital participants

% yes


Result 2: Go Digital participantsare successful ICT users as well

1 1,5 2 2,5 3 3,5 4 4,5 5

ICT reduced costs

ICT increased product quality

ICT increased turnover

ICT stimlated product innovation

ICT increased labor productivity

ICT improved communication

ICT improved customer satisfaction

ICT saved time

Go Digital participants non Go Digital participants

Fully disagree Fully agree


Reflection (1)

Standardized questions on ICT/e-business maturity were available by Dutch Statistics

Standardized questions on the added value, effectiveness and success ICT/e-business were not available, or ‘under construction’

Hence:– Two new surveys among Dutch SMEs and ICT

were conducted (n=900)– ‘Acceptable’ response rates (30%) among SMEs

were realized only after major investments– The validity, reliability and potential future use

of the new surveys are to be explored


Reflection (2)

The Ministry of Economic Affairs (and its policy partners) have little incentives to monitor survey methods and measurements

Dutch Statistics has little incentives to interfere or advise in survey methods and measurements of other survey projects

We as (academic) researchers do have an incentive to conduct new and customized survey methods and measurements

Hence:– Users, sponsors, researchers and statisticians of

surveys are loosely coupled, not optimally cooperating or integrating

– New ICT/e-business/Go Digital evaluations will lead to new survey projects


The STILE project 2001-2004

Goals:“to provide innovative methodologies and content for the statistical monitoring of

the European labor market in the eEconomy” “a critical assessment of the European statistical system and to increase awareness

in the international statistics and research community because of the current lack of measurement accuracy”

Research partners: Cork Telework Centre (CTC) and Central Statistical Office (CSO), Ireland Institute for Employment Studies (IES), United Kingdom CAMIRE Estadística y Análisis, S.L. (CAMIRE), Luxembourg Institut für Arbeitsmarkt- und Berufsforschung (IAB), Germany Istituto di Ricerche Economiche e Sociali (IRES), Italy Organisatie voor Strategisch Arbeidsmarktonderzoek (OSA), The Netherlands Institute of Sociology-Hungarian Academy of Sciences (ISB), Hungary

Products: Ramioul M., Huws U. & Bollen A. (eds.), Measuring the Information Society, STILE

report, HIVA-K.U.Leuven, Leuven www.stile.be

– Questionnaire database– Digital toolkit


The 11 STILE enterprise surveys

0 10 20 30 40 50 60 70 80 90

Germany-1987-1997

US-1987-2002

UE-1998

Canada-1996-1999

Sweden-1995-1997

Germany-1993-1000

US-1994-2000

Belgium-1992-1997

Netherlands-1987-2001

Belgium-1999

UK-1980-1989

•Longitudinal or panel

•Telephone, web, F2F, postal

•Sample size 1,000-23,000

•Stratified


The STILE survey and questionnaire database

Very informative overviews, however …– What would be the ‘best’ survey questions?

• Or: when to apply which questions?

– What would be the ‘best’ way to sample and approach enterprises for surveys?• Or: when to apply which approaches?

– What would be the ‘best’ method to conduct survey enterprise questionnaires?• Or: when to apply which methods?


Again: the actors involved(derived from STILE project)

Statisticians(academic)Researchers

Policy-makers Respondents

1

2 3


My Research Agenda

Three main problems to be tackled in the next five years:

1. To improve the use of questionnaire and survey standards, by researchers and research organizations

2. To improve the monitoring of questionnaire and survey standards, by statisticians

3. To improve the awareness of questionnaire and survey standards, by policy makers

19th Bled eConference, Surveying eValues Panel

Bled, 5 June 2006

e-Business W@tch in the context of e-business related activities of

DG Enterprise and Industry

Georgios KARAGEORGOS

European Commission, DG Enterprise & Industry

Overview of the presentation

• Background info and policy context (slides 3-5)

• What is the e-Business W@tch (slides 6-13)

• Lessons learned and future plans (slides 14-16)

• Support / Background info (slides 17-20)

Main lessons learned

• Fulfilling existing, real need => Great interest from stake-holders (even university from Mexico) + support to continue;

• Project unique (internationally?) in combining statistics (going one step further than ‘traditional’ surveys), qualitative analysis (incl. case studies) and policy recommendations;

• Strong points = Timeliness, quality and transparency!

• ‘Victims of success’: Growing expectations, demands and involvement BUT different interests => Can NOT satisfy all!

• Continuous trade-offs, e.g. increase sectoral focus => reduced survey’s capacity;

• Increased burden for flexibility and coordination capacities.

Future Plans: a ‘new’ eBW (within the policy context described in slide 5)

• Insights from on-going project, interim evaluation (by IDEA) and Cambridge conference (http://ec.europa.eu/enterprise/ict/conferences/cambridge.htm);

• Emphasis on 3 themes: Impacts, impacts and impacts!

• Focus on specific sectors, companies’ size classes & regions;

• Approach: Maintaining quantitative (data) and qualitative (case studies) information collection, combined with high quality analysis, leading to identification of policy challenges;

• ‘Common’ data coming from available official statistics (mainly Eurostat), complemented by purpose-made surveys;

• More flexibility of implementation arrangements.

For more information:

http://europa.eu.int/comm/enterprise/ict/index.htm

http://www.ebusiness-watch.org

[email protected] & [email protected]

[email protected]

Thank you for your attention!

June 5 2006Bled panel Surveying e-values1 Surveying eValues: Experiences and Challenges Panel at the...

Documents

Transcript of June 5 2006Bled panel Surveying e-values1 Surveying eValues: Experiences and Challenges Panel at the...