4th Quality Conference (4QC) Impact Assessment Study

Rui Sousa, PhDCatholic University of Portugal

Lisbon, 16 July 2007

Agenda

• Background to the study

• Model underlying questionnaire

• Pilot test results

• Survey methodology and key dates

Background to the Study

• Goal of Quality Conferences is to improve the quality of Public Services

• UK Evaluation of Quality Conferences Report– It is difficult to give examples of improvements to

service delivery made as a result of attending the QCs.

– There should be a greater emphasis on evaluating the QCs and a greater, more structured follow-up of the impact on:

• Participants work activities as practitioners and decision makers

• Direct or indirect inspiration for action

Questionnaire Development

• Review of the QCs documents.

• Review of the literature on Quality Management and Public Services.

• Exploratory interviews with 4QC participants in Portugal to capture specific context of 4QC.

• Discussions with DGAEP.

Target Respondents

• Participants in the 4QC:– Public Sector– Non-Profit Organization

• a not-for-profit organization with public interest goals but which does not belong to the Public Sector

– Private Sector• an organization which provides services to Public

Sector organizations (e.g., information technology, software applications, consulting), but which does not itself provide public services to citizens

Underlying Model – Public Sector

Participation in 4QC

B. Creation of an Environment Conducive to Change

C. Increase in the Use of Quality Practices

D. Barriers to Action and Suggestions for Improving Effectiveness of Future Conferences

A. Respondent Profile

B. Environment Conducive to Change

• Increased knowledge– Ex: Increased knowledge about best practices

in public services

• Motivation for change/improvement– Ex: Inspiration for change

• Implementation advice– Ex: Obtaining advice for implementing and

managing change

C. Quality Practices

• Adapted from EFQM framework:– Top management commitment– Citizen focus– Employee involvement– Process improvement and information

management– Formal organization for quality– Social responsibility

D. Barriers to action

• Specific to the organization

• Specific to the conference

Underlying model – Private and Non-Profit

Participation in 4QC

Improved Support to Public Sector Organizations

Suggestions for Improving Effectiveness of Future Conferences

Respondent Profile

-Increased knowledge of Public Sector needs

-Improved relationships

-Development of new services

Pilot Test ResultsRespondents Profile

• 32 respondents (2 July 2007)

• Public Sector 29 90.6%• Non-Profit Organization 1 3.1%• Private Sector 2 6.3%

• From 15 different countries• Germany - 6• Portugal - 4• Finland, France, Ireland, Latvia, Netherlands,

Poland, Slovenia, Spain, Sweden - 2 from each country

• Greece, Italy, Nigeria, Switzerland - 1 from each country

• Average completion time

16 min 7 sec

Pilot Test Results Respondents Profile

• Age • Under 36: 6 18.8%• 36-45: 9 28.1%• 46-55: 11 34.4%• Over 55 6 18.8%

• Female (11 – 34.4%) Male (21 – 65.6%)

• Intention to participate in the 5QC• Confirmed 56.3%• Don’t know yet 43.7%

• Within the public sector• Central administration 24 82.8%• Regional administration 1 3.5%• Local administration 4 13.8%• European institutions 0

• Manager 19 65.5%• Employee 10 34.5%

Pilot Test ResultsRespondents Profile

Selected Results

Overall, to what extent has the participation in the conference contributed to and/ or facilitated the implementation of any changes in your organization?

1 - 4 12.12%

2 - 5 15.15%

3 - 11 33.33%

4 - 11 33.33%

5 - 1 3.03%

NA - 1 3.03%

Selected ResultsWhat difficulties, if any, did you come across in transferring the knowledge and skills that you learned about in the 4QC into action? (e.g., what barriers did you find in proposing and/ or implementing some of the practices considered above?) a) Difficulties specific to my organization

Lack of resources (money, time, people) - 17 26.56%

Lack of top management commitment - 11 17.19%

Political changes - 10 15.63%

Fear and resistance to change - 9 14.06%

“Not invented here“ syndrome - 6 9.38%

Projects already under way that would not be modified - 8 12.50%

Organization too far behind, not ready for best practice - 3 4.69%

Selected Resultsb) Difficulties specific to the conference

A conference is not the proper forum to achieve effective knowledge transfer, regardless of how it is formatted and organized

- 4 10.26%

The conference format was not conducive to effective knowledge transfer

- 6 15.38%

The presented practices were not applicable in the context of my organization

- 6 15.38%

Inadequate information about how to adapt the practices and make them work

- 4 10.26%

Uncertainty about the merits of the presented practices and their likelihood of success

- 13 33.33%

Poor quality of the presentations - 5 12.82%

Did not have enough prior knowledge to adequately absorb the concepts discussed in the conference

- 1 2.56%

Main Changes Made

• Removal of the definitions for “Manager” and “Employee”.

• Question added on “Length of employment with organization”.

• A few improvements in the wording and sequence of questions.

• Shortening of section 7 of the questionnaire (2 questions deleted).

• Open-ended section soliciting suggestions for improving the effectiveness of future QCs.

• Modified questionnaire:

http://asp.infinivista.com/cesop/survey.php?sid=29

(pwd protected)

• Note: Old pilot test version is still accessible, but will not be acted on.

Survey Methodology

1) Obtain and validate list of e-mails of participantsa. Via IPSG members

b. By sending a letter by post, asking for a valid e-mail address

2) Invitation e-mail sent by web survey software with link to questionnaire.

3) Questionnaire open for 2 weeks- Automated and IPSG-driven follow-up

4) Analysis of results and final report

Key Dates

• 3 August: Validation of e-mail addresses completed.

• 3 September: Invitation e-mails sent.

• 3-19 September: Questionnaire active online.

Thank you for yourcollaboration