Graduate surveys as an outcome evaluation

Institut für Forschungsinformation und Qualitätssicherung

Graduate surveys as an outcome evaluation

Presentation EAIR Forum 2009, July 23-26, Vilnius, Lithuania

Dr. René Krempkow Institute for Research Information and Quality Assurance, Bonn (Germany)

(former project leader of Freiburg Graduate Surveys, University of Freiburg)



iFQ

OutlineOutline

1. Relevance of graduate surveys for Quality Management

2. Measuring outcomes of higher education – Influencing factors of professional success

3. Exemplary results

4. Interpretation of results

5. Prospects for analysis


Graduate Surveys


iFQ

When integrating graduate surveys in quality management:

1. Adequate interpretation of results only through comparison with other HEIs and more general national data (see Teichler 2003)

2. Account for individual characteristics through institution-specific and programme-specific surveys (see QM-literatur)

Nationwide “Core Questionnaire” with institution-/programme-specific adaptations

The Potential of graduate surveys The Potential of graduate surveys for Quality Management Ifor Quality Management I


iFQ

• Analyse data to identify determinants of professional success to feed back to Quality Management system

• Unit of observation: study programmes

• Account for specific context by interpreting data taking into account influencing factors (e.g. regional economy)

• Account for specific initial conditions, e.g. % of student parents, socioeconomic background of students (Schomburg/Teichler, 1998)

The Potential of graduate surveys The Potential of graduate surveys for Quality Management II for Quality Management II


Source: Schomburg/ Teichler (1998: 165)

• Two HEI stand out positively• To what degree can differences be attributed to different starting

conditions of HEIs? This will be discussed on the basis of Freiburg Graduate Surveys and Dresden Graduate Surveys.

The Potential of graduate surveys The Potential of graduate surveys for Quality Management III for Quality Management III


iFQ

Measuring Outcome with graduate Measuring Outcome with graduate surveys – what does professionell surveys – what does professionell success depend on?success depend on?

• Multivariate analysis allows identifying factors influencing professionell success

• This information is independant from (subjective) self-assessment of graduates (see Krempkow/Pastohr 2006)

• Example: job experience, study times abroad and specific competences appear to be more important than study duration and grades – That leads to the question: Can and (if yes) What can HEIs do to improve professional success of graduate?

• Multivariate (correlatory) analysis allows analysing relationships between professional success and aspects of study, including starting conditions and context!


iFQ

Influencing factors on professional Influencing factors on professional successsuccessObjective and subjective criteria of professional success as indicators

of HE outcomes (Teichler/Schomburg (1997: 248)

1. Objective indicators for professional success (e.g. income, hierarchical position)

2. Subjective indicators for professional success (e.g. job satisfaction, autonomy)

3. Objective indicators for transition HE-labour market (e.g. Time of job hunt)

4. Subjective assessment of goodness-of-fit HE – job (e.g. usefulness of study contents, employment adequacy).

• Freiburg and Dresden Survey: Gross income, job satisfaction and employment adequacy fitting criteria for professional success.


Basic model of influencing factors on professional Basic model of influencing factors on professional successsuccess

INDIVIDUAL CHARACTERIS-TICS + SOCIO-ECONOMIC BACKGROUND

CHARACTERISTCS OF HIGHER EDUCATION (PROGRAM)

LEARNING OUTCOMES (COMPETENCES)

JOB SEARCH, TRANSITION PHASE, JOB ENTRY

INDIVIDUAL STATE AT TIME OF SURVEY

INDIVIDUAL STATE AT TIME OF SURVEY

JOB SEARCH, TRANSITION PHASE, JOB ENTRY

PROFESSIONAL SUCCESS /

PROFESSIONAL SUCCESS

One time study:

First repetition of study:

1.

thre

sho

ld

2.

thre

sho

ld2

. th

resh

old


Example of results: stand. Regression coefficients for Example of results: stand. Regression coefficients for incomeincome

Regression coefficients for monthly income of current occupation

Technical studiesR2=.41

adj. R2=.00 n=105

Medical studiesR2=.25*

adj. R2=.11*n=270

Social sciencesR2=.20*

adj. R2=.08*n=316

Grade of matriculation standard ,179 ,223+ -,203*

Gender (male=1) -,320+ -,020 -,100

Finished vocational training before studying (yes=1) ,317+ ,051 ,114

Duration of stay abroad ,182 ,116 ,031

Part time job during semester break ,093 -,141 ,167

Part time job during lecture period -,063 ,104 -,130

Availability of program specific profiles ,053 ,111 ,057

Social and communication skills ,066 ,101 -,146+

Duration of study -,422* -,004 -,125

Job entrance age - ,028 ,266+

Final degree of study -,165 -,012 ,052

Foreign language skills -,025 ,080 ,106

Proficiency in own subject ,268 ,010 ,127

Ability to develop new ideas and problem solutions -,116 -,115 -,110

Ability to work efficiently towards a goal -,236 ,066 -,089

Ability to critically rethink own ideas and ideas of others

,181 -,042 -,058

Use of personal contacts for job search ,063 -,323** ,011

Employers searched for graduates with other study focus

-,043 - -,076

Importance of family -,119 -,004 ,101

Importance of career ,215 -,209* -,270**

Figure 3: Regression coefficients for the monthly income of current occupation; +/ */ ** = significant on the 10%-/ 5%-/ 1%-level (“-“ = variable was not included in the explanatory model)


Example of results: stand. Regression coefficients for Example of results: stand. Regression coefficients for incomeincome

Regression coefficients for monthly income of current occupation

Technical studiesR2=.41n=105

Medical studies

R2=.25*n=270

Social sciencesR2=.20*

n=316

Grade of matriculation standard ,179 ,223+ -,203*

Gender (male=1) -,320+ -,020 -,100

Social and communication skills ,066 ,101 -,146+

Duration of study -,422* -,004 -,125

Job entrance age - ,028 ,266+

Final degree of study -,165 -,012 ,052

Use of personal contacts for job search ,063 -,323** ,011

Importance of family -,119 -,004 ,101

Importance of career ,215 -,209* -,270**

Figure 3: Regression coefficients for the monthly income of current occupation; +/ */ ** = significant on the 10%-/ 5%-/ 1%-level (“-“ = variable was not included in the explanatory model)

• Similar Results in Dresden Graduate Survey with the same analysis scheme, but the social background was more important


iFQ

Interpretation of results (Freiburg & Interpretation of results (Freiburg & Dresden)Dresden)• Influencing income: Previous job training, GPA (Secondary school) and

gender, in Medicine: Personal contacts and career oriantation are important

• Influencing job satisfaction: Technical sciences: Socioeconomic background, previous job trainingEngineering: Tendendy for women to be less satisfied Humanities: Higher self-assessed expertise and family orientation: lower job satisfaction

• Influencing job adequacy:Engineering: gender - women have less adequate jobs

Prospects for analysis?


iFQ

Further informationFurther information

• IfQ Bonn: www.research-information.de

• Freiburg Graduate Surveys: www.qm.uni-freiburg.de/projekte

• Requirements of graduate surveys for measuring HE outcomes: „Leistungsbewertung, Leistungsanreize und die Qualität der Hochschullehre“ (2007) www.universitätsverlagwebler.de/krempkow.html, Onlinepublikation (2005) http://nbn-resolving.de/urn:nbn:de:swb:14-1129208825969-55860

• Determinants of professional success: Zeitschrift für Evaluation 1/2006 (www.zfev.de), S. 7-37, bzw. http://www.kfbh.de/downloads/Was_macht_Hochschulabsolventen_erfolgreich.pdf

• Dresden Graduate Survey: www.kfbh.de/absolventenstudie


Example of results: stand. Regression Example of results: stand. Regression coefficientscoefficients

Graduate surveys as an outcome evaluation

Documents

Transcript of Graduate surveys as an outcome evaluation