October 2008

Using qualitative data to prove and improve quality in Australian higher education

Geoff Scott, Leonid Grebennikov and Mahsood Shah Office of Planning and QualityUniversity of Western Sydney

2

• Introduction

– Limited use of qualitative data in institutional performance assessment

– Advantages and benefits of using qualitative data

– UWS experience in the systematic analysis of the qualitative data from student feedback surveys

• Method

– CEQuery qualitative analysis tool

– Comparative analysis of qualitative data generated by three key UWS student surveys

• Results and discussion

• Implications

Outline

3

Qualitative data in institutional performance assessment

• Receive limited attention

• Cover aspects of student experience which are untapped in existing evaluations

• Identify reasons for statistical results which may be different from what researchers assume

• Define in students’ own words what they find important

• Should complement quantitative data

4

UWS experience in the systematic analysis of the qualitative data from student feedback surveys

• Since 2006 all UWS student surveys covering

- overall experience at the University level - particular course or program - specific subjects

invite respondents to answer two questions in their own words:

• What were the best aspects of their course/unit?

• What aspects of their course/unit are most in need of improvement?

5

UWS experience in the systematic analysis of the qualitative data from student feedback surveys

• Written comments are automatically classified by the CEQuery qualitative analysis tool into five domains and 26 subdomains using a custom-tailored dictionary.

• CEQuery results are integrated into Annual Course and Unit Reports in order to better identify key ‘hot spots’ for improvement and actual solutions from the student perspective.

• Actual comments can be viewed once sorted into specific CEQuery domains and subdomains.

• High importance areas are used in course accreditation and review, and to validate rating items on surveys.

6

Comparative analysis of qualitative data from three key student surveys

• Survey 1: Covers total university experience; sample – 3,492 current students; 9,410 written comments

• Survey 2: The national CEQ covers graduate experience of the course just completed; sample – 2,734 respondents; 4,213 written comments

• Survey 3: Evaluates individual subjects each time they are offered; sample – about 200,000 students each year; 94,803 written comments

7

About CEQuery

• Best Aspect (BA) and Needs Improvement (NI) ‘hits’ are coded and sorted into domains then subdomains.

• 5 domains – Assessment, Course Design, Outcomes, Staff, and Support, and 26 subdomains

• Hit rate – 80%, allocation accuracy – 90%

• BA + NI = Importance

• BA / NI = Quality

• Custom-tailored dictionary

8

CEQuery subdomains

• Assessment

– Expectations– Feedback– Marking– Relevance– Standards

9

CEQuery subdomains

Assessment: ExpectationsProvision of clear assessment tasks and expectations on how to tackle and present them; clear submission deadlines, guidelines rules and grading criteria. Provision of examples of work, to give an operational picture of different grades and quality of work in each subject.

Typical NI commentsExpectations for assignments need to be clearerLack of clear criteria for marking More explanations than just expecting us to know or guess Better description of tasks

10

CEQuery subdomains

Assessment: FeedbackPromptness with which assignments are returned, use of staged deadlines, quality of the feedback received including the extent to which markers comment on what was done well, explicitly identify key areas for improvement and say how improvements could have been achieved – with specific attention to the grading criteria distributed at the start of the subject.

Typical NI commentsI’m still trying to get back an assignment over 5 months oldWhen returning essays tutors should give more detailed feedback so students know exactly how to improve workWe only received one assessment back before the subject finished

11

CEQuery subdomains

• Course Design

– Flexibility– Learning methods– Practice-theory links– Relevance– Structure

• Outcomes

– Further learning– Intellectual– Interpersonal– Personal– Knowledge/skills– Work application

12

CEQuery subdomains

• Staff

– Accessibility– Practical experience– Quality & attitude– Teaching skills

• Support

– Infrastructure– Learning resources– Library– Social affinity– Student administration– Student services

13

More information on CEQuery

Scott, G. (2006). Accessing the student voice: Using CEQuery to identify what retains students and promotes engagement in productive learning in Australian higher education.

http://www.dest.gov.au/sectors/higher_education/publications_resources/profiles/access_student_voice.htm

14

The CEQuery study of comments from students in 14 universities - key implications for student retention and

engagement

• It is the total experience that counts.

• Teaching is not learning• Learning is a profoundly social experience.• Need for more research on how various forms of IT-enabled learning

do and do not add value as part of a broader learning design

• 60 learning methods, especially active and practice oriented ones depending on FOE and level of study

• Traditional lectures and class-based methods must be seen as just one of the options not the sole one.

CEQuery Subdomain Hits and Ranks across Three Levels of Student Experience

Domains Subdomains Hits Rank Hits Rank Hits Rank

Expectations 79 19 114 17 2,862 10

Feedback 91 18 78 20 1,412 14

Marking 37 22 65 22 869 18

Relevance 39 21 75 21 1,184 16Standards 126 16 146 15 2,973 9

Flexibility 458 7 448 4 4,178 5

Methods of L & T 338 9 578 3 15,871 1

Practical-Theory Links 61 20 89 18 605 19

Relevance 127 15 215 10 3,082 7Structure 207 12 254 8 3,079 8

Further Learning 23 24 3 26 43 26

Intellectual 100 17 148 14 2,057 12

Interpersonal 35 23 41 23 258 20

Know ledge/Skills 130 14 196 12 2,341 11

Personal 10 26 31 25 215 22Work Application 137 13 203 11 910 17

Accessibility & Resp. 583 5 592 2 3,811 6

Practical Experience 12 25 41 24 113 24

Quality & Attitude 594 4 787 1 7,611 3Teaching Skills 292 11 405 5 4,480 4

Infrastructure 1,621 1 251 9 1,398 15

Learning Resources 618 3 255 7 7,884 2

Library 1,178 2 180 13 150 23

Social Aff inity 481 6 316 6 1,521 13

Student Administration 410 8 128 16 242 21Student Services 312 10 84 19 69 25

Course Design

Outcomes

Staff

Support

Total experience Course level Subject level

Assessment

CEQuery

CEQuery Subdomain BA/NI ratios and Ranks across Three Levels of Student Experience

Domains Subdomains Hits Rank Hits Rank Hits Rank

Expectations 0.4 22 0.2 24 0.3 26

Feedback 0.1 26 0.1 26 0.4 25

Marking 0.2 25 0.2 25 0.5 24

Relevance 1 12 1.7 8 3.7 15Standards 0.3 23 0.3 23 0.7 20

Flexibility 1 11 1.1 12 3.7 14

Methods of L & T 0.7 17 1.9 7 3.4 16

Practical-Theory Links 0.9 15 1.1 13 6.6 9

Relevance 0.6 20 1 14 5.6 12Structure 0.3 24 0.5 19 0.6 23

Further Learning 22.1 1 2.1 6 42.1 2

Intellectual 2.5 3 20.1 1 32.8 3

Interpersonal 2.2 5 4.8 3 12 6

Know ledge/Skills 9.1 2 4.2 4 9.1 8

Personal 0.9 14 9.1 2 214.1 1Work Application 1.6 7 0.9 15 15.3 4

Accessibility & Resp. 1.3 8 1.6 10 6.1 10

Practical Experience 1.2 10 1.6 9 15.1 5

Quality & Attitude 0.5 21 1.5 11 10.9 7Teaching Skills 2.2 6 0.7 17 4.7 13

Infrastructure 0.7 18 0.5 20 0.6 21

Learning Resources 0.8 16 0.6 18 1.1 18

Library 2.4 4 0.7 16 0.8 19

Social Aff inity 0.9 13 3 5 3.1 17

Student Administration 0.6 19 0.3 22 0.6 22Student Services 1.3 9 0.3 21 5.9 11

Support

Assessment

Course Design

Outcomes

Staff

CEQuery Total experience Course level Subject level

17

Discussion

• Why do very important CEQuery subdomains demonstrate patchy results in terms of quality?

- Variety of factors shaping student experience

- Extent of multi-campus university operation

18

Six areas of student experience that warrant an improvement focus

• Assessment (standards, marking, expectations management and feedback)

• Student Administration

• Course Structure

19

Six areas of student experience that warrant an improvement focus

‘High-hit’ CEQuery Subdomains with low BA / NI ratios

CEQuery Total experience Course level Subject level

Subdomains BA/NI BA/NI BA/NI

Course Design: Flexibility 0.7 1.9 3.4

Course D: Methods of L & T 0.9 1.1 6.6

Staff: Accessibility & Resp. 1.3 1.6 6.1

Staff: Quality & Attitude 0.5 1.5 10.9

Staff: Teaching Skills 2.2 0.7 4.7

Support: Infrastructure 0.7 0.5 0.6

Support: Learning Resources 0.8 0.6 1.1Support: Library 2.4 0.7 0.8

20

Six areas of student experience that warrant an improvement focus

• Assessment (standards, marking, expectations management and feedback)

• Student Administration

• Course Structure

• Staff: Quality and Attitude (at the overall university level)

• Student Support: Infrastructure (course and subject level)

• Student Support: Learning Resources (course level)

21

UWS improvement actions based on quantitative and qualitative data (analysed via CEQuery) from student

feedback surveys

• Introduction of online enrolment

• Implementation of the online complaints resolution system

• New assessment policy

• Introduction of assessment focused self-teaching guides for each subject

• A range of new, targeted transition support programs

• A number of new, free study assistance workshops and programs

• Use of the more interactive version of the online learning system.

• More opportunities for practice-based learning, e.g., through increased engagement with regional employers and industry bodies

• Results: Improvement in CEQ OS by 10% in three years, retention by 4%

22

Concisely, the systematic analysis of qualitative data helps:

• Generate a more focused and evidence-based set of ‘good practice’ guidelines and areas for quality improvement down to the course and unit level

• Ensure that course and subject design focus on what counts for students, as courses and units are implemented and reviewed

• Inform what is and is not tracked in quantitative surveys, validate the items in these surveys to ensure they cover what is really important to students

• Assist in making staff development programs more relevant by providing BA and NI comments regarding each course and unit to relevant teaching and administrative staff

• Complement the quantitative data that are typically used to inform decision-making for the area