Devising and Implementing a System for Online Module Evaluation

Questionnaires (MEQ) and Reporting

James Birkett, Head of Student Administration (Programmes)

Robert Hudd, Senior Analyst/ Developer

1

1. Context & Background

2. Developing a new approach

3. Implementation

4. Lessons Learnt

5. Top Tips

6. Future considerations

7. Discussion/ Any Questions

2

Agenda

• all eligible research-active staff are in a position to be submitted to the 2020

REF exercise

• the University increases its income to fund research activity

• we continue to provide a high quality teaching and learning experience for

students, measured by indicators such as progression rates, good degrees

and improved module evaluation feedback from students, commensurate

with a top ten league table position

• we generate sufficiently strong financial performance to enable us to invest in

our infrastructure, support new growth activities and effectively finance our

operations.

3

Looking back to March 2014: Vision 2020

Context & Background

4

League Table Position

Context & Background

• Surrey uses Tribal SITS for its Student Records System

• MEQs undertaken twice a year- Semester 1 (Autumn) and Semester 2

(Spring); end of summer MEQ for all year modules and those ending

after semester 2 (Nursing, Acting)

1. Teacher Support

2. Learning Experience & Pedagogy

3. Module Design

4. Student Interaction

5. Assessment & Feedback

- What were best aspects of the module?

- What aspects could be improved?

5

Surrey Terminology – Module Evaluation Questionnaires (MEQ)

Context & Background

1. Module Co-ordinators often had to wait up to 4 months before getting the

results for their module as the reports were all generated at once.

2. Lack of controls meant teaching staff could be changed at any time –

phantom results / incorrect attributions!

3. Survey fatigue: Students receiving an email for every MEQ issued, even

though most of these started on the same day.

4. Inconsistent generation of records – no standard definition of who received

an MEQ.

5. Hidden “black holes” in the existing Survey Manager implementation (e.g.

modules with “non-standard” MEQ dates missing from reports / odd route

code exclusions from MEQ role group).

6. Resitting students not accounted for.

7. Different occurrences of same module, e.g. Exec MBA v FT MBA

6

MEQs: Life before

Context & Background

Automate the MEQ process including the sending out of the survey and the subsequent chase emails

– Module start and finish dates included in SITS.

– All ‘standard’ modules have MEQs released at the same time.

– Collated emails to students. ie one email containing all of their modules

– Adjustable dates for modules so MEQs can be sent at the appropriate time

Ensure the data quality with regard identification and maintenance of teaching teams

– Automatic population of information for admin teams and programme leads to check online.

– Approval process in place if appropriate.

– Must be done at least 1 week prior to MEQ release date

– Standardised population definitions

– Defined staff roles

7

MEQs: Life after

Developing a new approach

Provide mechanism for data returns in a timely efficient and secure

manner

– Improved data security and a standard definition around confidentiality

– Standard online reporting for academic members of staff available

according to defined roles

– Reports available in a more timely manner

– Data exported for non-SITS reporting by Planning

Provide training for all staff where required

8

Developing a new approach

9

The survey - Example screen 1

Developing a new approach

10

The survey - Example screen 2

Developing a new approach

11

Example screen 2

Developing new approach

MEQ Set Up

- Admin staff

- Module Coordinator

- Programme Director

12

Design: Module maintenance hub

Implementation

13

Design: Module Coordinator

Implementation

14

Design: Programme Director - No interaction – simply a means of checking MEQ status

Implementation

15

Design: Reporting

Implementation

Top 4 reports are available to management only (e.g. Associate Deans)

Lack of Ownership

The initiation of the project had gone through several stages, including several

staff who left. The project generally suffered from a lack of ownership as the

outcomes are very politically sensitive (informing staff appraisals). This was only

adequately resolved post-implementation with the formation of a steering group.

Testing

How do you ensure the reports are accurate when everything is anonymised? It

was also difficult to volume test. Also impacted by the ownership issue.

This was (in part) mitigated by releasing some of the reports early to the

Associate Deans (though stipulating that the results were not yet final), in order

to allow them to ask questions about areas they felt did not “look right”.

Comparisons with historic data were also useful tests of the accuracy of the

reports.

16

Lessons learnt 1

Technical

The Snapshot Creation/Population batches caused issues with our batch server.

Historic Data

The A3 Staff Appraisal requires 2 year rolling averages. However, the 2013/4

calculated averages were only held in PDF reports – there was no tabular data

from which the correct historic values could be taken. In the end, the Planning

department had to recalculate these values and put them into a template that

was then imported. It was stipulated that if there were any doubt around these

historic values, staff should refer to the historic reports.

17

Lessons learnt 2

1. Be explicit about your data definitions. It can be sliced in many different

ways.

2. Think about the politics / sensitivity / data protection. For example, agreeing

who should have access to what needs to be very clearly established early

on!

3. Allocate some developer time post-implementation for tweaks

4. Depending on your culture, your administrators may be surprised when you

tell them it’s the Module Co-ordinator’s own fault if they don’t check their

module(s)…

5. Despite having implemented automated reminders in the project, the

steering group has since encouraged additional emails from administrators.

Make sure your process owner(s) understand that email saturation is not a

good thing nor an efficient use of time!

18

5 Top Tips

19

Report Access Grid

Planning SITS

R160 F (module

title by

Fac./dept.)

Open MEQ Report M20 (Module

Comparison)

M30 (Staff by

Module)

M40 (Module by

Staff)

M10 (Module

Detail)

M11 (Module

Comments)

A3 Staff Summary

Dean Yes Faculty Faculty Faculty Faculty Faculty Faculty 3 Faculty 1

ADLT Yes Faculty Faculty Faculty Faculty Faculty Faculty 3 Faculty 1

Head of Dept. Yes No AOU AOU AOU AOU AOU 3 AOU 1

Director of L&T Yes No AOU AOU AOU AOU AOU 3 No

HR Yes University University University University University University University

Programme

Leaders

No No No No Programme 2 No No

Module leaders No No No No Module 4 No No

Module

contributor

No No No No No No Own

Guest Speakers No No No No No No Via manager(s)

1 Staff teaching in the Faculty/AOU within the last 2 years 2 Modules in the programme’s AOU, taken by the programme(s)’ students in the past 2 years 3 Modules within the Faculty/AOU for which the user accessing the report is neither a contributor nor the leader 4 To protect student confidentiality it was decided that module leaders should not have direct access to freeform comments in modules of 10 or fewer students

1. Ownership: academic leadership, link to staff performance / appraisals

2. Maintaining / increasing response rates: 39% overall ?

3. Non standard modules: performance, dissertation ?

4. Modules with ≤10 students: confidentiality of freeform comments

5. Feedback loop: do what with analytics, qualitative feedback?

20

5 Future Considerations

Discussion/ Any questions