Post on 14-Dec-2015
Colour coding, KPIs and the Colour coding, KPIs and the Road to DamascusRoad to Damascus
Shedding new light on reporting the student experience at the
University of Sydney
Rachel Y. SymonsQuality Assurance Officer (Learning and Teaching)
“We can learn from experience if we are ready to adapt that
experience to changed conditions”
JC Masterton (1972): The Double Cross System in the war of 1939-1945
Scene setting
• Reports Marks 1 & 2• Colour coding• Taxonomy upgrade• NVivo7• Reports Mark 3• What next?
Reports Mark One (2003 – 2006)
• Areas of best practice• Suggested
improvements• Ordered by percentage
of comments received• Not linked to SCEQ
scales• Redundant with
introduction of KPIs
Reports Mark Two (2007)• Aligned with KPIs
– Generic Skills– Good Teaching– Overall Satisfaction
• List of key issues• Minimal sample comments• Results as raw numbers• Undergraduate and
postgraduate coursework combined on one page
Feedback and A new broom
• Dense• Not user friendly• Difficult to interpret• Percentages not raw
numbers• Confusing to novice• Need for succession
planning
Colour coding and KPIs
• Easy identification of related data across all documentation– Taxonomy
– Recording analysis
– Communicating information
• Linked to KPIs / SCEQ scales
Generic Skills
Good Teaching
Overall Satisfaction
Learning Community
Curriculum
Assessment
Workload
Clear Goals and Standards
Taxonomy enrichment
• Describes the concepts
• Coverage of each component
• Answers the question
• Illustrates with student comments
The Road to Damascus• NVivo7 + Boolean
logic = technological conversion
• Combines manual and automated analysis
• Use of pre-set base queries
• Faculty customisation
Benefits of NVivo7 methodology
• No need to print source documents
• Simultaneous coding• Time saving• Direct transfer of
comments into reports
• Reduces impact on environment
Revitalising the reports
• Consultation with stakeholders and communications experts
• Developed quality product
• Beneficial to all stakeholders
• More actionable than previous reports
Reports Mark Three (2008)• KPI alignment • SCEQ and CEQ data • Executive summary• Trends• Comparative quantitative and
qualitative data (2001 – 2007)• Key issues (2007)• Customisation by faculty• Separate reports
undergraduate and postgraduate coursework
Reports Mark Three (2008)Key results• Faculty scores for Good Teaching (KPI 15; Section 2)
– have increased by 8% since 2005 for current students (SCEQ) – have been consistently above the University average for current
students (SCEQ) since 2000– are lower than previous years for graduates (CEQ)– are lower than the University average for graduates (CEQ) – domestic and international students are recording similar experiences
• Analysis of the qualitative data from current students (SCEQ) indicates that experiences of teaching within the Faculty are evenly divided between students who consider this to be one of the best areas of their experience and those who suggested improvements. For graduates (CEQ), a higher percentage of comments received from respondents focused on areas of improvement rather than areas of best practice.
Reports Mark Three (2008)
5652
62
5157
6055 53
61
0
10
20
30
40
50
60
70
2003 2005 2007
DomesticInternationalAll
Reports Mark Three (2008)
SCEQ 2005
SCEQ 2007
CEQ 2006
CEQ 2007
Areas of best practice
Domestic 46% 43% 25% 30%
International 0% 50% 46% 33%
All 32% 46% 28% 30%
Suggested improvements
Domestic 49% 52% 25% 40%
International 38% 36% 45% 20%
All 46% 46% 28% 38%
Reports Mark Three (2008)Key issues for coursework students (SCEQ 2007; CEQ 2007)
Areas of best practice
Current students (SCEQ 2007) Domestic
(n=44)
International
(n=32)
All
(n=76)
Good teaching experiences•Use of innovative and up to date teaching methods•Ability to keep students interested in the subject•Learning resources are up to date
9% 19% 13%
Sample comments•There are many dedicated staff who put a lot of effort into their teaching•Lectures stimulate further thinking and understanding rather than just presenting facts and memory work
Have we succeeded?• Improved arrangement• Easy identification of areas
– colour coding – collation by KPI
• Used to inform – curriculum review– marketing
• Aids in prioritising areas for improvement
• Incorporation of qualitative and quantitative data plus executive summary particularly helpful
What next?• Investigate strategies for
“closing the loop”• Report on faculty/ student
administration• Cross university reports
– eLearning– Research-enhanced
learning and teaching– Library– International students
• Alignment with AUSSE Scales and survey items
What next?• Commitment to quality
improvement
– Reports continue to be actionable and accessible
– Taxonomy continuous w-i-p
• Flexibility of NVivo7 will accommodate range of analytical reports
“We can learn from experience if we are ready to adapt that
experience to changed conditions”
Useful information
• Taxonomy; methodology; reports http://www.usyd.edu.au/learning/evaluating/open_ended_responses.shtml
• Contact information
r.symons@usyd.edu.au
02 9351 6560