From a pyramid to traffic lights
Assessing the effectiveness of QAA
Douglas BlackstockDirector of Administration &
Company Secretary
Governance at QAA• QAA is a company and a charity• Regulated by Companies House
and two charity regulators• Has a board of 15 – seven
independents (one is a student) and nominees from the funding councils and institutions representative bodies from each of the four UK nations
• Is funded through contracts with funding councils and through institutional subscriptions
Key purposes of QAA
• Protection
• Communicate
• Improvement
• Understanding
More detail in our strategic plan at www.qaa.ac.uk/aboutus
About us
• 140 staff (125 full time equivalent)
• Up to 1000 auditors and reviewers
• Over 200 audits and reviews in 09-10
• Budget for 09-10 is €14.5million
• C. 200 corporate publications per annum plus review reports
• www.qaa.ac.uk/aboutus
Where I fit in
• Management of infrastructure servicesCommunications, Finance, Human Resources, Information systems, Office Services
• Cross QAA workBusiness development, information management, student engagement and
car parking
Assessing our effectiveness
• 2003 we considered EFQM but couldn’t secure commitment
• restarted 2006 suggesting a simple set of 10-12 Key Performance Indicators
• got complicated through 06 and 07 with a proposal for 120+ performance measures and the arrival of a pyramid
Mission & Purpose
External profile
Working worldwide
Offering expertise
Safeguarding standards
Reducing Regulation
Supporting and enhancing
quality
Management and Governance
ExecutiveBoard
SupportPeopleFinance
Core BusinessReviews and audits
Supporting development and enhancementWorking with others
Knowledge transfer and relationships (outputs)
QAA infrastructure and people (inputs)
Outcomes
Measurement
Assessment
But
Would this actually tell us if we are
meeting our purposes?
2008 - three tier system
An analysis of
Performance & Success
Would help us assess
Impactand produce a traffic light summary
The traffic lights(based on Committee of University Chairmen guidance)
Excellent: this is on track; performing well and is substantially effective.
Good; this is broadly on track; performance is satisfactory with one ortwo areas which could be addressed.
Mixed: some significant concerns which could be damaging and result in ineffective performance if not addressed.
Problematic: serious concerns threaten this area; this has the potential to become substantially ineffective.
We set about gathering data(in the style of Donald Rumsfeld)
• What we knew already• What we knew someone else
knew• What we thought someone
should know but were not sure they did know
• What we wanted to know and knew we could find out
• Does what we know or don’t know help us know whether we are achieving our aims?
QAA’s performance
Programmes of work (quarterly monitoring of operating plan)• Key contractual commitments met e.g. reviews and audits
completed, reports published• Progress against non-contractual activities e.g. papers published,
other publications, events held, presentations given, attendance at external events
• Exceptions and achievements• Any shift in risk External evaluation (some annual, some periodic)
• External auditors• Internal auditors• External standards – for Information security, human resources
management, website accessibility and our Welsh Language Scheme• ENQA, Higher Education Regulatory Review Group, quality groups in
Scotland and Wales
QAA’s performance (2)
Staff satisfaction (quarterly and annual)• Overall indicators from the annual staff survey• Quarterly staff turnover, absence, recruitment
analysis
Financial performance (quarterly and annual)• Monitoring and achievement against budget;
budget variances• Balance sheet• Forecasts from quarter 2 onwards
QAA’s success
Effectiveness of processes• Institutional and reviewer / auditor evaluations of
reviews and audits; reviewer / auditor evaluations of their training; participants’ evaluations of QAA events
• Renewal of contracts; repeat business• New business (from current clients and new clients)• Success of ‘engagement’ strategies – students,
employers, bodies representing the professions or with regulatory powers
QAA’s success (2)
Adding value• Demand for our ‘products’ – hard copy reports, web traffic,
external attendance at our events• Demand for our services – advice, contribution, participation,
expertise, briefing, presentations• Hot topics – our take, our engagement• Interactions at government level• Interactions with non-HEI stakeholders• External review of QAA, such as ENQA membership review• Institutions taking our reports seriously and putting right any
identified problems• External partners, the funding councils, student bodies and
institution representative bodies
QAA’s success (3)
The difficult to measure– relationships, mood, reception• The ‘mood’ of the regular meetings with the funding
councils and representative bodies (rectors councils?) in each of the countries
• The ‘mood’ of the twice-yearly meetings of the Sounding Board group ( made up of all QAA funders)
• The ‘mood’ of, and intelligence from, QAA liaison officers’ interactions with individual institutions
• The ‘mood’ of the QAA Annual Reception and the annual Subscribers’ meeting
• The amount, mood and tone of press coverage• The annual survey of Higher Quality readers
QAA’s impact
To what extent the performance and success data allowed the Board to make an assessment of the achievement of each of our Key Purposes.
The Board would then need to identify whether additional information is needed to help it assess QAA’s impact.
What did we find? Key purpose
Information
Safeguard the student and wider public interest in the maintenance of standards of academic awards and the quality of higher education
Communicate information on academic standards and quality to inform student choice and employers’ understanding, and to underpin public policy-making
Enhance the assurance and management of standards and quality in higher education
Promote wider understanding of the nature of standards and quality in higher education, including the maintenance of common reference points, drawing on UK, other European and international practice
Programmes
External evaluation
Staff satisfaction
Financial management
Effectiveness of processes
Adding value
Reception
Overall assessment Overall assessment Overall assessment Overall assessment
What did we learn?
• The process was very demanding of staff time and we could simplify the process
• The traffic lights help you focus on risk• In some areas we needed to improve
information management and sharing• Some of the information available did not
contribute to an assessment of the achievement of certain key purposes.
and finally
Whilst measurement can inform…
ultimately, any assessment of the achievement of key purposes is a
JUDGEMENT based on an array of evidence, not a measurement.
The Quality Assurance Agency for Higher Education. Registered charity numbers 1062746 and SC037786
Top Related