‘Research into Quality Assurance of Training Provision for the BANKSETA’ - November 9 th 2005 -

‘Research into Quality Assurance of Training Provision for the BANKSETA’

- November 9th 2005 -

2

The structure of this presentation is as follows …

The research map

The challenges of the BANKSETA

Quality Assurance from a process perspective

BANKSETA’s view on the definition of Quality Assurance

Selection of findings and conclusions of the research

General conclusions on Quality Assurance

Challenges Slide 3

Frame work Slide 4-12

Definition Slide 13-17

Findings Slide 18-35

Conclusions? Slide 36

- Index -

Research Map Slide 2

3

Practice 3

Practice 2

The research followed the following steps, partly in an iterative way …

- Research map -

Literature research

Challenges BANKSETA

Framework

Legislation / ETQA functions

Relevance (Filter)

‘Practice’ 1

‘Practice’ 2

‘Practice’ ..

‘Practice’ X

Initial research

Relevant Practice 1

Best Practice

+ -

In depth view

Fit to challengesImplementation

Acceptance

Information& Sources

Definition

ChallengesCountrySector

Legislation

Interviews

4

A summary of the main challenges that the BANKSETA was facing at the start of the research …

Accreditation: What is a user-friendly process for service providers and is it accessible and manageable for small providers?

Programme evaluation/approval: Should this be done and if so, what is the most effective and efficient way to do so?

Overlap: Certain scenarios result in overlaps with other SETAs with different systems in use. How can potential conflict in roles and responsibilities be avoided or resolved? Are MoUs the appropriate mechanisms?

Audits: What is the best combination of internal and external audits and how can the internal audit process be streamlined?

- Challenges BANKSETA -

5

The Primary Process: The Learning Value Chain

- Framework -

Needs Analysis

- SP / Empl -

Programme Development

- SP -

Learning- SP -

Assessment- Assessor -

Certification/Data

recording- ETQA -

Characteristics Volume driven process steps. The more programmes and the more

learners, the more often these steps have to be executed Steps are part of the Learning Value Chain. If the activities are not (well) executed this has a direct effect on the

Quality of Learning.

Pri

mary

6

Secondary Process: How do we check the Quality of the primary process?

- Framework -

Characteristics Volume driven activities. The more programmes and the more

learners, the more often these steps have to be executed Activities look directly into the steps of the Learning Value Chain. If the activities are not (well) executed this has an indirect effect on the

Quality of Learning.

Programme review ModerationLearning EvaluationInternal

External Programme review ModerationLearning Evaluation

Needs Analysis

- SP / Empl -


- SP -

Learning- SP -


Certification/Data

recording- ETQA -

Secon

dary

Pri

mary

7

The Assessor

The ETQA registers Assessors.

Tertiary Process: The capacitating of the entities involved in the primary and secondary processes. Who do we need

to capacitate?

- Framework -

Internal

External Learning Evaluation

Needs Analysis

- SP / Empl -


- SP -

Learning- SP -


Certification/Data

recording- ETQA -

Service Provider

The ETQA accredits, audits and empowers Service Providers

1

Programme review

2

The Moderator

The ETQA registers Moderators.

3

Moderation

Programme reviewSecon

dary

Pri

mary

Moderation

Learning Evaluation

8

Assessor

Tertiary Process: How is the right capacity of Service Providers, Assessors and Moderators ensured by the

ETQA?

- Framework -

Service Provider

Moderator

Characteristics Activities are not volume driven. Irrespective of the number of

programmes or learners, the activities will take place according to a certain frequency.

Activities do not look directly into the steps of the Learning Value Chain but check the capacity of the different entities involved in executing the steps of the Learning Value Chain and the Secondary (QA) processes connected to it.

If the activities are not (well) executed this has an indirect effect on the Quality of Learning.

Tert

iary

In take Audit Develop

Accreditation

Registration

Registration

Internal AuditExternal Audit

Monitoring visits

Workshops

?Forums

Workshops

?Forums

Workshops

9

Assessor

The total model is made up of Primary, Secondary and Tertiary Processes all working together to deliver quality

for the learner.

- Framework -

Service Provider

Moderator

Tert

iary


Accreditation

Registration

Registration


Monitoring visits

Workshops

?Forums

Workshops

?Forums

Workshops



Needs Analysis

- SP / Empl -


- SP -

Learning- SP -


CertificationData

recording- ETQA -

Secon

dary

Pri

mary

10

The first six of the ten ETQA functions can clearly be linked to the framework and fall within the scope of the QA

research.

ETQA functions linked to framework1. Accrediting Providers2. Promoting Quality amongst Providers3. Monitoring provision4. Evaluating assessment and facilitating moderation amongst providers5. Registering assessors6. Certificating learners

ETQA functions outside the framework1. Co-operation with relevant moderating bodies2. Recommending new standards or qualifications (or modifications to

existing standards and qualifications) to SGBs 3. Maintain a database4. Submitting reports to SAQA

- Framework -

11

All the ETQA functions are linked to three streams of the Training & Quality Assurance process

- Framework -

Primary process“from Needs Analysis

till Certification”

Needs Analysis & Programme

Development

Learning Assessment Certification

Secondary Process

“Assuring the Quality of the

actual process“

Int

Ext

Tertiary process


different entities

involved”

Entity Intake process Audit process Empowering

Service Provider

Assessor

Moderator

1 Accrediting Providers

2 Promoting Quality amongst Providers

3 Monitoring Provision

4 Evaluating assessment and facilitating

moderation amongst providers

4 Evaluating assessment and facilitating moderation amongst

providers

5 Registering Assessors

6 Certificate learners

12

Legislation indicates the following guidelines for the Quality Assurance processes …

- Framework -

Primary process“from learning till

certification”

Needs Analysis & Programme Development

Learning

Must be OBET.Must be according to

NQF standards.

Assessment

Guidelines are given by SAQA:

- assessor requirements-principles of assessment- assessment process

Certification

ETQA must keep database acceptable to

SAQA and have means of preventing fraud

Secondary Process


actual process“

IntNo relevant legislation.

Guidelines are given to check Assessment as well

as to do moderation internally

N/A

Ext

SP needs to proof ability to develop,

deliver and evaluate prog which culminate in spec reg std and

qualifications.

No relevant legislation.

SAQA may appoint moderating bodies to

ensure credible assessment decisions. ETQAs

need to have a system to manage assessments.

N/A

Tertiary process


different entities

involved”


Service Provider

QMS requirement, link qualification, relevant P&P,

resources, accr. by one ETQA, prov. accreditation

possible

Legislation does not specify, only that ETQAs should monitor and audit achievements in general


Assessor

Must have successfully completed assessor training

and have subject matter expertise.

ETQAs need to have a system to manage

assessments.


Moderator

Must have successfully completed moderation training + knowledge of

subject area.

ETQAs need to have a system to manage

moderation.No relevant legislation.

13

Currently the BANKSETA has the following Quality Assurance processes in place …

- Framework -

Primary process“from learning till

certification”

Needs Analysis & Programme Development

Learning

(Done by 32 SPs8562 registered learners currently on database)

Assessment

(4424 assessors on the database. Assessment

guide available for assessors.)

Certification

Certain parts devolved to providers.

Secondary Process


actual process“

IntProgramme Review informed by QMS

Training evaluation informed by QMS

Internal moderation based on SPs QMS (2% policy,

10% practice)

Ext

Evaluation of selection of programmes by site-visits (only format, e.g.

OBE)

Reviewing of internal moderating reports (part of site-visits) + in case of

problems

Tertiary process


different entities

involved”


Service Provider

‘Preliminary and formal’Criteria: similar to SAQA, focus on D&D control +

internal audit

Requires yearly internal audit by SP, External Audit only if

necessary based on appeals + Annual report on moderation and

assessment

Workshops, Consulting

AssessorCompetent against US 115753, have technical

expertise

ModeratorCompetent against US 115759, experienced registered assessor

14

The more we focus on the BANKSETA itself, the more the ‘general knowledge’ needs to be extrapolated to the

BANKSETA

Quality Assurance

Quality Assurance in training

Quality Assurance for SAQA

Quality Assurance for the BANKSETA

- Definition -

Scope

Precision

Rese

arch

Dis

cuss

ion

15

The General Definitions of Quality Assurance ….

ISO 9000:2000 “Part of quality management focused on providing confidence that

quality requirements will be fulfilled.”

Six Sigma “Work done to ensure that Quality is built into work products, rather

than Defects.” This is by (a) identifying what “quality” means in context: (b) specifying methods by which its presence can be ensured; and (c) specifying ways in which it can be measured to ensure conformance.

Q-Africa 2004 Conference Paper “The sum of activities/elements that assure the quality of products and

services.”

- Definition -

16

Quality Assurance for training is considered …

CHE “The underlying objective of the HEQC with regard to quality assurance

is: to ensure that institutions effectively and efficiently deliver education, training, research and community service which are of high quality and which produce socially useful end enriching knowledge…”

Engineering Council “Quality assurance: in general, a process which examines the quality of

a process and its product; when applied to an education and training programme, evaluating the programme and its assessment against standards using defined policies and procedures.”

Higher Education - Australia “(The) systematic management and assessment procedures adopted to

monitor performance and achievement and to ensure achievement of specified quality or improve quality.” (Harman & Meek, 2000, p. 11)

- Definition -

17

The view of SAQA and the SETAs …

SAQA’s definitions “Quality Assurance means the sum of activities that assure the quality

of products and services at the time of production or delivery. Quality assurance procedures are frequently applied only to the activities and products associated directly with the goods and services provided to external customers”

“Quality Assurance means the process of ensuring that the degree of excellence specified is achieved”

“Implicit in SAQA’s implementation of the quality system is the understanding that quality assurance, quality management and accreditation are not things or products; rather, quality is a process.” (Quality is an ongoing event)

- Definition -

18

What does this mean specifically for the BANKSETA?

The extrapolation to a definition of quality assurance for the BANKSETA:

= = = = = =

“Quality assurance refers to the integrated processes that evaluate the provision and

outcomes of education, training and development, so that compliance with SAQA

requirements is achieved and systems for continuous improvement are in place for

the benefit of learners.”

= = = = = =

- Definition -

Learning- SP -


CertificationData

recording- ETQA -

Pri

mary


- SP -

Needs Analysis

- SP / Empl -

In ScopeOutside Scope

19

The following sources gave input to the findings

Literature General literature on Quality Assurance Skills Development Legislation (National & International) SAQA documentation ETQA Guidelines and Policies

Interviews A total of 28 persons were interviewed representing the following

groups: Employers, Service Providers, ETQA managers, SAQA officials and independent experts

Practices 5 ETQAs were researched 3 International Practices were researched (Aus, New Zealand,

Malaysia/Scotland)

- Findings: Primary Process -

20

Practices of the Primary process


Needs Analysis

- SP / Empl -


- SP -

Learning- SP -


Certification /

Data recording

- ETQA -

Pri

mary

Programme Approval by ETQAs1) Yes, including SME and review committee2) Yes, mainly as part of Accreditation3) Yes, mainly as part of Accreditation4) Yes, mainly as part of Accreditation5) Yes, mainly as separate process, no SME needed

Programme Approval InternationalAustralia: Use of nationally endorsed Training Packages

+ Programme ≠ Material

Material ∈ Programme

21

The view of experts on the Primary processes


Needs Analysis

- SP / Empl -


- SP -

Learning- SP -


Certification /

Data recording

- ETQA -

Pri

mary

Employers:“Business Impact will inform the quality better than assessments”

Service Providers: “Too much focus on Assessment and not on Training Needs affects the performance negatively”“SPs tend more toward measuring business impact of training to improve themselves”

ETQAs: “There is a risk in the Employer – Assessor relationship”

Employers: “Huge problem with time that assessments take”

Scope issue

Scope issue

ETQAs: Programme approval is …- Very important- Not so important- SME needed- SME not needed

Independent Experts:“We don’t know how to measure the impact of the NQF and with that the ETQA function”

22

Programme Approval is a gate keeper for the end result, Subject Matter Expertise can be applied depending on the

context …


Safety

Service

Compliance

Banking Sector

Turn around time from sector in case of incorrect content of programme

Pote

nti

al Im

pact

of

err

ors

Thoroughness needed in

Programme

Approval

Possibilit

y of relying on se

lf regulatory fu

nction of th

e

market

Outcomes based

Aligned with US

Aligned with US according to SME

Material correct according to SME

E X A M P L E S

Approval by committee

23

Conclusions on the Primary process


Needs Analysis

- SP / Empl -


- SP -

Learning- SP -


Certification /

Data recording

- ETQA -

Pri

mary

“Garbage in is Garbage out”

“Performance Management is needed”

“Needs analysis falls outside the scope”

1

2

3

24

Way forward on the Primary process


Needs Analysis

- SP / Empl -


- SP -

Learning- SP -


Certification /

Data recording

- ETQA -

Pri

mary

Suggestions

1. Introduce a gate keeper in the form of Programme Approval:- Minimise the use of Subject Matter Expertise- Take samples based on ‘smart’ selection

2. Do further research into whole process of assessments. Is it possible to address the challenge of the sector?

3. Use the Performance Auditing that will be introduced by SAQA to start measuring the performance of the ETQA function from a client perspective (the learner). This will enable continuous improvement of the ETQA processes.

1 2 3

25

Practices of the Secondary process

- Findings: secondary process -

Moderation by ETQAs1) Is done incidentally, policy to minimise it.2) Is not done3) External moderation is done

Moderation InternationalAus: No secondary processes are performed



Secon

dary

Combined with Approval

Internal checksMostly directly or indirectly covered in Accreditation / QMS criteria

Does not existIncidentally considered

26

Experts on the Secondary process

ModerationLearning EvaluationInternal

External ModerationLearning Evaluation

Secon

dary


Independent Experts: “Too much emphasis on assessment and moderation, we should be putting systems in place”

Independent Experts: “Moderation and Program Evaluation should be built-in systems and part of self evaluation”

Experts: “Quality Assurance Mechanisms are being set up for the sake of compliance rather than to improve Quality”

SAQA: “The main degrees of freedom that occur where overlap between ETQAs take place, lie in the Secondary Processes”

Service Provider: “Let me pass the (accreditation) mark and let me then run my own processes”

Programme review

Programme review

27

Service Provider Qualification Sector

Normal CasesCase 1 I a, b, c A Case 2 II i, j, k BCase 3 III x, y, z C

Special CasesExtra-sectoral I a B Extra-qualification I i AExtra-sectoral/qualification I I i BExtra-sectoral/qualification II I x B


The overlap between different SETAs leaves us with four combinations with many more possible applications (1) …

28

The overlap between different SETAs leaves us with four combinations with many more possible applications (2) …

Service Provider QualificationSector/ETQA

Special Cases1. Extra-sectoral I a B 2. Extra-qualification I i A3. Extra-sectoral/qualification I I i B4. Extra-sectoral/qualification II I x B

ETQA activities 1 2 3 4

TertiaryAccreditation A A A A

Auditing A A A A

SecondaryProgram Review ? ? ? ?

Ext Moderation ? ? ? ?

Primary

Certification A B B C

DATA – NLRD A B B C

Data – DOL B A B B


Are these activities done? Who is doing them? Can they be outsourced to another

ETQA?

29

Approval

Conclusions on the Secondary process

“MoU challenge = secondary processes”

“Programme Review = Secondary Process”

“Programme Approval = Primary Process”

“Promotion of a Culture of Quality?”

1

2

3



Secon

dary


ETQA activities 1 2 3 4

TertiaryAccreditation B B B B

Auditing B B B B

Secondary

Program Review ? ? ? ?

Ext Moderation ? ? ? ?

Primary

Certification B A B B

DATA – ELRC

Data – DOL B A B B

° Freedom


- SP -

Learning- SP -

Policing? / Redoing? / Empowering?

Programme Review

30

Way forward on the Secondary process

ModerationLearning EvaluationInternal

External ModerationLearning Evaluation

Secon

dary


Suggestions

1. Minimise the effort put into Secondary Processes

2. Rather focus on Tertiary Processes that enable Service Providers to perform internal Secondary Processes

3. Work in conjunction with SAQA towards clarity concerning the degrees of freedom that occur around the Secondary Processes in case of overlap between ETQAs

Programme review

Programme review

31

Practices of the Tertiary process

- Findings: Tertiary Processes -

Assessor

Service Provider

Moderator

Tert

iary


Accreditation

Registration

Registration


Monitoring visits

Workshops

?Forums

Workshops

?Forums

Workshops

ETQAs - AccreditationProvisional option often usedAll different categories / Criteria within SAQA framework

International AccreditationAus: larger scope, includes advertising and marketing Can, Scotland: incl. e.g. Marketing and Needs Analysis

ETQAs - AuditingInternal Audit not always applicableMostly aligned with AccreditationMixed with Secondary Processes

International AuditingAus: 1-1 alignment with Accreditation

32

Experts on the Tertiary process

Assessor

Service Provider

Moderator

Tert

iary


Accreditation

Registration

Registration


Monitoring visits

Workshops

?Forums

Workshops

?Forums

Workshops


Service Provider: “There is not always a clear link between Accreditation Criteria and how it can positively affect business processes”

Service Provider: “Internal Audits can be a helpful tool to create an internal platform for change and performance improvement”

Service Provider: “Accreditation is still a cumbersome process”

ETQAs: “Accreditation is the key-element to ensure performance”

Independent Experts: “Accreditation process is often not fast enough, results in bottlenecks”

General: “It is possible to get accredited and not perform”

33

Although alignment is important, the Demming wheel suggests an accent on the “Act” phase when performing

external audits


Accreditation

=

Int. Audit Ext. Audit

Checking the results of the Primary Process

Taking Corrective Actions

Plan

Do

Check

Act

Primary Processes

Needs Analysis

- SP / Empl -

Program Development

- SP -

Learning- SP -


Certification /Data

recording- ETQA -

Planning the Primary Process

What is the procedure for Programme Development?

What is the business planning in terms of Programme Focus?

What is the procedure for Programme Evaluation?

What are the corrective actions based on Plan, Do,

Check?

34

The categories for accreditation/auditing are informed by the Primary and Secondary processes where the SP is

involved.



Needs Analysis

- SP / Empl -


- SP -

Learning- SP -


Certification/Data

recording- ETQA -

11 22

33 44

5566 77

88

Secon

dary

Pri

mary

SA

QA

99

……

Secto

r S

pecifi

c

xx

e.g. overhead processes

35

Accreditation and Audit Criteria could be summarised and structured as follows …

General QMS requirements, e.g.- Management responsibility- Follow-up systems

Specific Policies, Procedures and Processes, e.g.- Core

Programme Development, Programme Review (+ requirements)Provision, Learning Evaluation (+ requirements)Assessment, Moderation

- OverheadHR (+ requirements) Financial (+ requirements)…

Static Criteria, e.g.- Admin and Financial Capacity, Legal Entity, …


36

Conclusions on the Tertiary process

Assessor

Service Provider

Moderator

Tert

iary


Accreditation

Registration

Registration


Monitoring visits

Workshops

?Forums

Workshops

?Forums

Workshops

“thorough processes are needed”

“simple and aligned processes”

1

2


37

Way forward on the Tertiary process

Assessor

Service Provider

Moderator

Tert

iary


Accreditation

Registration

Registration


Monitoring visits

Workshops

?Forums

Workshops

?Forums

Workshops

Suggestions Establish rhythm of Accreditation, Internal Audit and External Audit Align accreditation 1-on-1 with Audits and site visits Focus on a Culture of Quality that goes beyond compliance. Look for

‘real’ evidence in the “Act” phase The strength of the sector suggests that Provisional Accreditation is

currently not a necessity. Rather focus on empowerment.


38

The following bullet points remain open for discussion …

Can one size fit all?

Is a sector specific model the route to follow?

Given the degrees of freedom, is it possible to make sensible comparisons?

How close is Best Practice to Common Practice?

- Conclusions? -

‘Research into Quality Assurance of Training Provision for the BANKSETA’ - November 9 th 2005 -

Documents

Transcript of ‘Research into Quality Assurance of Training Provision for the BANKSETA’ - November 9 th 2005 -