O TIPS ON BUILDING - Labelmaster€¦ · Assessment stations Journal articles Case studies Concept...

TIPS ON BUILDING AND USING COMPETENCY ASSESSMENTS

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 3

OF 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

7

Credential Evaluation

6

Collection of Evidence

1

Written Examinations

2

Structured Interviews

3 4

Portfolio

5

Reflective, Self Analysis

Observation,Demonstrations,

Simulations

Types of Assessment Tools / Strategies

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 6

OF 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Written Examinations

Types of Assessment Tools

DESCRIPTION

“Individuals respond to a series of specific questions, often in the form of essays, short answer or multiple-choice test items. They are most often used to assess knowledge about a specific subject.” – THE MANUAL

EXAMPLES ON HOW IT IS USED

Various formats, e.g. Multiple-choice exams, abstract, articles, critical incidents reports, dissertations, essays

Summative, ‘high stakes’, e.g. placement tests, certification exams

Formative, e.g. test of learning offered in a course

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 7

OF 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Structured Interviews



Various formats, e.g. job or placement interviews, component of simulation or demonstration

Summative, ‘high stakes’, e.g. element of certification or licensure exam

Formative, e.g. preparing for job interviews

DESCRIPTION

“Interviews allow individuals to describe in detail what they know and can do, how they have solved problems, and responded to contingencies, etc.” – THE MANUAL

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 8

OF 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Demonstrations, Simulations, Observations



Various formats, e.g. Objective Structured Clinical Evaluation (OSCE), performance evaluations

Summative, ‘high stakes’, e.g. element of certification or licensure exam

Formative, e.g. practicing skills in a learning setting

DESCRIPTION

“Individuals are assessed against the assessment criteria as they perform the task. They are observed in either a ‘natural’ setting or in a simulated situation.” – THE MANUAL

Observation,Demonstrations,

Simulations

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 9

OF 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

DESCRIPTION

“A portfolio is generally a more formal and organized collection of evidence developed by an individual. It may be paper based or digital.” – THE MANUAL

Portfolio


Evidence of achievement of learning outcomes/occupational standards and experience


To prepare for, and use in various opportunities, e.g. entrance requirements, academic credit, job interviews

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 1

0 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

DESCRIPTION

“Self-assessments allow an individual to match specific knowledge, skills, and abilities against established criteria.” – THE MANUAL

Reflection, Self-Analysis, Self-Assessment



Various formats, e.g. training and development needs assessment, ‘interest inventory’

Formative, e.g. part of end-of course study/preparation for exam

Diagnostic, e.g. towards the development of a learning plan

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 1

1 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

DESCRIPTION

“A collection of proof of learning. It will demonstrate an individual’s knowledge, skills and abilities.” – THE MANUAL

Collection of Evidence



Various formats, e.g. log books, notations, product samples, skills passport

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 1

2 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Credential or Qualification Evaluation



For example, a transcript that articulates credential equivalency for the specific application

DESCRIPTION

“An individual’s education and training credentials, or professional qualifications are evaluated to determine authenticity and relevance to Canadian standards. This process compares the credential documents to Canadian standards for a course, program, professional license or employment expectations.” – THE MANUAL

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 1

3 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Plot of General Benefits, Limitations

RIGOUR

CO

ST /

RES

OU

RC

E D

EMA

ND

S

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 1

4 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Further Examples of Assessment Tools

Types of Assessment Tools/Strategies

Abstract

Annotated bibliographies

Assessment stations

Journal articles

Case studies

Concept maps

Critical incident accounts

Debates, oral arguments, mock court

Dissertation

Essay

Field report

In-basket/tray exercises

Lab report

Learning logs/records

Make or design something

Multiple-choice questions

Observation

Online discussion boards

Oral presentations

Problem sets

Portfolios

Reflective diaries

Research projects/group projects

Role play

Simulations

Wiki / blogs

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 1

5 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Who is Doing the Assessing?

Peer AssessmentSelf Assessment External, Third-Party Assessment

Multiple Forms of Assessment

(This linked to the purpose/objectives of the assessment)

Diagnostic, Learning, Professional Development

Diagnostic, Learning, PD, Performance Evaluation

Measure of achievement/ learning, compliance

Complexity, risk, consequence requires multiple evidence

LOW STAKES HIGH STAKES

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 1

6 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

What form of Assessment is the Right One?

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 1

7 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

© 2017 Tourism HR Canada, All Rights Reserved › STEC JUNE 2017 EDUCATORS FORUM › Slide 19 of 84

Creating an Assessment StrategyDue Diligence

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 1

8 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Typical Assessment PlanTesting Specifications / Testing Blueprint

Introduction, Statement of Purpose, Reason

Pre-requisites, Required Conditions

Assessment Process

Structural Variables

Content Weighting and Coverage

Bloom’s Taxonomy

Situational and Contextual Variables

Standard Setting and Scoring, Data Analysis Procedures

Suggested Renewal/Maintenance Requirements

Who

Why

What

Where

When

How

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 1

9 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Typical Testing Blueprint ContentTesting Specifications

Introduction, Statement of Purpose


Assessment Process



Bloom’s Taxonomy




The context, e.g.:

› Purpose, e.g. formative, summative, instructional, guidance

› Intended uses and outcomes

› The assessment criteria that informs the assessment

Compliance with regulations

High stakes: failure to perform can lead to fatality

Assessment of the worker: the specific function they undertake

as part of the supply chain

Based on competencies that support compliance

requirements

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 2

0 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED




Assessment Process



Bloom’s Taxonomy




Statement on minimum requirements of:

› The candidate (i.e. what qualifies them to be eligible for assessment or designation)

› The process, e.g. what must be completed or supplied in order to be eligible

Education or training?

Experience (type, quantity)?

Language proficiency?

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 2

1 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED




Assessment Process



Bloom’s Taxonomy




Selected methods and rationale, e.g.

› Type and length of exam

› Time to complete› Administrative

conditions› Method of delivery› Scoring procedures› Assessment aids

Order of operation

Think about the conditions and constraints that inform the process…

• Employer is doing the assessment

• Assessing current employees on select functions/skills, unique to each worker

• Done while on the job in a normal work context

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 2

2 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED




Assessment Process



Bloom’s Taxonomy




Presentation/format of the assessment

Length of the assessment

Cognitive domains addressed

Range and conditions, where applicable, e.g.: time constraints, location

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 2

3 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED




Assessment Process



Bloom’s Taxonomy




Explicit summary of content areas (from assessment criteria), showing the relative importance and weighting of content

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 2

4 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED




Assessment Process



Bloom’s Taxonomy




Statement on the cognitive abilities represented by the assessment criteria and how this is concordant to the assessment strategy

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 2

5 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED




Assessment Process



Bloom’s Taxonomy




Typical job incumbent profile

Job context and setting

Other profession-related contexts

Factors to consider:Worker profile, e.g. • Type of experience• Level of education• Language proficiency• Cultural factorsJob context, e.g. • Environment• Pace• Level of supervision• Required tools/aids used

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 2

6 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED




Assessment Process



Bloom’s Taxonomy




Scoring rules and method, and the methodology/ies used to establish protocol

Where applicable, decision making rules for:

› handling multiple source of evidence

› determining authenticity, currency, sufficiency of evidence

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 2

7 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED




Assessment Process



Bloom’s Taxonomy




Recommendations for maintenance of the credential, where applicable

Description of recommended administrative practices to monitor assessment results and to make decisions on changes, e.g. address errata, make updates

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 2

8 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Administratively feasible• For example: centralized approach• For example: supported by current LMS and staff resources

Publicly credible• Viewed as important to the public: ‘should be up-to-date’• Increased trust in services rendered

Professionally acceptable• Provides competitive advantage (for employment, contracts)• Meaningful approach and appropriate effort: value proposition

Legally defensible• Conforms to psychometric requirements, where necessary• Lawful

Economically affordable• Cost effective• Priced at a level the market can bear; sees as ‘value’, fair

National Council of State Boards of Nursing, Institute for Credentialing Excellence

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 2

9 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

1 2 3 4 5

Planning Development

Assess Candidate Support

Steps for Development of Effective Assessments

Administration

DETERMINE AND PREPARE

ASSESSMENT STRATEGY AND SPECIFICATIONS

ADAPT OR DEVELOP AND

VALIDATE ASSESSMENT

CONTENT, INSTRUMENTS

AND SUPPORTING TOOLS

PREPARE ANDCONDUCT

ASSESSMENT;GATHER FURTHER

EVIDENCE, AS NEEDED;

INTERPRET RESULTS AND

PREPARE REPORT

FEEDBACK TO CANDIDATE

GUIDANCE, SUPPORT

REVIEW OF TEST RESULTS, MISUSE ISSUES, SECURITY OF ASSESSMENTS,

ENACTING REVISIONS

* This is an example of the phases, e.g. not all applies where standardized testing is in place

Monitoring Results, Quality

Assurance

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 3

0 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Putting it to the TestUsing an Assessment Tool

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 3

1 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Mini Multiple-Choice Exam

1

2

Answer the six questions

5 MINUTES

Debrief as a Group

A FEW TAKE-AWAYS

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 3

2 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Role of Assessor

Plan assessment activities

Develop / prepare assessment

strategies, tools

Assess competence

Follow up from assessment

process

Contribute to quality assurance,

review

On-going professional development

Professionalism, Interpersonal Skills Knowledge

Communication skills, dealing with difficult situations… Assessment tools, system, profession/domain…

3

12

4

Technical, Legal and Ethical Responsibilities

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 3

3 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

A+?

OVERVIEW

Rater/Assessor

BiasHalo Effect, Halo Error

Central Tendency

Leniency Bias

Strictness Bias

Cross-Cultural Biases

Personal Prejudice

Recency Effect/Error

Impaired Judgment

Self-Rater Bias

Time Delay, Waning Memory

Structural / Systemic Bias

Calibration Bias

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 3

4 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

”

“

DEFINITIONSACCOMMODATION

Adaptations to test format or administration that maintain the same construct and produce results that

are comparable to those obtained by individuals who do not use accommodations

MODIFICATIONAdaptations to test format or administration that change the construct being measured in order to

make it accessible for designated students while retaining as much of the original construct as possible

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 3

5 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

GuidanceAccommodation Examples

1 2 3 4

5 6 7 8

Scribe Assistance Technology

Adapted Language

Recorded Responses

Spoken Examination

Administration

Use of Interpreter

Adjusted Physical

Environment

Allotment of More Time

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 3

6 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

Safeguards Against Inappropriate Use

Assessment Instruments, Results

Explicit, published statement on what the assessment is for and where it

should not be used

Ongoing evidencethe assessment is

‘performing’ for the intended purpose

and audience

Corroborating evidence when assessment is

used for special purposes

Published resultsshould be

descriptive to help interpret meaning and application

Follow consistent and prescribed administrative

protocol (no exceptions)

Educate/support users of the results,

e.g. information, tools, guidance

5 S

EP

2018

CH

ICA

GO

〔LAB

ELM

ASTER

DA

NG

ER

OU

S G

OO

DS S

YM

PO

SIU

M〕

SLID

E 3

7 O

F 3

9 ˑ ©

2018

TO

UR

ISM

HR

CA

NA

DA

ALL R

IGH

TS R

ESER

VED

LEAD PRESENTER/FACILITATOR

Philip Mondor, President › PrésidentTourism HR Canada › RH Tourisme CanadaAnd Emerit Consulting

© 2018, Tourism HR Canada, All Rights Reserved4 – 71 rue Bank Street › Ottawa ON Canada K1P 5N2

[email protected]@emeritconsulting.ca

Tourism HR Canada works on behalf of 1.8 millionworkers and 200,000 businesses across Canada, servingone of Canada’s most prosperous and growing industries.Its main goals include improving the quality and mobilityof the tourism workforce, and undertaking workforcedevelopment initiatives that help individual businessesand regions maintain the skilled workforce they need toprosper.

Emerit Consulting is the business division of Tourism HRCanada that specializes in labour market projects. Withover 25 years of experience, Emerit is a world leader in thedevelopment of competency-based training, peopleassessment (psychometric services), competency/standards-setting, voluntary certification and designationprograms, accreditation programs, and specialized labourmarket research. Emerit Consulting has a long history inworking with Essential Employability Skills, LanguageBenchmarks, Occupational Standards, and other jobanalyses models, including the development of world-recognized methodologies and managing the application oruse of these products in a wide range of labour marketproducts.

The consultancy team has a strong understanding of a widenumber of sectors and industry groups, with projects thatspan 18 sectors. Emerit Consulting has had the privilege ofworking for clients in 15 countries (and growing).

O TIPS ON BUILDING - Labelmaster€¦ · Assessment stations Journal articles Case studies Concept...

Documents

Transcript of O TIPS ON BUILDING - Labelmaster€¦ · Assessment stations Journal articles Case studies Concept...