DEVELOPMENT OF SURVEY FROM AN ITEM BANKFor Counselors Motivational Interviewing Performance Management and Vocational Rehabilitation Program Evaluation

By Ifeoma Chika Iyioke, PhD.

PERFORMANCE MANAGEMENT(PM)

Process by which an agency plans and monitors organizational practices and progress towards goals

PM involves assessment and measurement

Assessment and measurement are activities for collecting, analyzing and/or reporting information about the performance of an individual, group, organization, system or component.

PM IN VOCATIONAL REHABILITATION: CONDITION Emphases on Program Outcome Measures Standards and Indicators Outcome-Based Indicato

r No.Description RSA Criteria

PI 1.1: Change in number of employment outcomes

≥ 0  

PI 1.2: Percent employed > 55.8

PI 1.3: Competitively employed > 72.6

PI 1.4: Significant disability > 62.4

PI 1.5: Earnings ratio > 0.52

PI 1.6: Self-support > 0.52

PI 2.1: Minority ratio > 0.80

PM IN VOCATIONAL REHABILITATION: GAPS

Many confounding factors with outcome measures

Difficult to compare performance of state –agencies and counselors

Lack of focus on the program process and the primary function of the vocational rehabilitation

PURPOSE OF PRESENTATION Introduce a strategy for vocational

rehabilitation counselor PM by monitoring program process

An Item Bank for development of surveys for management of counselor performance of counseling

Connection with vocational rehabilitation program evaluation

FOOD FOR THOUGHT Why measure?

Who or object to measure?

What to measure?

How to measure?

COUNSELOR FUNCTIONS

Consultant

Career/Vocational Counseling

Service Coordinator

COUNSELING FRAMEWORKS Directive Counseling

Trait-Factor (Parson, 1909, 1989) Person-Environment Fit (Dawis & Lofquist, 1984;

Lofquist & Dawis, 1991) Motivational Interviewing (Miller & Rollnick,

1991)

Non-Directive Counseling - Client-Centered (Carl Rogers, 1951)

WHAT IS AN ITEM BANK? A repository for survey/test questions

Survey/test questions organized and catalogued in a similar way to books in a library by topics

Contains also measurement characteristics of the survey/test questions (e.g., their reliability and validity)

EXAMPLE OF AN ITEM BANKItem No

Construct Source Stem Response Scale

Item Statistics

1 Value clarification

Intellectual property

 

The counselor asked about what I really want from life

 

Yes, No, I don't know.

 

 

2 Develop discrepancy

Intellectual property

 

The counselor highlighted inconsistency between my actions and the things I consider most important in life

Every meeting, Half or more than half of the meetings, Fewer than half of the meetings, Never, I don't know

 

 

See

WHY AN ITEM BANK

An assessment tool for standardized performance management

Provides program administrators access to a variety of survey/test questions to cover a whole variety of situations

Can be used to develop effective and efficient surveys/tests for meeting the needs of criterion referenced evaluation

Facilitates assessing the performance of individual employees, monitoring changes, and for evaluating program innovations

DEVELOPING ITEM BANK FOR MOTIVATIONAL INTERVIEWING PM Started with Literature Review - of

Motivational Interviewing

What, Why, and How

Item Bank Design

Populating Item Bank - Creation of Items

MI: DEFINITION

An Assessment Method Interview on motivation via open-ended

questions)

An Intervention (stage-based and tailored) A counseling method (directive and person-

centered ) A method of communication (collaborative

conversation and partner-like relationship)

MI: WHY

Motivation

Readiness or priority

Willing or Importance

Ambivalence - Approach-avoidance conflict

Able or Confidence

MI: WHYImportance/Confidence

Low High

Low Group A Group B

High Group C Group D•Group A: Low Importance, low confidence - These people neither see change as sufficiently important nor believe that they could succeed in making such a change if they tried•Group B: Low importance, high confidence – these people are confident that they could make the change if they thought it were important to do so but are not persuaded that they want to change•Group C: high importance, low confidence – here the problem is not in willingness to change, for these people express desire to do so. The problem is low confidence that they could succeed if they tried•Group D: High importance, high confidence – these people see it is important to change and also believe that they could succeed

MI: PRINCIPLES

Explore motivation

Express empathy

• Develop discrepancy

Roll with resistance

Support self-efficacy

MI: METHODS

Enhance Importance

OARS Open-ended questions Affirmations Reflections Summaries Eliciting change talk

MI: METHODSEnhance Confidence

Review and affirm his or her own past successes in changing behavior

Give a choice of strategies to try, it is more likely that a person may find one that is acceptable as a starting point

DESIGN OF AN ITEM BANK

Define the purpose – formative or summative performance management and evaluation

Define content domain – Question classification and organization Scheme of an item bank

Motivational Interviewing: Why, Components, Principles and Methods

MI: ITEM BANK DESIGN Why MI Component MI Principles MI Methods

Importance

[Ambivalence ]

Assessment Explore Open-ended questions

Confidence Assessment Explore Open-ended questionsImportance

[Ambivalence

]

Assessment Express empathy Reflective listening

Confidence Assessment Express empathy Reflective listeningImportance

[Ambivalence]

Intervention Develop discrepancy

Eliciting change talk

Importance

[Ambivalence ]

Intervention Develop discrepancy

Summarizing

Importance

[Ambivalence ]

Intervention Develop discrepancy

Affirmation

Confidence Intervention Support self-efficacy

Elicit confidence talk

Confidence Intervention Support Self-efficacy

Affirmation

HOW TO POPULATE THE ITEM BANK

Compile and share previous questions

Create new survey questions

HOW TO USE THE ITEM BANK FOR COUNSELOR PM

Build the Plan for performance management into the customer satisfaction survey

The counselor should keep a record of the customers they serve and the level of motivation presented at the time they began to be counseled and at the end of the planning process.

HOW TO USE THE ITEM BANK FOR COUNSELOR PM

Question sampling and survey delivery procedures Administration electronic vs. paper

The surveys are to be administered to the customers served by the counselor at intervals to monitor the counselor delivery of counseling services.

HOW TO USE THE ITEM BANK FOR COUNSELOR PM

Grading – Criterion-Referenced vs. Norm-referenced

Temporal Feedback on areas of strengths and weaknesses

Follow-up with training and remediation of areas of weaknesses

MI AND VOCATIONAL REHABILITATION PROGRAM EVALUATION: MODEL

Engagement & performance of motivational interviewing consumer motivation consumer engagement consumer outcomes (e.g. employment, satisfaction)

MI PROGRAM EVALUATION QUESTIONS

To what extent did the counselor deliver the intended counseling style of motivational interviewing to the consumers?

Does relationship exist between the level of consumer perception of counselor engagement and performance of MI with their motivation, engagement, and outcomes (e.g. employment, satisfaction)?

EXERCISE

The counselor asked about the things I consider most important in life - ?

The counselor highlighted inconsistency between my actions and the things I consider most important in life?

The counselor listened to what I had to say -?

The counselor asked about the justifications for my actions-?

The counselor praised my efforts towards accomplishing my goal(s) -?

QUESTIONS ?

Thank you for listening