Competency Validation: An Advisory Board Approach

Competency Validation An Advisory Board Approach

by Tammy Barben, RN, MSN

Why Focus on Competency?Over the past decade competency has gone from buzzword to

requirement in healthcare. The Joint Commission, along with

federal and state regulatory agencies, has increasingly focused

on staff competency with each coming year. Why this focus on

competency? The clinician and humanitarian in all of us would

love to say it is about raising the standard of care, but the realist

in us knows that it is also about liability and money. Clinical jobs

are complex and require solid, mastery-level knowledge and

skills relevant to the discipline. Competency exams drill deep

into the clinician’s skill-set and knowledge, which leaves little

room for impression when properly developed, administered and

scored (Biddle 2006, p. 84). Ildiko Hock defined competence as the

intrinsic, tacit knowledge that underlies actual performance (Hock

2003, p. 65). Thus, competency exams have become a necessary

standard for healthcare facilities, agencies and employers.

Creating a validation program on their own can be financially draining for employers.

Competency Validation Program OptionsMany companies struggle to meet

regulatory requirements concerning

staff competency. Developing a

competency program is not only labor

intensive, it requires a team of experts

to develop a program with validity. The

use of substandard test development

and validation procedures can greatly

increase the vulnerability of a test to

legal challenges based on inadequate

evidence for validity (McPhail 2007,

p. 307). Content validation generates

two major benefits for employers:

measuring key, relevant job knowledge,

and dissuasion of possible litigation

(Biddle 2006, p. 29). Creating a

validation program on their own can

be financially draining for employers.

Job experts must be clinically

knowledgeable, licensed, educated

and have significant experience in

order to have the skill set for test

development. Employers do not always

have a team of clinical experts readily

available and struggle with having

a diverse ethnic group of clinical

experts to develop a valid competency

program that is free of bias.

Alternatives to developing your own

competency program can prove to be

financially burdensome as well. Hiring

a consultant group, sending employees

to classes or paying monthly or per-

test fees for online testing can quickly

deplete any company’s profits and time.

API Healthcare developed a competency program that has clinical validity and is free of testing bias.

API Healthcare’s Clinical Validation ProgramHealthcare workforce management

technology vendor API Healthcare

developed a competency program that

has clinical validity and is free of testing

bias. They have partnered with a team

of clinical experts that have expertise in

nursing, therapy, respiratory, imaging,

laboratory, counseling, advanced

practice professionals, pharmacy and

the specialties within these disciplines.

The development, validation and

analysis of the program content is

conducted under the direction of an

Advisory Board which includes the

Clinical Advisory Team, software end-

users and API Healthcare staff.

The Clinical Advisory Team is made up of

clinical experts, many of whom taught

at the college level. The advisory team

includes several Master’s prepared

and Nationally Certified Registered

Nurses, a Pharm D, a Master’s

prepared Social Worker, Licensed

Vocational Nurses, a Certified Wound

Care Nurse, a Master’s prepared

Respiratory Therapist, a Radiology

Technician and Master’s prepared

Medical Technologist. Software end-

users consist of organizations that

have licensed and use API Healthcare’s

Recruiting Solution (ARS) and provide

feedback on the content and direction

of the application. API Healthcare staff

includes members of the Executive

team as well as implementation,

training and engineering resources at

API Healthcare.

API Healthcare programming experts developed internal tools to perform testing validation and analysis.

The first step in this process was to

conduct research on test writing,

validation methods and test analysis

strategies. Validating and analyzing

content and exams to ensure a

bias-free, quality product is a multi-

staged and multi-stepped process. To

address the professional standards,

API Healthcare programming experts

developed internal tools to perform

testing validation and analysis.

The Clinical Advisory Team conducted

the validation process in which each

test question and the complete exams

underwent two separate analyses. The

first review, KSAPCs (Knowledge, Skill,

Ability and Personal Characteristics),

is a technical, job-related review that

ensures questions selected for the

written tests cover information that is

1. Needed day one on the job.

2. Important or critical for

performance of the job and linked to

one or more critical job duties.

3. Requires analysis that draws on the

memory of the test taker, not just

information that can be looked up

(Biddle 2006, p. 89).

Each test question has been

categorized according to this analysis:

Basic Knowledge, Critical Thinking,

Medications/Therapeutic Interventions

and Assessment/Patient Care.

Mitigating Bias in Competency TestingThe second step in the exam review covered a seven-point inspection

to ensure the questions and exams were bias free.

1. Is the question clear and understandable?

2. Is the question written at an appropriate difficulty level?

3. Does the question measure the skill set needed the first day of the job?

4. Is the question fair to all groups of people?

5. Is the question job related?

6. What job skill is measured by this question?

7. Will the applicant be able to provide a sufficient response without employer-specific knowledge?

These court-endorsed ratings allow scoring and analysis of

test results that represent the test taker’s knowledge and skill level.

Competency Testing Analysis Post Validation

Analysis of competency testing results is as important to a bias-free competency

program as the validation of the content by the clinical experts. The Advisory

Board used four primary sources of professional and legal guidance as

guidelines for the ARS software program during the development, validation and

analysis of the competencies.

1. Title VII of the Civil Rights Act of 1964, as amended by the Civil Rights Act of 1991.

2. The Uniform Guidelines on Employee Selection Procedures (Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor and Department of Justice, 1978).

3. The Standards for Education and Psychological Testing (American Educational Research Association, American Psychological Associates and National Council on Measurement in Education, 1999).

4. The Principles for validation and use of Employee Selection Procedures (Society for Industrial and Organizational Psychology, 2003) [Alternative pg. 350].

In the next stage of the competency

project, clinical experts shifted their

focus from development and validation

of content to analysis of the program

and testing of content. Test analysis

included review of the content and

each individual question. The passing

score for each exam was set at 80%,

as this is the standard across the

healthcare industry. Two statistical

tests, the 80% test and the two-

tail fisher exact probability statistic,

were used to identify any testing

bias between groups using the TVAP

analysis software. Content reviews

were also completed, looking at the

Males

• TotalMalestaking exam 30

• Totalpassingexam27

• Totalwhofailedexam3

• Percentageofmalespassing the exam 88%

Females

• TotalFemalestakingexam 181



• Percentageoffemalespassing the exam 85%

Caucasians

• TotalWhitestaking exam 138



• PercentageofCaucasianspassing the exam 86%

African Americans

• TotalAfricanAmericanstaking exam 41



• PercentageofAfricanAmericans passing the exam 85%

Hispanics

• TotalHispanicstakingexam 12



• PercentageofHispanicspassing the exam 83%

Asians

• TotalAsianstaking exam 12



• PercentageofAsianspassing the exam 83%

percentage of test takers correctly

answering each question. All questions

with less than 70% of test takers

answering the question correctly

were removed from the test or the

question was rewritten for clarification

of the question and/or answers.

Feedback was also received from ARS

users and clinicians concerning length

of tests, specific questions, overlapping

content between tests and general

questions concerning competencies.

As this sample data shows, the

exams have undergone extensive

analysis on many levels. The

information gained from this analysis

has been used to enhance both

the test format and the content.

Data

Native Americans

• TotalNativeAmericans taking exam: 2

• Totalpassingexam:2

• Totalwhofailedexam:0

• PercentageofNativeAmericans passing the exam: 100%

Others

• Totalotherstaking exam: 6



• Percentageofotherspassing the exam: 87%

Total Exam Takers: 211



• Percentageoftotaltaking exam: 82%

The following graph displays the

comparison of group scores as

reported by TVAP for the Nursing

Medication Exam. The analysis is based

on a total of 211 scores. As you can see

in the following graph and bullet points,

the groups are compared using both

gender and ethnicity.

0

50

100

150

200

Caucasian

86%

Pas

sed

85%

Pas

sed

83%

Pas

sed

83%

Pas

sed

100%

Pas

sed

87%

Pas

sed

88%

Pas

sed

AfricanAmerican

Hispanic Asian NativeAmerican

Other Males Females

Pass

Num

ber o

f te

sts

take

n

Fail

85%

Pas

sed

Annual Clinical ValidationHansen (1998) conceptualizes ongoing

monitoring as a feedback loop that

provides important information to

revise if necessary. Therefore, ongoing

test analysis, using the Department

of Labor Statistics for each discipline,

will be conducted by The Clinical

Advisory Team along with the API

Healthcare Validation Committee. To

ensure bias-free and valid content,

the analysis will be used to identify

any questions or content that might

need to be revised or updated.

Using both clinical and technological

expertise, API Healthcare has

developed a library of online reading

material and validated exams

focusing on regulatory requirements,

trends in healthcare, standards of

care and basic skill sets specific

to clinical disciplines. This is in

addition to the discipline-focused

checklists that allow employees

to complete a self-evaluation.

Categorizing of exam questions to

match the checklist substantiates

the skills in which the clinician

has experience and expertise.

Competency Testing Reading Materials

Written April 21, 2008; Amended September 25, 2014© Copyright 2015 API Healthcare Corporation. All rights reserved. © 2015 General Electric Company – All Rights Reserved. GE Healthcare, a division of General Electric Company.

Competency Validation: An Advisory Board Approach

Healthcare

Transcript of Competency Validation: An Advisory Board Approach