170.315(g)(3) Safety-Enhanced Design -...

12/20/2016 1 of 23

170.315(g)(3) Safety-Enhanced Design

1636 Lookout Cir Waxhaw, NC 28173 (704) 843-2374

Report based on ISO/IEC 25062:2006(E) Common Industry for Usability Test Reports. Product: eNotes® 4.1 Dates Of Usability Test: November 18 – 22, 2016

Date Of Report: 12/20/2016 Report Prepared By: eMed Solutions LLC

Atul Kukreja, President (704) 843-2374 [email protected] 1636 Lookout Cir Waxhaw, NC 28173

12/20/2016 2 of 23



Contents Executive Summary ................................................................................................... 3

Introduction ................................................................................................................ 6

Participants ................................................................................................................. 7

Participant Demographics .......................................................................................... 8

Study Design .............................................................................................................. 9

Tasks ........................................................................................................................ 10

Procedures ................................................................................................................ 11

Test Location & Test Environment.......................................................................... 12

Usability Metrics ...................................................................................................... 13

Data Analysis And Reporting .................................................................................. 14

System Usability Scale............................................................................................. 16

Major Findings ......................................................................................................... 17

Areas For Improvement ........................................................................................... 17

Appendix A: Introduction & Instructions ................................................................ 18

Appendix B: Tasks ................................................................................................... 19

Appendix C: Optimal Paths ..................................................................................... 21

12/20/2016 3 of 23



Executive Summary

A usability study of eNotes® 4.1, an EHR used for mental and behavioral health was

conducted November 18 – 22, 2016 in Waynesville, NC and Winterville, NC by eMed

Solutions LLC.

The purpose of this study was to test and validate the usability of the current user

interface, and provide evidence of usability in the EHR Under Test (EHRUT). During the

usability test, eleven healthcare providers matching the target demographic criteria

served as participants and used the EHRUT in simulated, but representative tasks.

This study collected performance data on eight tasks typically conducted with an EHR:

1. Add New Client

2. Add Medication Order

3. Add Laboratory Order

4. Add Imaging Order

5. Add Medication Allergy

6. Add Medication List

7. Check Drug-Drug And Drug-Allergy Interactions

8. Add To Problem List

During the 30 minute one-on-one usability study, each participant was greeted by the

administrator and they were instructed that they could withdraw at any time. Most

participants had prior experience with the EHR. The administrator introduced the test,

and instructed participants to complete a series of tasks (given one at a time) using the

EHRUT.

During the testing, the administrator timed the test and recorded user performance data.

The administrator did not give the participant assistance in how to complete the task.

No additional training or help materials were provided.

12/20/2016 4 of 23



The following types of data were collected for each participant:

• Task successes and failures

• Task performance time

• Path deviations

• Participant’s verbalizations

• Participant’s satisfaction ratings of the system

All participant data was de-identified – no correspondence could be made from the

identity of the participant to the data collected. Following the conclusion of the testing,

participants were asked to complete a post-test questionnaire. Various recommended

metrics, in accordance with the examples set forth in the NIST Guide to the Processes

Approach for Improving the Usability of Electronic Health Records, were used to

evaluate the usability of the EHRUT.

While there were a few path deviations, all participants were able to complete all tasks

successfully. However, for this study , tasks that took longer than twice the expected

(optimal) completion time were considered failures.

Deviations in time taken to complete tasks are attributed primarily to familiarity with the

specific module / function within the EHR, and to the participants’ comfort levels with the

use of technology.

Following is a summary of the performance and rating data collected on the EHRUT.

12/20/2016 5 of 23



Time (Seconds) Rating

Measure Task N#

Task Success Mean %

Task SuccessStd Dev

%

Task Errors%

Task Errors Std Dev

%

Steps Optimal/ Overserved Optimal Mean

Std Dev Dev Mean

StdDev

§ 170.315 (a)(1) CPOE ‐ Medications Add Medication Order 11 100 0 0 0 10/10 75 66 42 9 4 1

§ 170.315 (a)(2) CPOE ‐ Laboratory Add Laboratory Order 11 100 0 0 0 10/10 45 30 16 15 4 1

§ 170.315 (a)(3) CPOE – diagnostic imaging Add Imaging Order 11 100 0 0 0 10/10 30 22 9 8 5 0

§ 170.315 (a)(4) Drug‐drug, Drug‐allergy Interaction Checks for CPOE

Check Drug‐Drug And Drug‐Allergy Interactions 11 100 0 0 0 6/6 60 54 23 6 5 0

§ 170.315 (a)(5) Demographics Add New Client 11 100 0 0 0 5/5 50 43 26 7 5 0

§ 170.315 (a)(6) Problem List Add To Problem List 11 100 0 0 0 6/6 40 29 16 11 5 0

§ 170.315 (a)(7) Medication List Add Medication List 11 80 7.71 20 7.71 7/8 60 77 37 17 4 1

§ 170.315 (a)(8) Medication Allergy List Add Medication Allergy 11 80 7.71 20 7.71 6/7 60 50 43 10 4 1

12/20/2016 6 of 23



Introduction

The EHRUT(s) tested for this study was eNotes® 4.1.

Designed to capture and present medical information to healthcare providers in mental /

behavioral health agencies, the EHRUT consists of a clinical, practice management and

administrative modules accessed via a web browser.

The usability testing attempted to represent realistic exercises and conditions.

The purpose of this study was to test and validate the usability of the current user

interface, and provide evidence of usability in the EHR Under Test (EHRUT). To this

end, measures of effectiveness, efficiency and user satisfaction, such as time to

compete tasks and path deviations were captured during the usability testing

12/20/2016 7 of 23



Participants

During the usability test, 11 healthcare providers matching the target demographic

criteria served as participants and used the EHRUT in simulated, but representative

tasks.

Participants were recruited by eMed Solutions from a pool of existing users at various

client sites. They were not compensated for their time. In addition, participants had no

direct connection to the development of or organization producing the EHRUT(s).

Participants were not from the testing or supplier organization. Participants were given

the opportunity to have the same orientation and level of training as the actual end

users would have received.

Recruited participants had a mix of backgrounds and demographic characteristics. The

following is a table of participants by characteristics, including demographics,

professional experience, computing experience and user needs for assistive

technology. Participant names were replaced with Participant IDs so that an individual’s

data cannot be tied back to individual identities.

12/20/2016 8 of 23



Participant Demographics Part.

Id Sex

Age

(Yrs) Education Role

Professional

Experience

(Months)

Computer

Experience

(Months)

Product

Experience

(Months)

Assistive Technology

Needs

1 F 30-39 Master’s degree

Clinical Manager

180 240 72 None

2 F 30-39 Bachelor’s

degree Quality Mgmt

108 180 6 None


Clinical Manager

264 360 72 None


degree CMS 12 300 12 None

5 M 40-49 Bachelor’s

degree OP Tech 144 180 0 None


degree Medical Records

24 264 12 None

7 M 40-49 High School

Diploma Manager 72 240 36 None

8 F 20-29 High School

Diploma Billing 98 132 12 None


Therapist 42 132 42 None


degree Admin 114 276 42 None


degree Billing 300 300 42 None

12/20/2016 9 of 23



Study Design

The objective of this study was to determine areas where the application performed well

(effectively, efficiently, and to the user’s satisfaction), and areas where the application

failed to meet the needs of the participants. The data from this test may serve as a

baseline for future tests with an updated version of the same EHR and/or comparison

with other EHRs provided the same tasks are used. In short, this testing serves as both

a means to record or benchmark current usability, but also to identify areas where

improvements must be made.

The selected tasks were a representation of the normal tasks performed in the course of

using the application.

During the usability study, participants interacted with the EHR in their regularly

scheduled location / office, and were provided with identical instructions. The system

was evaluated for effectiveness, efficiency and satisfaction as defined by measures

collected and analyzed for each participant:

• Task successes and failures

• Number and types of errors

• Task performance time

• Path deviations

• Participant’s verbalizations

• Participant’s satisfaction ratings of the system

12/20/2016 10 of 23



Tasks

A number of tasks were constructed that would be realistic and representative of the

kinds of activities a user might perform while using this EHR:

Measure Task Associated Risk of User Error

(1 Lowest, 5 Highest)

§ 170.315 (a)(1) CPOE - Medications Add Medication Order 4

§ 170.315 (a)(2) CPOE - Laboratory Add Laboratory Order 3

§ 170.315 (a)(3) CPOE – diagnostic imaging Add Imaging Order 3

§ 170.315 (a)(4) Drug-drug, Drug-allergy Interaction Checks for CPOE

Check Drug-Drug And Drug-Allergy Interactions 2

§ 170.315 (a)(5) Demographics Add New Client 1

§ 170.315 (a)(6) Problem List Add To Problem List 1

§ 170.315 (a)(7) Medication List Add Medication List 2

§ 170.315 (a)(8) Medication Allergy List Add Medication Allergy 2

12/20/2016 11 of 23



Procedures

To ensure that the study results were captured accurately, each participant was

administered the test individually.

Each participant was assigned a participant ID, and the test administrator explained the

purpose of the test, set expectations and provided detailed instructions.

Participants were instructed to perform the tasks:

As quickly as possible making as few errors and deviations as possible.

Without assistance; administrators were allowed to give immaterial guidance and

clarification on tasks, but not instructions on use.

Without using a think aloud technique.

For each task, the participants were given a written copy of the task.

The administrator then began the test, one scenario at a time, and monitored task

times, post-task rating data, and took notes on participant comments.

For each task, the participants were also given verbal instructions of the task, and task

timing began once the administrator said “Begin”. The task time was stopped once the

participant indicated they had successfully completed the task.

Following completion of all tasks, the administrator gave the participant the post-test

questionnaire and thanked each individual for their participation.

Participants' demographic information, task success rate, time on task, errors,

deviations, verbal responses, and post-test questionnaire were recorded into a

spreadsheet.

(Participant Instructions are enumerated in Appendix A)

12/20/2016 12 of 23



Test Location & Test Environment

The eNotes EHR is typically used in a mental /behavioral healthcare office or agency

facility.

Testing was conducted in the participants’ main office or conference room (if privacy

was a concern), on the same workstation (typically a laptop) used by the participant for

their day to day tasks.

The laptops ran either Windows 7 or Windows 8.1 operating system. Internet Explorer 9

or later web browser was used to access the application, as eNotes EHR is a web-

based solution.

The participants used a mouse and keyboard when interacting with the EHRUT.

The laptops had screen size of either 14.1 inches or 15.6 inches (measured diagonally)

and screen resolutions of 1366 x 768, 1600 x 900 or 1920 x 1080 pixels and standard

color settings.

The internet connection was wireless and connected to the participants’ WAN account.

The production (live) application was used in a test environment running a test

database.

Technically, the system performance (i.e., response time) was representative to what

actual users would experience in a field implementation. Participants were permitted to

change any of the default system settings (such as control of font size), to suit their

individual preferences.

12/20/2016 13 of 23



Usability Metrics

The goals of the study were to assess:

1. Effectiveness of EHR by measuring participant success rates and errors

2. Efficiency of EHR by measuring the average task time and path deviations

3. Satisfaction with EHR by measuring ease of use ratings

Measures Rationale and Scoring

Effectiveness: Task Success

A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. The expected amount of time for each task was derived by timing a fully-trained user and multiplying the result by 2 to allow for users who may not be as familiar with the system.

Effectiveness: Task Errors

For each task, if the user failed to complete the task or exceeded the allotted time, that task was recorded as an error.

Efficiency: Path Deviations

The participant’s path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation.

Efficiency: Task Time

Each task was timed from when the administrator said “Begin” until the participant said, “Done.” If he or she failed to say “Done,” the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task.

12/20/2016 14 of 23



Data Analysis And Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. (Table of result metrics are in Page 5) Effectiveness

All participants completed all tasks successfully.

In the allergies module, some users initially had difficulty with selecting the allergen and

reaction. The type of allergy (Drug, food, pet etc.) was easily found since it was a

selection list. However, once the type was selected, the specific allergen and reaction

lists are displayed only on double clicking the respective fields. While this was indicated

via a tooltip on those fields, some participants did not read the tooltips and attempted to

initially type in the information. This failed since the fields are read only. Thereafter the

participants noticed the tooltips and tried double clicking and found the lists to select

from.

This issue will be alleviated by (a) training, and (b) showing the text ‘Double click to

select” in the allergen and reaction fields, so that it is obvious to users unfamiliar with

the module.

When checking Drug-drug and drug-allergy interactions, participants unfamiliar with the

medications module were initially unsure of how to select a medication (from an auto

complete list as the first few letters are typed). Once they started typing the information

displayed and they were able to select the medications.

This issue will be alleviated by training.

12/20/2016 15 of 23



Efficiency

Participants followed the optimal paths throughout most of the testing to complete each

task. There were 15 path deviations across 8 tasks and 11 participants, 8 of which were

for two participants with little or no familiarity with the EHR.

Deviations in time to complete were primarily a result of differences in familiarity with the

specific module of the EHR.

All the observed deviations are easily remediated with training. For the participants

whose primarily responsibility does not include one or more of the assigned tasks being

tested, a deviation is not unusual. For example, one participant’s regular responsibilities

does not include ordering medications. Thus it is not surprising that she clicked on the

wrong toolbar icon initially, before selecting the correct one (the Orders module).

Satisfaction

A System Usability Scale survey was done at the end of each test, with 9 questions.

The overall usability score was 75.91%.

The survey along with the average score of each question is attached:

12/20/2016 16 of 23



System Usability Scale Strongly

Disagree StronglyAgree

1 2 3 4 5 Avg

1. I think that I would like to use this system frequently 4.27

2. I found the system unnecessarily complex 1.36

3. I thought the system was easy to use 4.55

4. I think that I would need the support of a technical person to be able to use this system

1.64

5. I found the various functions in this system were well integrated

4.27

6. I thought there was too much inconsistency in this system

1.55

7. I would imagine that most people would learn to use this system very quickly

3.82

8. I found the system very cumbersome to use 1.45

9. I felt very confident using the system 4.45

12/20/2016 17 of 23



Major Findings

Overall, the users rated EHR highly. There were no major complaints or serious

actionable comments made by participants in the study. While there were some minor

user interface issues that will be reviewed for, there were no major findings during this

usability.

Areas For Improvement The study highlighted some areas of improvement, especially in the allergies module.

More user-friendly and obvious data entry directions on the various fields would help,

especially for new users.

Also the need for additional cross training amongst various users was evident, to allow

familiarity with all modules within the EHR, not just the ones used most by the

participant.

12/20/2016 18 of 23



Appendix A: Introduction & Instructions

Thank you for taking the time today to participate in this usability study.

We are interested in how easy (or how difficult) this system is to use, what in it would be

useful to you, and how we could improve it.

Please remember that we are testing the system and not you. Any problems you

encounter are a result of the user interface being unclear. Your name will not be will not

be associated with any of your comments.

You will be asked to complete a few tasks on your own, trying to do them as quickly as

possible with the fewest possible errors or deviations. Do not do anything more than

asked. If you get lost or have difficulty I cannot answer or help you with anything to do

with the use of the system.

Please start each task only when I say ‘Start’ and say ‘Done’ when you finish

I will be documenting the results of each task before moving on to the next task.

Each of you will be assigned a client named Pink PantherX, where X will be a number

from 1 to 11. Use only that client for all tasks.

The tasks you will be performing are:

1. Add New Client 2. Add Medication Order 3. Add Laboratory Order 4. Add Imaging Order 5. Add Medication Allergy 6. Add Medication List 7. Check Drug-Drug And Drug-Allergy Interactions 8. Add To Problem List

Each task indicates a toolbar icon or menu option to get started.

This study should take approximately 30 minutes. Let’s get started!

12/20/2016 19 of 23



Appendix B: Tasks

Task 1: Add new client You have a new client, Pink PantherX (X = 1, 2, 3 4, 5 etc.). Add the client to the EMR. Only the first name, last name and date of birth and gender are required.

Task 2: Add Medication Allergy The client informs you that he is severely allergic to Penicillin, and has Anaphylaxis reaction to it. Document this drug allergy and the reaction.

Task 3: Add Medication Order You have decided to treat the new client’s depression with Prozac. Add a new order for PROzac 10 MG Oral CAPSULE to be given orally at bedtime. The Dx for this symptom is F33.0 (Major depressive disorder, recurrent, mild). Assign the task to any staff. Do not exit the orders module.

Task 4: Add Laboratory Order You suspect the client may be abusing cocaine. Order a Cocaine Toxicity Workup. The Dx for this symptom is F14.10 (Cocaine abuse, uncomplicated). Assign the task to any staff. Do not exit the orders module.

Task 5: Add Imaging Order The client has been coughing during the session. Add a new order for Chest X-rays. The Dx for this symptom in R91.8 (Other nonspecific abnormal finding of lung field) . Assign the task to any staff. Exit the orders module.

Task 6: Add Medication List The client informs you that he is currently taking Coumadin 10 MG tablets for approximately a month, to treat blood clots. He got it at CVS Pharmacy on Main St. (Add the medication to the client’s medication list. Since you are not prescribing the medication clear your name from the prescriber field by double clicking on your name)

12/20/2016 20 of 23



Task 7: Check Drug-drug, Drug-allergy Interactions You are considering prescribing Amoxicillin for client’s chest infection, indicated by the dry cough, and Xarelto to treat blood clots. Select the medication Amoxicillin 250 MG Oral CAPSULE and observe what happens. Clear the medication by double clicking the medication name. Next, select Xarelto 10 MG Oral TABLET and observe what happens.

Task 8: Add To Problem List (Lists->Clients->Problem List) During the session the client’s guardian shared that he experiences hallucinations, delusions and depression. Add F25.1: (Schizoaffective disorder, depressive type) to the client’s problem list.

12/20/2016 21 of 23



Appendix C: Optimal Paths (Used only by the administrator, not shared with participants)

Task 1: Demographics – Add new client You have a new client, Pink PantherX (X = 1, 2, 3 4, 5 etc.). Add the client to the EMR. Only the first name, last name and date of birth and gender are required.

(1) Navigate to the client portal by clicking on the icon on the toolbar. (2) Type in the last name, first name, date of birth and sex. (3) Click the ‘Save’ button (4) The new client should be saved with a system generated clientId (5) Exit the screen by clicking on ‘Cancel’

Task 2: Add Medication Allergy The client informs you that he is severely allergic to Penicillin, and has Anaphylaxis reaction to it. Document this drug allergy and the reaction.

(1) Navigate to the client portal by clicking on the icon on the toolbar. (2) Select your client. (3) Click the ‘ icon (next to the Allergies text box) (4) In the Allergies module, add a new Drug allergy Penicillin (5) For the Reaction, select ‘Anaphylaxis’. (6) Exit the screen by clicking on ‘Cancel’

Task 3: Add Medication Order You have decided to treat the new client’s depression with Prozac. Add a new order for PROzac 10 MG Oral CAPSULE to be given orally at bedtime. The Dx for this symptom is F33.0 (Major depressive disorder, recurrent, mild). Assign the task to any staff.

(1) Navigate to the clinical portal by clicking on the icon on the toolbar. (2) Select your client (3) Click on the ‘Orders’ tab (4) Click ‘New’ (5) In the new order module, select ‘Medications’ (6) From the Order Desc drop down list select ‘PROzac 10 MG Oral CAPSULE’ (7) In the comments section, type in ‘PO, At Bedtime’ (8) Assign the task to any staff (9) In the Dx1 type in F33.0 and then tab to select Dx ‘Major depressive disorder, recurrent,

mild’ (10) Click the ‘Save’ button

12/20/2016 22 of 23



Task 4: Add Laboratory Order You suspect the client may be abusing cocaine. Order a Cocaine Toxicity Workup. The Dx for this symptom is F14.10 (Cocaine abuse, uncomplicated). Assign the task to any staff.

(1) Navigate to the clinical portal by clicking on the icon on the toolbar. (2) Select your client (3) Click on the ‘Orders’ tab (4) Click ‘New’ (5) In the new order module, select ‘Labs’ (6) From the Order Desc drop down list select ‘Cocaine Toxicity’ (7) Assign the task to any staff (8) In the Dx1 type in F14.10 and then click to select ‘Cocaine abuse, uncomplicated’ (9) Click the ‘Save’ button (10) Close this module by clicking the ‘Cancel’ button.

Task 5: Add Imaging Order The client has been coughing during the session. Add a new order for Chest X-rays. The Dx for this symptom in R91.8 (Other nonspecific abnormal finding of lung field) . Assign the task to any staff.

(1) Navigate to the clinical portal by clicking on the icon on the toolbar. (2) Select your client (3) Click on the ‘Orders’ tab (4) Click ‘New’ (5) In the new order module, select ‘Imaging’ (6) From the Order Desc drop down list select ‘Chest X-Ray’ (7) Assign the task to any staff (8) In the Dx1 type in R91.8 and then click to select ‘Other nonspecific abnormal finding of lung

field’ (9) Click the ‘Save’ button (10) Close this module by clicking the ‘Cancel’ button.

Task 6: Add Medication List The client informs you that he is currently taking Coumadin 10 MG tablets for approximately a

month, to treat blood clots. He got it at CVS Pharmacy on Main St.

(1) Navigate to the medications module by clicking on the icon on the toolbar. (2) Select your client. (3) Select the medication Coumadin 10 MG Oral TABLET (4) In the Date Prescibed field, enter 10/01/2016 (5) In the ‘Prescribed By’ field, double click on your name to clear your name and Id (6) In the Pharmacy field, type in ‘CVS Main St.’ (7) Click on ‘Save’.

12/20/2016 23 of 23



Task 7: Drug-drug, Drug-allergy Interaction Checks You are considering prescribing Amoxicillin for client’s chest infection, indicated by the dry

cough, and Xarelto to treat blood clots.

(1) Select the medication Amoxicillin 250 MG Oral CAPSULE (2) Notice the warnings received for the potential allergic reaction to Penicillin, including blood

disorders and anaphylaxis (3) Double click on the medication to clear the field. (4) Select the medication Xarelto 10 MG Oral TABLET (5) Observe the Drug To Drug Interaction warning for increased bleeding (6) Close out of the Medications module

Task 8: Add To Problem List During the session you observed that the client suffers from of schizophrenia symptoms (

hallucinations and delusions) and depression.

(1) Navigate to the Problem List from the menu Lists->Client->Problem List (2) Select your client (3) Select Dx F25.1: Schizoaffective disorder, depressive type (4) In the ‘Date Reported’ field double click for today (5) Click ‘Add’ to save (6) Exit the screen

170.315(g)(3) Safety-Enhanced Design -...

Documents

Transcript of 170.315(g)(3) Safety-Enhanced Design -...