CONOR HANRAHAN, PharmD, BCPS

Medical University of South Carolina

South Carolina College of Pharmacy

Charleston, South Carolina

TIMOTHY DY AUNGST, PharmD

MCPHS University School of Pharmacy

Worcester, Massachusetts

SABRINA COLE, PharmD, BCPS

Wingate University School of Pharmacy

Levine College of Health Sciences

Wingate, North Carolina

Any correspondence regarding this publication should be sent to the publisher, American Society of Health-System Pharmacists, 7272 Wisconsin Avenue, Bethesda, MD 20814, attention: Special Publishing.

The information presented herein reflects the opinions of the contributors and advisors. It should not be interpreted as an official policy of ASHP or as an endorsement of any product.

Because of ongoing research and improvements in technology, the information and its applications contained in this text are constantly evolving and are subject to the professional judgment and interpretation of the practitioner due to the uniqueness of a clinical situation. The editors and ASHP have made reasonable efforts to ensure the accuracy and appropriateness of the information presented in this document. However, any user of this information is advised that the editors and ASHP are not responsible for the continued currency of the information, for any errors or omissions, and/or for any consequences arising from the use of the information in the document in any and all practice settings. Any reader of this document is cautioned that ASHP makes no representation, guarantee, or warranty, express or implied, as to the accuracy and appropriateness of the information contained in this document and specifically disclaims any liability to any party for the accuracy and/or completeness of the material or for any damages arising out of the use or non-use of any of the information contained in this document.

Director, Special Publishing: Jack Bruggeman

Acquisitions Editor: Robin Coleman

Editorial Project Manager: Ruth Bloom

Production Manager: Kristin Eckles

Cover and Page Design: David Wade and Carol Barrer

© 2014, American Society of Health-System Pharmacists, Inc. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage and retrieval system, without written permission from the American Society of Health-System Pharmacists.

ASHP is a service mark of the American Society of Health-System Pharmacists, Inc.; registered in the U.S. Patent and Trademark Office.

ISBN: 978-1-58528-458-0

4 Evaluating Mobile Medical Applications

CONTENTS

IntroductionRise of Mobile Applications (Apps)

Table 1: Key terms and definitionsFigure 1: Number of apps by operating system

Smart Devices and MedicineRelevance to Pharmacy Practice

Mobile Medical Apps in Clinical PracticeDangers and Pitfalls of Mobile Medical AppsRegulation of Mobile Medical Apps

Table 2: FDA regulation of medical appsApproach to Critiquing Medical Apps

Traditional Methods for Critiquing ResourcesTable 3: HONcode’s 8 principles for certification of websitesFigure 2: HONcode seal of approvalTable 4: AHRQ criteria for evaluating health information on the Internet

Evaluating Mobile Medical AppsFigure 3: Common information icons used in mobile appsFigure 4: Obtaining details about mobile appsFigure 5: Assessing the design of mobile apps

Tools for Evaluating Mobile Medical AppsTable 5: Rubric for evaluating mobile drug information appsTable 6: Checklist for evaluating mobile drug information and medical reference appsTable 7: Checklist for evaluating mobile medical calculators

SummaryReferencesAnnotated Outside SourcesAppendix 1: Worksheet for Evaluating Mobile Medical Apps

INTRODUCTION

Within the past decade, great strides have been made in the advancement of mobile devices. While the early part of the decade saw the rise of personal digital assistants (PDAs) and mobile phones, recent advancements have led to an evolution of mobile devices into so called smart devices (e.g., smartphones, tablet computers) with further features not present before. Several factors helped contribute to this change, including increased processing power, greater memory storage, a reduction in the size of components, and lower costs to consumers. In addition, cellular services have greatly increased along with the availability of Internet connections through Wi-Fi networks, allowing greater access to information on the Internet via smart devices. Furthermore, most smart devices now incorporate sensors such as accelerometers, global positioning satellite components, and cameras, which have greatly impacted their scope and utilization.

Taking into consideration the technical growth of smart devices, many products are now capitalizing on mobile software applications (apps). These apps are tailored to a specific mobile platform and allow users to perform actions that use one or many functions built into the smart device. Such apps enable users to engage in forms of social media; pursue leisure activities (e.g., photography, shopping, travel, dining);

Evaluating Mobile Medical Applications 5

assist with everyday tasks (e.g., calendars, event planning, navigation); and much more. Many companies have also recognized the ability to use apps as both a marketing tool and to enhance services provided to their customers. As such, smart devices are no longer relegated as merely a direct communication tool but as a source of social engagement and personal productivity. Table 1 describes some of the key terms and concepts associated with smart devices.

Rise of Mobile Apps The current market of mobile apps is reliant on the operating system (OS) associated with the device. The most widely adopted systems are Apple’s iOS, Google’s Android OS, Windows Mobile, and Blackberry. Each system supports their own native and third-party apps available from the device’s mobile app store (e.g., iTunes®, Google Play™). Although some apps are available across systems, others are specific to a particular OS. Current data demonstrate a significant amount of mobile apps available on the market, as seen in Figure 1.

The Apple iTunes® and Android Google Play™ stores currently lead the market in terms of available apps, with Windows and Blackberry having the least amounts of apps. With respect to mobile medical apps, the marketplace is again dominated by iTunes® and Google Play™, both of which have a medical category. Although the true number of mobile medical apps is not currently known, data suggest that more of these apps are available on the iTunes® store compared with its competitors. This market share is due, in part, to the fact that Apple’s iOS was the first novel mobile app supportive device and was adopted early by those in the medical field.

Smart Devices and MedicineThe medical field has always had an interest in the incorporation of technological advances into practice. Many early adopters of PDAs sought to evaluate their impact on clinical care as a medical reference. Several studies throughout the early twenty-first century evaluated the benefits of mobile devices as a way to increase communication, improve access to the medical literature, and streamline productivity and clinical workflow.1-9 At the time, ownership of such devices was rather limited due to their price and functionality.

Table 1: Key terms and definitions

Term Definition

Mobile application (app) Software for use on a mobile platform

Native mobile app Mobile app that comes equipped on a device (e.g., contacts, camera)

Downloadable mobile app Mobile apps that are developed by third-party organizations and then downloaded to the device hardware

Mobile device A handheld computing device characterized by a touch-screen display for input, streamlined operating system, and apps

Smartphone Small portable mobile device focused on communication through messaging and voice-to-voice communication supported by cellular services

Short messaging service (SMS) Services dedicated toward communication through messages consisting of text (i.e., texting)

Social media Networks of communication focused on electronic interactions between users where content is shared, created, and ideas exchanged

Operating system (OS) Overarching device software that manages its memory, processes, and overall performance

Mobile store Online store where mobile apps may be purchased and downloaded to the device

6 Evaluating Mobile Medical Applications

It was also relegated to those institutions that purchased such devices for their members or relied on users to bring their own devices. However, with the advent of recent smart devices, there has been a renewed interest in the implication of mobile devices as an adjunctive tool in medical practice.10-13

Similarly, the practice of pharmacy has been interested in the utilization of mobile devices. Notably, several studies published in the early 2000s sought to identify the quality of information dealing with medication references in comparison with the currently available infrastructure.1-3,14-16 Early adopters found a means to replace the encumbering amount of printed literature on drug information with data that could be compressed into an easily accessible handheld device. In addition, a portable means of documenting interventions by clinical pharmacists via mobile devices was also further investigated.6-9 Today, smart devices and their concomitant apps offer further opportunities in clinical practice for pharmacists.

RELEVANCE TO PHARMACY PRACTICE

Mobile Medical Apps in Clinical PracticeMobile medical apps fulfill multiple roles, often taking advantage of built-in features found within smart devices. For instance, the amount of clinical information that can be stored on smart devices is substantial, with many offering 8 to 64 gigabytes of internal memory. In addition, storage space is enhanced by the availability of cloud computing, whereby content can be stored on external servers and then accessed directly from the mobile device. With such availability, many companies of clinical information have created apps and electronic books that enhance already existing product information for use on a mobile device screen.

Incorporation of other sensors on mobile devices, such as a camera, has also been used to help capture and share clinical images.17 Several developers are exploring the use of the camera and augmented reality technology as a means to conduct diagnostic activities, such as evaluating a skin mole for melanoma. Another ongoing project includes utilizing the camera of a smart device to conduct pill identification. Apps are also creating novel ways to communicate among health care professionals, while safeguarding patient health information and allowing members to have ongoing relationships similar to social media.

*Data current through December 1, 2013.

Figure 1: Total number of apps by operating system*

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

Apple Google Windows BlackBerry

Total Number of Apps

Evaluating Mobile Medical Applications 7

Furthermore, many companies and hospitals are investing in apps to serve as patient education tools, incorporating videos and touch-based teaching methods. Additionally, mobile apps are being used in the education of medical students through the use of quizzes and virtual flashcards. They are also being utilized as digital simulations of clinical situations or surgical interventions, taking advantage of the touch-based display on the smart device. Some examples include demonstration of orthopedic surgical techniques or cardiac interventions, such as coronary artery bypass surgery, in order to show a patient what will be done.

In addition to mobile apps functioning as standalone elements, many upcoming mobile apps are integrating with other peripheral devices. In fact, several companies are creating patient-centric ambulatory monitoring tools to be used in the outpatient setting. These apps capitalize on the ability of smart devices to synchronize with other peripheral devices or systems via Bluetooth (e.g., motor vehicle, scales, watches). Such peripheral devices include health fitness trackers, blood pressure cuffs, glucose monitors, heart monitors, and medication adherence systems. These devices record outpatient vitals, which can be then shared with health care professionals on a regular basis.

Pharmacy practice stands to benefit from the diversity of mobile medical apps and associated peripheral devices as a new means to enhance daily clinical activities and patient care. Future research will investigate the utilization of mobile apps to improve patient health through diet and exercise trackers, and as ways to increase medication adherence. The integration of apps and their ability to track data in the outpatient setting may pose a significant boon to ambulatory care pharmacists working with patients that require chronic monitoring. These apps may also play an integral role in the rise of telemedicine activities. In addition, studies have identified the use of mobile apps as a means to enhance pharmacists’ access to the literature and medication information.1-3,5,14,15

This veritable explosion in mobile medical apps may poise itself to help medical professionals in all ranges of practice; however, there have been significant issues leading to questions about the legitimacy of medical apps.

Dangers and Pitfalls of Mobile Medical AppsWith the huge inroads smart devices have made into society and the medical field, researchers have begun to evaluate the quality, efficacy, and safety of their apps. While initial reports suggested that there were many benefits to using apps in clinical practice,18,19 there have also been studies and reports detailing their numerous shortcomings.20-24

One of the most galling issues that pose the most significant risk to the public is apps that falsely claim to help cure or modify a disease. One early instance occurred in 2011, when a company created an app meant to cure acne by emitting a blue light from the screen of a smartphone. At the time, there were over 10,000 downloads for the app, which sold for $1.99 on the iTunes® and Google Play™ store. The Federal Trade Commission (FTC) stepped in and stopped all marketing of the product and the app quickly disap-peared from the app stores.25 Although no patient was ever harmed by the product, it is troubling that such an app was able to make it to market without proof of efficacy.

Perhaps the next significant risk mobile medical apps may pose is the accuracy of their diagnostic abilities. Currently, there are numerous apps on the market that claim to analyze moles and determine the risk of melanoma using the built-in camera of a smartphone. In a recent study, Wolf and colleagues evaluated several of these dermatology apps and found a startling sensitivity range of 6.8% to 98% and specificity range of 30.4% to 93.7%.26 Several other studies have also shown similar results.27,28 This wide range of accuracy is concerning for the overall industry of mobile medical apps and may pose a significant risk to patients.

8 Evaluating Mobile Medical Applications

While apps used for diagnosing disease may pose a more immediate danger to patients, there are also noted issues with the overall quality of medical apps in general. One serious issue has been a focus on the information provided by apps. Many investigators have found that some third-party developers do not cite or provide references for the content included in the app. In one study investigating the source of infor-mation provided in apps related to cancer, the authors found that less than half actually cited their source of information.29 Another study evaluating the reliability of opioid conversion apps found that only half of the apps evaluated cited the source of their dose conversion guides.30 The significance of these studies demonstrates that, while apps may be beneficial to keep as a medical reference, the quality of information found within them could pose a risk to patient care.

Along with the quality of information found within an app, there is cause for concern when dealing with those who created the app. Similar to other areas of medicine, it is often important to demonstrate that the author has the clinical knowledge to present quality information. For that reason, it is surprising and concerning to find that many medical apps are either created by people with no medical training, or do not include the background of the creator.31 In one study of apps related to vascular surgery, only 27% of the apps reported that a medical professional was involved with its development.32 There have also be several documented instances of plagiarism within medical apps, where a developer has simply taken information from a textbook and transposed it into an app without permission. Several of these cases have been taken to court, though the issue remains that this may occur due to lack of oversight.33

Regulation of Mobile Medical AppsDespite the potential dangers associated with mobile medical apps, most do not undergo formal review or evaluation before entering the market. Currently, most developers must first submit their program for review by the app store (e.g., iTunes®, Google Play™). Although a review process is conducted to ensure the app is functional and has no major technical issues, the clinical content in medical apps is not assessed. As such, it is understandable that many apps of lesser quality can slip through the review process. While this lack of review by those responsible for the app marketplace is concerning, there is also a general lack of federal oversight. In fact, the federal regulation of most software products has proven to be difficult due to their complexity and diversity.

Historically, software products intended for use in the diagnosis or treatment of disease have been classified as a medical device by the U.S. Food and Drug Administration (FDA). The regulation of medical devices differs from that of drugs since it is based on a three-tier classification system. Specifically, devices are designated as either Class I, II, or III, depending on their potential risk. Class I devices are considered to be the lowest risk and are generally exempt from FDA review. Class II devices, however, are considered an intermediate level of risk and developers are usually required to submit a premarket notification (or 510[k] notification). Under this process, developers must show that the product is “substantially equivalent” to a similar device already on the market. Class III devices are considered to be the highest risk level and must generally undergo a more complex, time-consuming, and expensive premarket approval process.34

The regulation of medical apps was not specifically addressed until 2011, when the FDA released a draft guidance on the topic. The guidance, which was updated and finalized in September 2013, outlines how the FDA will apply its regulatory authority to mobile medical apps.35 Table 2 summarizes the types of apps that the FDA intends to regulate. In brief, oversight will apply only to a small subset of medical apps that the FDA considers to be high risk if it does not function as intended. These include apps designed to control other medical devices (e.g., an app that controls the delivery of insulin on an insulin pump) as well as apps that display, store, analyze, or transmit patient-specific data from another medical device (e.g.,

Evaluating Mobile Medical Applications 9

an app that displays live data from a bedside monitor). In addition, the FDA will also apply oversight to apps that use attachments to transform the mobile platform into an already regulated medical device (e.g., attachment of a glucose strip reader to transform the mobile platform into a blood glucose meter). Lastly, apps that perform sophisticated analysis of patient-specific data to provide a patient-specific diagnosis or treatment recommendation will also be subject to regulation (e.g., an app that uses patient data to create a dosage regimen for radiation therapy).35

The guidance also discusses the types of apps for which the FDA plans to exercise “enforcement discretion,” meaning that their regulatory authority would not be applied except in special circumstances (Table 2). This category mainly includes patient-oriented apps, such as those that help patients track and manage health information. Unfortunately, this category also includes many of the medical apps used by pharmacists and other clinicians in daily practice. For example, the FDA will not regulate apps that provide contextually-relevant access to medical information used in clinical practice (e.g., apps that check for drug–drug or drug–allergy interactions). Similarly, the FDA will not review apps that provide clinicians with a summary of best practice guidelines or other therapy recommendations for a particular medical condition (e.g., an app presenting a contextually-relevant antibiotic treatment algorithm based on site of infection). Mobile medical calculators are another type of commonly used app for which the FDA will exercise enforcement discretion. As such, programs used to calculate creatinine clearance, body mass index, CHADS

2 score, etc., are not subject to FDA review.35 Given that many of these apps are used to assist with

treatment decisions (e.g., determining drug choice, drug dose), it is concerning that developers will not be held accountable for the quality and accuracy of the information provided.

In addition to guidance from the FDA, the FTC has released a guide to help mobile app developers remain truthful in advertising and basic privacy principles. Specifically, they suggest that developers avoid making false or misleading claims, avoid omitting important details in advertisements, and have “competent

Table 2: FDA regulation of medical apps35

Regulated apps

• Control other medical devices

• Display, store, analyze, or transmit patient-specific medical data from another device

• Use attachments, display screens, or sensors to transform the mobile platform into a medical device

• Perform patient-specific analysis and provide a patient-specific diagnosis or treatment recommendation

Apps subject to enforcement discretion

• Provide or facilitate supplemental care by coaching or prompting patients

• Tools to help patients organize or track health information

• Provide access to information related to health conditions or treatments

• Allow patients to communicate medical conditions with providers

• Perform simple calculations used in clinical practice

• Enable individuals to interact with electronic health records

Apps that will not be regulated

• Electronic copies of medical textbooks, teaching aids, or other reference materials

• Intended as educational tools for medical training

• Facilitate patient access or understanding

• Automate general office operations

• Not specifically designed or intended for medical purposes

10 Evaluating Mobile Medical Applications

and reliable evidence” that the app functions as intended. Disclosures should also be clear and transparent, as should any data practices regarding privacy (e.g., collecting and sharing user information).36

Although the FTC has made a concerted effort to address deceptive and unfair practices surrounding mobile technology, they are unable to proactively review the large influx of medical apps entering the marketplace. Similarly, as of November 2013, only 100 of the over 10,000 medical apps available on the marketplace were cleared by the FDA.37 As such, it is obvious that clinicians cannot rely on government oversight alone to ensure the safety of mobile apps. Considering the potential dangers, it is imperative that individual users possess the knowledge and skill needed to evaluate these programs.

APPROACH TO CRITIQUING MEDICAL APPS

Traditional Methods for Critiquing ResourcesThe concept of critiquing electronic medical resources has been around for years, originating with various quality initiatives for evaluating health information on the Internet. In 2002, the Institute of Medicine likened the Internet to the “Wild West” stating “it has vast amounts of unregulated territory and no one in charge.”38 In many ways, the growing landscape of mobile medical apps, and the concerns regarding the quality of information, parallels this Wild West mentality. As such, it may be helpful to adapt and apply the criteria for evaluating Internet resources to mobile apps.39

One of the most well-known evaluation tools for Internet resources is the Health on the Net Foundation (HON). First established in 1995, HON was founded to support the access of health information to patients and health care professionals via the Internet.40 The original HON Code of Conduct (HONcode), published in 1996, was intended to enhance the quality of information available, and the current version of the code was published in 1997. The HONcode for medical and health websites consists of the following eight principles: authority, complementarity, confidentiality, attribution, justifiability, transparency, financial disclosure, and advertising. Detail on each principle is provided in Table 3.

Upon request, websites determined to adhere to these eight principles are awarded the HONcode seal (see Figure 2) for placement on the site. HON also performs regular monitoring of certified sites to ensure compliance with the code. The presence of the HONcode seal is a sign to users that the website publisher adheres to an ethical standard. It is important to consider that the seal does not ensure the accuracy of all information contained on the site, nor does it replace the need for professional judgment to apply health infor-mation to a particular patient. With the vast amounts of health information on the Internet and the assurance of continued growth in this area, users must employ critical appraisal of the information found. Criteria and questions have been posed by various organizations that allow users to conduct such critical assessment.

Table 3: HONcode’s 8 principles for certification of websites40

Authority Give qualifications of authors

Complementarity Information to support, not replace

Confidentiality Respect the privacy of site users

Attribution Cite the sources and dates of medical information

Justifiability Justification of claims/balanced and objective claims

Transparency Accessibility, provide valid contact details

Financial disclosure Provide details of funding

Advertising Clearly distinguish advertising from editorial content

Evaluating Mobile Medical Applications 11

The Agency for Healthcare Research and Quality (AHRQ) also provides support for Internet users to critically evaluate health information on the Internet.41 The criteria proposed by AHRQ include credibility, content, disclosure, links, design, interactivity, and caveats. These criteria reinforce the need for health infor-mation to be current, relevant, accurate, thorough, well-organized, and nonbiased. Additional description for each of these criteria is detailed in Table 4.

Several other quality initiatives for evaluating Internet resources have been published to help address the concerns with the quality of information on the Web.42 Although a review of every Internet evaluation tool is beyond the scope of this article, the following key questions have been shown to be useful when evaluating a website43,44:

• Who runs the website?

• What is the purpose of the website?

• Who is responsible for the information?

• How is the information documented?

• What are the credentials of the contributions or reviewers?

• Is the information current?

• What is the website’s linking policy?

• What is the website’s privacy policy?

• Is contact information readily available?

• Who monitors the chat room (if available)?

Table 4: AHRQ criteria for evaluating health information on the Internet41

Credibility Includes the source, currency, relevance/utility, and editorial review process for the information

Content Must be accurate and complete, and an appropriate disclaimer provided

Disclosure Includes informing the user of the site’s purpose, as well as any profiling or collection of information associated with using the site

Links Evaluated according to selection, architecture, content, and back linkages

Design Encompasses accessibility, logical organization (navigability), and internal search capability

Interactivity Includes feedback mechanisms and means for exchange of information among users

Caveats Clarification of whether site’s function is to market products and services or is a primary information content provider

Figure 2: HONcode seal of approval40

12 Evaluating Mobile Medical Applications

Thus, the evaluation of information on the Internet requires a critical assessment of the integrity of the content, careful examination and attention to the source, and a detailed appraisal of the intent for dissemi-nating the information. Fortunately, these same guiding principles can serve as a foundation for assessing the credibility, accuracy, and intent of mobile apps.

Evaluating Mobile Medical AppsMobile medical apps come in a variety of forms, each with their own unique purpose. As such, the evalu-ation of medical apps must vary based on the specific product in question. For example, an app designed to provide antibiotic recommendations for the treatment of pneumonia is very different than one designed to measure a patient’s blood glucose levels. With the antibiotic app, one might be interested in evaluating the clinical references used to support treatment recommendations; however, when evaluating the blood glucose app, one might be more interested in evaluating aspects related to operability. Nevertheless, most apps can be evaluated based on several common principles including their usefulness, accuracy, authority, objectivity, timeliness, functionally, design, security, and value.

Usefulness. One of the first things to consider when evaluating a medical app is its overall usefulness in a particular practice setting. Ideally, apps should help improve one’s efficiency and knowledgebase. An app that is truly useful should make life easier and help streamline job responsibilities. It should be relevant and pertinent to one’s area of practice and have the potential to be used regularly. For example, a pediatric pharmacy specialist may find that an app used to calculate calorie requirements for infants requiring total parenteral nutrition is useful, since it saves him or her time from performing the calculations by hand. Not only does the app streamline the specialist’s job responsibilities, but it is also something that he or she would use on a regular basis. This same app, however, would be of little use to an adult cardiology pharmacy specialist. Thus, the usefulness of an app will largely depend on the purpose of that app relative to one’s practice setting. Evaluating this aspect is often a good starting point because it may not be worthwhile to fully critique an app that would not be useful in daily practice.

Accuracy. After evaluating usefulness, the accuracy of medical apps should be thoroughly examined. This part of the evaluation, however, may vary depending on the intent of the app. For apps designed to provide drug or medical information, the source material used to develop the content will be an important consideration. One should evaluate whether or not the clinical information is well referenced, and then further determine if these references are appropriate (e.g., are they up-to-date and applicable to the content being discussed). For instance, one may determine that the accuracy of an app for warfarin reversal would be better if it cites the most recent guidelines from the American College of Chest Physicians. Similarly, an app used to calculate a CHADS2

risk score may be considered more accurate if it provided details on how to interpret the score and referenced the original studies used to validate the calculation.

The accuracy of the drug or medical content itself should also be checked. One should evaluate the completeness of the information presented and determine if there are inconsistencies or mistakes in the text. For example, one would question the accuracy of an app on antibiotics if it stated that methicillin-resistant Staphylococcus aureus infections could be treated with cephalexin—a drug known to be ineffective against this strain of bacteria. Depending on the app in question, accuracy may also refer to how well it performs its intended purpose. For instance, with the CHADS

2 calculator described above, it is important that the

calculation is accurate and does not over or underestimate the risk score. Likewise, it is important that an app designed to measure blood pressure accurately captures the systolic and diastolic readings within an acceptable margin of error.

Evaluating Mobile Medical Applications 13

Authority. The authority, or authorship, of an app is another important consideration. Given that medical apps can be developed by anyone, it is critical to assess whether the authors and developers are reputable, qualified, and authoritative enough to create the medical content in question. To this end, one must first determine if the content experts of an app are listed. In addition, contact information for the developer should be available in the event that a user has a question or wants to provide feedback about the app. Determining this information can be challenging; however, it can often be accessed in the “about” or “contact” section of an app, as well as various information buttons (Figure 3). The download page within the app marketplace is also a good source of information for authors and developers. Figure 4 demonstrates locations to obtain additional details about an app.

If authors are found, it is important to assess their qualifications and expertise. Considerations should include whether or not the author has medical training, his or her profession (e.g., physician, pharmacist, computer programmer), level of education (e.g., postdoctoral degree, residency training), field of specialty (e.g., cardiology, infectious diseases), previous contributions to the medical literature, previously developed apps, potential sources of bias (e.g., professional affiliations), and years of experience. For instance, one might consider that an app for calculating opioid conversions would be more credible if it were developed by a well-published palliative care physician, as opposed to a software engineer with no medical training.

One must also consider the authority of the developer, especially if specific authors and content experts are not disclosed. Well-recognized and reputable infectious disease resources such as The Sanford Guide® and Johns Hopkins ABX Guide®, for example, will likely hold more authoritative weight than infectious disease apps developed by an unknown entity. Regardless, it is wise to research developers to determine the scope and purpose of the company, other apps they have developed, and the people or organizations with whom the company is affiliated.

Objectivity. Apps that provide drug and medical information should also be evaluated on their objectivity, meaning that the content within the app is fair, balanced, and unbiased. Although making this determination can be difficult—especially if one lacks clinical knowledge or background in a certain area—it is often easy to recognize. For example, an app that is marketed to help clinicians choose an antidepressant medication, but only includes drugs made by a certain manufacturer, is obviously biased since it steers practitioners

Figure 3: Common information icons used in mobile apps

14 Evaluating Mobile Medical Applications

Figure 4: Obtaining details about mobile apps

4 5

1

2

3

6

2

1

3

4

5

6

Mobile app store description tells the user what the ultimate purpose of the app is and how it can be utilized.

The store’s description tells the user about who made the app and where they can be reached, along with details about how it may be used on what devices.

Based on link provided in the store, the user can then view the website where information about the app is stored prior to purchase.

Settings allows the user to access troubleshoot information, alter the app to their specific needs, and allow users to contact the developer with questions.

The help button will display additional information related to clinical database.

The help button may also provide references pertinent to the data or function being provided.

Evaluating Mobile Medical Applications 15

toward recommending certain medications. Similarly, apps that contain promotional advertisements for products or services may have less objectivity than similar apps without advertisements. Ideally, the data and clinical content of the app should be its primary focus.

One should also consider if the funding source or developers of an app could influence its overall objec-tivity. However, it is important to note that while some of an app’s objectivity can be related to authorship, it would be inappropriate to characterize an app as biased based solely on its developer. An app developed by a pharmaceutical company may be perfectly fair and balanced, just as an app developed by a national organization may be biased and misleading. Regardless, the developers or authors of an app should be closely examined for any potential conflicts of interests.

Timeliness. Given that medical information is continually changing, mobile medical apps must also be evaluated based on the timeliness of its content. For instance, an app for picking an anticoagulation regimen may provide inappropriate treatment recommendations if it is based on outdated guidelines or does not include recently approved medications. In addition, one should consider how regularly app updates will occur (e.g., scheduled or on an as-needed basis) and if these updates will be manual (i.e., initiated by the user) or automatic. The Lexicomp® mobile drug information app, for instance, uses a manual process, meaning that the clinical content can become out-of-date if the user forgets to perform regular updates. Determining when an app was last updated can be difficult, but can usually be found within the “about” or “information” section of the app, or on the app download page within the marketplace.

Functionality. An app may contain appropriate clinical information or may be able to measure a patient’s vitals within a small margin of error; however, if that app does not install, launch, and operate consistently then its overall utility will suffer. Thus, examining the functionality and operability of an app is an important part of the evaluation process. Like many of the other components outlined above, this portion of the evaluation may change depending on the particular app.

For example, functionality for an app that provides asthma treatment algorithms may simply mean that it downloads materials appropriately, does not crash/freeze, and has few technical glitches. However, evaluating functionality for an app designed to read an international normalized ratio (INR) may mean that the peripheral device connects consistently and that the sensor is able to interpret INR values under various light conditions. User reviews on the app marketplace are often a good source of information regarding app functionality and overall performance.

Design. Apps should also be evaluated on their design, since well-designed apps are generally more user-friendly. Although there is little formal guidance on evaluating mobile apps, in 2012 the mobile health technology division of the Healthcare Information and Management Systems Society (mHIMSS) released their standards for evaluating app usability.45 In it, they discuss several tenets that health care professionals should consider when selecting and designing mobile apps.

In short, mHIMSS recommends that the interface of the app should be simple and easy to learn, with minimal or no training involved. Buttons should be easy to understand and data should be presented in a clean, uncluttered arrangement. Furthermore, the graphics, layout, and terminology should be consistent and unified across the app. Doing so should help streamline navigation within the app and minimize extra-neous gestures/steps. The terminology used in the app should also be appropriate for the intended audience (e.g., complicated medical terminology should be avoided in apps intended for patients). Text should be sized appropriately, uppercase lettering should be avoided, and a sufficient amount of white space should be present to help improve readability.45 Figure 5 provides an example regarding factors to consider when assessing app design.

16 Evaluating Mobile Medical Applications

Security. Security is another important consideration when evaluating apps. Many apps now require users to create an account with a username and password, as well as enter personal information, such as profession or place of employment. These data have the potential to be collected and sold to third parties for marketing and advertising purposes. Thus, users should make sure that these apps disclose their privacy policy and are provided with an explanation as to why personal data are being collected.

Apps that collect personal information (e.g., passwords, credit cards) should encrypt these data and store them securely to guard against theft. If personal health information is being collected (e.g., medical conditions, test results), then the app must follow compliance rules set forth by the Health Insurance Portability and Accountability Act (HIPAA), as well as the Health Information Technology for Economic and Clinical Health (HITECH) Act.46 More information about the HIPAA and HITECH regulations can be found on the U.S. Department of Health and Human Services website.

Additionally, apps should not compromise the security or functionality of the mobile device being used. The app itself, as well as any advertisements in the app, should not contain viruses, spyware, or other malicious software.

Value. Lastly, the value of the app should be assessed. This aspect should incorporate the cost of the app in relation to its overall strengths and limitations. For instance, suppose a user is evaluating two drug information apps. Both apps provide essentially the same information, but one requires a $50 annual fee while the other is available at no cost. Considering both content and price, the free app is clearly the better value. However, now suppose that the first app with the annual fee provides significantly more information and features. Compared with the free app, the app with the annual fee may now be the better value. The assessment of value is therefore quite subjective.

Figure 5: Assessing the design of mobile apps

Heart Failure

Overview

Clinical Presentation

Diagnosis

Treatment

References

Text is sized appropriately for

easier reading

>

>

>

>

>

Buttons are intuitive and

have a consistent placement on the screen

Sufficient white space is present to separate headings

Headings are simple, clear, and direct

Evaluating Mobile Medical Applications 17

Tools for Evaluating Mobile Medical AppsGiven that there are currently no validated tools for assessing the quality of mobile medical apps, several different rubrics and checklists have been developed for this review. These instruments incorporate the principles outlined in the preceding sections to help users evaluate the relevance, quality, functionality, and security of medical apps. Table 5 places these criteria into a quantitative rubric form, grading them on a scale from 1 (indicating major deficiencies) to 4 (indicating no deficiencies). Although this rubric has not been validated in studies, it may prove particularly useful as a quantitative way to compare similar medical apps with each other. An in-depth worksheet has also been developed to help users evaluate the relevance, quality, functionality, and security of a mobile medical app (Appendix 1). This worksheet is particularly useful for those who want to follow a stepwise approach to the evaluation process.

Tables 6 and 7 place the evaluation criteria into a more simplistic checklist format for use with mobile apps for drug information and medical calculators, respectively. These latter tools were designed to be a simple “yes” or “no” evaluation that can be performed in a short amount of time. It is important to remember, however, that all of the tools provided are only meant to aid users in their evaluation of medical apps. Ultimately, it is the user’s decision as to whether the potential deficiencies of an app are great enough to deter its use.

In addition to the evaluation rubric and checklists provided with this review, several online resources exist to help users assess medical apps. Two of the more commonly used websites include iMedicalApps.com and the Medical App Journal. Both are independent resources created by medical professionals to review, index, and research health-related mobile apps. Happtique, which was founded by the Greater New York Hospital Association, is another website that aims to help users evaluate medical, health, and fitness-related apps. Specifically, Happtique has created a certification program that uses a set of standards to evaluate apps in terms of operability, privacy, security, and content.46 To date, however, the results and utility of this certification program have yet to be seen.

SUMMARY

Mobile medical apps are quickly becoming one of the most important tools in clinical practice. They are an efficient and convenient means to provide real-time medical information at the patient’s bedside, assist with the diagnosis of disease, help patients manage their medical conditions, and serve as educational tools for clinicians and patients. Furthermore, these apps may play an important role in the rise of telemedicine services and other mobile health initiatives. It has even been suggested that one day smartphones, and their associated apps, will be just as necessary as a stethoscope in the clinicians’ medical armamentarium.47

However, despite these many benefits there are potential dangers with the use of mobile medical apps. Perhaps most important is the serious risks to patient safety if these apps do not function as intended or if they are based on inaccurate information. Although the FDA has said it will apply its regulatory authority to a certain subset of high-risk mobile medical apps, they do not plan to regulate many of the medical apps used by clinicians in daily practice. As such, it is imperative that individual users understand how to critically appraise and properly identify apps that are safe to use in patient care. This ability to evaluate resources and ensure their quality has been, and will remain, an essential skill for health care professionals—especially in the ever changing world of mobile technology.

18 Evaluating Mobile Medical Applications

Table 5: Rubric for evaluating mobile drug information appsCriteria 4 points = no deficiencies 1 point = major deficiencies

Usefulness

Points: ____ / 4

App is relevant and would be very useful in daily practice; will improve efficiency or knowledgebase

App is somewhat relevant and could be useful in practice; may improve efficiency or knowledgebase

App is not very relevant and probably won’t be useful in daily practice; may or may not improve efficiency or knowledge

App is irrelevant and would not be useful in daily practice; may hurt efficiency or knowledgebase

Accuracy

Points: ____ / 4

Source material is appropriate and cited throughout; clinical content is thorough/comprehensive

Most source material is appropriate and cited; clinical content relatively thorough

Some material is inappropriate or has few citations; app lacks some important clinical data

No references to source material; missing important content; deficiencies may cause patient harm

Authority

Points: ____ / 4

Publisher and/or authors clearly listed; app developers are considered to be content experts

Publisher and/or authors are listed; developers seem trustworthy and qualified

Publisher and/or authors difficult to locate; app developers may or may not be qualified

Publisher and/or authors not listed; app developers are not qualified/reputable

Objectivity

Points: ____ / 4

Content is fair and balanced; no bias evident; app is only for clinical purposes

Content is relatively fair and balanced; no overt promotion of products is noticed

Content may be biased; some product promotion is evident

Content is heavily biased; app is only for promotional purposes

Timeliness

Points: ____ / 4

Clinical content is current and will be updated regularly

Content is relatively current; may lack some new data but will likely be updated

Content is somewhat old, but still useful to clinical practice; future updates unclear

Clinical content is out of date and irrelevant or harmful to practice; will not be updated in the future

Functionality

Points: ____ / 4

Installs and functions perfectly; no technical problems are evident or anticipated

Rarely crashes, freezes, or has other technical problems

Occasionally crashes, freezes, or has other technical problems

Repeatedly crashes, freezes, or has other technical problems; contains malware

Design

Points: ____ / 4

Incredibly easy to use and navigate; all design elements are consistent and easy to understand

Relatively easy to use and navigate; most design elements are consistent and easy to understand

Often difficult to use and navigate; design elements may hurt some of the app’s usability

Very difficult to use and navigate; design elements definitely hinder usability

Security

Points: ____ / 4

Free of malicious software; privacy statement available; personal data are encrypted/protected

Free of malicious software; privacy statement might be available; personal data are probably protected

May contain malicious software; privacy statement difficult to find; unclear if personal data are protected

Contains malicious software; privacy statement unavailable; personal data are not protected

Value

Points: ____ / 4

Price of app is appropriate, given its content and features

Price of app is reasonable, considering its content and features

Price of app is a barrier and may not be worth the content provided

App is overpriced and not worth the cost

Total: ____ / 36 Comments

Evaluating Mobile Medical Applications 19

Table 6: Checklist for evaluating mobile drug information and medical reference apps

Criteria Assessment Description

Usefulness ❑ Yes ❑ No App will be useful in daily practice

Accuracy ❑ Yes ❑ No Clinical content is based on evidence and is verifiable

Authority ❑ Yes ❑ No Developers are reputable and qualified

Objectivity ❑ Yes ❑ No Content is fair, balanced, and unbiased

Timeliness ❑ Yes ❑ No Content is current and will be updated regularly

Functionality ❑ Yes ❑ No Functions well with no technical glitches

Design ❑ Yes ❑ No Design elements help make the app easy to use

Security ❑ Yes ❑ No Protects user data and is free of malicious software

Value ❑ Yes ❑ No Price is appropriate considering the content and features

Table 7: Checklist for evaluating mobile medical calculatorsCriteria Assessment Description

Usefulness ❑ Yes ❑ No Calculator will be useful in daily practice

Accuracy❑ Yes ❑ No Performs calculations accurately, formulas for equations are verifiable, and results are

explained appropriately

Authority ❑ Yes ❑ No Developers are reputable and qualified

Timeliness ❑ Yes ❑ No Content is current and will be updated regularly

Functionality ❑ Yes ❑ No Functions well with no technical glitches

Design ❑ Yes ❑ No User interface makes the calculator easy to use

Value ❑ Yes ❑ No Price is appropriate considering the content and features

REFERENCES 1. Barrons R. Evaluation of personal digital assistant software for drug interactions. Am J Health-Syst Pharm. 2004;61(4):380-5.

2. Dallenbach MF, Bovier PA, Desmeules J. Detecting drug interactions using personal digital assistants in an out-patient clinic. QJM. 2007;100(11):691-7.

3. Enders SJ, Enders JM, Holstad SG. Drug-information software for Palm operating system personal digital assistants: breadth, clinical dependability, and ease of use. Pharmacotherapy. 2002;22(8):1036-40.

4. Baumgart DC. Personal digital assistants in health care: experienced clinicians in the palm of your hand? Lancet. 2005;366(9492):1210-22.

5. Honeybourne C, Sutton S, Ward L. Knowledge in the Palm of your hands: PDAs in the clinical setting. Health Info Libr J. 2006;23(1):51-9.

6. Collins MF. Measuring performance indicators in clinical pharmacy services with a personal digital assistant. Am J Health-Syst Pharm. 2004;61(5):498-501.

7. Ford S, Illich S, Smith L, et al. Implementing personal digital assistant documentation of pharmacist interventions in a military treatment facility. J Am Pharm Assoc (2003). 2006;46(5):589-93.

20 Evaluating Mobile Medical Applications

8. Fox BI, Felkey BG, Berger BA, et al. Use of personal digital assistants for documentation of pharmacists’ interventions: a literature review. Am J Health-Syst Pharm. 2007;64(14):1516-25.

9. Raybardhan S, Balen RM, Partovi N, et al. Documenting drug-related problems with personal digital assistants in a multisite health system. Am J Health-Syst Pharm. 2005;62(17):1782-7.

10. Sclafani J, Tirrell TF, Franko OI. Mobile tablet use among academic physicians and trainees. J Med Syst. 2013;37(1):9903.

11. Franko OI, Tirrell TF. Smartphone app use among medical providers in ACGME training programs. J Med Syst. 2012;36(5):3135-9.

12. Payne KB, Wharrad H, Watts K. Smartphone and medical related app use among medical students and junior doctors in the United Kingdom (UK): a regional survey. BMC Med Inform Decis Mak. 2012;12:121.

13. Aungst TD, Clauson KA, Misra S, et al. How to identify, assess and utilize mobile medical applications in clinical practice. Int J Clin Pract. 2014;68(2):155-162.

14. Perkins NA, Murphy JE, Malone DC, et al. Performance of drug-drug interaction software for personal digital assistants. Ann Pharmacother. 2006;40(5):850-5.

15. Lowry CM, Kostka-Rokosz MD, Mccloskey WW. Evaluation of personal digital assistant drug information databases for the managed care pharmacist. J Manag Care Pharm. 2003;9(5):441-8.

16. Clauson KA, Seamon MJ, Clauson AS, et al. Evaluation of drug information databases for personal digital assistants. Am J Health-Syst Pharm. 2004;61(10):1015-24.

17. Gurrin C, Qiu Z, Hughes M, et al. The smartphone as a platform for wearable cameras in health research. Am J Prev Med. 2013;44(3):308-13.

18. Mosa AS, Yoo I, Sheets L. A systematic review of healthcare applications for smartphones. BMC Med Inform Decis Mak. 2012;12:67.

19. Ozdalga E, Ozdalga A, Ahuja N. The smartphone in medicine: a review of current and potential use among physicians and students. J Med Internet Res. 2012;14(5):e128.

20. Misra S, Lewis TL, Aungst TD. Medical application use and the need for further research and assessment for clinical practice: creation and integration of standards for best practice to alleviate poor application design. JAMA Dermatol. 2013;149(6):661-2.

21. van Velsen L, Beaujean DJ, van Gemert-Pijnen JE. Why mobile health app overload drives us crazy, and how to restore the sanity. BMC Med Inform Decis Mak. 2013;13:23.

22. McCartney M. How do we know whether medical apps work? BMJ. 2013;346:f1811.

23. Visser BJ, Korevaar DA, Nolan T. Mobile medical apps: dangers and potential solutions. J Telemed Telecare. 2013;19(4):229-30.

24. Buijink AW, Visser BJ, Marshall L. Medical apps for smartphones: lack of evidence undermines quality and safety. Evid Based Med. 2013;18(3):90-2.

25. “Acne Cure” mobile app marketers will drop baseless claims under FTC settlements [Internet]. Washington (DC): Federal Trade Commission; 2011 Sept 8 [cited 2014 Jan 28]. http://www.ftc.gov/news-events/press-releases/2011/09/acne-cure-mobile-app-marketers-will-drop-baseless-claims-under.

26. Wolf JA, Moreau JF, Akilov O, et al. Diagnostic inaccuracy of smartphone applications for melanoma detection. JAMA Dermatol. 2013;149(4):422-6.

27. Robson Y, Blackford S, Roberts D. Caution in melanoma risk analysis with smartphone application technology. Br J Dermatol. 2012;167(3):703-4.

28. Ferrero NA, Morrell DS, Burkhart CN. Skin scan: a demonstration of the need for FDA regulation of medical apps on iPhone. J Am Acad Dermatol. 2013;68(3):515-6.

29. Pandey A, Hasan S, Dubey D, et al. Smartphone apps as a source of cancer information: changing trends in health information-seeking behavior. J Cancer Educ. 2013;28(1):138-42.

30. Haffey F, Brady RR, Maxwell S. A comparison of the reliability of smartphone apps for opioid conversion. Drug Saf. 2013;36(2):111-7.

31. Rosser BA, Eccleston C. Smartphone applications for pain management. J Telemed Telecare. 2011;17(6):308-12.

32. Carter T, O’Neill S, Johns N, et al. Contemporary vascular smartphone medical applications. Ann Vasc Surg. 2013;27(6):804-9.

33. Dyer C. Doctors are accused of plagiarizing a medical guide to produce a smartphone app. BMJ. 2013;347:f5426.

Evaluating Mobile Medical Applications 21

34. Overview of device regulation [Internet]. Silver Spring , MD: US Food and Drug Administration; [updated 2013 Mar 5; cited 2014 Jan 28]. http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/overview/.

35. Mobile medical applications: guidance for industry and Food and Drug Administration Staff [Internet]. Rockville, MD: US Food and Drug Administration; 2013 Sept 25 [cited 2014 Jan 28]. http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM263366.pdf.

36. Marketing your mobile app: get it right from the start [Internet]. Washington, DC: Federal Trade Commission; 2013 Apr [cited 2014 Jan 28]. http://business.ftc.gov/documents/bus81-marketing-your-mobile-app.

37. Dolan B. Analysis: 103 FDA regulated mobile medical apps [Internet]. MobiHealthNews; 2013 Nov 26 [cited 2014 Jan 28]. http://mobihealthnews.com/27645/analysis-103-fda-regulated-mobile-medical-apps/.

38. National Research Council. Speaking of Health: Assessing Health Communication Strategies for Diverse Populations. Washington, DC: The National Academies Press; 2002.

39. Lewis TL. A systematic self-certification model for mobile medical apps. J Med Internet Res. 2013;15(4):e89.

40. HONcode [Internet]. Switzerland: Health on the Net Foundation; [updated 2013 Jun 13; cited 2014 Jan 28]. http://www.hon.ch/HONcode/Pro/Visitor/visitor.html.

41. Assessing the quality of Internet health information: summary [Internet]. Rockville, MD: Agency for Healthcare Research and Quality; 1999 June [cited 2014 Jan 28]. http://www.ahrq.gov/research/data/infoqual.html.

42. Risk A, Dzenowagis J. Review of internet health information quality initiatives. J Med Internet Res. 2001;3(4):E28.

43. Brunetti L, Hermes-DeSantis E. The Internet as a drug information resource. US Pharm [Internet]. 2010;35(1):Epub. http://www.uspharmacist.com/content/d/web_exclusive/c/19130/.

44. Evaluating Internet health information: a tutorial from the National Library of Medicine [Internet]. Bethesda, MD: US National Library of Medicine; [updated 2012 Apr 19; cited 2014 Jan 28]. http://www.nlm.nih.gov/medlineplus/webeval/webeval.html.

45. mHIMSS App Usability Work Group. Selecting a mobile app: evaluating the usability of medical applications [Internet]. Healthcare Information and Management Systems Society; 2012 July [cited 2014 Jan 28]. http://www.mhimss.org/sites/default/files/resource-media/pdf/HIMSSguidetoappusabilityv1mHIMSS.pdf.

46. Health app certification program: certification standards [Internet]. New York, NY: Happtique Inc; 2013 Dec [cited 2014 Jan 28]. http://www.happtique.com/docs/HACP_Certification_Standards.pdf.

47. Visser BJ, Bouman J. There’s a medical app for that: students should be wary when using medical apps, but a few questions can help sift the good from the bad. Student BMJ. 2012;20:e2162.

ANNOTATED OUTSIDE SOURCES

Mobile medical applications: guidance for industry and Food and Drug Administration Staff [Internet]. Rockville, MD: US Food and Drug Administration; 2013 Sept 25 [cited 2014 Jan 28]. http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM263366.pdf.

This document describes how the FDA will apply its regulatory authority to mobile medical apps. According to the FDA, only a select group of high-risk apps will be subject to the medical device regulations under the Federal Food and Drug Act. Unfortunately, many apps that clinicians use in daily practice (e.g., medical reference, drug–drug interaction checkers, medical calculators) are excluded from this high-risk category and will not be subject to regulation. The FDA also goes into detail describing the types of apps that they will and will not regulate, and many different examples are provided. The guidance also includes a list of apps that have currently received FDA approval.

22 Evaluating Mobile Medical Applications

mHIMSS App Usability Work Group. Selecting a mobile app: evaluating the usability of medical applica-tions [Internet]. Healthcare Information and Management Systems Society; 2012 July [cited 2014 Jan 28]. http://www.mhimss.org/sites/default/files/resource-media/pdf/HIMSSguidetoappusabilityv1mHIMSS.pdf.

The Healthcare Information and Management Systems Society (HIMSS) is a not-for-profit organi-zation that seeks to optimize health through information technology. As part of this mission, in 2012 the mobile division of HIMSS (mHIMSS) released its standards for evaluating the usability of medical apps. The document primarily focuses on the areas of efficiency, effectiveness, and user satisfaction. They also outline several key design tenets (or usability principles) that all mobile medical apps should follow.

Mosa AS, Yoo I, Sheets L. A systematic review of healthcare applications for smartphones. BMC Med Inform Decis Mak. 2012;12:67.

Mosa and colleagues provide an interesting article that gives good insight into the scope and effectiveness of mobile medical apps in health care. The authors performed a systematic literature search to identify articles discussing the use, design, or evaluation of smartphone-based software for health care professionals and patients. Several different types of mobile apps are described, such as apps for diagnosing disease, obtaining drug information, performing medical calculations, etc. The authors also provide a nice summary table describing each app and its functions. In addition, the article reviews the rise of smartphones and the various operating systems on which they function.

Haffey F, Brady RR, Maxwell S. A comparison of the reliability of smartphone apps for opioid conversion. Drug Saf. 2013;36(2):111-7.

This article does a good job highlighting the potential dangers associated with medical apps. The authors reviewed 23 different opioid conversion apps across various operating systems. Several startling findings were noted, including a large degree of variability in the dosage conversion between apps, an overall lack of stated medical professional involvement in app development, and a general lack of data sources on which the calculations were based. The authors go on to discuss similar reports that highlight the danger of medical apps and the need for health care professionals to be educated about these risks.

Aungst TD, Clauson KA, Misra S, et al. How to identify, assess and utilize mobile medical applications in clinical practice. Int J Clin Pract. 2014;68(2):155-62.

The authors provide information on how to identify and utilize mobile medical apps. Specifically, they discuss the challenges associated with finding apps for health care professionals in the mobile app store and provide strategies for conducting a more efficient, targeted search. They also describe additional methods for identifying apps, aside from the app store (e.g., published literature, web-based resources). Lastly, the authors mention the need for critically appraising apps, as well as methods for maintaining one’s mobile device as a clinical tool (e.g., securing personal data, institutional support for software apps).

Evaluating Mobile Medical Applications 23

APPENDIX 1: WORKSHEET FOR EVALUATING MOBILE MEDICAL APPS

Determine RelevanceIs this an app worth taking the time to review? If the answer to any of these questions is “No,” it

may be better to look at other apps.

Based on the app store description:

1. Does the app provide information that will benefit you as a clinician or help you in your daily activities?

Yes No (Stop)

2. Are you willing to pay or subscribe to the services provided by the app? Yes No (Stop)

3. Does the app claim to accomplish any of the following:

a. To be used as an accessory to a regulated medical device? Yes No

b. To transform a mobile platform into a regulated medical device? Yes No

If Yes to any of the above, is the app cleared by the FDA? Yes No (Stop)

Determine QualityIf the answers to questions 1 and 2 above are “Yes,” then continued assessment of the app is

mandatory.

4. Information Accuracy

a. Is the information provided accurate and verifiable? Yes No (Stop)

b. Is the information cited within the app or developer website? Yes No

c. Is the information updated regularly? Yes No

5. Authorship and Objectivity

a. Are the authors of the app identified in the app or via the developer website? Yes No

b. Do the authors disclose any conflicts of interest? Yes No

c. Do the authors have the clinical background to be trusted in the integrity of the app? Yes No

Determine FunctionalityIf the app is clinically meaningful with supported information, the following should be reviewed for

continual use:

6. App Stability

a. Does the app download and install correctly? Yes No

b. Does the app perform correctly and is stable? Is it free from any crashing, freezing, or other technical problems?

Yes No

7. App Support

a. Is the app continually updated based on software updates? Yes No

b. Is there a mechanism in place to contact developers with any technical issues? Yes No

8. Design and Usability

a. Is the app easy to use and navigate? Yes No

b. Is there a mechanism or guide on how to use the app? Yes No

Determine SecurityIf relevant to personal practice and data protection is important to user preference, the following

should be assessed:

9. App Access

a. If applicable, does the app offer the ability to password protect any sensitive information stored within?

Yes No

10. App Data Utilization

a. Does the app explain what information is collected from the user and what is done with it?

Yes No

To purchase ASHP eReports and eBooks, please click here.

Was this ASHP eReport helpful? Let us know by completing a quick survey.