Evaluation-of-Telehealth-Implementations-in-Australia copy

www.broadband.unimelb.edu.au

A Unified Approach for the Evaluation of Telehealth Implementations in Australia

September 2013

Project team Ambica DattakumarHealth and Biomedical Informatics Centre, University of Melbourne

Kathleen GrayHealth and Biomedical Informatics Centre, University of Melbourne

Susan JuryRoyal Children’s Hospital

Beverley-Ann BiggsRoyal Melbourne Hospital

Anthony MaederSchool of Computing, Engineering & Mathematics, University of Western Sydney

David NoblePrivate Anaesthetic Consultant

Ann BordaVictorian eResearch Strategic Initiative (VeRSI)

Tom SchulzRoyal Melbourne Hospital

Henry GaskoRoyal Melbourne Hospital

AcknowledgementsThis paper was written as part of a project on the evaluation of Australian telehealth implementations. Support for this project entitled An open research initiative to improve the evaluation of Australian telehealth implementations is provided by the Institute for a Broadband Enabled Society (IBES).

Further InformationAmbica Dattakumar: [email protected]

Institute for a Broadband-Enabled SocietyLevel 4, Building 193The University of Melbourne, Victoria 3010, Australia

ISBN 978 0 7340 4865 3

© The University of Melbourne 2013

This work is copyright. Apart from any use as permitted under the Copyright Act 1968 (Cth), no part may be produced by any process without prior written permission from the University of Melbourne.

Unified approach for the evaluation of telehealth implementations in Australia 1

Executive Summary

This paper was produced as part of a one year study, funded by the University of Melbourne interdisciplinary seed grant. This paper will firstly provide a conceptual framework that incorporates the key dimensions, criteria and measures that need to be considered in the evaluation of telehealth implementations in Australia.

Telehealth evaluation can be considered to be the examination of the effectiveness, appropriateness and cost of a telehealth service, by answering four fundamental questions 1) does the intervention work; 2) for whom; 3) at what cost and 4) how does it compare with the alternatives?1!In helping to address these questions for telehealth evaluation in the Australian context, this framework is linked back to a national, validated health performance framework (Australian Institute of Health and Welfare, 2009). The AIHW framework was also used to form a link between the evaluation criteria and measures described in international literature, to health performance indicators. This resulting conceptual framework will be modified and validated with 3 to 5 case studies involving interviews, focus groups with key stakeholders involved in telehealth implementations. This framework will make it more efficient to undertake evaluation of any Australian telehealth implementation, to produce more widely applicable findings, to share these and to improve practice based on the collective results. This paper will be of interest to decision makers, coordinators of telehealth programs or others who are either involved in or concerned about the evaluation of telehealth implementations in Australia. It is a timely and valuable resource, especially in light of the recent recommendations put forth by the Health Innovation and Reform Council, Department of Health, Victoria.2

This paper also provides an evidence base that illustrates the current state of telehealth evaluation on an international scale. A systematic review of systematic reviews on telehealth implementations and evaluations indicates that:

The research around telehealth evaluation is plenty. Some focus on telehealth outcomes in particular specialties, some on specific outcomes such as cost-effectiveness, and others on frameworks and guidelines to support telehealth evaluation.

Telehealth evaluation can be complex with a great many potential inputs, outputs, outcomes and stakeholders. This may be one reason for a lack of established telehealth evaluation protocols, which in turn has hindered decision-making to implement wide scale initiatives.3

• There is no standardisation of definitions, criteria, measures across the literature leading to ambiguity and confusion.

• There is no link between the telehealth evaluation criteria and health performance indicators. Therefore, the overall impact of telehealth on the healthcare system cannot be judged.

• The methodologies for conducting telehealth evaluations are not consistent.

The paper is divided into three sections. Section 1 provides an overview of the literature, method and proposed framework. Section 2 provides the proposed framework for telehealth evaluation. Section 3 describes the results of grouping the various criteria and measures mentioned in the literature. This paper also has two Appendices. Appendix A should be used to support the reading of Section 3. Appendix B provides an annotated bibliography of the papers reviewed, as well as further information on the literature.


Contents

Executive Summary .................................................................................................................................................................................................. 1

1 Telehealth Evaluation: International Evidence ............................................................................................................................ 3

1.1 Method ............................................................................................................................................................................................................... 3

1.2 Insights .............................................................................................................................................................................................................. 3

1.3 Lessons ............................................................................................................................................................................................................... 4

1.4 Conclusion ........................................................................................................................................................................................................ 4

2 Proposed Framework for Telehealth Evaluation in Australia ............................................................................................. 5

2.1 Dimensions, Criteria & Measures for Proposed Framework ............................................................................................ 5

2.2 Scope of Proposed Framework ........................................................................................................................................................... 5

3 Grouping of evaluation criteria and measures identified in literature ...................................................................... 8

3.1 Results ............................................................................................................................................................................................................... 8

Appendix A: Grouping of criteria and measures from literature under key terms ........................................................ 9

Appendix B: Systematic review of systematic reviews of literature on evaluation of telehealth implementations ..................................................................................................................................................................................................... 31

4 References ......................................................................................................................................................................................................... 71


1 Telehealth Evaluation: International Evidence This section offers a summary of the systematic review of reviews of evaluation of telehealth (more detail provided in Appendix C) that has been completed by the project team. The literature review aimed to identify:

• Dimensions, criteria and measures that have been used to evaluate telehealth implementations. • Insights into how (methods) telehealth evaluation has been conducted in the past. • Future directions for telehealth evaluation.

1.1 Method

A systematic review of systematic reviews was conducted of literature published internationally, between 2008 and 2013, sourced through Google Scholar. The search was carried out over a 2 month period. Search terms included:

Telehealth evaluation, telemedicine evaluation, "telehealth evaluation" "Australia", "telehealth" "evaluation" "models", benefits evaluation frameworks telehealth, systematic review telehealth, systematic review telehealth telemedicine, telemedicine and telehealth evaluation "systematic review", "systematic review" "telehealth" "Australia", systematic review telehealth assessment models, systematic review of systematic reviews telehealth.

Over 90 papers were found as a result of these searches. The titles and abstracts of each paper were assessed and this narrowed down the number of papers to 70. Upon reading the whole article, and determining whether the paper contained anything about outcomes, evaluation guidelines, methods or frameworks, 57 articles were chosen for review. Of these 57 articles, 7 were primarily focused on the frameworks and methods. These were reviewed separately. Therefore, 50 articles were part of the systematic review of reviews.

1.2 Insights

There is a significant amount of literature discussing telehealth evaluation, across various specialties in health care. However, the criteria and measures are ambiguous, and ill defined. Criteria such as ‘patient satisfaction’, ‘quality of life’, ‘effectiveness’, which are mentioned several times (please see Section 2) are not defined and a method of measure, not identified. This makes it difficult to identify the outcomes of telehealth use and to compare these outcomes to other telehealth projects.

Recent systematic reviews (over the past 5 years) of telehealth and/or telehealth evaluation in Australia are lacking. In 2000, a methodology for telehealth evaluation in Australia was put forth by the Australian New Zealand Telehealth Committee (no longer in existence), but this has since not been updated or widely used (with the exception of the paper written by Hughes, King and Kitt in 20024). This presents a big gap in the literature.

Patients are mostly, not at the forefront of evaluations. With the exception of 1 review !Verhoeven, Tanja-Dijkstra, Nijland, & Eysenbach, 2010)5, the focus of evaluations has mainly been on clinical aspects, and costs to run telehealth and savings in healthcare provision costs. This needs to change as we move into a more patient centred approach to providing care.


Methods used to conduct telehealth evaluations are very varied. Randomised control trials, case studies, and case control trials have been applied, to study small sample sizes in pilot projects (reviews can be found in Ekeland, Bowes & Flottorp, (2012), Kummervold, Johnsen, Skrovseth & Wynn (2012), Rubin, Wellik, Channer & Demaerschalk, (2012))6. Frameworks and guidelines for evaluations have been developed with no link to health performance indicators (for example of frameworks, see Masella & Zanaboni, (2008), Rowe, Jonsson & Terio (2011), Chen, Chen, Weng Shang, Yu et al, (2012), Kidholm, Ekeland, Hensen, Rasmussen, pefersen et al. (2012))7. They provide extensive lists of criteria and measures that are at times, repetitive across various dimensions, and not defined or justified. Therefore, no method provided an accurate, thorough picture of the evaluation outcomes of telehealth projects, including the economic benefits, and their impact on the overall healthcare system.

The findings above are summarised by the work done by Haily, Ohinmaa and Roine in 20048. They developed a way to classify the literature to assess the quality of telecardiology evaluation studies. The classification was based on study performance and study design. Study performance was assessed against the following criteria: Patient selection and randomization, variances in intervention versus control group; Description of interventions – for both intervention and control groups; Specification and analysis – including sample sizes, statistical methodology and clear outcome measures; Patient disposal – such as drop-out, follow-up and compliance and Outcomes reported – comprehensively, accurately, wholly (positive and negative), consistent with data. Study design was related to the method used to collect data. Depending on the score assigned to these two aspects, the quality of the study was assessed (11.5-15 being a high quality study and a score < 5, poor quality. It was found that the quality of studies remains low. Out of 44 papers, over half were rated to be poor or poor to fair. From a decision maker’s perspective, there was very little evidence of the benefits of telecardiology. Unfortunately, this trend has continued within the telehealth evaluation field.

1.3 Lessons

Externally validated instruments, frameworks and guidelines for telehealth evaluations are few and lacking. Although telehealth is supposed to improve the efficiencies of healthcare systems, the outcomes of its use are not related to healthcare performance indicators. There is no consensus on which criteria are most important and how these should be measured. This leads to the development of very complex frameworks that contain several criteria and measures. This is further exacerbated by the number of stakeholders involved, various outcomes and other variables. This means that evaluating telehealth projects become a challenging task, and the results of evaluations are frequently not comparable or shareable.

1.4 Conclusion

The evaluation of telehealth is well documented on an international scale, but very varied in quality. It is very challenging to generalise/compare telehealth evaluation criteria and measures.


2 Proposed Framework for Telehealth Evaluation in Australia

This section provides an outline of the proposed framework for telehealth evaluation in Australia. This framework:

1. Takes into consideration the views of stakeholders including clinicians, and telehealth project coordinators, currently involved in various telehealth projects, in three different organisations in Australia.

2. Is linked to a validated, national based framework that provides precise health performance indicators (Australian Institute of Health and Welfare National Health Performance Framework, 20099). The technical dimensions of the framework are linked to the ‘technical aspects of telehealth’ in the framework put forth by the Australian College of Remote and Rural Medicine (ACRRM Telehealth Advisory Committee (ATHAC) telehealth standards framework, 201210).

3. Is based on a systematic review of systematic reviews of literature related to telehealth evaluations on an international scale (please refer to Appendix B for an annotated bibliography and further information on the papers reviewed). It is important to note that the two frameworks mentioned in point 2 above, were also used to link the evaluation criteria and measures in the literature to health performance indicators.

4. Is based on the grouping of the evaluation criteria and measures, extracted from the literature review. These were grouped into key themes (please refer to section 3 and Appendix A for more detail).

2.1 Dimensions, Criteria & Measures for Proposed Framework

The proposed dimensions of the framework, criteria and measures are provided in Table 1 below. The dimensions are defined according to the AIHW framework and ACRRM framework (discussed in Section 2). The criteria and measures are arranged as follows: patient control, followed by the clinical quality of care, then organisation sustainability and lastly, technology capacity/capability. The patient is a crucial stakeholder, and this is followed closely by the provision of safe and effective care. The measures are broad enough to capture data across various specialities. This allows for comparison across various telehealth projects. A description of the criteria and measures are also provided in the table.

2.2 Scope of Proposed Framework

The definition of telehealth in the context of this work considers the consultations between clinicians and patients, using real-time video conferencing. Therefore the use of telehealth equipment for continued professional development activities is out of scope of this work. Also, the store and forward or asynchronous mode of delivery (storage of information before sending it to clinician for further investigation), is not investigated, unless it is somehow, a part of the real-time interaction between the patients and clinicians.


Table 1: Proposed framework for telehealth evaluation in Australia

Dimension/Components Link to AIHW framework health performance indicators (2009) or ACRRM framework (2012) Criteria Measures Description of criteria and measures

Patient Control

Responsiveness: Healthcare service is patient oriented. The client is treated with dignity, confidentiality and encouraged to participate in choices related to their care. Accessibility: People can obtain healthcare at the right place, at the right time irrespective of incomes, physical location and cultural backgrounds. Continuity of care: Ability to receive uninterrupted coordinated care or service across programs, practitioners, organisations and levels over time.

Changes in individual’s productivity

Number of days of leave for health reasons

Changes in an individuals’ productivity, and days of leave for health reasons, can be a simple and clear measure of improved quality of life.

Changes in access to required healthcare service

Number of in person appointments, number of telehealth appointments

Access to required healthcare services is proposed to be a significant benefit of implementing telehealth. Time and cost savings may not be accurate measures. However, a change in the number of in person appointments could help in explaining whether or not telehealth has improved access to healthcare.

Clinician Quality of Care

Effectiveness: Care/intervention/action provided is relevant to the client’s needs and based on established standards. Care, intervention or action achieves desired outcome. Safety: The avoidance or reduction to acceptable limits of actual or potential harm from healthcare management or the environment in which healthcare is delivered.

Mortality rate

Number of deaths of patients using telehealth in comparison to in person delivery of care.

This is a vital measurement of how safe and effective the treatment provided via telehealth, is to the patient. This is an important determinant of the quality of care provided via telehealth.

Clinical indicators

The accuracy of a key measure in any illness – BP, blood sugar, physical activity, movement etc.

This criterion accommodates for changes to various determinants of good health. These determinants will vary across specialities, but provide a good way to measure the effectiveness of diagnostics / treatment provided via telehealth consultations.


Dimension/Components Link to AIHW framework health performance indicators (2009) or ACRRM framework (2012) Criteria Measures Description of criteria and measures

Organisation sustainability

Efficiency & Sustainability: Achieving desired results with most cost effective use of resources. Capacity of system to sustains workforce and infrastructure, to innovate and respond to emerging needs.

Cost to run the telehealth service for healthcare provider.

Fixed Cost in comparison to alternative modes of treatment. Variable Cost in comparison to other modes of treatment.

Fixed costs, such as capital investment in equipment and variable costs include maintenance and repairs, telecommunication costs, administrative support and supplies, training, wages to technicians and clinical staff. Measuring both these fixed and variable costs will provide an overall picture of how much has been invested into the telehealth project.

Savings in Cost for healthcare provider.

Savings per patient per year to healthcare service provider.

This criteria accommodates for telehealth application in a range of settings, such as hospitals, clinics, nursing homes, etc. The measures here could therefore vary from number of hospital admission, readmissions, number of visits to emergency department, to the number of visits to the clinics after using telehealth.

Technology capability/capacity

Link to ACRRM telehealth advisory committee standards framework (2012): Technical aspects of telehealth: Adequate performance: equipment works reliably and well over available network and bandwidth. Equipment is compatible with equipment used at other sites. Standards relevant to security of storage and transmission are met. Peripheral devices are fit-for-purpose. Commissioning of equipment: Equipment installed according to producer’s guidelines. Equipment and connectivity are tested with other participating healthcare organisations. Risk management: Risk analysis is performed. Procedures for detecting, diagnosing and fixing equipment are in place. Technical support services are available. Backup plan to cope with equipment or connectivity failure.

Reliability

Number of successful consultations.

The reliability of telehealth technology must be evaluated as it can affect the quality of care. The number of successful consultations is a good measure of the impacts of speed, distance and connectivity.

Data quality

Number of instances data was re-sent during/after teleconsultation (post measure).

The quality of the data being transmitted, whether it be audio, or images, is important as again, it can have an impact on the decision making and quality of care. A simple measure, that is not technical in nature, can be used to measure data quality; the number of instances when data was re-sent during/after the teleconsultation.


3 Grouping of evaluation criteria and measures identified in literature

The following section provides a grouping of the criteria and measures that have been mentioned in telehealth evaluation literature. This can be useful when trying to identify and short list the criteria that can be use to evaluate a telehealth project. This section presents the results of categorising and grouping the various criteria and measures presented in the reviewed literature using the dimensions of the proposed framework. As mentioned in section 2, there are 4 different dimensions (patient control, clinical quality of care, organisation sustainability and technical capability/capacity). The criteria and measures pertaining to each dimension have been listed, grouped and categorised by key themes that best described these groupings. The criteria and measures under each key theme are organised by year and author. This has been done in the form of 4 tables (Tables 2 to 5), in Appendix A. The purpose and relevance of the last table, Table 6, is explained below. For further information on the papers, please refer to Appendix B.

3.1 Results

• The biggest group of criteria and measures (45 in total) was under organisation sustainability. The title or theme of the grouping was Utilisation of healthcare services and included criteria that focused on reduced hospital admissions, emergency visits and so on. The next biggest group was Cost to run telehealth service (28 in total).

• In relation to patient control, the criteria and measures grouped under the theme Time and cost savings was the biggest group (20 in total). This was closely followed by the Patient satisfaction group (18 in total). The measures for both of these criteria are varied and at times, unclear.

• In relation to clinical quality of care, changes in clinical indicators, such as glucose levels, BP measurements were the biggest group (27 in total). This was closely followed by quality of life measures (20 in total). However, quality of life measures were either not specified or too vague (NB: the paper by Hawthorne & Osborne (2005) can be used to measure quality of life in the Australian context)11. Due to the lack of definition, it is also likely that quality of life was confused with quality of care.

• The technology capability/capacity dimension contained the least number of criteria and measures in comparison to the other three dimensions of the framework. Of these, security issues (5 in total) were the top grouping, followed by connectivity and technology issues (3 in each grouping).

• There were some criteria and measures discussed in the literature that could not be clearly linked to any of the dimensions of the conceptual framework. When grouped under key themes, the themes are caregiver satisfaction/perceptions, health professionals’ satisfaction, and usability of the device. Even though these criteria are important, their impact on the quality of care provided via telehealth is not direct or very clear. These criteria and measures are listed in Table 6.


Appendix A: Grouping of criteria and measures from literature under key terms

Table 2: Grouping of Criteria and Measures from literature under Patient Control dimension of proposed framework

Criteria Measures

Time/ cost savings for patients

Time and cost (Rojas & Gagnon, 2008, p.899) Average travel time and cost (p.899, p.900) for patients (minutes), lost productivity for patient (p.899)

Hospitalisation charges (Seto, 2008, p.680) Usual care vs. telemonitoring costs.

Reduced travel costs (Seto, 2008, p.681) Pre and post telemonitoring comparison of costs.

Medical costs (Davalos, French, Burdick, & Simmons, 2009, p.938) Out of pocket expenses, avoided travel expenditure (p.939)

Economic viability for patients (Jaana, Pare, & Sicotte, 2009, p.318) Cost-benefits analysis

Cost effectiveness (Ekeland, Bowes, & Flottorp, 2010, p.741) Travel time

Cost savings (Wade, Karnon, Elshaug, & Hiller, 2010, p.6) Travel reduction (p.6), reduced time off work (p.9)

Cost savings to patient (Verhoeven, Tanja-Dijkstra, Nijland & Eysenbach, 2010, p.677) Travel costs

Travel cost (Warshaw, Hillman, Greer, Hagel, MacDonald et al., 2011, p.769) Distance travelled by the patients.

Decreased costs (Backhaus, Agha, Maglione, Repp, Ross et al. 2012,p.118) Travel costs

Less travel and associated costs (Hilgart, Hayward, Coles, & Iredale, 2012, p. 773) Comparison of time and cost of travel between control group and telegenetics group.

Reduced waiting times (Hilgart, Hayward, Coles, & Iredale, 2012, p. 773)

Comparison of waiting time to get an appointment with genetics specialist, between control group and telegenetics group.

Direct non-medical costs (Mistry, 2012, p.4) Do not mention types of cost, but state that direct non-medical costs are those incurred by the patient.

Savings in cost (Rietdjik, Togher, & Power, 2012, p.918) Cost of telehealth intervention in comparison to impatient costs.

Cost savings (Wallace, Hussain, Khan, & Wilson, 2012, p.467) Travel time


Criteria Measures

Reduced cost of treatment (Hall, Boisvert & Steele,2013, p.32) No measures have been specified.

Reduced travel costs (Bradford, Armfield, Young & Smith, 2013, p.9) Comparison between home telehealth and a control group receiving care through another medium.

Reduced waiting time at appointments (Bradford, Armfield, Young, & Smith, 2013,p.9) Comparison between home telehealth and a control group receiving care through another medium.

Time savings (Edirippulige, Martin-Khan, Beattie, Smith, & Gray, 2013, p.129) Travel time.

Travel costs (Hall, Boisvert, & Steele,2013, p.32) No measures have been specified.

Patient Satisfaction

Patient satisfaction (Jaana, Pare, & Sicotte, 2009, p.315) Not specified

Patient satisfaction (Kairy, Lehoux, Vincent, & Visintin, 2009, p.442) Interpersonal relationship between patient and clinician

Patient satisfaction (Polisena, Tran, Cimon, Hutton, McGill, et al., 2009, p.924) Comparison to conventional care

Patient satisfaction (Ekeland, Bowes, & Flottorp, 2010, p.741) More confidence and empowerment, better clinician relationships

Patient satisfaction (Garcia-Lizana, Munoz-Mayorga, 2010, p.123) Comparison to patients receiving traditional treatment.

Patient satisfaction (Garcia-Lizana, Munoz-Mayorga, 2010, p.5)

Comparison of satisfaction between those receiving treatment through video conferencing and those receiving traditional treatment.

Patient satisfaction (Johansson & Wild, 2010, p.151) No measure has been specified.

Patient satisfaction (Polisena, Tran, Cimon, Hutton, McGill, et al., 2010, p.126)

Not specified as various instruments used to measure this criteria, and the outcomes could not be pooled together.

Patient satisfaction (Shulman, O’Gorman & Palmert, 2010, p.4) Comparison between telemedicine group and control group.

Patient satisfaction (Peeters, Mistiaen & Francke, 2011, p.409) Measures are not identified.

Patient preference (Warshaw, Hillman, Greer, Hagel, MacDonald, et al., 2011, p.768) Comparison between teledermatology and clinic dermatology.


Criteria Measures

Patient satisfaction (Warshaw, Hillman, Greer, Hagel, MacDonald et al., 2011, p.768)

Either a comparison between the teledermatology group and clinic dermatology group or just feedback from teledermatology group.

Patient satisfaction (Backhaus, Agha, Maglione, Repp, Ross et al. 2012, p.119) Comparing groups using videoconferencing with in-person psychotherapy

Patient satisfaction (Hilgart, Hayward, Coles, & Iredale, 2012, p. 773) Not specified clearly – each aspect used to describe this can stand alone as a separate criterion.

Patient satisfaction (Mistry,2012, p.5) Convenience for patient, patient acceptability, patient willingness to use telemedicine

Patient Satisfaction (Young, 2012, p.50) Initiation of consultation by patient, length of intervention

Patient Satisfaction (Edirippulige, Martin-Khan, Beattie, Smith, & Gray, 2013, p.129) No measure has been specified

Satisfaction (Rubin, Wellik, Channer, & Demaerschalk, 2013, p.4) Comparison between telemedicine and in-person counselling.

Patient self care/knowledge

Increased health knowledge/ability to self –care (Davalos, French, Burdick & Simmons, 2009, p.942) Not specified

Improved care (Durrani, & Khoja, 2009, p.178) Increased patient's knowledge and maintaining good health.

Patient behavioural and psychological effects (Jaana, Pare, & Sicotte, 2009, p. 315)

Increased feelings of security and reassurance and control over their medical condition; better knowledge and awareness about their disease; improved communication with health professionals; adherence to medication.

Improved interaction (Verhoeven, Tanja-Dijkstra, Nijland, & Eysenbach, 2010, p.675)

Frequency of metabolic data transmission, increased intensity of contact between provider and patient (p.676)

Improved self care (Verhoeven, Tanja-Dijkstra, Nijland & Eysenbach, 2010, p.676)

Frequency of monitoring blood sugar levels or blood pressure, adherence to medication, improvement in knowledge, better understanding of medical condition.

Knowledge of the illness (Peeters, Mistiaen & Francke, 2011, p.409) Measures are not identified.

Self measurement of BP (Verberk, Kessles & Thien, 2011, p.150) Comparison between telecare group and usual care group.

Blood glucose self-monitoring (Cassimatis & Kavanagh, 2012, p.449) Frequency of self reported glucose levels.


Criteria Measures

Improvement in knowledge (Ciere, Cartwright, & Newman, 2012, p.389)

Proportion of patients stating the purpose and side effect of medication between telehealth group and control group.

Improvement in self-care (Ciere, Cartwright, & Newman, 2012, p.389) Compliance with medication and monitoring of weight, diet and BP in comparison to control group.

Improvement in self-efficacy (Ciere, Cartwright, & Newman, 2012, p.389) Levels of confidence in performing self care between telehealth group and control group.

Affective outcomes (Hilgart, Hayward, Coles & Iredale, 2012, p.774) Pre-post comparison of patient’s knowledge after telegenetics consultation (p.775)

Knowledge of stroke signs and symptoms (Rubin, Wellik, Channer, & Demaerschalk, 2013,p.4) Comparison between telemedicine and in-person counselling.

Stroke awareness education for a rural community (Rubin, Wellik, Channer, & Demaerschalk, 2013,p.4) Comparison between telemedicine and in-person counselling.

Likelihood to change habits (Rubin, Wellik, Channer, & Demaerschalk, 2013,p.4) Comparison between telemedicine and in-person counselling.

Patient empowerment (Van den Berg, Schumann, Kraft, & Hoffmann, 2012, p.112) No measures specified.

Patient outcomes (Vedel, Mignerat, Saksena & Lapointe, 2013, p.226)

Patient’s knowledge or self-care, patient transfer or travel time, social support/functioning, patient worry, patient satisfaction.

Access to required healthcare services

Access to healthcare (Davalos, French, Burdick, & Simmons, 2009, p.942) Change in QALYs

Access to care (Durrani, & Khoja, 2009, p.178) decreased travel, getting required care or access to a specialist

Access (Johansson & Wild, 2010, p.152) Ability to access services by those based in remote areas.

Accessibility (Ray-Moreno, Reigadas, Villalba, Vinagre & Fernandez, 2010, p.118) Accessibility to services for people in deprived areas.

Improved equity (Verhoeven, Tanja-Dijkstra, Nijland & Eysenbach, 2010, p.678) Better access to specialised health care, timely follow up (p.678)

Increased accessibility (Wade, Karnon, Elshaug, & Hiller, 2010, p.10) Access to different types of specialist service in rural/remote areas


Criteria Measures

Increased access to healthcare (Verberk, Kessles, & Thien, 2011, p.153) Comparison between telecare group and usual care group.

Increased access to healthcare (Backhaus, Agha, Maglione, Repp, Ross, et al., 2012, p.118) Increased access to care for rural, underserved populations

Access to care (De Waure, Cadeddu, Gualano & Ricciardi, 2012, p.326) Comparison between group using telemedicine and control group.

Increased access to healthcare services (Hall, Boisvert & Steele, 2013, p.32) This is presented in the discussion section of the paper with no measures provided.

Quality of care (Vedel, Mignerat, Saksena, & Lapointe, 2013, p.226) Accessibility to health services

Patient adherence/compliance

Increased medication adherence (Davalos, French, Burdick, & Simmons, 2009, p.942) Change in QALYs

Treatment adherence (Garcia-Lizana & Munoz-Mayorga, 2010, p.123) Comparison to patients receiving traditional treatment.

Medication compliance (Peeters, Mistiaen & Francke, 2011, p.409) Measures are not identified.

Adherence to treatment (Verberk, Kessles, & Thien, 2011, p.150) Comparison between telecare group and usual care group.

Medication adherence (Cassimatis & Kavanagh, 2012, p.449) ASK-20 items, Morisky Adherence Scale.

Dietary adherence (Cassimatis & Kavanagh, 2012, p.449) Measurement straight after active intervention period.

Improved compliance (Giamouzis, Mastrogiannis, Koutrakis, Karyannis, Parisis et al., 2012, p.5) Recording of weight, blood pressure

Improved medical adherence (Giamouzis, Mastrogiannis, Koutrakis, Karyannis, Parisis et al., 2012, p.5) World Health Organisation’ multidimensional adherence model.

Behavioral endpoints (Van den Berg, Schumann, Kraft & Hoffmann, 2012, p.112)

Adherence to medication or diet, physical activity, daily life activities. Better self-efficacy and management of disease.

Improved adherence to intervention protocol (Hall, Boisvert & Steele, 2013, p.32) This is presented in the discussion section of the paper with no measures provided.


Criteria Measures

Improved productivity for patients

Employment/leisure/classroom time (Davalos , French, Burdick & Simmons, 2009, p.938) Missed days or hours, avoided absences, increased leisure time, Increased productivity (p.942)

Clinical outcomes (Kairy, Lehoux, Vincent & Visintin, 2009, p.430) Returning to work

Indirect costs (Shulman, O’Gorman & Palmert, 2010, p.4) Missed days of work or school

Reduction of urgent visits to the school nurse (Shulman, O’Gorman & Palmert, 2010, p.3) Comparison between telemedicine group and control group.

Return to work (Rietdjik, Togher & Power, 2012, p.918) No measure has been specified.

Patient Acceptance/Awareness

Awareness of telehealth (Swanepoel,& Hall, 2010, p.6) Not specified

Acceptance/ satisfaction (Van den Berg, Schumann, Kraft, & Hoffmann, 2012, p.112) No measures specified.

Perceptions of telehealth (Bradford, Armfield, Young, & Smith, 2013, p.10) Feasibility, acceptability, satisfaction with telehealth.

Acceptance (Edirippulige, Martin-Khan, Beattie, Smith & Gray, 2013, p.129) No measure specified.

Willingness to use

Use among patients (Jaana, Pare & Sicotte, 2009, p.315) Adherence to data transmission.

Willingness to use (Swanepoel,& Hall, 2010, p.6) No measures have been specified.


Table 3: Grouping of Criteria and Measures from literature under Clinical quality of care dimension of proposed framework

Criteria Measures

Changes in clinical/ other relevant indicators

Clinical outcomes (Kairy, Lehoux, Vincent & Visintin, 2009, p.430) Quality of life, Function in activities of daily living, range of motion, pain, gait, exercise capacity, cognitive tasks, speech quality, skin integrity, falls efficacy, fatigue, anxiety and depression, compliance with exercise plan.

Changes in modifiable risk factors (Neubeck, Redfern, Fernandez, Briffa, Bauman et al. , 2009, p. 284)

total cholesterol, blood pressure, Body Mass Index, Smoking status, amount of physical activity, nutritional status.

psychosocial state(Neubeck, Redfern, Fernandez, Briffa, Bauman et al., 2009, p.285)

depression, stress measured using cardiac depression scale, center for epidemiologic studies depression scale, perceived stress scale

Glycaemic control (Polisena, Tran, Cimon, Hutton, McGill, et al. 2009, p.924) Measurement and changes in hba1c.

Evaluation of symptoms (Garcia-Lizana & Munoz-Mayorga, 2010, p.5) Changes in current symptoms.

Changes in glycemic control (Pare, Moqadem, Pineau & St-Hilaire, 2010,p.7) Comparison between telemonitoring group and group receiving usual care

Changes in asthma symptoms (Pare, Moqadem, Pineau, & St-Hilaire, 2010, p.8)

Comparison of telemonitoring group to one being treated by GP and one being treated by specialists. Changes in nighttime symptoms, daytime symptoms, improved peak expiratory flow in the morning and night.

Changes in blood pressure levels for BP patients (Pare, Moqadem, Pineau, & St-Hilaire, 2010, p.8) Comparison between telemonitoring group and group receiving alternative care.

Changes in blood pressure levels, and total and LDL cholesterol levels (Pare, Moqadem, Pineau, & St-Hilaire, 2010, p.8) Comparison between telemonitoring group and group receiving usual care.

Effect on HbA1C (Shulman, O’Gorman, & Palmert, 2010, p.2) Comparison between telemedicine group and control group.

Effect on hypoglycaemia (Shulman, O’Gorman, & Palmert, 2010, p.3) Comparison between telemedicine group and control group.

Changes in clinical values (Verhoeven, Tanja-Dijkstra, Nijland & Eysenbach, 2010, p.672)

Haemoglobin levels (p.672), lipid profiles (p.673), stable ulcer healing (p.673), cardiovascular risk factor control (p.673), glucose levels (p.673), glycemic control (p.675)

Changes in BP (Verberk, Kessles, & Thien, 2011, p.150) Comparison between telecare group and usual care group.

Clinical outcomes (Warshaw, Hillman, Greer, Hagel, MacDonald, et al., 2011, p.768) Comparison between teledermatology and clinic dermatology.


Criteria Measures

Changes in clinical outcomes (Backhaus, Agha, Maglione, Repp, Ross, et al., 2012, p.119) Depression and or/anxiety, eating disorders, physical problems, addictions

Glycaemic control (Cassimatis & Kavanagh, 2012, p.448) Measurement straight after active intervention period.

Physical activity (Cassimatis, & Kavanagh, 2012, p.449) Effect on glycaemic control

Disease exacerbation (Kamei, Yamamoto, Kajii & Nakayama, 2012, p.8) Comparing group receiving care via telehealth with group receiving conventional care, after a period of 3 months of using telehealth care.

Clinical outcomes (Oliver, Demiris, Wittenberg-Lyles, Washington, Day et al., 2012, p.45) Patient anxiety, communication anxiety.

Psychological well being (Rietdjik, Togher & Power, 2012, p.918) Comparison of telehealth group with group receiving usual care.

Behavioral status (Rietdjik, Togher & Power, 2012, p.918) Comparison of telehealth group with group receiving usual care.

Cognitive function (Rietdjik, Togher, & Power, 2012, p.918) Comparison of telehealth group with group receiving usual care.

Reduction in substance use (Young, 2012, p. 50) Addiction severity index, timeline follow up method, urine screening, alcohol use disorders identification test.

Changes in motor and cognitive symptoms (Edirippulige, Martin-Khan, Beattie, Smith, & Gray, 2013, p.130) Comparison between patients assessed via telemedicine and in-person assessment.

Improvement in stimuli (Hall, Boisvert, & Steele, 2013, p.32) Improvement of patient’s stimuli to material presented on computers

Visual cues and stimuli (Hall, Boisvert, & Steele, 2013, p.32) Comparison to traditional settings for assessment.

Health indicators (Vedel, Mignerat, Saksena, & Lapointe, 2013, p.226) Impact on glucose or HbA1c blood level, hyper glycemic events, BMI or weight, cholesterol or trigyceride blood level, blood pressure, quality of life, physical activity, Framingham risk score, depression/mental health, nutrition intake, pain.

Improvements to quality of life

Patient outcomes (Seto, 2008, p.679) Improved quality of life.

Quality of life (Maric, Kaan, Ignaszewski & Lear, 2009, p.507) A measure of this is not specified.

Quality of life (Polisena, Tran, Cimon, Hutton, McGill et al., 2009, p.924) Diabetes quality of life score

Quality of life (Ekeland, Bowes & Flottorp, 2010, p.741) Not specified


Criteria Measures

Quality of life (Garcia-Lizana & Munoz-Mayorga, 2010, p.5) Comparison of satisfaction between those receiving treatment through video conferencing and those receiving traditional treatment.

Quality of life (Pare, Moqadem, Pineau, & St-Hilaire, 2010, p.12) Quality of life measures not specified. It is only stated that questionnaires were use to collect this information

Quality of life (Polisena, Tran, Cimon, Hutton, McGill, et al., 2010, p.126)


Diabetes-related quality of life (Shulman, O’Gorman & Palmert, 2010, p.4) Comparison between telemedicine group and control group.

Quality of Life (Verhoeven, Tanja-Dijkstra, Nijland & Eysenbach, 2010, p.675) Diabetes Quality of Life measure , rate of depression

Health status (Peeters, Mistiaen & Francke, 2011, p.409) Measures are not specified.

Quality of life (Peeters, Mistiaen, & Francke, 2011, p.409) Measures are not identified.

Decreased risk of rehospitalization, revascularization, subsequent myocardial infarction, and/or death (De Waure, Cadeddu, Gualano, & Ricciardi, 2012, p.325)

Comparison between group using telemedicine and control group.

Health related quality of life (Kamei, Yamamoto, Kajii & Nakayama, 2012, p.9)

St George Respiratory Questionnaire (SGRQ). A score of more that 4 on this was claimed to be clinically significant.

Reduced hospitalization risk (Kamei, Yamamoto, Kajii & Nakayama, 2012, p.4) Comparing group receiving care via telehealth with group receiving conventional care.

Quality of life (Rietdjik, Togher, & Power, 2012, p.918) Comparison of telehealth group with group receiving usual care.

Overall functioning (Rietdjik, Togher & Power, 2012, p.918) Comparison of telehealth group with group receiving usual care.

Improvement of functional status (Van den Berg, Schumann, Kraft & Hoffmann, 2012, p.96) No measures specified.

Quality of life (Van den Berg, Schumann, Kraft & Hoffmann, 2012, p.96) General improvement or disease-specific improvements.

Quality of life (Bradford, Armfield, Young, & Smith, 2013, p.8) Comparison between home telehealth and a control group receiving care through another medium.


Criteria Measures

Quality of life (Edirippulige, Martin-Khan, Beattie, Smith & Gray, 2013, p.130) Comparison between patients assessed via telemedicine and in-person assessment.

Accuracy of diagnosis/treatment/intervention/assessment

Diagnostic use (Kairy, Lehoux, Vincent & Visintin, 2009, p.442) Quality of video transmission, use of technology to treat certain patients, eg. Shy children.

Correct treatment decisions (Johansson & Wild, 2010, p.152) Measures are not specified.

Diagnostic accuracy (Johansson, & Wild, 2010, p.152) Measures are not specified.

Diagnostic accuracy (Ray-Moreno, Reigadas, Villalba, Vinagre & Fernandez, 2010, p.118) Tele-ECG compared to normal ECG.

Diagnostic accuracy (Wade, Karnon, Elshaug, & Hiller, 2010, p.7) Authors mention a reduction in diagnostic accuracy as a result of using telehealth for dermatology, but do not discuss how this was measured.

Diagnostic accuracy (Warshaw, Hillman, Greer, Hagel, MacDonald, et al., 2011, p.763)

Matching teledermatology diagnosis to histopathology diagnosis or other laboratory tests. Comparing in person dermatology diagnosis with teledermatology.

Diagnostic concordance (Warshaw, Hillman, Greer, Hagel, MacDonald et al., 2011, p.765) Comparison between teledermatology and clinic dermatology.

Management accuracy (Warshaw, Hillman, Greer, Hagel, MacDonald, et al., 2011, p.767) Comparison between teledermatology and clinic dermatology.

Management concordance (Warshaw, Hillman, Greer, Hagel, MacDonald, et al., 2011, p.767)

Comparison between teledermatology and clinic dermatology management decisions, eg. ‘Refer or not refer’, ‘biopsy or no biopsy’

Diagnostic accuracy (Hilgart, Hayward, Coles & Iredale, 2012, p.774) No new diagnosis/ no difference when compared to face-to-face appointment.

Medical outcomes (Van den Berg, Schumann, Kraft & Hoffmann, 2012, p.96) Accuracy of diagnostic results between telemedical group and usual care.

Accuracy of diagnosis (Wallace, Hussain, Khan & Wilson, 2012, p.470) Adequacy of image, image quality

Adequacy of decision making (Wallace, Hussain, Khan & Wilson, 2012, p.473)

Comparison of telehealth consultation to face to face consultation: appearance and fell of scar, contracture, range of motion, activity status, wound breakdown problems.

Reliability of assessment (Edirippulige, Martin-Khan, Beattie, Smith & Gray, 2013, p.129) Comparison between patients assessed via telemedicine and in-person assessment.


Criteria Measures

Accuracy of assessment (Hall, Boisvert & Steele, 2013, p.32) Clinician’s ability to administer language assessment protocols.

Accuracy of diagnosis (Rubin, Wellik, Channer & Demaerschalk, 2013, p.121) Tracking diagnosis made by neurologist in teleconsult and face-to-face visits.

Diagnosis (Rubin, Wellik, Channer & Demaerschalk, 2013, p.4) Performing transcranial Doppler (TCD) and carotid duplex (CD) through telemedicine and comparing to in-person examination

Quality of care – timely treatment & effectiveness of treatment/intervention/assessment

Effectiveness of care(Rojas & Gagnon, 2008, p.899) Quality of life

Improved quality of care (Durrani, & Khoja, 2009, p.178) On-time hospitalisation

Effectiveness of assessment (Boisvert, Lang, Andrianopoulos & Boscardin, 2010, p.431) Positive behavioural changes

Therapeutic effectiveness (Ekeland, Bowes & Flottorp, 2010, p.739) Examples of telehealth applications that are therapeutically effective are provided but evaluation measures are not specified.

Improved quality of care (Johansson & Wild, 2010, p.151) No measures have been specified.

Time between onset of stroke symptoms to admission (Johansson, & Wild, 2010, p.152) Onset-to-hospital, door-to-needle, onset-to-needle

Clinical significance (Hailey, Roine, Ohinmaa & Dennett, 2011, p.285) Measures are not specified.

Effectiveness of intervention (Hailey, Roine, Ohinmaa & Dennett, 2011, p.283)

Comparison between home and hospital based rehabilitation, specifically, how frequently patient was contacted and provided with additional services.

Time to treatment (Warshaw, Hillman, Greer, Hagel, MacDonald et al., 2011, p.768)

Comparison between teledermatology and clinic dermatology. Time taken from GP consult to dermatology clinic.

Rapid medical decision making and timely intervention (De Waure, Cadeddu, Gualano & Ricciardi, 2012, p.325) Comparison between group using telemedicine and control group.

Family member characteristics moderated treatment efficacy (Rietdjik, Togher & Power, 2012, p.918)

Family socio-economic status, parent-reported child behaviour outcomes, ethnicity, carer’s experience with technology

Effective interventions (Young, 2012, p.50) Intervention length and its effect to substance use outcomes.

Effectiveness (Hall, Boisvert, & Steele, 2013, p.32) Comparison of telepractice group to in-person assessment.

Effectiveness (Marino & Ghanim, 2013, p.181) Comparison to alternative modes of treatment.


Criteria Measures

Morbidity/ Mortality rate

Reduced mortality (Seto, 2008, p.679) A specific measure is not discussed.

Morbidity (Davalos, French, Burdick & Simmons, 2009, p.942) Change in QALYs

Mortality (Davalos, French, Burdick & Simmons, 2009, p.942) Change in QALYs

All-cause mortality (Neubeck, Redfern, Fernandez, Briffa, Bauman et al. 2009, p.283) mortality of any patient taking part in the telehealth program.

Better health outcomes (Johansson & Wild, 2010, p.151) Reduced dependency, reduced mortality

Mortality rate (Pare, Moqadem, Pineau, & St-Hilaire, 2010, p.8) Comparison between telemonitoring group and usual care group

Patient mortality (Wade, Karnon, Elshaug & Hiller, 2010, p.7) Change in mortality rate after using telehealth.

12-month survival rate (De Waure, Cadeddu, Gualano, & Ricciardi, 2012, p.325) Comparison between group using telemedicine and control group.

In-hospital mortality (De Waure, Cadeddu, Gualano, & Ricciardi, 2012, p.325) Comparison between group using telemedicine and control group.

All-cause mortality (Giamouzis, Mastrogiannis, Koutrakis, Karyannis, Parisis et al., 2012,p.2) Difference between intervention and control groups.

Mortality rate (Kamei, Yamamoto, Kajii, & Nakayama, 2012, p.9) Comparing group receiving care via telehealth with group receiving conventional care.

Mortality (Rubin, Wellik, Channer & Demaerschalk, 2013, p.121) Change in symptoms over a 24 week period in the hospital and within 3 months from hospital discharge.

Mortality (Van den Berg, Schumann, Kraft & Hoffmann, 2012, p.96) Comparison between telemedical group and usual care.

Reliability of clinical data

Data quality (Jaana, Pare & Sicotte, 2009, p. 314) Spirometric measures obtained in a clinical setting and those taken at home.

Adequate display of data (Luxton, & Mishkin, 2010, p.709) Amount of characters that can be displayed, in this case, characters contained in the Patient Health Questionnaire.

Availability/reliability of service (Garg, & Brewer, 2011, p.773) Frequency of reporting glucose levels.

Data quality (Garg, & Brewer, 2011, p.772) Completeness of data being transmitted

Reliability/accuracy of data (Verberk, Kessles & Thien, 2011, p.152) Comparison between telecare group and usual care group.

Communicating issues - Audio/video quality (Hall, Boisvert, & Steele, 2013, p.32) Comparison to traditional settings for assessment.


Criteria Measures

Safety of care

Safety (Garcia-Lizana, & Munoz-Mayorga, 2010, p.5) Occurrence of complications in patients receiving care via telehealth.

Adequate safety plans (Luxton & Mishkin, 2010, p.707)

Presence of exclusion criteria for patients who are at risk of harming themselves, plan to monitor symptoms during treatment protocol (eg. Patient health questionnaire), reduction of adverse events (p.708)

Treatment credibility (Swanepoel & Hall, 2010, p.7) Comparison of treatment provided face to face and through teleaudiology.

Safety (Garg, & Brewer, 2011, p.769) Measuring physical safety, eg. Detection of falls.

Adverse events (Van den Berg, Schumann, Kraft & Hoffmann, 2012, p.96) Number of adverse events occurring in telemedical group and usual care group.

Completeness of care plans

Clinical effects on patient medical condition (Jaana, Pare, & Sicotte, 2009, p.315) Ability to monitor and change care plan.

Clinical process (Kairy, Lehoux, Vincent & Visintin, 2009, p. 442) Completeness of care plan.

Improved transparency (Verhoeven, Tanja-Dijkstra, Nijland, & Eysenbach, 2010, p.678) Completeness of patient record, better coordination of treatment

Trust between patient and practitioner

Therapeutic relationship (Backhaus, Agha, Maglione, Repp, Ross et al., 2012, p.118) Therapeutic alliance in videoconferencing and face to face sessions

Trust between patient and provider (Garg & Brewer, 2011, p.769) Accurate self reporting of blood glucose levels, revealing complete and accurate information.


Table 4: Grouping of Criteria and Measures from literature under Organisation sustainability dimension of proposed framework

Criteria Measures

Utilisation of healthcare services

Other effectiveness indicators (Rojas & Gagnon, 2008, p.899)

Number of telehomecare visits, number of home visits, number of ER visits, number of hospitalizations, number of readmissions, average travel time for professionals (minutes), average distance covered by the professionals (km), average length of hospitalization (days)

Improved outcomes (Seto, 2008, p.679) Reduced hospital readmissions, reduced lengths of hospital stay, reduced bed days of care.

Emergency department visits (Seto, 2008, p.680) Pre and post telemonitoring comparison of costs.

Reduced home visits (Seto, 2008, p.680) Reduction of nurse visits to home compared to usual care.

Physician office visits (Seto, 2008, p.680) Pre and post telemonitoring comparison of costs.

Duration of hospitalization (Kamei, Yamamoto, Kajii & Nakayama, 2012, p.6) Comparing group receiving care via telehealth with group receiving conventional care.

Health services and other (Davalos, French, Burdick, & Simmons, 2009, p.943) Avoided referral (p.943)

Health services and other (Davalos, French, Burdick, & Simmons, 2009, p.943) Reduced length of consultations (p.943)

Health services and other (Davalos, French, Burdick, & Simmons, 2009, p.943) Reduced length of hospital stay in days (p.943)

Health services and other (Davalos, French, Burdick, & Simmons, 2009, p.943) Avoided hospital readmissions (p.943)

Health services and other (Davalos, French, Burdick, & Simmons, 2009, p.943) Avoided hospitalizations (p.943)

Health services and other (Davalos, French, Burdick, & Simmons, 2009, p.943) Avoided emergency room visits (p.943)

Health services and other (Davalos, French, Burdick, & Simmons, 2009, p.943) Avoided laboratory tests (p.943)

Health services and other (Davalos, French, Burdick, & Simmons, 2009, p.943) Avoided physician’s office visits (p.943)

Health services utilisation (Jaana, Pare & Sicotte, 2009, p.315) Office visits, emergency department visits, hospital admissions, length of stay


Criteria Measures

Healthcare utilization (Kairy, Lehoux, Vincent & Visintin, 2009, p.442) Number and days of hospitalizations, visits to the emergency room, doctor visits.

Hospital visits (Maric, Kaan, Ignaszewski, & Lear, 2009, p.507) Reduced hospitalisation, reduced re-hospitalisation, number of hospitalised days.

Hospital visits (Polisena, Tran, Cimon, Hutton, McGill et al. 2009, p.924) Number of patients hospitalised, number of patients visiting emergency department, bed days of care (p.924)

Visits to clinic (Polisena, Tran, Cimon, Hutton, McGill, Palmer, 2009, p.924)

Number of patients visiting primary care clinics (p.916), number of patients visiting specialist clinics (p.924), number of office visits (p.924)

Effectiveness of service (Boisvert, Lang, Andrianopoulos, & Boscardin, 2010, p.430)

Comparison of service provided via telehealth and telephone. It is noted in the paper that this measure was related to whether telepractice is a “viable service delivery model” (p.430), rather than the clinical quality of care provided by telehealth.

Health service efficiency (Ekeland, Bowes & Flottorp, 2010, p.739) Reduced use of hospitals (p.746)

Process indicators (Johansson & Wild, 2010, p.152) Median length of hospital stay, number of hospital transfers

Service utilisation (Pare, Moqadem, Pineau, & St-Hilaire, 2010, p.9)

Fewer calls to hospital (p.9), fewer hospital readmissions (p.11), fewer days in hospital (p.12), number of emergency room visits (p.13)

Clinical outcome (Polisena, Tran, Cimon, Hutton, McGill et al., 2010, p.125) Number of hospitalizations per month

Clinical outcome (Polisena, Tran, Cimon, Hutton, McGill et al., 2010, p.125) Number of emergency department visits

Clinical outcome (Polisena, Tran, Cimon, Hutton, McGill et al., 2010, p.125) Number of office visits

Clinical outcome (Polisena, Tran, Cimon, Hutton, McGill et al., 2010, p.125) Bed days of care

Clinical outcome ((Polisena, Tran, Cimon, Hutton, McGill et al., 2010, p.126) Number of home care visits

Improvement in diagnostic capacity (Ray-Moreno, Reigadas, Villalba, Vinagre & Fernandez, 2010, p.118)

Reduction in hospital admissions.


Criteria Measures

Reduction in urgent calls to diabetic centre (Shulman, O’Gorman & Palmert, 2010, p.3) Comparison between telemedicine group and control group.

Reduction in hospital admissions (Verberk, Kessles, & Thien, 2011, p.153) Comparison between telecare group and usual care group.

Reduction in hospital visits (Verberk, Kessles & Thien, 2011, p.153) Comparison between telecare group and usual care group.

Avoided visits to clinic (Warshaw, Hillman, Greer, Hagel, MacDonald, et al., 2011, p.769) Number of avoided visits to dermatology clinic.

Rate of hospitalization and rehospitalisation (Giamouzis, Mastrogiannis, Koutrakis, Karyannis et al., 2012, p.2)

Difference between intervention and control groups.

Rate of hospitalization (Kamei, Yamamoto, Kajii & Nakayama, 2012, p.4) Comparing group receiving care via telehealth with group receiving conventional care.

Number of emergency department visits (Kamei, Yamamoto, Kajii & Nakayama, 2012, p.8) Comparing group receiving care via telehealth with group receiving conventional care.

Bed days of care (Kamei, Yamamoto, Kajii, F & Nakayama, 2012, p.9) Comparing group receiving care via telehealth with group receiving conventional care.

Clinical effectiveness (Wallace, Hussain, Khan, & Wilson, 2012, p.473) Rate of emergency transfers, number of referrals.

Quicker discharge rate (Bradford, Armfield, Young & Smith, 2013, p.8) Comparison between home telehealth and a control group receiving care through another medium.

Reduction in hospital admissions (Bradford, Armfield, Young & Smith, 2013, p.9) Comparison between home telehealth and a control group receiving care through another medium.

Substitution of home visits (Bradford, Armfield, Young & Smith, 2013, p.9) Retrospective chart review.

Effectiveness (Edirippulige, Martin-Khan, Beattie, Smith& Gray, 2013, p.129) Changes in number of clinic visits.

Resource utilization(Rubin, Wellik, Channer & Demaerschalk, 2013, p.121) Use of neuroimaging, number of transfer, outpatient appointments.

Health service use-cost-productivity (Vedel, Mignerat, Saksena & Lapointe, 2013, p.226) Health service use, healthcare costs, time spent by clinicians.


Criteria Measures

Cost to run telehealth service

Costs- telemonitoring equipment (Seto, 2008, p.680) Comparison between usual care and telemonitoring.

Direct costs (Rojas, & Gagnon, 2008, p.900) Remuneration of professionals: visit time, travel/driving time, Remuneration and cost of technical support , ER cost hospitalization costs, readmission costs, software licenses, purchase of equipment, installation, support and maintenance of equipment, remuneration of project coordinator, costs specific to specialty

Variable costs (Davalos, French, Burdick & Simmons, 2009, p.938)

Maintenance and repairs, telecommunication costs, administrative support and supplies, training, wages to technicians, wages to staff.

Fixed costs (Davalos, French, Burdick, & Simmons,2009, p.938) Capital investment in equipment, depreciation, office space

Economic viability for organisation (Jaana, Pare & Sicotte, 2009, p.318) Cost-benefits analysis

Cost (Kairy, Lehoux, Vincent & Visintin, 2009, p.442)

Duration of sessions, hourly salaries of therapists, travel time, equipment installation costs, staff training, technical support

Cost (Neubeck, Redfern, Fernandez, Briffa, Bauman et al., 2009, p. 287) Cost of telehealth intervention against readmission cost incurred in usual care group.

Implementation expenses (Boisvert, Lang, Andrianopoulos, & Boscardin, 2010, p.430) Technical support personnel, maintenance, material costs (p.430)

Investment costs (Johansson & Wild, 2010, p.152) Capital investment in equipment, education and technical support, personnel cost.

Cost per patient (Johansson & Wild, 2010, p.152)

Median cost per patient in telemedicine videoconferencing group, compared to a telephone group and teleradiology group.

Technology cost (Wade, Karnon, Elshaug & Hiller, 2010,p.8) Capital cost, hourly cost of connectivity, time taken to fix technical issues.

Setup cost (Wade, Karnon, Elshaug & Hiller, 2010, p.10) Facility space cost.

Costs and financial benefits (Peeters, Mistiaen & Francke, 2011, p.405) Technology cost

Cost of equipment (Verberk, Kessles & Thien, 2011, p.153) No measure has been specified.


Criteria Measures

Cost (Warshaw, Hillman, Greer, Hagel, MacDonald et al. 2011, p.769) Number of teledermatology sessions, cost of clinic dermatology

Cost of hospitalization per patient (Giamouzis, Mastrogiannis, Koutrakis, Karyannis, Parisis et al., 2012, p.6)


Technology cost (Mistry, 2012, p.2) Depreciation rate

Technology life (Mistry, 2012, p.2) Equipment lifetime

Direct medical costs (Mistry, 2012, p.5) Do not mention types of cost, but state that direct non-medical costs are those incurred by the health service.

Cost (Oliver, Demiris, Wittenberg-Lyles, Washington, Day, et al., 2012, p.45) Cost of telehospice visits

Resource utilization (Young, 2012, p. 49) number of participants using the telemedicine intervention, number of contacts [consultations] per participant, comparison to face to face interventions

Cost of telemedicine sessions (Edirippulige, Martin-Khan, Beattie, Smith, & Gray, 2013, p.129)

Changes in staff workload.

Real estate (Kumar, Falk, Bonello, Kahn, Perencevich, et al., 2013, p.22) Rental costs, space leasing costs.

Staffing (Kumar, Falk, Bonello, Kahn, Perencevich, et al., 2013, p.22) Central monitoring site staff costs, health professional costs, technician costs.

Technology costs (Kumar, Falk, Bonello, Kahn, Perencevich, et al., 2013, p.22) Costs to purchase, install, and maintain, hardware, software and licenses, equipment and networking, technical support

Hospital variable costs (Kumar, Falk, Bonello, Kahn, Perencevich, et al., 2013, p.22)

Nursing supplies, pharmacy, laboratory, pathology, radiology and bedside diagnostics, interventional services, ancillary services.

Feasibility/ Cost savings

Cost-effectiveness/minimisation (Durrani & Khoja, 2009, p.178) Specific cost measures not discussed.

Cost savings to organisation (Verhoeven, Tanja-Dijkstra, Nijland & Eysenbach, 2010, p.677)

Number of hospitalizations, decreased number of emergency department visits number of bed days of care, decrease in consultation times, number of discharges to home care, decrease of recertification of patients, replacement of conventional visits by videoconferencing, Savings per year per patient, reduction in overall utilization and charges after 1 year, treatment time of caregivers (p.677)


Criteria Measures

Cost savings (Wade, Karnon, Elshaug & Hiller, 2010, p.7) Reduced hospital admissions, reduced transfers

Costs and financial benefits (Peeters, Mistiaen & Francke, 2011, p.405) Reduced cost of visits, Reduced cost of admissions

Costs and financial benefits (Peeters, Mistiaen & Francke, 2011, p.405) Savings in staff salaries

Feasibility (Backhaus, Agha, Maglione, Repp, Ross et al. 2012, p.118) Reduced intervention costs (p.118)

Cost savings (Wallace, Hussain, Khan & Wilson, 2012, p.474) Transport costs, avoided transfers (air and land)

Cost savings (Bradford, Armfield, Young & Smith, 2013, p.9) Comparison between home telehealth and home visits.

Feasibility/cost-effectiveness (Edirippulige, Martin-Khan, Beattie, Smith & Gray, 2013, p.130)

Increase in the number of patients being treated through telemedicine.

Changes in time and productivity of clinicians

Productivity (Wade, Karnon, Elshaug & Hiller, 2010, p.9) Medical staff time, avoided transfers (p.10), specialist referrals (p.10)

Decreased workload for physicians (Verberk, Kessles & Thien, 2011, p.152) Comparison between telecare group and usual care group.

Time taken to train patient (Verberk, Kessles & Thien, 2011, p.153) Time taken by physician to train patient on how to use telecare system.

Time to complete assessment (Rubin, Wellik, Channer, & Demaerschalk, 2013, p.4) Comparison between telemedicine and in-person consultations.


Table 5: Grouping of Criteria and Measures from literature under Technology capability/capacity dimension of proposed framework

Criteria Measures

Security issues

Security of network (Boisvert, Lang, Andrianopoulos & Boscardin, 2010, p.430) Protection of data, data origin authentication, replay protection and access control.

Legal and ethical issues, specifically informed consent (Boisvert, Lang, Andrianopoulos & Boscardin, 2010, p.430) Presence of guidelines on risks and benefits on using telehealth.

Confidentiality and security of documentation and storage (Johansson & Wild, 2010, p.153) Transfer of information to required clinical professional only. (p.154)

Standards and policies to ensure that technology is reliable and safe (Garg & Brewer, 2011, p.772) Not discussed.

Training of personnel, including care givers, in order to reduce changes of security breaches (Garg & Brewer, 2011, p.772) Not discussed.

Connectivity issues

Network transmission (Clarke & Thiyagarajan, 2008, p.173) Time delay measurement, video-clip transfer time, live video transmission rate, live video latency.

Network connection (Boisvert, Lang, Andrianopoulos & Boscardin, 2010, p.430) Speed of network connection

Connectivity (Hall, Boisvert & Steele, 2013, p.32) Speed of connection

Technical difficulties

Technical problems and errors (Jaana, Pare & Sicotte, 2009, p.314) Connection and transmission problems, cable damage

Technical difficulties (Maric, Kaan, Ignaszewski & Lear, 2009, p.508) Missing or interrupted data transmission.

Technical issues (Luxton & Mishkin, 2010, p.709) Computer, network problems

Hardware/software functionality

System functionality (Clarke & Thiyagarajan, 2008, p.173) Software functionality.

information acquisition and information display (Clarke & Thiyagarajan, 2008, p.173)

Resolution, colour contrast, colour discrimination, colour hue, colour saturation, image clarity, compression effect on image quality.


Table 6: Grouping of Criteria and Measures from literature under ‘Other’. These criteria and measures could not be directly related to any of the proposed framework dimensions.

Criteria Measures

Caregiver perception/ satisfaction

Lost productivity for relatives (Rojas & Gagnon, 2008, p.900) Comparison between face to face and telehealth consultations.

Improvements to caregiver's knowledge (Jaana, Pare & Sicotte, 2009, p.315) Comparison between face to face and telehealth consultations.

Caregiver’s quality of life (Oliver, Demiris, Wittenberg-Lyles, Washington, Day, et al., 2012, p.45) Measure not specified.

Caregiver’s perceptions of pain medication (Oliver, Demiris, Wittenberg-Lyles, Washington, Day, et al., 2012, p.45) Measure not specified.

Families perceptions of technology (Oliver, Demiris, Wittenberg-Lyles, Washington, Day, et al., 2012, p.45) Measure not specified.

Application of knowledge learnt from intervention (Rietdjik, Togher, & Power, 2012, p.918) Follow up with caregivers after a certain period of time.

Caregiver’s burden and needs (Rietdjik, Togher, & Power, 2012, p.918) Pre-post differences

Carer’s satisfaction (Rietdjik, Togher, & Power, 2012, p.918) Recommend intervention to others, preference to face-to-face interventions, ease and comfort of using technology, helpfulness or value of the intervention.

Family member’s psychological well being (Rietdjik, Togher, & Power, 2012, p.918) Depression, anxiety, stress and mood, pre-post differences.

Peer support for family members (Rietdjik, Togher, & Power, 2012, p.919) Comparison of access between face to face and telehealth consultations.

Increase of access to healthcare services for families in rural areas (Rietdjik, Togher, & Power, 2012, p.919) Comparison between face to face and telehealth consultations.

Carers quality of life (Bradford, Armfield, Young & Smith, 2013, p.9) Measure not specified.

Families feel a sense of security (Bradford, Armfield, Young & Smith, 2013, p.8) Measure not specified.

Reduction in parental anxiety (Bradford, Armfield, Young & Smith, 2013, p.8) Comparison between face to face and telehealth consultations.

Health professionals satisfaction

Provider satisfaction (Garcia-Lizana & Munoz-Mayorga, I. 2010, p.5) Measure is not specified.

Healthcare provider satisfaction (Johansson & Wild, 2010, p.151) Measure is not specified.


Criteria Measures

Provider satisfaction (Backhaus, Agha, Maglione, Repp, Ross et al., 2012, p.119) Measure is not specified.

Clinician satisfaction (Hilgart, Hayward, Coles & Iredale, 2012, p.773) Measure is not specified.

Provider attitudes towards telehospice technologies (Oliver, Demiris, Wittenberg-Lyles, Washington, Day, et al., 2012, p.45) Measure is not specified.

Nurse-staff satisfaction (Wilcox & Adhikari, 2012, p.9) Measure is not specified.

Clinicians’ perceptions of telehealth (Bradford, Armfield, Young & Smith, 2013, p.10) Measure is not specified.

Health professional confidence in using videoconferencing (Rubin, Wellik, Channer & Demaerschalk, 2013, p.5) Measure is not specified.

Usability of the devices

Ergonomics of the devices used, usability of device by patients who suffer from cognitive and visual impairments. (Van den Berg, Schumann, Kraft & Hoffmann, 2012, p.112) Measure is not specified.

Usability of the technology for patients and providers (Verhoeven, Tanja-Dijkstra, Nijland & Eysenbach, 2010, p.677) Measure is not specified.


Appendix B: Systematic review of systematic reviews of literature on evaluation of telehealth implementations

Appendix B provides a detailed review of the literature. The literature is organised by year and then by author. Each paper’s reference is provided and highlighted in gray, followed by a succinct summary of the paper. The evaluation criteria and measures are categorised under the relevant dimension of the proposed framework. It is important to note that the criteria and measures are not presented in order of importance. They are presented in the order presented in the relevant paper.

The non-technical criteria are linked back to the Health Performance Framework (AIHW, 2009). The technical criteria are linked back to the ACRRM telehealth Advisory Committee Standards Framework. This supports the classification of the criteria and measures under the relevant dimensions of the framework.

Year of Publication – 2008

Clarke, M., & Thiyagarajan, C.A.(2008). A systematic review of technical evaluation in telemedicine systems. Telemedicine and e-health, 14(2), 170-183

This paper is related to the technical evaluation of telemedicine systems. 47 articles were reviewed dated between the years 1966 and 2005.

Technical capability/capacity

Criteria Measures ACRRM TeleHealth Advisory Committee Standards Framework (2012)

information acquisition and information display (p.173)

Resolution, colour contrast, colour discrimination, colour hue, colour saturation, image clarity, compression effect on image quality.

Related to adequate performance of the technology. This is related to the image resolution and image quality. This will have an impact on the effectiveness of care, but these measures are related more to the technology that any other health performance measure.

Network transmission (p.173)

Time delay measurement, video-clip transfer time, live video transmission rate, live video latency.

Related to adequate performance of the technology. This is about the network connectivity

System functionality (p.173)

Software functionality.

Related to adequate performance of the technology, specifically the software functionality.


Rojas, S.V., & Gagnon, M.P. (2008). A systematic review of the key indicators for assessing telehomecare cost-effectiveness. Telemedicine and e-health, 14(9), 896-904

A systematic review to identify common indicators of cost effectiveness of telehomecare. Articles published between 1997 and 2007 were reviewed. 23 articles were selected for this review.

Patient Control

Criteria Measures Link to Health Performance Framework (AIHW, 2009)

Time and cost (p.899)

Average travel time and cost (p.899, p.900) for patients (minutes), lost productivity for patient (p.899)

Accessibility, continuity of care: this is related to being able to access timely and required care.

Clinical quality of care

Effectiveness of care(p.899) Quality of life

Effectiveness: this is related to whether the quality of care is effective and therefore, achieves the desired outcomes.

Organisation Sustainability

Other effectiveness indicators (p.899)

Number of telehomecare visits, number of home visits, number of ER visits, number of hospitalizations, number of readmissions, average travel time for professionals (minutes) average distance covered by the professionals (km), average length of hospitalization (days)

Efficiency and sustainability: this is related to providing healthcare services in the most cost effective and efficient manner.

Direct costs (p.900)

Remuneration of professionals: visit time, travel/driving time, Remuneration and cost of technical support , ER cost hospitalization costs, readmission costs software licenses, purchase of equipment, installation, support and maintenance of equipment, remuneration of project coordinator, costs specific to specialty


Other (p.900)

Lost productivity for relatives.


Seto, E. (2008). Cost comparison between telemonitoring and usual care of heart failure: a systematic review. Telemedicine and e-health , 14 (7), 679-686.

Conducted a systematic review over a 7 month period (reviewed by 2 independent reviewers) to determine whether telemonitoring of patients who have suffered from heart failures results in a decrease of costs for the patient and organisation. 10 articles were reviewed.

Patient Control


Hospitalisation charges (p.680)

Usual care vs. telemonitoring costs.

Accessibility: being able to access the required healthcare service via telemonitoring at the right time at a reasonable cost.

Reduced travel costs (p.681)

Pre and post telemonitoring comparison of costs.

Accessibility, continuity of care: being able to access required healthcare service without having to travel.


Patient outcomes (p.679) Improved quality of life.

Effectiveness: Effectiveness: care/ intervention achieved desired outcome, in this case, improved quality of life.

Reduced mortality (p.679) Not specified. Effectiveness: care/ intervention achieved desired

outcome, in this case, reduced mortality.


Improved outcomes (p.679)

Reduced hospital readmissions, reduced lengths of hospital stay, reduced bed days of care.

Efficiency and sustainability: reduced or avoided hospital readmissions means that the hospital and staff are more efficient in providing care to those who need it the most.

Reduced home visits (p.680)

Reduction of nurse visits to home compared to usual care.

Efficiency and sustainability: more productive workforce.

Costs- telemonitoring equipment (p.680)

Comparison between usual care and telemonitoring.

Efficiency and sustainability: achieving results with most cost effective use of resources.

Physician office visits (p.680)



Emergency department visits (p.680)




Year of Publication - 2009

Davalos, M. E., French, M. T., Burdick, A. E., & Simmons, S. C. (2009). Economic Evaluation of Telemedicine: Review of literature and research guidelines for benefit-cost analysis. Telemedicine and e-health , 15 (10), 933-948.

This paper is not a systematic review but there are guidelines for economic evaluation of telemedicine from the perspective of various stakeholders. Specifically, the authors present research guidelines for conducting benefit-cost analysis of telemedicine programs.

Patient Control


Employment/leisure/ classroom time(p.938)

Missed days or hours, avoided absences, increased leisure time, Increased productivity (p.942)

Continuity of care, accessibility: this is related to being able to access timely and uninterrupted care.

Medical costs (p.938) Out of pocket expenses, avoided travel expenditure (p.939)

Accessibility: related to the timely access to healthcare.

Access to healthcare (p.942) Change in QALYs Accessibility: related to the timely access to

healthcare.

Increased health knowledge/ability to self –care (p.942)

Not specified

Responsiveness: this is related to the patient being treated with respect, dignity and being given some control over their care.

Increased medication adherence (p.942) Change in QALYs

Responsiveness, accessibility: this is related to the patient being given some control over their care and also being able to access health professional advice when the need be.

Clinical Quality of Care

Morbidity (p.942) Change in QALYs

Effectiveness: this is related to the effectiveness of the care plan, whether it is built around the patient’s needs as well as achieves the desired outcome.

Mortality (p.942) Change in QALYs

Effectiveness: this is related to the effectiveness of the care plan, whether it is built around the patient’s needs as well as achieves the desired outcome.


Health services and other (p.943) Avoided referral (p.943)


Health services and other (p.943)

Reduced length of consultations (p.943)


Fixed costs (p.938) Capital investment in equipment, depreciation, office space



Patient Control

Variable costs (p.938)

Maintenance and repairs, telecommunication costs, administrative support and supplies, training, wages to technicians, wages to staff


Health services and other (p.943)

Reduced length of hospital stay in days (p.943)


Health services and other (p.943) Avoided hospital readmissions (p.943)


Health services and other (p.943) Avoided hospitalizations (p.943)


Health services and other (p.943) Avoided emergency room visits (p.943)


Health services and other (p.943) Avoided laboratory tests (p.943)


Health services and other (p.943) Avoided physician’s office visits (p.943)


Durrani, H., & Khoja, S. (2009). A systematic review of the use of telehealth in Asian countries. Journal of telemedicine and telecare , 15 (4), 175-181.

Systematic review of telehealth applications in Asian countries. 109 articles, dated between 1997 to 2007 were reviewed. The article focused on a rangeof modalities, methods etc.

Patient Control


Access to care (p.178)

decreased travel, getting required care or access to a specialist

Accessibility: related to being able to access the required service at the right time.

Improved care (p.178)

Increased patient's knowledge and maintaining good health.

Accessibility, responsiveness: related to being able to access the required service at the right time. Also relates to patient education and better understanding of their own care.


Improved quality of care (p.178) On-time hospitalisation !""#$%&'#(#))*+,#-.%#/+%0+12#%2#,+%2#+$.,#+3,0'&/#/+&)+,#-#'.(%+.(/+

.$2&#'#)+%2#+/#)&,#/+04%$05#6+&(+%2&)+$.)#6+0(+%&5#+20)3&%.-&).%&0(7+


Cost-effectiveness/ minimisation (p.178)

Specific cost measures not discussed.

Efficiency and sustainability: related to the sustainability of the telehealth program.


Jaana, M., Pare, G., & Sicotte, C. (2009). Home telemonitoring for respiratory conditions: A systematic review. The American Journal of Managed Care , 15 (5), 305-323.

Systematic review to evaluate the existing evidence on the effects of home telemonitoring for respiratory conditions (page 314). 24 articles dated between 1966 and 2007 were selected and reviewed.

Patient Control


Patient satisfaction (p.315) Not specified

Responsiveness, continuity of care: if patient makes choices about his/her treatment, and is receiving uninterrupted care, then they should be satisfied.

Patient behavioural and psychological effects (p. 315)

Increased feelings of security and reassurance and control over their medical condition; better knowledge and awareness about their disease; improved communication with health professionals; adherence to medication.

Responsiveness, continuity of care: if patient makes choices about his/her treatment, and is receiving uninterrupted care, then they should feel better about their health and well being.

Use among patients (p.315) Adherence to data transmission.

Responsiveness: the frequency of data transmission is under the patient’s control. It allows them to monitor their health and get advice from the practitioner if and when required.

Economic viability for patients (p.318) Cost-benefits analysis

Accessibility: being able to access required healthcare service at the right time, in a cost effective way.


Clinical effects on patient medical condition (p.315)

Ability to monitor and change care plan. Effectiveness, safety: allows the health professional to monitor the care plan and alerts the clinician of any abnormal events.

Data quality (p. 314)

Spirometric measures obtained in a clinical setting and those taken at home.

Effectiveness: the quality and accuracy of the data will affect the care provided to the patient.


Health services utilisation (p.315)

Office visits, emergency department visits, hospital admissions, length of stay


Economic viability for organisation (p.318)

Cost-benefits analysis Sustainability: related to the cost of the technology and whether telemonitoring is cost effective.

Technical capacity/capability

Technical problems and errors (p.314)

Connection and transmission problems, cable damage

Adequate performance of the equipment. (from ACRRM telehealth advisory committee standards framework, 2012)

Other (p.315)

Improvements to caregiver's knowledge.


Kairy, D., Lehoux, P., Vincent, C., & Visintin, M. (2009). A systematic review of clinical outcomes, clinical process, healthcare utilization and costs associated with telerehabilitation. Disability rehabilitation , 31 (6), 427-447.

Systematic review to identify clinical outcomes, clinical process, healthcare utlisation and costs associated with telerehabilitation. Articles dated from the earliest date of database articles to 2007. 28 articles were selected and analysed.

Patient Control


Clinical outcomes (p.430)

Returning to work

Accessibility, continuity of care: if patient is able to receive care when they need it, in a way that suits them, this can lead to less time being taken off from work and also lead to uninterrupted provision of healthcare services.

Patient satisfaction (p.442)

Interpersonal relationship between patient and clinician

Responsiveness: this relates to whether the patient is treated with dignity, given a choice when it comes to their treatment as the absence of this can affect the interpersonal relationships between the patient and clinician.



Quality of life, Function in activities of daily living, range of motion, pain, gait, exercise capacity, cognitive tasks, speech quality, skin integrity, falls efficacy, fatigue, anxiety and depression, compliance with exercise plan.

Effectiveness: related to whether the care provided is relevant and achieves the desired outcome, in this case, an improvement in exercise capacity etc.

Diagnostic use (p.442)

Quality of video transmission, use of technology to treat certain patients, eg. Shy children.

Effectiveness, safety: related to whether the care provided is relevant and safe as issues with video quality and other technology constraints can affect the outcome of the telehealth session.

Clinical process (p. 442)

Completeness of care plan.

Effectiveness: this relates to the provision of care in a timely manner and also in an uninterrupted way. Better coordination of care leads to more effective care plans.


Cost (p.442) Duration of sessions, hourly salaries of therapists, travel time, equipment installation costs, staff training, technical support

Efficiency and sustainability: related to whether care is being provided in an efficient and cost effective way.

Clinical process (p. 430)

Drop out rates, consultation duration, contact time.

Efficiency and sustainability: related to the sustainability of the telehealth program.

Healthcare utilization (p.442)

Number and days of hospitalizations, visits to the emergency room, doctor visits.

Efficiency and sustainability: related to whether care is being provided in an efficient and cost effective way.


Maric, B., Kaan, A., Ignaszewski, A., & Lear, S.A. (2009). A systematic review of telemonitoring technologies in heart failure. European journal of heart failure. 11(5), 506-517.

Systematic review of articles related to the use of telemonitoring for patients suffering from heart failure. 56 articles dated before 2007 were selected and reviewed.



Quality of life (p.507)

A measure of this is not specified.

Effectiveness: this relates to the provision of care in a timely manner and also in an uninterrupted way which can lead to a better quality of life.


Hospital visits (p.507)

Reduced hospitalisation, reduced re-hospitalisation, number of hospitalised days.



Technical difficulties (p.508)

Missing or interrupted data transmission.

Related to adequate performance of the technology. This is about the network connectivity or software issues. (from ACRRM telehealth advisory committee standards framework, 2012)

Neubeck, L., Redfern, J., Fernandez, R., Briffa, T., Bauman, A., & Freedman, S. B. (2009). Telehealth interventions for the secondary prevention of coronary heart disease: a systematic review. European Journal of cardiovascular prevention and rehabilitation, 16 (3), 281-289.

Systematic review of the effectiveness of telehealth interventions for coronary heart disease. 11 papers were selected and reviewed. Literature reviewed was dated between 1990 and 2008, but focus was on RCTs.



All-cause mortality (p.283)

mortality of any patient taking part in the telehealth program.

Effectiveness, safety: this relates to whether the care provided is relevant to the patient, and also the safety of the care provided.

psychosocial state(p.285)

depression, stress measured using cardiac depression scale, center for epidemiologic studies depression scale, perceived stress scale

Effectiveness: this relates to whether the care provided is relevant to the patient, and is achieving desired outcomes.

Changes in modifiable risk factors (p. 284)

total cholesterol, blood pressure, Body Mass Index, Smoking status, amount of physical activity, nutritional status..

Effectiveness: this relates to the relevance of the care and whether the desired outcome is achieved, in this case a lowering of the measures stated in the previous column.


Cost (p. 287) Cost of telehealth intervention against readmission cost incurred in usual care group.

Efficiency and sustainability: this is related more to the sustainability of the telehealth program and providing healthcare in a cost effective way.


Polisena, J., Tran, K., Cimon, K., Hutton, B., McGill, S., Palmer, K.(2009). Home telehealth for diabetes management: a systematic review and meta-analysis, Diabetes, Obesity and Metabolism. 11(10), 913-930

Systematic review that compares telemonitoring for diabetes management with usual care. 26 articles, dated between 1998 and 2008 were selected and reviewed. NB: both telephone support as well as telemonitoring are included in this review. The summary below only includes home telemonitoring evaluation results.

Patient Control



Comparison to conventional care

Accessibility, responsiveness, continuity of care: this is related to whether the patient is able to access timely, uninterrupted care, that caters to their needs.


Quality of life (p.924) Diabetes quality of life score

Effectiveness: this is related to a care plan being catered to the patients needs which can achieve the desired outcomes, in this care an improvement in quality of life.

Glycaemic control (p.924) Measurement and changes in hba1c.

Effectiveness: if the care plan is catered to the patients needs, the desirable outcomes can be achieved, in this case, lower glycaemia levels.


Hospital visits (p.924)

Number of patients hospitalised, number of patients visiting emergency department, bed days of care (p.924)


Visits to clinic (p.924)

Number of patients visiting primary care clinics (p.916), number of patients visiting specialist clinics (p.924), number of office visits (p.924)




Boisvert, M., Lang, R., Andrianopoulos, M., & Boscardin, M.L. (2010). Telepractice in the assessment and treatment of individuals with autism spectrum disorders: a systematic review. Developmental neurorehabilitation. 13(6). 423-432

Systematic review of the use of telehealth for the treatment of individuals suffering from autism spectrum disorders. 8 studies were identified. There was no limit on the date of publication.


Criteria Measures Link to AIHW framework (2009)

Effectiveness of assessment (p.431) Positive behavioural changes

Effectiveness: if the treatment provided via telehealth is that required by the patients and meets their needs, there should be a positive change in behaviour.


Effectiveness of service (p.430)

Comparison of service provided via telehealth and telephone. It is noted in the paper that this measure was related to whether telepractice is a “viable service delivery model” (p.430), rather than the clinical quality of care provided by telehealth.

Efficiency and sustainability: this is related to whether this is a viable service delivery model and whether it is sustainable in the long run.

Implementation expenses (p.430)

Technical support personnel, maintenance, material costs (p.430)

Sustainability: this is related to technology and other expenses related to telehealth and whether the service is sustainable.


Network connection (p.430) Speed of network connection

Adequate performance: This is related to the adequacy of the technology. (from ACRRM telehealth advisory committee standards framework, 2012)

Security of network (p.430)

Protection of data, data origin authentication, replay protection and access control.

Risk management: this is related to establishing protocols that prevent any security breaches. (from ACRRM telehealth advisory committee standards framework, 2012)

Legal and ethical issues, specifically informed consent. (p.430)

Presence of guidelines on risks and benefits on using telehealth.

Risk management: this is related to establishing protocols that ensure that legal and ethical procedures are adhered to by the organisation during a telehealth consultation. (from ACRRM telehealth advisory committee standards framework, 2012)


Ekeland, A. G., Bowes, A., & Flottorp, S. (2010). Effectiveness of telemedicine: A systematic review of reviews. International Journal of Medical Informatics , 79 (11), 736-771.

Systematic review of reviews on the impact and costs and effectiveness of telemedicine services. 80 articles published from 2005 onwards were analysed.

Patient Control


patient satisfaction (p.741)

More confidence and empowerment, better clinician relationships

Responsiveness: patients feel more empowered and in control of their care.

cost effectiveness (p.741) Travel time Accessibility: easier and timely access to the

required health service.

Improved patient compliance (p.741) Not specified

Accessibility, continuity of care: If the patient is able to access health services when required, and coordinated care, this can encourage treatment adherence.


Quality of life (p.741) Not specified

Effectiveness: related to whether the care being provided is effective and catered to the patients needs.

Therapeutic effectiveness (p.739)

Examples of telehealth applications that are therapeutically effective are provided but evaluation measures are not specified.

Effectiveness, safety: this is related to whether the care being provided is effective, and safe for the patients.


Health service efficiency (p.739) Reduced use of hospitals (p.746) Efficiency and sustainability: this is related to a

cost effective approach of providing care.

Garcia-Lizana, F., & Munoz-Mayorga, I.(2010). Telemedicine for depression: a systematic review. Perspectives in psychiatric care. 46(2). 119-126

Systematic review of 10 papers published between 1997 and2008, were reviewed to understand the effectiveness of ICT use for the treatment of depression. The studies employed RCTs.

Patient Control



Comparison to patients receiving traditional treatment.


Treatment adherence (p.123)


Accessibility, continuity of care: If the patient is able to access health services when required, and coordinated care, this can encourage treatment adherence.




Effectiveness: if the treatment provided is catered to the patient’s need, it should bring about better quality of life.

Changes in symptoms (p.123)

Changes in depression and anxiety symptoms (p.124)

Effectiveness: if the treatment provided is catered to the patient’s need, it should bring about positive changes to these measures.


Garcia-Lizana, F., & Munoz-Mayorga, I. (2010). What about telepsychiatry? A systematic review. Primary care companion to the journal of clinical psychiatry. 12(2). 10 pages

Systematic review to evaluate the effectiveness of video conferencing for treatment of mental illness. 10 articles, focused on RCTs, published between 1997 and 2008 were analysed.

Patient Control








Effectiveness: if the treatment provided is catered to the patient’s need, it should bring about positive changes to the quality of life.

Safety (p.5) Occurrence of complications in patients receiving care via telehealth.

Safety: this is related to whether the treatment provided is safe as well as effective.

Evaluation of symptoms (p.5) Changes in symptoms

Effectiveness: if the treatment provided is catered to the patient’s need, it should bring about positive changes to these measures.

Other (p.5)

Provider satisfaction. It is unclear as to whether the authors are referring to the satisfaction level with regard to the technology. This criteria was studied in 2 RCTs that were reviewed.

Johansson, T., & Wild, C.(2010). Telemedicine in acute stroke management: systematic review. International Journal of technology assessment in health care. 26(2). 149-155.

Systematic review of the feasibility, acceptability and delivery of treatment through telemedicine for acute stroke management. 18 studies dated between 1995 and 2008 were selected and reviewed.

Patient Control


Patient satisfaction (p.151) No measure has been specified.


Access (p.152) Ability to access services by those based in remote areas.



Improved quality of care (p.151) No measures have been specified. Effectiveness: this is related to the provision of the

required treatment and care to the patient.

Better health outcomes (p.151)

Reduced dependency, reduced mortality

Effectiveness, safety: this is related to the provision of the required treatment and care to the patient in a safe way.


Patient Control


Time between onset of stroke symptoms to admission (p.152)

Onset-to-hospital, door-to-needle, onset-to-needle

Effectiveness: this is related to the provision of the required treatment and care to the patient, in a timely manner.

Diagnostic accuracy (p.152) Measures are not specified. Effectiveness: this is related to the provision of the


Correct treatment decisions (p.152) Measures are not specified. Effectiveness: this is related to the provision of the



Process indicators (p.152)

Median length of hospital stay, number of hospital transfers

Efficiency: this is related to providing healthcare services in a more efficient manner.

Investment costs (p.152)

Capital investment in equipment, education and technical support, personnel cost.

Efficiency and sustainability: related to the cost of care.

Cost per patient (p.152)

Median cost per patient in telemedicine videoconferencing group, compared to a telephone group and teleradiology group.



Confidentiality and security of documentation and storage (p.153)

Transfer of information to required clinical professional only. (p.154)

Risk management: this is related to whether there are adequate risk management plans in place to deal with security and confidentiality issues. (from ACRRM telehealth advisory committee standards framework, 2012)

Other (p.151)

Healthcare provider satisfaction

Luxton, D.D, & Mishkin, M.C.(2010). Safety of telemental healthcare delivered to clinically unsupervised settings: A systematic review. Telemedicine and e-health. 16(6). 705-711

Systematic review of safety issues related to providing telemental care in unsupervised settings, such as the patient’s home. 9 studies dated between 1982 and 2009 were selected and reviewed.



Adequate safety plans (p.707)

Presence of exclusion criteria for patients who are at risk of harming themselves, plan to monitor symptoms during treatment protocol (eg. Patient health questionnaire), reduction of adverse events (p.708)

Safety: related to either providing or in some cases, not providing care as this may result in unsafe actions by the patient.

Adequate display of data (p.709)

Amount of characters that can be displayed, in this case, characters contained in the Patient Health Questionnaire.

Effectiveness, safety: related to providing the required care and having access to data that can determine the symptoms of the patient.


Technical issues (p.709) Computer, network problems

Adequate performance: related to the performance of the system and network.


Pare, G., Moqadem, K., Pineau, G., & St-Hilaire, C.(2010). Clinical Effects of Home Telemonitoring in the Context of Diabetes, Asthma, Heart Failure and Hypertension: A Systematic Review. Journal of Medical Internet Research. 12(2): e21

Systematic review of the clinical effects of home telemonitoring for chronic diseases. 62 articles published between 1966 and 2008 were analysed.



Changes in glycemic control (p.7)

Comparison between telemonitoring group and group receiving usual care

Effectiveness: if the care plan is catered to the patients needs, the desirable outcomes can be achieved, in this case, lower glyacaemic levels.

Changes in blood pressure levels, and total and LDL cholesterol levels (p.8)

Comparison between telemonitoring group and group receiving usual care.

Effectiveness: if the care plan is catered to the patients needs, the desirable outcomes can be achieved.

Changes in asthma symptoms (p.8)

Comparison of telemonitoring group to one being treated by GP and one being treated by specialists. Changes in nighttime symptoms, daytime symptoms, improved peak expiratory flow in the morning and night.


Changes in blood pressure levels for BP patients (p.8)

Comparison between telemonitoring group and group receiving alternative care.


Mortality rate (p.8) Comparison between telemonitoring group and usual care group

Effectiveness, safety: if the care plan is catered to the patients needs, the desirable outcomes can be achieved.

Quality of life (p.12) Quality of life measures not specified. It is only stated that questionnaires were use to collect this information

Effectiveness, safety: if the care plan is catered to the patients needs, the desirable outcomes can be achieved.


Service utilisation (p.9) Fewer calls to hospital (p.9), fewer hospital readmissions (p.11), fewer days in hospital (p.12), number of emergency room visits (p.13)



Polisena, J., Tran, K., Cimon, K., Hutton, B., McGill, S., Palmer, K., et al. (2010). Home telehealth for chronic obstructive pulonary disease: a systematic review and meta-analysis. Journal of telemedicine and telecare , 16 (3), 120-127.

Conducted a systematic review and meta analysis of clinical outcomes, patient quality of life and the use of health care services when using telehealth in comparison to "usual care" (p.120). Results on care telephone care excluded. Only results on remote monitoring care are provided. 9 articles dated from 1998 onwards were selected and reviewed.

Patient Control










Clinical outcome (p.125)

Number of hospitalizations per month Efficiency and sustainability: achieving results with most cost effective use of resources.


Number of emergency department visits Efficiency and sustainability: achieving results with most cost effective use of resources.


Number of office visits Efficiency and sustainability: achieving results with most cost effective use of resources.


Bed days of care Efficiency and sustainability: achieving results with most cost effective use of resources.


Number of home care visits Efficiency and sustainability: achieving results with most cost effective use of resources.


Ray-Moreno, C., Reigadas, J. S., Villalba, E. E., Vinagre, J. J., & Fernandez, A. M. (2010). A systematic review of telemedicine projects in Colombia. Journal of telemedicine and telecare. 16(3). 114-119.

Systematic review of telehealth projects in Columbia. 32 articles dated between 2001 and 2003 were selected and analysed.

Patient Control


Accessibility (p.118) Accessibility to services for people in deprived areas.



Diagnostic accuracy (p.118) Tele-ECG compared to normal ECG.

Effectiveness: being able to provide relevant and correct care/report to patients.


Improvement in diagnostic capacity (p.118)

Reduction in hospital admissions.

Efficiency and sustainability: related to the provision of cost effective healthcare.

Shulman, R.M., O’Gorman, C.S., & Palmert, M.R. (2010). The impact of telemedicine interventions involving routine transmission of blood glucose data with clinician feedback on metabolic control in youth with type 1 diabetes: A systematic review and meta-analysis. International journal of pediatric endocrinology. Article ID: 536957, 9 pages.

Systematic review to determine impact of telemedicine on type 1 diabetes management (T1DM) in youth. 10 studies were selected and reviewed. This was the result of searching databases from their inception to 2009. Only RCTs and CCTs were chosen for this review.

Patient Control



Comparison between telemedicine group and control group.


Indirect costs (p.4) Missed days of work or school

Accessibility, responsiveness, continuity of care: this is related to whether the patient is able to access timely, uninterrupted care, that caters to their needs and therefore allows them to be more productive.

Reduction of urgent visits to the school nurse (p.3)


Accessibility, responsiveness, continuity of care: this is related to whether the patient is able to access timely, uninterrupted care, that caters to their needs and therefore allows them to be more productive.


Effect on HbA1C (p.2) Comparison between telemedicine group and control group.

Effectiveness: related to whether the patient is receiving the required care and therefore, achieving the relevant outcomes.

Effect on hypoglycaemia (p.3)




Patient Control

Diabetes-related quality of life (p.4)




Reduction in urgent calls to diabetic centre (p.3)



Swanepoel de, W., & Hall, J. 3rd. (2010). A systematic review of telehealth applications in audiology. Telemedicine journal and e-health , 16 (2), 181-200.

Systematic review of telehealth applications in audiology from the earliest date of database till 2009. The paper focuses on the various features that are supported by telehealth, eg. Diagnosis, rather than the evaluation or outcomes of using telehealth.

Patient Control


Awareness of telehealth (p.6) Not specified Responsiveness: related to whether the patient feels that by using

telehealth, they are in more control over their care.

Willingness to use (p.6) Not specified

Accessibility, responsiveness, continuity of care: related to whether the use of telehealth has improved access to the required healthcare service, provides patient some control over their care and whether they are able to received uninterrupted care.


Treatment credibility (p.7)

Comparison of treatment provided face to face and through teleaudiology.

Effectiveness, safety: this is related to whether the treatment provided is relevant and safe.


Verhoeven, F., Tanja-Dijkstra, K., Nijland, N., & Eysenbach, G. (2010). Asynchronous and Synchronous Teleconsultation for Diabetes Care: A Systematic Literature Review. Journal of diabetes science and technology , 4 (3), 666-684.

A systematic literature review, covering 90 publications between 1994 and 2009 was done to determine the effects of teleconsultations on the clinical, behavioural and care coordination outcomes of diabetes care.

Patient Control


Improved self care (p.676)

Frequency of monitoring blood sugar levels or blood pressure, adherence to medication, improvement in knowledge, better understanding of medical condition.

Responsiveness: this is related to the patient being treated with respect, dignity and being given some control over their care.

Cost savings to patient (p.677)

Travel costs Accessibility: related to the timely access to healthcare.

Improved interaction (p.675)

Frequency of metabolic data transmission, increased intensity of contact between provider and patient (p.676)

Responsiveness, continuity of care: if patient makes choices about his/her treatment, and is receiving uninterrupted care, then this should encourage to participate more in their care plan.

Improved equity (p.678)

Better access to specialised health care, timely follow up (p.678)



Changes in clinical values (p.672)

Haemoglobin levels (p.672), lipid profiles (p.673), stable ulcer healing (p.673), cardiovascular risk factor control (p.673), glucose levels (p.673), glycemic control (p.675)

Effectiveness: changes to these measures will highlight that the treatment plan is effective and caters to the patients needs.

Quality of Life (p.675) Diabetes Quality of Life measure , rate of depression

Effectiveness: changes to these measures will highlight that the treatment plan is effective and caters to the patients needs.

Improved transparency (p.678)

Completeness of patient record, better coordination of treatment

Effectiveness: this relates to the provision of care in a timely manner and also in an uninterrupted way. Better coordination of care leads to more effective care plans.


Cost savings to organisation (p.677)

Number of hospitalizations, decreased number of emergency department visits number of bed days of care, decrease in consultation times, number of discharges to home care, decrease of recertification of patients, replacement of conventional visits by videoconferencing, Savings per year per patient, reduction in overall utilization and charges after 1 year, treatment time of caregivers (p.677)


Other (p.677)

Usability of the technology for patients and providers. This is related to training, complexity of the technology.


Wade, V. A., Karnon, J., Elshaug, A. G., & Hiller, J. E. (2010). A systematic review of economic analyses of telehealth services using real time video communication. BMC Health services research , 10 (233), 13 pages.

Systematic review to assess the economic value of real time video communication. 36 articles, dated from the commencement of the databases till 2009, were selected and analysed.

Patient Control


Cost savings (p.6)

Travel reduction (p.6), reduced time off work (p.9)

Accessibility: this is related to the increased access to the required healthcare service.

Increased accessibility (p.10)

Access to different types of specialist service in rural/remote areas

Accessibility: this is related to the availability of care to remote/rural patients who under normal circumstances would not have the opportunity for such care.

Clinical Quality of care

Diagnostic accuracy (p.7)

Authors mention a reduction in diagnostic accuracy as a result of using telehealth for dermatology, but do not discuss how this was measured.

Effectiveness: if diagnostic accuracy is poor, then the possibility of providing the required care plan is reduced.

Patient mortality (p.7)

Change in mortality rate after using telehealth.

Effectiveness, safety: this is related to the safety of care as well as the effectiveness of the care plan.


Cost savings (p.7)

Reduced hospital admissions, reduced transfers

Efficiency and sustainability: this is related to savings in the cost of care as a result of using telehealth.

Technology cost (p.8)

Capital cost, hourly cost of connectivity, time taken to fix technical issues.

Efficiency and sustainability: this is related to the total cost of telehealth and its sustainability.

Productivity (p.9)

Medical staff time, avoided transfers (p.10), specialist referrals (p.10)

Efficiency and sustainability: this is a measure of productivity and sustainability of telehealth.

Setup cost (p.10) Facility space cost

Efficiency and sustainability: this is related to the total cost of telehealth and its sustainability. If the service is being provided in a rural/remote area, there may be a need for telecenters which will cost more.



Garg, V., & Brewer, J. (2011). Telemedicine security: a systematic review. Journal of diabetes science and technology. 5(1). 768-777

A systematic review of telemedicine security. The authors focus on papers related to the application of telemedicine to treat chronic diseases. 58 articles, published between 1994 and 2009 were selected and reviewed.



Trust between patient and provider (p.769)

Accurate self reporting of blood glucose levels, revealing complete and accurate information.

Effectiveness: this is related to whether the data being reported is accurate and complete. This then affects the type of care provided.

Safety (p.769) Measuring physical safety, eg. Detection of falls.

Safety: related to the provision of care in a safe environment.

Data quality (p.772) Completeness of data being transmitted

Effectiveness: this is related to whether the data being sent is accurate and complete. This then affects the type of care provided.

Availability/reliability of service (p.773)

Frequency of reporting glucose levels.

Effectiveness: this is related to the frequency of reporting data. If the service is reliable and the transmission of data is frequent, the type of care provided will be of high quality.


Standards and policies to ensure that technology is reliable and safe (p.772).

Not discussed.

Risk management: This is part of risk management as it is relate to procedures related to safety of the technology. (from ACRRM telehealth advisory committee standards framework, 2012)

Training of personnel, including care givers, in order to reduce chances of security breaches (p.772)

Not discussed.

Risk management: This is part of risk management as it is relate to procedures related to safety of the technology. (from ACRRM telehealth advisory committee standards framework, 2012)


Hailey, D., Roine, R., Ohinmaa, A., & Dennett, L. (2011). Evidence of benefit from telerehabilitation in routine care: a systematic review. Journal of telemedicine and telecare , 17 (6), 281-287.

Systematic review of evidence related to the effectiveness of telerehabilitation, in particular, a review of studies that provide an indication of the use or potential of telerehabilitation in routine practice. 61 studies, dated from the inception of the database till 2009, were selected and reviewed. Paper focused more on identifying weakness in papers reviewed than the effectiveness of care.



Effectiveness of intervention (p.283)

Comparison between home and hospital based rehabilitation, specifically, how frequently patient was contacted and provided with additional services.

Effectiveness: this is related to the relevance of the care provided to the patient and whether it achieves the desired outcomes.

Clinical significance (p.285) Measures are not specified.

Effectiveness: related to the provision of relevant care, and achieving the desired health outcomes.

Peeters, J. M., Mistiaen, P., & Francke, A. L. (2011). Costs and financial benefits of video communication compared to usual care at home: a systematic review. Journal of telemedicine and telecare , 17 (8), 403-411.

A systematic review of 36 articles on video communications for home care. Studies had to include costs of video communication as well as financial benefits (avoided costs). Studies published from the inception of database till 2009.

Patient Control


Knowledge of the illness (p.409)

Measures are not specified.

Responsiveness: the patient is provided knowledge about their illness and can thus take some control over their care.




Medication compliance (p.409)







Health status (p.409)




Costs and financial benefits (p.405) Technology cost Efficiency and sustainability: this is related to providing healthcare

services in the most cost effective and efficient manner.

Costs and financial benefits (p.405)

Reduced cost of visits, Reduced cost of admissions


Costs and financial benefits (p.405) Savings in staff salaries Efficiency and sustainability: this is related to providing healthcare

services in the most cost effective and efficient manner.


Verberk, W. J., Kessles, A. G., & Thien, T. (2011). Telecare is a valuable tool for hypertension management, a systematic review and meta-analysis. Blood Pressure Monitoring , 16 (3), 149-155.

A systematic review of the use of blood pressure measurements in telecare. 9 articles, specifically on Randomised Control Trials (RCTs) were chosen after a search was conducted on Medline/PubMed, Embase and Cochrane library. The duration of this search is not specified in the article.

Patient Control


Adherence to treatment (p.150)

Comparison between telecare group and usual care group.

Responsiveness, continuity of care: if patient makes choices about his/her treatment, and is receiving uninterrupted care, then health status should improve.+

Self measurement of BP(p.150)


Responsiveness, continuity of care: if patient makes choices about his/her treatment, and is receiving uninterrupted care, then health status should improve.+

Increased access to healthcare (p.153)


Accessibility: this is related to the increased access to the required healthcare service.

Clinical quality of care!

Changes in BP (p.150) Comparison between telecare group and usual care group.

Effectiveness: this is related to the relevance of the care provided to the patient and whether it achieves the desired outcomes.+

Reliability/accuracy of data (p.152)


Effectiveness: this is related to the relevance of the care provided to the patient and whether it achieves the desired outcomes.+


Decreased workload for physicians (p.152)



Reduction in hospital visits (p.153)



Reduction in hospital admissions (p.153)



Cost of equipment (p.153)

No measure has been specified.


Time taken to train patient (p.153)

Time taken by physician to train patient on how to use telecare system.



Warshaw, E. M., Hillman, Y. J., Greer, N. L., Hagel, E. M., MacDonald, R., Rutks, I. R. & Wilt, T.J. (2011). Teledermatology for diagnosis and management of skin conditions: a systematic review. Journal of the American Academy of Dermatology , 64 (4), 759-772.

A systematic review of teledermatology that was focused on diagnostic accuracy, management accuracy, clinical outcomes and costs. 78 studies, dated between 1990 and 2009 were selected and reviewed.

Patient Control!



Either a comparison between the teledermatology group and clinic dermatology group or just feedback from teledermatology group.

Responsiveness, accessibility: this relates to the convenient access to healthcare as well as providing a client oriented service.

Patient preference (p.768)

Comparison between teledermatology and clinic dermatology.

Accessibility, responsiveness: this is related to whether the patient can access timely and required care as well as whether they are treated with respect, dignity and have some control over their care.

Travel cost (p.769) Distance travelled by the patients. Accessibility: this is related to the increased

access to the required healthcare service.



Matching teledermatology diagnosis to histopathology diagnosis or other laboratory tests. Comparing in person dermatology diagnosis with teledermatology.

Effectiveness: this is related to the provision of the required care to the patients and therefore achieving the desired outcomes.

Diagnostic concordance (p.765)



Management accuracy (p.767)



Management concordance (p.767)

Comparison between teledermatology and clinic dermatology management decisions, eg. ‘Refer or not refer’, ‘biopsy or no biopsy’





Time to treatment (p.768)

Comparison between teledermatology and clinic dermatology. Time taken from GP consult to dermatology clinic.



Avoided visits to clinic (p.769)

Number of avoided visits to dermatology clinic.

Efficiency: this is related to being able to run a more efficient practice.

Cost (p.769) Number of teledermatology sessions, cost of clinic dermatology

Sustainability: related to the sustainability of the telehealth program.


Year of Publication - 2012

Backhaus, A., Agha, Z., Maglione, M. L., Repp, A., Ross, B., Zuest, D., et al. (2012). Videoconferencing psychotherapy: a systematic review. Psychological Services , 9 (2), 111-131.

A systematic review of the use of videoconferencing for psychotherapy.65 articles were selected and analysed. Articles were dated between 1996 and 2010.

Patient Control


Decreased costs (p.118) Travel costs Accessibility: related to the timely access to healthcare.

Increased access to healthcare (p.118)

Increased access to care for rural, underserved populations Accessibility: related to the timely access to healthcare.


Comparing groups using videoconferencing with in-person psychotherapy



Therapeutic relationship (p.118)

Therapeutic alliance in videoconferencing and face to face sessions

Effectiveness: this is related to the provision of the care required by the patient.

Changes in clinical outcomes (p.119)

Depression and or/anxiety, eating disorders, physical problems, addictions

Effectiveness: if the required care is provided and the care plan meets the patient’s needs, there should be positive changes in these measures.


Feasibility (p.118) Reduced intervention costs (p.118)

Efficiency and sustainability: if there are lesser interventions, this means that the workforce is more productive and healthcare system more efficient.

Other (p.119)

Provider satisfaction. This is related to provider satisfaction with using telemedicine for psychotherapy sessions.


Cassimatis, M. & Kavanagh, D.J. (2012). Effects of type 2 diabetes behavioural telehealth interventions on glycaemic control and adherence: a systematic review. Journal of telemedicine and telecare. 18(8). 447-450

Systematic review of behavioural telehealth interventions on glycaemic control and diabetes self-management in patients suffering from type 2 diabetes. 14 studies were selected and reviewed. There was no limit on the publication dates.

Patient Control


Dietary adherence (p.449)

Measurement straight after active intervention period.

Responsiveness: related to allowing the patient to have control over their care.

Blood glucose self-monitoring (p.449)

Frequency of self reported glucose levels.


Medication adherence (p.449)

ASK-20 items, Morisky Adherence Scale.



Glycaemic control (p.448)

Measurement straight after active intervention period.


Physical activity (p.449) Effect on glycaemic control


Ciere, Y., Cartwright, M., & Newman, S.P. (2012). A systematic review of the mediating role of knowledge, self-efficacy and self-care behaviour in telehealth patients with heart failure. Journal of telemedicine and telecare. 18(7). 384-391.

Systematic review of benefits of telehealth, related to increased in knowledge, self-efficacy and self-care. 12 papers, dated between 2003 and 2010 were selected and reviewed.

Patient Control


Improvement in knowledge (p.389)

Proportion of patients stating the purpose and side effect of medication between telehealth group and control group.


Improvement in self-care (p.389)

Compliance with medication and monitoring of weight, diet and BP in comparison to control group.


Improvement in self-efficacy (p.389)

Levels of confidence in performing self care between telehealth group and control group.



De Waure, C., Cadeddu, C., Gualano, M.R. & Ricciardi, W.(2012). Telemedicine for the reduction of myocardial infarction mortality: A systematic review and a meta-analysis of published studies. Telemedicine and e-health, 18(5), 323-328.

A systematic review and meta-analysis on telemedicine systems that can improve the health outcomes for patients suffering from myocardial infarction. 5 articles, dated from the inception of different databases till 2010, were selected and reviewed.

Patient Control


Access to care (p.326) Comparison between group using telemedicine and control group.

Accessibility: this is related to being able to access timely and required healthcare services.


In-hospital mortality (p.325) Comparison between group using telemedicine and control group.

Effectiveness, safety: treatment is relevant and safe for the patient and achieves the desired outcomes.

12-month survival rate (p.325) Comparison between group using telemedicine and control group.


Rapid medical decision making and timely intervention (p.325)



Decreased risk of rehospitalization, revascularization, subsequent myocardial infarction, and/or death (p.325)




Giamouzis, G., Mastrogiannis, D., Koutrakis, K., Karyannis, G., Parisis, C., Rountas, C., et al. (2012). Telemonitoring in chronic heart failure: a systematic review. Cardiology research and practice , 1 (1), 7 pages.

Systematic review of the use of telemonitoring for chronic heart failure. 12 articles were selected and reviewed. The databases were searched from the time of their inception to 2011.

Patient Control


Improved compliance (p.5) Recording of weight, blood pressure


Improved medical adherence (p.5)

World Health Organisation’ multidimensional adherence model.



All-cause mortality (p.2) Difference between intervention and control groups.



Rate of hospitalization and rehospitalisation (p.2)


Efficiency and sustainability: this is related to creating a more efficient way of providing care.

Cost of hospitalization per patient (p.6)



Hilgart, J. S., Hayward, J. A., Coles, B., & Iredale, R. (2012). Telegenetics: a systematics review of telemedicine in genetics services. Genetics in Medicine , 14 (9), 765-776.

Systematic review of 14 telegenetics consultations, carried out through video conferencing, published between 2000 and 2011 were reviewed to determine the value of video conferencing for genetic consultations. Various methods used in the studies reviewed.

Patient Control


Patient satisfaction (p. 773)

Not specified clearly – each aspect used to describe this can stand alone as a separate criterion.


Less travel and associated costs (p. 773)

Comparison of time and cost of travel between control group and telegenetics group.

Accessibility, continuity of care: being able to access required healthcare service without having to travel.

Reduced waiting times (p. 773)

Comparison of waiting time to get an appointment with genetics specialist, between control group and telegenetics group.

Accessibility, continuity of care: being able to access required healthcare at the right time.

Affective outcomes (p.774)

Pre-post comparison of patient’s knowledge after telegenetics consultation (p.775)

Responsiveness: educating the patient and thus enabling them to make choices about their treatment and be educated about their care plan.


Patient Control



No new diagnosis/ no difference when compared to face-to-face appointment.

Effectiveness: treatment is relevant to the patient and achieves the desired outcomes.

Other (p.773)

Clinician satisfaction. The amount of interaction with other clinicians such as nurses, and the ability to observe non verbal behaviour.

Kamei, T., Yamamoto, Y., Kajii, F., & Nakayama, Y. (2012). Systematic review and meta-analysis of studies involving telehome monitoring telenursing for patients with chronic obstructive pulmonary disease. Japan journal of nursing science , DOI:10.1111/j.1742-7924.2012.00228.x, 1-13.

Systematic review of the effects of telemonitoring-based telenursing for patients suffering from chronic obstructive pulmonary disease. 9 articles, dated between 1927-2011 were selected and reviewed.



Reduced hospitalization risk (p.4)

Comparing group receiving care via telehealth with group receiving conventional care.


Disease exacerbation (p.8)

Comparing group receiving care via telehealth with group receiving conventional care, after a period of 3 months of using telehealth care.


Mortality rate (p.9) Comparing group receiving care via telehealth with group receiving conventional care.


Health related quality of life (p.9)

St George Respiratory Questionnaire (SGRQ). A score of more that 4 on this was claimed to be clinically significant.



Rate of hospitalization (p.4)



Duration of hospitalization (p.6)



Number of emergency department visits (p.8)



Bed days of care (p.9) Comparing group receiving care via telehealth with group receiving conventional care.



Mistry, H. (2012). Systematic review of studies of the cost-effectiveness of telemedicine and telecare. Changes in the economic evidence over twenty years. Journal of telemedicine and telecare , 18 (1), 1-6.

Systematic review to identify 80 studies that reported on the cost effectiveness of telemedicine and telecare interventions, from the inception day of database to 2010. The authors suggests that the outcome measures and benefits such as mortality, patient satisfaction are all indicators of cost-effectiveness of telemedicine. The focus is on papers where full economic evaluations are reporters including cost-minimisation, cost-consequences, cost-effectiveness, cost-utility and cost-benefit analyses.

Patient Control



Convenience for patient, patient acceptability, patient willingness to use telemedicine


Direct non-medical costs (p.4)

Do not mention types of cost, but state that direct non-medical costs are those incurred by the patient.

Continuity of care, Accessibility: costs related to being able to access health care that is uninterrupted in nature.


Direct medical costs (p.5)

Do not mention types of cost, but state that direct non-medical costs are those incurred by the health service.

Efficiency and sustainability: related to providing effective healthcare services at a reasonable cost.

Effectiveness (p.5)

Number of cases averted (p.3, p.5), Number of cases detected (p.5), Number of cases requiring further treatment (p.5)


Technology cost (p.2) Depreciation rate

Efficiency and sustainability: this is related to the cost of technology, and therefore, the sustainability of telehealth.

Technology life (p.2) Equipment lifetime

Efficiency and sustainability: this is also related to cost and in some ways, the frequency of maintenance required.

!


Oliver, D. P., Demiris, G., Wittenberg-Lyles, E., Washington, K., Day, T., & Novak, H. (2012). A systematic review of the evidence base for telehospice. Telemedicine and e-health , 18 (1), 38-46.

Systematic review of the state of evidence related to telehospice services. 26 articles dated between 2000 and 2010 were selected and reviewed.



Clinical outcomes (p.45) Patient anxiety, communication anxiety.



Cost (p.45) Cost of telehospice visits Efficiency and sustainability: related to providing effective healthcare services at a reasonable cost.

Other (p.45)

Provider attitudes towards telehospice technologies.

Families perceptions of technology

Caregiver’s quality of life

Caregiver’s perceptions of pain medication


Rietdjik, R., Togher, L., & Power, E.(2012). Supporting family members of people with traumatic brain injury using telehealth: A systematic review. Journal of rehabilitation medicine. 44(11). 913-921

Systematic review of the effectiveness of telehealth programs to provide training and support to family members of people suffering from traumatic brain injuries. 24 articles were selected and reviewed. No dates of publications were specified. The authors report outcomes for carers and patients.

Patient Control


Savings in cost (p.918) Cost of telehealth intervention in comparison to impatient costs.

Accessibility, continuity of care: related to whether the patient is able to access timely care and continued support and care without travelling long distances.

Return to work (p.918) No measure has been specified. Accessibility, continuity of care: this is related to being able to access timely and required care.


Overall functioning (p.918)

Comparison of telehealth group with group receiving usual care.


Behavioral status (p.918) Comparison of telehealth group with group receiving usual care.


Psychological well being (p.918)



Quality of life (p.918) Comparison of telehealth group with group receiving usual care.


Cognitive function (p.918)



Family member characteristics moderated treatment efficacy (p.918)

Family socio-economic status, parent-reported child behaviour outcomes, ethnicity, carer’s experience with technology

Effectiveness: education on treatment is relevant and useful to the carers and achieves the desired outcomes.

Other (p.918)

Carer’s satisfaction (p.918)

Recommend intervention to others, preference to face-to-face interventions, ease and comfort of using technology, helpfulness or value of the intervention.

N/A

Family member’s psychological well being (p.918)

Depression, anxiety, stress and mood, pre-post differences. N/A

Application of knowledge learnt from intervention (p.918)

Follow up with caregivers after a certain period of time. N/A

Caregiver’s burden and needs (p.918) Pre-post differences. N/A

Increase of access to healthcare services for families in rural areas (p.919)

Peer support for family members (p.919)


Van den Berg, N., Schumann, M., Kraft, K., & Hoffmann, W. (2012). Telemedicine and telecare for older patients - a systematic review. Maturitas , 73 (2), 94-114.

Systematic review of telemedicine and telecare applications for older patients in various aspects of care. 68 articles, dated between 2007 and 2012 were selected and reviewed.

Patient Control


Acceptance/ satisfaction (p.112) No measures specified.


Patient empowerment (p.112)

No measures specified. Responsiveness: as well as whether they are treated with respect, dignity and have some control over their care.

Behavioral endpoints (p.112)

Adherence to medication or diet, physical activity, daily life activities. Better self-efficacy and management of disease.



Quality of life (p.96) General improvement or disease-specific improvements.


Medical outcomes (p.96)

Accuracy of diagnostic results between telemedical group and usual care.


Mortality (p.96) Comparison between telemedical group and usual care.


Improvement of functional status (p.96)

No measures specified. Effectiveness: treatment is relevant to the patient and achieves the desired outcomes.

Adverse events (p.96)

Number of adverse events occurring in telemedical group and usual care group.

Effectiveness, safe: treatment is relevant and safe for the patient and achieves the desired outcomes.


Economic outcomes (p.112)

Total cost, cost-effectiveness, hospitalization


Utlisation of healthcare system (p.112)

Primary care practice visits, emergency department visits

Efficiency: related to improving the way in which healthcare is provided and workforce productivity.

Other (p.112)

Ergonomics of the devices used, usability of device by patients who suffer from cognitive and visual impairments.


Wallace, D. L., Hussain, A., Khan, N., & Wilson, Y. T. (2012). A systematic review of the evidence for telemedicine in burn care: with a UK perspective. Burns , 38 (4), 466-475.

Systematic review of the use of telemedicine for burn care in the UK. 24 articles published between 1993 and 2010 were selected and reviewed. Studies reviewed applied various methodologies.

Patient Control


Cost savings (p.467) Travel time



Accuracy of diagnosis (p.470) Adequacy of image, image quality

Effectiveness: being able to provide relevant and correct care to patients and achieve the correct outcomes.

Adequacy of decision making (p.473)

Comparison of telehealth consultation to face to face consultation: appearance and fell of scar, contracture, range of motion, activity status, wound breakdown problems.



Cost savings (p.474) Transport costs, avoided transfers (air and land) Efficiency and sustainability: related to

the cost of care.

Clinical effectiveness (p.473)

Rate of emergency transfers, number of referrals. Efficiency and sustainability: related to the efficiency of providing services and workforce efficiencies.

Wilcox, M. E., & Adhikari, N. K. (2012). The effect of telemedicine in critically ill patients: systematic review and meta-analysis. Critical Care , 16: R127, 12 pages.

Systematic review of the impact of telemedicine for critically ill patients. 11 studies, dated between 2001 and 2012 were selected an reviewed.



Mortality rate (p.4)) ICU mortality, Hospital mortality


Length of hospital stay (p.7) ICU length of stay


Effect of interventions (p.7)

Type of intervention (active or high-intensity passive telemedicine intervention) and effect on ICU mortality


Other (p.9)

Nurse-staff satisfaction


Young, L. B. (2012). Telemedicine interventions for substance-use disorder: a literature review. Journal of telemedicine and telecare , 18 (1), 47-53.

Systematic review of telemedicine interventions for substance-use disorders, focusing on the evaluation of these interventions. 50 articles were reviewed and were dated between 1998 and 2010. No standard methodology in the literature reviewed.

Patient Control


Patient Satisfaction (p.50)

Initiation of consultation by patient, length of intervention

Accessibility, responsiveness: allowing the patient to access the healthcare service when required, and also allowing the patient to be in control of their care.


Reduction in substance use (p. 50)

Addiction severity index, timeline follow up method, urine screening, alcohol use disorders identification test.

Effectiveness: this relates to providing the relevant healthcare service to the client and therefore, achieving the desired outcomes.

Effective interventions (p.50)

Intervention length and its effect to substance use outcomes.

Effectiveness: related to whether the use of telemedicine results in the desired outcome, in this case reduction in substance use.


Resource utilization (p. 49)

number of participants using the telemedicine intervention, number of contacts [consultations] per participant, comparison to face to face interventions

Efficiency and sustainability: related to the sustainability of the telemedicine service.



Bradford, N., Armfield, N. R., Young, J., & Smith, A. C. (2013). The case for home based telehealth in pediatric palliative care: a systematic review. BMC palliative care , 12 (4), 13 pages.

Systematic review to evaluate the use of home based telehealth in pediatric palliative care. 33 articles, dated from the inception of various databases till 2012, were selected and reviewed.

Patient Control


Reduced travel costs (p.9)

Comparison between home telehealth and a control group receiving care through another medium.

Accessibility: this is related to the increased access to healthcare services, which can lead to a reduction in travel costs.

Reduced waiting time at appointments (p.9)


Accessibility: this is related to the increased access to healthcare services, which can lead to a reduction in the amount of waiting time.

Perceptions of telehealth (p.10)

Feasibility, acceptability, satisfaction with telehealth.

Responsiveness, continuity of care: this is related to whether the patient is in control of their care, and also whether they are able to access uninterrupted care services.




Effectiveness: This is related to the provision of care that meets that patient’s needs and achieves the desired outcomes.


Quicker discharge rate (p.8)


Efficiency: this is related to whether telehealth is providing an efficient way to provide care to the patients.

Substitution of home visits (p.9) Retrospective chart review.


Cost savings (p.9) Comparison between home telehealth and home visits.


Reduction in hospital admissions (p.9)



Other

Reduction in parental anxiety (p.8)

Families feel a sense of security (p.8)

Carers quality of life (p.9)

Clinicians’ perceptions of telehealth (p.10)


Edirippulige, S., Martin-Khan, M., Beattie, E., Smith, A. C., & Gray, L. C. (2013). A systematic review of telemedicine services for residents in long term care facilities. Journal of telemedicine and telecare , 19 (3), 127-132.

Systematic review of telemedicine use in long-term care facilities (review consisted of various specialties). 22 papers, dated between 1990 and 2012 were selected and reviewed.

Patient Control


Patient Satisfaction (p.129)

No measure has been specified

Accessibility, responsiveness: allowing the patient to access the healthcare service when required, and also allowing the patient to be in control of their care.

Time savings (p.129) Travel time. Accessibility: this is related to the increased access to healthcare services, which can lead to a reduction in travel time.

Acceptance (p.129) No measure specified. Accessibility, responsiveness: allowing the patient to access the healthcare service when required, and also allowing the patient to be in control of their care.


Reliability of assessment (p.129)

Comparison between patients assessed via telemedicine and in-person assessment.


Quality of life (p.130) Comparison between patients assessed via telemedicine and in-person assessment.


Changes in motor and cognitive symptoms (p.130)

Comparison between patients assessed via telemedicine and in-person assessment.



Cost of telemedicine sessions (p.129) Changes in staff workload.

Efficiency: this is related to whether telehealth is providing an efficient and productive way to provide care to the patients.

Effectiveness (p.129) Changes in number of clinic visits.


Feasibility/cost-effectiveness (p.130)

Increase in the number of patients being treated through telemedicine.

Efficiency and sustainability: this is related to whether telehealth is providing an efficient way to provide care to the patients and whether it is sustainable in the long term.


Hall, N., Boisvert, M., & Steele, R. (2013). Telepractice in the assessment and treatment of individuals with Aphasia: A systematic review. International journal of telerehabilitation. 5(1). 27-38.

Systematic review of the use of telepractice procedures to assess and treat patients suffering from Aphasia. 10 studies, dated from the inception of the relevant databases till 2012, were selected and reviewed.

Patient Control


Travel costs (p.32)

This is presented in the discussion section of the paper with no measures provided.

Accessibility: this is related to the increased access to healthcare services, which can lead to a reduction in travel costs.

Reduced cost of treatment (p.32)


Accessibility, responsiveness, continuity of care: being able to access timely, uninterrupted healthcare services that are catered to the patient’s need can lead to lower costs.

Increased access to healthcare services (p.32)


Accessibility: this is related to the increased access to the required healthcare services.

Improved adherence to intervention protocol (p.32)




Effectiveness (p.32) Comparison of telepractice group to in-person assessment.


Accuracy of assessment (p.32)

Clinician’s ability to administer language assessment protocols.


Improvement in stimuli (p.32)

Improvement of patient’s stimuli to material presented on computers


Visual cues and stimuli (p.32)

Comparison to traditional settings for assessment.


Communicating issues - Audio/video quality (p.32)

Comparison to traditional settings for assessment.

Effectiveness, safety: This is related to the provision of care that meets that patient’s needs and achieves the desired outcomes. Also related to the provision of treatment that is safe for the patient.


Connectivity (p.32) Speed of connection

Related to adequate performance of the technology. This is related to the audio and video quality. This will have an impact on the effectiveness of care, but this measure is related more to the technology that any other health performance measure. (from ACRRM telehealth advisory committee standards framework, 2012)


Kumar, G., Falk, D.M., Bonello, R.S., Kahn, J.M., Perencevich, E., & Cram, P. (2013). The costs of critical care telemedicine programs. A systematic review and analysis. CHEST.143(1). 19-29.

A systematic review of the costs related to implementing tele-ICUs. The authors collected detailed. data on implementing a tele-ICU in a network of Veterans Health Administration hospitals. 8 studies, dated between 1990 and 2011 were selected and reviewed.



Technology costs (p.22)

Costs to purchase, install, and maintain, hardware, software and licenses, equipment and networking, technical support


Staffing (p.22) Central monitoring site staff costs, health professional costs, technician costs.


Real estate (p.22) Rental costs, space leasing costs


Hospital variable costs (p.22)

Nursing supplies, pharmacy, laboratory, pathology, radiology and bedside diagnostics, interventional services, ancillary services.


Marino, R. & Ghanim, A. (2013). Teledentistry: a systematic review of the literature. Journal of telemedicine and telecare. 19(4). 179-183

A systematic review of teledentistry. 59 articles, dated between 1992 and 2012 were selected and reviewed. The authors placed more emphasis on the various areas of applying teledentistry, rather than the outcomes.



Effectiveness (p.181) Comparison to alternative modes of treatment.



Feasibility of teledentistry (p.181)

No measures are specified or discussed.



Rubin, M.R., Wellik, K.E., Channer, D.D. & Demaerschalk, B.M.(2013). Systematic review of telestroke for post-stroke care and rehabilitation. Current atherosclerosis reports. 15(8):343. 7 pages.

Systematic review on the use of telemedicine for the purposes of providing post-stroke care. 24 articles, dated between 1996 and 2012, were selected and reviewed.

Patient Control


Knowledge of stroke signs and symptoms (p.4)

Comparison between telemedicine and in-person counselling.

Accessibility: being able to access education through a medium such as telemedicine can increase patient’s knowledge of stroke signs.

Likelihood to change habits(p.4)


Accessibility, responsiveness, continuity of care: related to being able to access the required service in a timely manner, patients having control over their health and access to uninterrupted care and advice.

Satisfaction (p.4) Comparison between telemedicine and in-person counselling.


Stroke awareness education for a rural community (p.4)


Accessibility: being able to access education through a medium such as telemedicine can increase patient’s knowledge of stroke signs.


Diagnosis (p.4)

Performing transcranial Doppler (TCD) and carotid duplex (CD) through telemedicine and comparing to in-person examination



Time to complete assessment (p.4)

Comparison between telemedicine and in-person consultations.


Other (p.5)

Health professional confidence in using videoconferencing


Rubin, M. N., Wellik, K. E., Channer, D. D., & Demaerschalk, B. M. (2013). Systematic Review of Teleneurology: Neurophospitalist neurology. The neurohospitalist , 3 (3), 120-124.

A systematic review of teleneurologic consultations in hospital neurology. 4 articles were chosen for the review. No dates were specified.



Accuracy of diagnosis (p.121)

Tracking diagnosis made by neurologist in teleconsult and face-to-face visits.


Mortality (p.121)

Change in symptoms over a 24 week period in the hospital and within 3 months from hospital discharge.



Resource utilization (p.121)

Use of neuroimaging, number of transfer, outpatient appointments.


Vedel, I., Mignerat, M., Saksena, A. & Lapointe, L.(2013). Using telecare for diabetic patients: A mixed systematic review. eTelemed 2013: The fifth international conference on eHealth, telemedicine and social medicine. Pp.224-227.Nice, France. ISBN: 978-1-61208-252-3

Systematic review of telecare interventions used to managed type 2 diabetes mellitus. 50 articles, dates between 2000 and 2011 were selected and reviewed.

Patient Control


Patient outcomes (p.226)

Patient’s knowledge or self-care, patient transfer or travel time, social support/functioning, patient worry, patient satisfaction.


Quality of care (p.226) Accessibility to health services

Accessibility: this is related to being able to access timely and required healthcare services.


Health indicators (p.226)

Impact on glucose or HbA1c blood level, hyper glycemic events, BMI or weight, cholesterol or trigyceride blood level, blood pressure, quality of life, physical activity, Framingham risk score, depression/mental health, nutrition intake, pain.



Health service use-cost-productivity (p.226)

Health service use, healthcare costs, time spent by clinicians.



4 References

Reference list

A reference list sorted by year of publication and author.

Year: 2008

Clarke, M., & Thiyagarajan, C.A.(2008). A systematic review of technical evaluation in telemedicine systems. Telemedicine and e-health, 14(2), 170-183

Rojas, S.V., & Gagnon, M.P. (2008). A systematic review of the key indicators for assessing telehomecare cost-effectiveness. Telemedicine and e-health, 14(9), 896-904

Seto, E. (2008). Cost comparison between telemonitoring and usual care of heart failure: a systematic review. Telemedicine and e-health , 14 (7), 679-686.

Year: 2009

Davalos, M. E., French, M. T., Burdick, A. E., & Simmons, S. C. (2009). Economic Evaluation of Telemedicine: Review of literature and research guidelines for benefit-cost analysis. Telemedicine and e-health , 15 (10), 933-948.

Durrani, H., & Khoja, S. (2009). A systematic review of the use of telehealth in Asian countries. Journal of telemedicine and telecare , 15 (4), 175-181.

Jaana, M., Pare, G., & Sicotte, C. (2009). Home telemonitoring for respiratory conditions: A systematic review. The American Journal of Managed Care , 15 (5), 305-323.

Kairy, D., Lehoux, P., Vincent, C., & Visintin, M. (2009). A systematic review of clinical outcomes, clinical process, healthcare utilization and costs associated with telerehabilitation. Disability rehabilitation , 31 (6), 427-447.

Maric, B., Kaan, A., Ignaszewski, A., & Lear, S.A. (2009). A systematic review of telemonitoring technologies in heart failure. European journal of heart failure. 11(5), 506-517.

Neubeck, L., Redfern, J., Fernandez, R., Briffa, T., Bauman, A., & Freedman, S. B. (2009). Telehealth interventions for the secondary prevention of coronary heart disease: a systematic review. European Journal of cardiovascular prevention and rehabilitation , 16 (3), 281-289.

Polisena, J., Tran, K., Cimon, K., Hutton, B., McGill, S., Palmer, K.(2009). Home telehealth for diabetes management: a systematic review and meta-analysis, Diabetes, Obesity and Metabolism. 11(10), 913-930

Year: 2010

Boisvert, M., Lang, R., Andrianopoulos, M., & Boscardin, M.L. (2010). Telepractice in the assessment and treatment of individuals with autism spectrum disorders: a systematic review. Developmental neurorehabilitation. 13(6). 423-432

Ekeland, A. G., Bowes, A., & Flottorp, S. (2010). Effectiveness of telemedicine: A systematic review of reviews. International Journal of Medical Informatics , 79 (11), 736-771.

Garcia-Lizana, F., & Munoz-Mayorga, I.(2010). Telemedicine for depression: a systematic review. Perspectives in psychiatric care. 46(2). 119-126

Garcia-Lizana, F., & Munoz-Mayorga, I. (2010). What about telepsychiatry? A systematic review. Primary care companion to the journal of clinical psychiatry. 12(2). 10 pages

Johansson, T., & Wild, C.(2010). Telemedicine in acute stroke management: systematic review. International Journal of technology assessment in health care. 26(2). 149-155.

Luxton, D.D, & Mishkin, M.C.(2010). Safety of telemental healthcare delivered to clinically unsupervised settings: A systematic review. Telemedicine and e-health. 16(6). 705-711

Pare, G., Moqadem, K., Pineau, G., & St-Hilaire, C.(2010). Clinical Effects of Home Telemonitoring in the Context of Diabetes, Asthma, Heart Failure and Hypertension: A Systematic Review. Journal of Medical Internet Research. 12(2): e21


Polisena, J., Tran, K., Cimon, K., Hutton, B., McGill, S., Palmer, K., et al. (2010). Home telehealth for chronic obstructive pulonary disease: a systematic review and meta-analysis. Journal of telemedicine and telecare , 16 (3), 120-127.

Ray-Moreno, C., Reigadas, J. S., Villalba, E. E., Vinagre, J. J., & Fernandez, A. M. (2010). A systematic review of telemedicine projects in Colombia. Journal of telemedicine and telecare. 16(3). 114-119.

Shulman, R.M., O’Gorman, C.S., & Palmert, M.R. (2010). The impact of telemedicine interventions involving routine transmission of blood glucose data with clinician feedback on metabolic control in youth with type 1 diabetes: A systematic review and meta-analysis. International journal of pediatric endocrinology. Article ID: 536957, 9 pages.

Swanepoel de, W., & Hall, J. 3rd. (2010). A systematic review of telehealth applications in audiology. Telemedicine journal and e-health , 16 (2), 181-200.

Verhoeven, F., Tanja-Dijkstra, K., Nijland, N., & Eysenbach, G. (2010). Asynchronous and Synchronous Teleconsultation for Diabetes Care: A Systematic Literature Review. Journal of diabetes science and technology , 4 (3), 666-684.

Wade, V. A., Karnon, J., Elshaug, A. G., & Hiller, J. E. (2010). A systematic review of economic analyses of telehealth services using real time video communication. BMC Health services research , 10 (233), 13 pages.

Year: 2011

Garg, V., & Brewer, J. (2011). Telemedicine security: a systematic review. Journal of diabetes science and technology. 5(1). 768-777

Hailey, D., Roine, R., Ohinmaa, A., & Dennett, L. (2011). Evidence of benefit from telerehabilitation in routine care: a systematic review. Journal of telemedicine and telecare , 17 (6), 281-287.

Peeters, J. M., Mistiaen, P., & Francke, A. L. (2011). Costs and financial benefits of video communication compared to usual care at home: a systematic review. Journal of telemedicine and telecare , 17 (8), 403-411.

Verberk, W. J., Kessles, A. G., & Thien, T. (2011). Telecare is a valuable tool for hypertension management, a systematic review and meta-analysis. Blood Pressure Monitoring , 16 (3), 149-155.

Warshaw, E. M., Hillman, Y. J., Greer, N. L., Hagel, E. M., MacDonald, R., Rutks, I. R. & Wilt, T.J. (2011). Teledermatology for diagnosis and management of skin conditions: a systematic review. Journal of the American Academy of Dermatology , 64 (4), 759-772.

Year: 2012

Backhaus, A., Agha, Z., Maglione, M. L., Repp, A., Ross, B., Zuest, D., et al. (2012). Videoconferencing psychotherapy: a systematic review. Psychological Services , 9 (2), 111-131.

Cassimatis, M. & Kavanagh, D.J. (2012). Effects of type 2 diabetes behavioural telehealth interventions on glycaemic control and adherence: a systematic review. Journal of telemedicine and telecare. 18(8). 447-450

Ciere, Y., Cartwright, M., & Newman, S.P. (2012). A systematic review of the mediating role of knowledge, self-efficacy and self-care behaviour in telehealth patients with heart failure. Journal of telemedicine and telecare. 18(7). 384-391.

De Waure, C., Cadeddu, C., Gualano, M.R. & Ricciardi, W.(2012). Telemedicine for the reduction of myocardial infarction mortality: A systematic review and a meta-analysis of published studies. Telemedicine and e-health, 18(5), 323-328.

Giamouzis, G., Mastrogiannis, D., Koutrakis, K., Karyannis, G., Parisis, C., Rountas, C., et al. (2012). Telemonitoring in chronic heart failure: a systematic review. Cardiology research and practice , 1 (1), 7 pages.

Hilgart, J. S., Hayward, J. A., Coles, B., & Iredale, R. (2012). Telegenetics: a systematics review of telemedicine in genetics services. Genetics in Medicine , 14 (9), 765-776.

Kamei, T., Yamamoto, Y., Kajii, F., & Nakayama, Y. (2012). Systematic review and meta-analysis of studies involving telehome monitoring telenursing for patients with chronic obstructive pulmonary disease. Japan journal of nursing science , DOI:10.1111/j.1742-7924.2012.00228.x, 1-13.

Mistry, H. (2012). Systematic review of studies of the cost-effectiveness of telemedicine and telecare. Changes in the economic evidence over twenty years. Journal of telemedicine and telecare , 18 (1), 1-6.

Oliver, D. P., Demiris, G., Wittenberg-Lyles, E., Washington, K., Day, T., & Novak, H. (2012). A systematic review of the evidence base for telehospice. Telemedicine and e-health , 18 (1), 38-46.

Rietdjik, R., Togher, L., & Power, E.(2012). Supporting family members of people with traumatic brain injury using telehealth: A systematic review. Journal of rehabilitation medicine. 44(11). 913-921


Van den Berg, N., Schumann, M., Kraft, K., & Hoffmann, W. (2012). Telemedicine and telecare for older patients - a systematic review. Maturitas , 73 (2), 94-114.

Wallace, D. L., Hussain, A., Khan, N., & Wilson, Y. T. (2012). A systematic review of the evidence for telemedicine in burn care: with a UK perspective. Burns , 38 (4), 466-475.

Wilcox, M. E., & Adhikari, N. K. (2012). The effect of telemedicine in critically ill patients: systematic review and meta-analysis. Critical Care , 16: R127, 12 pages.

Young, L. B. (2012). Telemedicine interventions for substance-use disorder: a literature review. Journal of telemedicine and telecare , 18 (1), 47-53.

Year: 2013

Bradford, N., Armfield, N. R., Young, J., & Smith, A. C. (2013). The case for home based telehealth in pediatric palliative care: a systematic review. BMC palliative care , 12 (4), 13 pages.

Edirippulige, S., Martin-Khan, M., Beattie, E., Smith, A. C., & Gray, L. C. (2013). A systematic review of telemedicine services for residents in long term care facilities. Journal of telemedicine and telecare , 19 (3), 127-132.

Hall, N., Boisvert, M., & Steele, R. (2013). Telepractice in the assessment and treatment of individuals with Aphasia: A systematic review. International journal of telerehabilitation. 5(1). 27-38.

Kumar, G., Falk, D.M., Bonello, R.S., Kahn, J.M., Perencevich, E., & Cram, P. (2013). The costs of critical care telemedicine programs. A systematic review and analysis. CHEST.143(1). 19-29.

Marino, R. & Ghanim, A. (2013). Teledentistry: a systematic review of the literature. Journal of telemedicine and telecare. 19(4). 179-183

Rubin, M.R., Wellik, K.E., Channer, D.D. & Demaerschalk, B.M.(2013). Systematic review of telestroke for post-stroke care and rehabilitation. Current atherosclerosis reports. 15(8):343. 7 pages.

Rubin, M. N., Wellik, K. E., Channer, D. D., & Demaerschalk, B. M. (2013). Systematic Review of Teleneurology: Neurophospitalist neurology. The neurohospitalist , 3 (3), 120-124.

Vedel, I., Mignerat, M., Saksena, A. & Lapointe, L.(2013). Using telecare for diabetic patients: A mixed systematic review. eTelemed 2013: The fifth international conference on eHealth, telemedicine and social medicine. Pp.224-227.Nice, France. ISBN: 978-1-61208-252-3


Endnotes

1 Goodman, C.S. (2004). HTA 101: Introduction to Health Technology Assessment. Canada: The Lewin Group. Retrieved from http://www.nlm.nih.gov/archive//20040831/nichsr/ta101/ta101_c1.html 2 Health Innovation and Reform Council. (2013). Summary of recommendations for telehealth. Retrieved from http://docs.health.vic.gov.au/docs/doc/Health-Innovation-and-Reform-Council-Summary-of-telehealth-recommendations 3 Zanaboni, P., & Lettieri, E. (2011). Institutionalizing telemedicine applications: the challenge of legitimizing decision making. Journal of Medical Internet Research. 13(3): e72. doi: 10.2196/jmir.1669 4 Hughes, E., King, C.& Kitt, S.(2002). Using the Australian and New Zealand Telehealth Committee Framework to Evaluate Telehealth: Identifying Conceptual Gaps. Journal of telemedicine and telecare. 8(3), 36-38. 5 Verhoeven, F., Tanja-Dijkstra, K., Nijland, N., & Eysenbach, G. (2010). Asynchronous and Synchronous Teleconsultation for Diabetes Care: A Systematic Literature Review. Journal of diabetes science and technology , 4 (3), 666-684. 6 Ekeland, A. G., Bowes, A., & Flottorp, S. (2012). Methodologies for assessing telemedicine: a systematic review of reviews. International journal of medical informatics , 8 (1), 1-11. Kummervold, P. E., Johnsen, K. J.-A., Skrovseth, S. O., & Wynn, R. (2012). Using Noninferiority Tests to Evaluate Telemedicine and E-Health Services: Systematic Review. Journal of medical internet research , 14 (5), e132. Rubin, M. N., Wellik, K. E., Channer, D. D., & Demaerschalk, B. M. (2012). Systematic review of teleneurology: methodology. Frontiers in neurology , 8 (3), 4 pages. 7 Masella, C., & Zanaboni, P. (2008). Assessment models for telemedicine services in national health systems. International journal of healthcare technology management , 9 (5/6), 446-472., Rowe, E., Jonsson, S., & Terio, H. (2011). PENG Analysis for Evaluation of Telemedicine Projects. In K. Dremstrup, S. Rees, & M. O. Jensen (Ed.), 15th NBC on biomedical engineering & medical physics. 34, pp. 249-252. Aalborg, Denmark: Springer Link., Chen, L. C., Chen, C. W., Weng, Y. C., Shang, R. J., Yu, H. C., Chung, Y., & Lai, F. (2012). An information technology framework for strengthening telehealthcare service delivery. Telemedicine and e-health , 18 (8), 596-603., Kidholm, K., Ekeland, A. G., Jensen, L. K., Rasmussen, J., Pefersen, C. D., Bowes, A., Flottotp, S.A. & Bech, M. (2012). A model for assessment of telemedicine applications: mast. International journal of technology assessment in healthcare , 28 (1), 44-51. 8 Hailey D, Ohinmaa A, Roine R. Evidence for the benefits of telecardiology applications: a systematic review. Edmonton, AB, Canada: Alberta Heritage Foundation for Medical Research. Health Technology Assessment; 34. 2004. Retrieved from http://www.ihe.ca/hta/publications.html?Category=HTA%20Series%20A 9 Australian Institute of Health and Welfare. (2009). National Health Information Standards and Statistics Committee (NHISSC) 2009. The National Health Performance Framework (2nd Edition) http://www.aihw.gov.au/health-indicators/ Link to “changes to the NHPF” 10 Australia College of Rural & Remote Medicine. (2012). ACRRM telehealth advisory committee standards framework. Retrieved from http://www.ehealth.acrrm.org.au/system/files/private/ATHAC%20Telehealth%20Standards%20Framework_0.pdf 11 Hawthorne, G & Osborne, R. (2005). Population norms and meaningful differences for the Assessment of Quality of Life (AQoL) measure. Australian and New Zealand journal of public health. 29.2: 136-142

Institute for a Broadband-Enabled SocietyLevel 4, Building 193The University of Melbourne, Victoria 3010e: [email protected]

www.broadband.unimelb.edu.au

Evaluation-of-Telehealth-Implementations-in-Australia copy

Documents

Transcript of Evaluation-of-Telehealth-Implementations-in-Australia copy