Subjective self-mapping in the Mental Health Continuum ... · Les participants du groupe 1 ont...

Defence Research and Development Canada Scientific Report DRDC-RDDC-2018-R294 June 2019

CAN UNCLASSIFIED

CAN UNCLASSIFIED

Subjective self-mapping in the Mental Health Continuum Model (MHCM) Examining mental health self-mapping and its relation to validated measures

Madeleine D'Agata, PhD Anthony Nazarov, PhD Joshua A. Granek, PhD DRDC – Toronto Research Centre

CAN UNCLASSIFIED

Template in use: EO Publishing App for SR-RD-EC Eng 2018-12-19_v1.dotm © Her Majesty the Queen in Right of Canada (Department of National Defence), 2019 © Sa Majesté la Reine en droit du Canada (Ministère de la Défense nationale), 2019

CAN UNCLASSIFIED

IMPORTANT INFORMATIVE STATEMENTS

This document was reviewed for Controlled Goods by Defence Research and Development Canada (DRDC) using the Schedule to the Defence Production Act.

Disclaimer: Her Majesty the Queen in right of Canada, as represented by the Minister of National Defence ("Canada"), makes no representations or warranties, express or implied, of any kind whatsoever, and assumes no liability for the accuracy, reliability, completeness, currency or usefulness of any information, product, process or material included in this document. Nothing in this document should be interpreted as an endorsement for the specific use of any tool, technique or process examined in it. Any reliance on, or use of, any information, product, process or material included in this document is at the sole risk of the person so using it or relying on it. Canada does not assume any liability in respect of any damages or losses arising out of or in connection with the use of, or reliance on, any information, product, process or material included in this document.

The data collected as part of this study was approved either by Defence Research and Development Canada’s Human Research Ethics Board or by the Director General Military Personnel Research & Analysis’ Social Science Research Review Board.

Endorsement statement: This publication has been peer-reviewed and published by the Editorial Office of Defence Research and Development Canada, an agency of the Department of National Defence of Canada. Inquiries can be sent to: [email protected].

DRDC-RDDC-2018-R294 i

Abstract

Background: To complement the Canadian Armed Forces’ (CAF) Road to Mental Readiness (R2MR) training program, a mobile application (i.e., “app”) was developed to provide users with an online opportunity to review and practice strategies to cope with stress and increase resilience. One of the components of the mobile app is the Mental Health Continuum Model (MHCM), designed to allow users to self-assess on six mental health and well-being domains (i.e., Mood, Attitude & Performance, Sleep, Physical Symptoms, Social Behaviour, Alcohol & Gambling) using a visual spectrum of the continuum that includes four anchors: healthy, reacting, injured, and ill. More specifically, for each functional domain the app involves self-rating on scales guided by different descriptors, which have been grouped to represent the four anchors of the MHCM. The current research was undertaken in order to inform and guide future R2MR mobile app development and training content. Specifically, we sought to understand 1) whether MHCM self-mapping aligns with established validated physical and mental health scales; and 2) how CAF members perceive the MHCM content (i.e., the degree of concurrence or consistency between endorsed anchors and their descriptors). Thus, the current research aimed to examine the concurrent validity of the MHCM compared to validated, well-established measures. Additionally, using a separate matching task, our research assessed if the current wording of the descriptors from the MHCM correspond to the four anchors as well as the in-app visual representation of the continuum.

Methods: Data were collected online from 392 Regular Force CAF members. Participants self-mapped onto each of the functional domains of the MHCM, and were randomly assigned to be presented with one of four versions of the MHCM. Participants assigned to Group 1 were presented with the descriptors separately (i.e., without the visual spectrum); they were also presented with the visual spectrum separately (i.e., without the descriptors) and were instructed to self-map onto each of the functional domains of the MHCM. Group 2 were also presented with the descriptors and visual spectrum separately; however, the visual spectrum included the four anchors. Group 3 was presented with the MHCM as is in the mobile app (i.e., the descriptors and spectrum were presented together), and Group 4 was presented with the same version as Group 3, however, the spectrum included the four anchors. All of the participants completed demographic information and validated measures of physical and mental health that assessed similar or the same constructs of each of the functional domains. Finally, all participants completed a matching task to assess the accuracy rates of matching each descriptor to its corresponding anchor.

Results: Overall, there was an adequate amount of agreement between MHCM self-mapping and the validated measures providing some support for the MHCM self-mapping approach, although the degree of agreement differed across functional domains. Importantly, of those participants screening positive for clinically significant symptoms of depression/anxiety, suicide ideation, or hazardous and harmful alcohol use, at least one-fifth self-assessed as healthy on the MHCM. Additionally, the rate at which each validated scale predicted group membership to the four MHCM anchors was high for healthy, however, the rates decreased for reacting, injured, and ill. Approximately 15% (Alcohol and Gambling domain) to 45% (Sleep domain) of participants were inconsistent in self-mapping across the functional domains. Similarly, the results of the matching task indicated that accuracy was high for identifying the descriptors that are healthy, but rates decreased noticeably for reacting, injured, and ill.

Discussion: The results provide initial support for the MHCM self-mapping approach, such that there was a fair amount of agreement between the MHCM self-mapping task and the validated measures. Moreover, for most of the functional domains, a large proportion of participants’ descriptor and spectrum self-mapping

ii DRDC-RDDC-2018-R294

were consistent. Nonetheless, our results indicate that there is a noticeable rate of false-negatives on the MHCM self-assessment tool, indicating some participants are underestimating their symptom severity on the continuum. The results of the predicted group membership analyses on the self-mapping task indicate a potential discrepancy between some of the descriptors, particularly the descriptors for the reacting, injured, and ill anchors, and the validated measures. Finally, for most of the functional domains, a large proportion of participants’ descriptor and spectrum self-mapping were consistent. However, our findings from the self-mapping task indicate that some of the MHCM descriptors do not accurately reflect how participants self-assess on the visual spectrum and that participants’ ability to distinguish and assess symptom severity between descriptors was restricted, especially for reacting, injured, and ill. A similar pattern emerged for the matching task such that accuracy was high for identifying the descriptors that are healthy, but rates decreased noticeably for reacting, injured, and ill. Future research should focus on further assessing the source of inconsistency and discrepancy between self-mapping and between the MHCM and well-established, validated measures.

Significance to Defence and Security

The psychosocial well-being of CAF personnel is a priority for the Department of National Defence and R2MR training is a primary tool that is employed to promote mental health and well-being in our members, as is the recently developed R2MR app. Overall, our results indicate that self-mapping on the MHCM is consistent, however, it is also associated with some inaccuracies when comparing the self-mapping tool to validated measures. These results are also supported by the results of a separate matching task indicating that discriminating symptom severity between some of the descriptors is hindered. Findings from the current research highlight some of the ways in which the MHCM could be improved upon for the mobile app tool as well as for the R2MR training.

DRDC-RDDC-2018-R294 iii

Résumé

Contexte : On a élaboré une application mobile comme complément au programme de formation En route vers la préparation mentale (RVPM) des Forces armées canadiennes (FAC). Cette application permet aux utilisateurs de passer en revue et de mettre en pratique virtuellement des stratégies pour composer avec le stress et augmenter leur résilience. L’une des composantes de l’application est le modèle de continuum de la santé mentale (MCSM). Cette composante permet aux utilisateurs d’évaluer leur bien-être et leur santé mentale en fonction de six critères (l’humeur, l’attitude et le rendement; la qualité du sommeil; les symptômes physiques; le comportement en société; la dépendance à l’alcool et au jeu) selon un spectre de couleurs comportant quatre catégories – en santé, en réaction, blessé et malade. Plus précisément, pour chaque domaine fonctionnel, l’application s’appuie sur une autoévaluation sur des échelles guidées par différents descripteurs, qui ont été regroupés pour représenter les quatre critères du MCSM. Les travaux de recherche actuels ont été entrepris pour faire connaître et orienter le développement futur de l’application mobile RVPM et du contenu de la formation. En particulier, on a cherché à comprendre 1) si la méthode d’autoévaluation du MCSM cadrait bien avec les échelons de santé mentale et physique déjà validés et reconnus; et 2) la façon dont les membres des FAC perçoivent le contenu du MCSM (pour évaluer le degré de concordance ou de cohérence entre les caractéristiques et les critères avalisées). En somme, on a voulu vérifier la validité du MCSM en le comparant à des mesures reconnues et validées. Un exercice de jumelage distinct a également été réalisé pour évaluer la qualité de la formulation existante des caractéristiques du MCSM et déterminer dans quelle mesure celles-ci correspondaient bien aux quatre critères et à la représentation visuelle du continuum dans l’application mobile.

Méthodes : Les données ont été recueillies en ligne auprès de 392 membres de la Force régulière des FAC. Les participants ont procédé à une autoévaluation en fonction des domaines fonctionnels du MCSM et de la version du modèle leur ayant été attribuée de façon aléatoire parmi les quatre versions possibles. Les participants du groupe 1 ont reçu les descripteurs séparément (c.-à-d. sans le spectre de couleurs). Les participants du groupe 2 ont aussi reçu les descripteurs et le spectre de couleurs séparément, mais ce dernier comportait les quatre critères. Le groupe 3 a reçu la version du MCSM qui figure actuellement dans l’application mobile (c.-à-d. les descripteurs et le spectre de couleurs sont présentés ensemble). Le groupe 4 a reçu la même version que celle du groupe 3, sauf que le spectre comportait les quatre critères. Tous les participants ont fourni des renseignements démographiques et validé les mesures de la santé physique et mentale qui visaient à évaluer des concepts semblables ou identiques à ceux associés à chaque domaine fonctionnel du MCSM. Tous les participants ont également effectué un exercice de jumelage afin de déterminer le degré de précision du taux de correspondance entre les descripteurs et les critères.

Résultats : Dans l’ensemble, on a observé un degré de concordance adéquat entre l’autoévaluation du MCSM et les mesures validées et ayant servi de point de comparaison, ce qui, dans une certaine mesure, appuie le MCSM. On a toutefois constaté que le degré de concordance était très variable d’un domaine fonctionnel à un autre. Par ailleurs, l’un des principaux points qui est ressorti de l’étude a été que, parmi les participants présentant des symptômes cliniques importants associés à la dépression ou à l’anxiété, à des idées suicidaires ou à une consommation dangereuse ou nocive d’alcool, au moins un cinquième d’entre eux ont déclaré être en santé dans leur autoévaluation. En outre, pour chacun des échelons validés qui prévoyait des membres du groupe dans les quatre critères du MCSM, les taux étaient élevés pour en santé, mais ils diminuaient pour en réaction, blessé et malade. Approximativement, de 15 % (dépendance

iv DRDC-RDDC-2018-R294

à l’alcool et au jeu) à 45 % (qualité du sommeil) des participants ont été incohérents dans leur autoévaluation en fonction des différents domaines fonctionnels. Dans le même ordre d’idée, dans le cadre de l’exercice de jumelage, le taux de réussite a été très bon pour l’association des descripteurs avec la catégorie en santé, alors qu’il a été considérablement moins élevé pour les autres catégories (en réaction, blessé et malade).

Analyse : Les résultats appuient la méthode d’autoévaluation associée au MCSM, car le degré de concordance observé entre l’autoévaluation et les mesures validées et reconnues était relativement élevé. De plus, pour la plupart des domaines fonctionnels, une grande proportion des descripteurs des participants et de l’autoévaluation à l’aide du spectre étaient cohérents. Néanmoins, on a observé un taux évident de faux négatifs parmi les autoévaluations, ce qui signifie que certains participants sous-estimaient la gravité de leurs symptômes dans le continuum. Les résultats des analyses des membres du groupe prévu pour l’exercice d’autoévaluation indiquent un écart potentiel entre certains descripteurs – notamment ceux associés aux catégories en réaction, blessé et malade – et les mesures validées. Enfin, pour la plupart des domaines fonctionnels, une grande proportion des descripteurs et des autoévaluations à l’aide du spectre de couleurs étaient cohérentes. Toutefois, les résultats de l’exercice d’autoévaluation ont révélé que certains descripteurs du MCSM ne reflétaient pas précisément la façon dont les participants s’étaient autoévalués selon le spectre de couleurs et que leur capacité à discerner et à évaluer la gravité de leurs symptômes à l’aide des caractéristiques s’en est trouvée limitée, surtout dans les catégories en réaction, blessé et malade. On a observé une tendance similaire dans l’exercice de jumelage. Le taux de réussite du jumelage des descripteurs avec la catégorie en santé s’est avéré élevé, mais il était beaucoup plus modeste dans les catégories en réaction, blessé et malade. De prochains projets de recherche devraient porter sur l’évaluation de la source des incohérences et des divergences entre les autoévaluations, le MCSM et les mesures reconnues et validées.

Importance pour la défense et la sécurité

Le bien-être psychosocial du personnel des FAC est une priorité pour le ministère de la Défense nationale. Le programme de formation En route vers la préparation mentale (RVPM) est l’un des principaux outils utilisés pour promouvoir la santé mentale et le bien-être des militaires, tout comme l’application mobile mise au point récemment. Dans l’ensemble, nos résultats indiquent que l’autoévaluation du MCSM est cohérente, mais la comparaison avec les mesures validées nous indique qu’elle comporte aussi des inexactitudes. Ces résultats sont également corroborés par ceux obtenus en réalisant un exercice de jumelage distinct, lesquels ont révélé que, pour certains symptômes, le degré de gravité était sous-estimé lorsqu’on l’évaluait à l’aide des descripteurs. Les résultats de la recherche actuelle mettent en lumière certaines des façons d’améliorer le MCSM, autant pour l’application mobile que pour le programme de formation RVPM.

DRDC-RDDC-2018-R294 v

Table of Contents

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i Significance to Defence and Security . . . . . . . . . . . . . . . . . . . . . . . . . ii Résumé . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Importance pour la défense et la sécurité . . . . . . . . . . . . . . . . . . . . . . iv Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Mental Health Training in the Canadian Armed Forces: The Road to Mental Readiness (R2MR) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 The Mental Health Continuum Model . . . . . . . . . . . . . . . . . . 1

1.2 The R2MR Mobile App . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Current Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1 Participants & Data Collection . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2.2 MHCM Self-Mapping Task . . . . . . . . . . . . . . . . . . . . . . 7 2.2.3 Matching Task . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.3 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3.1 Derived Variables to Assess Self-Mapping Consistency . . . . . . . . . . . 9 2.3.2 Validity Measures . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3.2.1 Mood Domain: Kessler Psychological Distress Scale & The Suicide Behaviors Questionnaire-Revised . . . . . . . . . . . . . . . 10

2.3.2.2 Attitude & Performance Domain: Utrecht Work Engagement Scale . . 10 2.3.2.3 Sleep Domain: Sleep Disturbance Subscale, Patient-Reported Outcomes

Measurement Information System (PROMIS) . . . . . . . . . . 11 2.3.2.4 Physical Symptoms Domain: Global Health Scale, Patient-Reported

Outcomes Measurement Information System (PROMIS) . . . . . . 11 2.3.2.5 Social Behaviour Domain: Multidimensional Scale of Perceived Social

Support (MSPSS) . . . . . . . . . . . . . . . . . . . . . 11 2.3.2.6 Alcohol & Gambling Domain: Alcohol Use Disorders Identification Test

(AUDIT) & Problem Gambling Severity Index . . . . . . . . . . 11 2.4 Secondary Measures . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.4.1 Positive Mental Health . . . . . . . . . . . . . . . . . . . . . . 12 2.4.2 Social Desirability & Self-Concealment . . . . . . . . . . . . . . . . 12 2.4.3 Demographic, Mental Health & Training Questions . . . . . . . . . . . . 12

vi DRDC-RDDC-2018-R294

2.5 Statistical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.1 A Comparison of Subjective Self-Mapping in the MHCM to Validated Measures . . . 14 3.1.1 Correlation Analyses . . . . . . . . . . . . . . . . . . . . . . . 14 3.1.2 Mean Scores of the Validated Measures & MHCM Anchors . . . . . . . . 15 3.1.3 Mental Health Screening Scales with Established Cut-Off Scores . . . . . . . 18 3.1.4 Predicted Group Membership of Validated Measures . . . . . . . . . . . 21

3.2 The Role of Subjective Self-Mapping in the MHCM . . . . . . . . . . . . . . 22 3.2.1 Self-Mapping Consistency . . . . . . . . . . . . . . . . . . . . . 22

3.2.1.1 R2MR Training . . . . . . . . . . . . . . . . . . . . . 23 3.2.1.2 Mental Health Service Use . . . . . . . . . . . . . . . . . 24

3.2.2 MHCM Visual Spectrum Scores as a Function of Embedding Text Anchors . . . 24 3.3 Matching Task Accuracy . . . . . . . . . . . . . . . . . . . . . . . . 25

4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.1 A Comparison of Subjective Self-Mapping in the MHCM to Validated Measures . . . 32

4.1.1 Correlation Analyses . . . . . . . . . . . . . . . . . . . . . . . 32 4.1.2 Mean Scores & MHCM Anchors . . . . . . . . . . . . . . . . . . . 33 4.1.3 Mental Health Screening Scales with Established Cut-Off Scores . . . . . . . 33 4.1.4 Predicted Group Membership . . . . . . . . . . . . . . . . . . . . 33

4.2 The Role of Subjective Self-Mapping in the MHCM . . . . . . . . . . . . . . 33 4.2.1 Self-Mapping Consistency . . . . . . . . . . . . . . . . . . . . . 33

4.2.1.1 R2MR Training & Mental Health Service Use . . . . . . . . . . 34 4.2.2 MHCM Visual Spectrum Scores as a Function of Embedding Text Anchors . . . 34

4.3 Matching Task Accuracy . . . . . . . . . . . . . . . . . . . . . . . . 34 4.4 Limitations & Future Directions . . . . . . . . . . . . . . . . . . . . . . 35 4.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Annex A Visual Representations of the Experiment . . . . . . . . . . . . . . . . . . 41

A.1 MHCM Instructions Presented Prior to Self-Mapping Task . . . . . . . . . . . . 41 A.2 Group 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 A.3 Group 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 A.4 Group 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 A.5 Group 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Annex B Visual Representations of the Matching Task . . . . . . . . . . . . . . . . 45 B.1 Separated Version of the Matching Task . . . . . . . . . . . . . . . . . . . 45 B.2 Combined Version of the Matching Task . . . . . . . . . . . . . . . . . . 46

Annex C Correlation Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 47

DRDC-RDDC-2018-R294 vii

List of Figures

Figure 1: Visual representation of the ordering of the experimental design based on groups. . . . 7

Figure 2: Example of the matching task for the Mood domain. . . . . . . . . . . . . . . . 9

Figure 3: Mean scores on the K10 by MHCM Mood anchors. [Note. Error bars represent standard error; dotted line represents depression/anxiety cut-off scores.] . . . . . . 15

Figure 4: Mean scores on the SBQ-R by MHCM Mood anchors. [Note. Error bars represent standard error; dotted line represents suicide ideation cut-off score.] . . . . . . . 16

Figure 5: Mean scores on the UWES-9 by MHCM Attitude & Performance anchors. [Note. Error bars represent standard error.] . . . . . . . . . . . . . . . . . . . . 16

Figure 6: Mean scores on the Sleep Disturbance Short Form by MHCM Sleep anchors. [Note. Error bars represent standard error.] . . . . . . . . . . . . . . . . . . . . 17

Figure 7: Mean scores on the Global Health Short Form by MHCM Physical Symptoms anchors. [Note. Error bars represent standard error.] . . . . . . . . . . . . . . . . . 17

Figure 8: Mean scores on the MSPSS by MHCM Social Behaviour anchors. [Note. Error bars represent standard error.] . . . . . . . . . . . . . . . . . . . . . . . . . 18

Figure 9: MHCM Mood anchor self-mapping among participants who screened positive for depression/anxiety (n = 138). . . . . . . . . . . . . . . . . . . . . . . . 19

Figure 10: MHCM Mood anchor self-mapping by K10 categories (n = 389). . . . . . . . . 20

Figure 11: MHCM Mood anchor self-mapping among participants who screened positive for suicide ideation (n = 54). . . . . . . . . . . . . . . . . . . . . . . . . . 20

Figure 12: MHCM Alcohol & Gambling anchor self-mapping among participants who screened positive for hazardous and harmful alcohol use (n = 64). . . . . . . . . . . . . 21

Figure 13: Mean spectrum scores by functional domain comparing spectrum with anchors to no anchors. [Note. A higher spectrum score reflects impaired state within each domain. Error bars represent standard error.] . . . . . . . . . . . . . . . . . . . . 25

Figure 14: Mean scores for accuracy on the matching task (N = 391). [Note. Error bars represent standard error] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Figure 15: Separated matching task (n = 192) for MHCM Mood descriptors by anchors. [Note. Percentages outlined with a box represent the proportion of the accurately selected anchor for its corresponding descriptor.] . . . . . . . . . . . . . . . . . . 28

Figure 16: Separated matching task (n = 192) for MHCM Attitude & Performance descriptors by anchors. [Note. Percentages outlined with a box represent the proportion of the accurately selected anchor for its corresponding descriptor.]. . . . . . . . . . . 29

Figure 17: Separated matching task (n = 195) for MHCM Sleep descriptors by anchors. [Note. Percentages outlined with a box represent the proportion of the accurately selected anchor for its corresponding descriptor.] . . . . . . . . . . . . . . . . . . 29

viii DRDC-RDDC-2018-R294

Figure 18: Separated matching task (n = 193) for MHCM Physical Symptoms descriptors by anchors. [Note. Percentages outlined with a box represent the proportion of the accurately selected anchor for its corresponding descriptor.]. . . . . . . . . . . 30

Figure 19: Separated matching task (n = 194) for MHCM Social Behaviour descriptors by anchors. [Note. Percentages outlined with a box represent the proportion of the accurately selected anchor for its corresponding descriptor.]. . . . . . . . . . . 30

Figure 20: Separated matching task (n = 193) for MHCM Alcohol & Gambling descriptors by anchors. [Note. Percentages outlined with a box represent the proportion of the accurately selected anchor for its corresponding descriptor.]. . . . . . . . . . . 31

Figure A.1: Visual representation of the MHCM that was presented alongside the instructions. . 41

Figure A.2: Visual representation of Group 1 participants who either completed the spectrum first followed by the descriptors, or completed the descriptors followed by the spectrum for each functional domain. . . . . . . . . . . . . . . . . . . . . . . . . . 42

Figure A.3: Visual representation of Group 2 participants who either completed the spectrum with anchors first followed by the descriptors, or completed the descriptors followed by the spectrum with anchors for each functional domain. . . . . . . . . . . . . . . 43

Figure A.4: Visual representation of Group 3 participants who completed the spectrum and descriptors together. . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Figure A.5: Visual representation of Group 4 participants who completed the spectrum with anchors and descriptors together. . . . . . . . . . . . . . . . . . . . . . 44

Figure B.1: Visual representation of matching task for participants assigned to the separated version (i.e., each domain was presented separately). . . . . . . . . . . . . . 45

Figure B.2: Visual representation of matching task for participants assigned to the combined version (i.e., all domains were presented together). . . . . . . . . . . . . . . 46

DRDC-RDDC-2018-R294 ix

List of Tables

Table 1: The R2MR Mental Health Continuum Model (taken from R2MR mobile app). . . . . 2

Table 2: Sample demographic, mental health, and training information. . . . . . . . . . . . 5

Table 3: Visual representation of the experimental design. . . . . . . . . . . . . . . . . 8

Table 4: Bivariate correlations among validated measures and MHCM spectrum scores (n = 364). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Table 5: Polyserial correlations of participants’ functional domain anchors and its corresponding validated scale. . . . . . . . . . . . . . . . . . . . . . . . 14

Table 6: Percentage of participants who screened positive on the K10, SBQ-R, and AUDIT. . 18

Table 7: Correct classification of participants’ functional domain anchors based on its corresponding validated scale(s) for 4 anchors. . . . . . . . . . . . . . . . . 21

Table 8: Correct classification of participants’ functional domain anchors based on its corresponding validated scale(s) for 3 anchors. . . . . . . . . . . . . . . . . 22

Table 9: Percentage of participants who self-mapped onto each of the four mental health categories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Table 10: Percentage of participants with consistent descriptor and visual spectrum self-mapping. 23

Table 11: Mean scores for inconsistent self-mapping by functional domain. . . . . . . . . 23

Table 12: Percentages of frequencies for the separated version of the matching task. . . . . . 26

Table 13: Percentages of frequencies for the combined version of the matching task. . . . . . 27

Table 14: Percentage of participants (n = 188) who selected “healthy” for the ill descriptor 0 to 6 times. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Table C.15: Bivariate correlations among all validated measures (n = 351). . . . . . . . . . 47

Table C.16: Bivariate correlations among validated measures and MHCM spectrum scores (n = 364). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

x DRDC-RDDC-2018-R294

Acknowledgements

The authors acknowledge Hamid Boland, LCol Suzanne Bailey, and Kimberly Guest for their insight and feedback on this research, and Dr. Peter Kwantes and Dr. Zhigang Wang for their guidance on data analysis. The authors also acknowledge Megan Thompson for her insight on this report. The authors would like to thank all of the CAF personnel who volunteered their time to participate in the study.

DRDC-RDDC-2018-R294 1

1 Introduction

1.1 Mental Health Training in the Canadian Armed Forces: The Road to Mental Readiness (R2MR)

The mental health and well-being of military personnel are key contributors to operational readiness and force sustainability (DND, 2017). In support of the fitness of the Canadian Armed Forces (CAF), an evidence-informed comprehensive mental health training and education program was developed, the Road to Mental Readiness (R2MR) (Bailey, 2015). Based on a similar program implemented by the United States (US) Navy SEALs, R2MR was developed as a classroom-based training program designed to increase mental health literacy and enhance resilience, in order to ultimately improve short-term performance and well-being outcomes. The R2MR program includes detailed information about stress and its potential effect on performance and decision-making processes. CAF members (and civilians) are provided with a detailed explanation of the types of stressors in a given environment and are taught a series of skills to manage such stressors. The R2MR curriculum includes six basic mental health management skills. The first four are skills (i.e., goal setting, self-talk, mental rehearsal, and tactical breathing) that are aligned with some of the tenets of Cognitive-Behavioural Therapy (CBT). The two additional skills, attentional control and psychological self-monitoring, were added at a later stage, following feedback from CAF personnel teaching the course.

1.1.1 The Mental Health Continuum Model

One of the tools presented within R2MR for psychological self-monitoring is the Mental Health Continuum Model (MHCM) which is designed to teach individuals to recognize symptoms of stress and distress in themselves and others. Adapted from the US Marine Corps, the MHCM is intended to allow individuals to self-monitor and to promote awareness of their current mental health and to characterize their mental health and well-being on a colour-coded continuum (see Table 1). Mental health self-monitoring also provides an opportunity for the identification of mental health concerns and consequently prompts individuals to take the appropriate action (e.g., seeking mental treatment, seeking social support) to confront such concerns (Kauer et al., 2012).

As presented in Figure 1, the MHCM depicts a mental health continuum across four anchors healthy to ill, including the possibility of falling somewhere in between (i.e., reacting or injured) across six functional domains: Mood, Attitude and Performance, Sleep, Physical Symptoms, Social Behaviour, and Gambling and Alcohol. In addition, the MHCM describes the behaviours indicative of each functional domain under each of the health continuum anchors; for instance, in the Alcohol and Gambling functional domain, the following descriptor is presented next to the colour yellow: “Regular but controlled alcohol use / gambling to cope.”

2 DRDC-RDDC-2018-R294

Table 1: The R2MR Mental Health Continuum Model (taken from R2MR mobile app). FUNCTIONAL DOMAINS

HEALTH CONTINUUM ANCHORS

MOOD Normal mood fluctuations and/or Calm & takes things in stride

Irritable / Impatient and/or Nervous and/or Sadness / Overwhelmed

Anger and/or Anxiety and/or Generally sad / Hopeless

Angry outbursts / aggression and/or Excessive anxiety / panic and/or Depressed / suicidal thoughts

ATTITUDE & PERFORMANCE

Good sense of humour and/or Performing well and/or In control mentally

Displaced sarcasm and/or Procrastination and/or Forgetfulness

Negative attitude and/or Poor performance / Workaholic and/or Poor concentration / decisions

Overt insubordination and/or Can’t perform duties, control behaviour or concentrate

SLEEP Normal sleep patterns and/or Few sleep difficulties

Trouble sleeping and/or Disruptive thoughts and/or Nightmares

Restless disturbed sleep and/or Repeated images / nightmares

Can’t fall asleep or stay asleep and/or Sleeping too much or too little

PHYSICAL SYMPTOMS

Physically well and/or Good energy level

Muscle tension / Headaches and/or Low energy

Increased aches and pains and/or Increased fatigue

Physical illnesses and/or Constant fatigue

SOCIAL BEHAVIOUR

Physically and socially active

Decreased activity / socializing

Avoidance and/or withdrawal

Not going out or answering the phone

ALCOHOL & GAMBLING

No/limited alcohol use / gambling

Regular but controlled alcohol use / gambling to cope

Increased alcohol use/gambling – hard to control with negative consequences

Frequent alcohol or gambling use – inability to control with severe consequences

1.2 The R2MR Mobile App

To date, the primary education delivery mode for R2MR has been classroom-based learning. Preliminary research examining the current classroom delivery of R2MR with CAF recruits suggests the slide-based training may prevent attitudes and intentions surrounding mental health service use from becoming more negative over time (Fikretoglu, Liu, & Blackler, 2016). Moreover, CAF recruits who are not exposed to R2MR training self-report significantly more negative intentions towards mental health service use from the beginning to the middle of their basic military training suggesting there may be important protective factors associated with receiving the training (Fikretoglu et al., 2016). Empirical evidence suggests that repeated application and practice of the skills in the training environment improves retention and effectiveness (e.g., Bouchard, Bernier, Boivin, Morin, & Robillard, 2012; Driskell, Copper, & Moran, 1994). Thus, in an effort to complement the CAF’s R2MR mental health and resilience training program, a mobile application (i.e., “app”) was recently developed and launched (i.e., via iTunes and Google Play), which allows CAF personnel, civilians, and our allies to adapt the R2MR instructional content to their particular missions and life situations, making it available anywhere and anytime. The R2MR mobile app aims to provide users with information and strategies, and to reinforce the skills taught in the CAF R2MR

DESCRIPTORS


training (for details, see Granek, Jarmasz, Boland, Guest, & Bailey, 2016). Similar to the classroom-based training, components of the R2MR app were developed based on CBT practices. As is the case in the R2MR course, the app provides various modules such as goal-setting, self-talk, mental rehearsal, tactical breathing, attention control, and working memory which are geared towards building CBT-based personal training scenarios in order to help users achieve their mental health objectives. One of the benefits of the app is that it allows for an assessment of the effectiveness of augmenting the classroom-based training with advanced smartphone features such as location services, gamification, biofeedback and so on. For instance, utilizing features, such as the users’ current location (i.e., GPS), can provide access to nearby mental health resources via the maps and dialling apps, which ultimately may help reduce the stigma- and cost-related barriers associated with remote access to care (Königbauer, Letsch, Doebler, Ebert, & Baumeister, 2017). Another benefit of the app is that it allows for continued practice and on-the-go training as needed. Moreover, the app provides users with reminders to encourage long-term engagement and consistency with R2MR training, with the aim of preparation for stressful life events, both within military and civilian contexts.

As mentioned prior, one of the components of the R2MR app, and the focus of the current research, consists of the MHCM. Consistent with the training in the R2MR curriculum, the MHCM module of the app enables individuals to self-monitor and promotes awareness of their current state of mental health and remote access to resources available by telephone or by navigation. As with the classroom version of the MHCM, in the R2MR app, users rate themselves (i.e., self-assess) on the functional domain(s) of interest: (a) Mood, (b) Attitude and Performance, (c) Sleep, (d) Physical Symptoms, (e) Social Behaviour, and (f) Alcohol and Gambling, by indicating their current subjective state using a slider that represents the four health anchors of the continuum (i.e., healthy, reacting, injured, and ill). Descriptors are presented alongside the four health anchors to aid in selecting the appropriate part of the colour spectrum (i.e., ranging from green to red).

1.3 Current Research1

Although the creation of mental health mobile apps is a rapidly growing area of technological development, research examining the efficacy of mental health interventions to reduce symptomatology through mobile apps is limited (e.g., Firth et al., 2017; Firth, Torous, Nicholas, Carney, Rosenbaum, & Sarris, 2017). Furthermore, research assessing the impact of self-monitoring using mental health mobile apps, as opposed to intervention-based research, is lacking (Rickard, Arjmand, Bakker, & Seabrook, 2016). Thus, the current research represents the first assessment of the validity of the presentation of the MHCM in the R2MR mobile app. No prior validity research has been conducted on the MHCM in general, thus, our results may be broadly applicable to the MHCM, not solely the in-app version of the continuum.

To explore the validity of the MHCM app, the first stage of our assessment involved two sets of self-mapping analyses that explored (i) the extent to which MHCM self-mapping responses are associated with other validated measures of health and well-being in the same or similar domains presented in the MHCM; and (ii) the relation of MHCM self-mapping responses to those validated mental health measures that have established clinical cut-off scores.

1 Portions of the results that are included in this Scientific Report were presented to our client group in two Scientific Letters (see D’Agata, Granek, Nazarov, Boland, & Hendriks, 2018; D’Agata, Nazarov, Granek, & Kwantes, 2018).


Once the initial validity of the MHCM app was established in this way, the second stage of these analyses was devoted to examining aspects of the MHCM and the mobile app itself in more detail. Here we further investigated the validity of the MHCM mobile app by assessing the concordance (i.e., the consistency) of respondents’ self-mapping onto the four mental health categories (i.e., healthy, reacting, injured, ill), and the in-app visual representation (i.e., the spectrum slider) of the continuum. This allowed for an examination of how participants may perceive the colour-coding associated with the MHCM and its effect on the self-mapping process.

We also examined whether self-mapping consistency was related to prior R2MR training and mental health service use. We hypothesized that (i) prior exposure to R2MR training, and (ii) a history of mental health service use would be associated with higher levels of self-awareness and thus would be associated with higher levels of self-mapping consistency.

Finally, in the third section of the analyses, we explored whether there may be issues related to the construction and design of the MHCM itself. More specifically, we examined if participants associate each of the descriptors with its corresponding anchor. For instance, it may be that certain descriptors are consistently interpreted as being associated with the “wrong” anchor. This allows us to better understand if the descriptors associated with each anchor are interpreted as such by our sample. In other words, such a task could provide information surrounding the overall face validity of the design of the MHCM (i.e., each descriptor is rightly matched with each anchor for each domain). In order to assess this, participants also completed a matching task in which their goal was simply to determine which anchor (healthy, reacting, injured, ill) is associated with each MHCM descriptor (e.g., “Decreased activity/socializing” = reacting). We also examined if accuracy was better when each functional domain and its corresponding descriptors were presented one at a time, as opposed to viewing all of the descriptors together as one list. We expected that accuracy would be higher when the descriptors were presented alongside its corresponding functional domain (as is in the mobile app and the R2MR training), supporting the approach that each functional domain should be presented independently.


2 Method

2.1 Participants & Data Collection

We used stratified random sampling in order to acquire a representative sample of the CAF. Invitations to participate were sent to 2367 Regular Force personnel and 826 Reservists. Although 599 individuals consented to participate in our study, 66 were removed for incomplete data, 22 participants were removed because they screened positive for colour blindness, one participant was removed for not being a current member of the CAF, 26 participants were removed for failing three or more validity checks, and 92 Reservists were removed due to insufficient sampling across multiple strata. Thus, our final sample consisted of Regular Force CAF members (N = 392) across elements (i.e., land, air, and, sea) and ranks (i.e., junior Non-Commissioned Members (NCM), senior NCMs, junior officers, and senior officers). Most participants were male, between the ages of 30–39, and had completed post-secondary education (see Table 2), with an average length of service of 12.87 years (SE = .34). Only 9.0% of participants reported that their trade or education was related to mental health; 59.5% of the sample had sought mental health service or treatment, and 54.5% had previously received R2MR training.

Table 2: Sample demographic, mental health, and training information.2

Characteristic Weighted %/na (95% Confidence Interval)

p value for χ2

Sex Male 77.3/49047 (72.3 to 81.6) <0.0001 Female 22.7/14429 (18.4 to 27.7)

Age Under 20 0.5/331 (0.1 to 3.6) <0.0001 20–29 19.8/12554 (15.6 to 24.7) 30–39 42.4/26927 (37.1 to 48.0) 40–49 27.2/17243 (22.8 to 32.0) 50–59 10.1/6419 (7.9 to 12.9)

Educational Attainment Less than post-secondary 24.8/15766 (20.3 to 30.0) <0.0001 Post-secondary or higher 75.2/47709 (70.0 to 79.7)

Trade/Education Relates to Mental Health

Yes 9.0/5692 (6.1 to 13.0) <0.0001 No 91.0/63476 (87.0 to 93.9)

Sought Mental Health Treatment/Service

Yes 59.5/37745 (54.0 to 64.7) <0.001 No 40.5/25731 (35.3 to 46.0)

Received R2MR Training Yes 54.3/34486 (48.9 to 59.7) 0.118 No 45.7/28989.28 (40.3 to 51.1) a Weighted cell counts were rounded to the nearest whole number. Unweighted sample: N = 392.

2 Current rank and current element were used to calculate the weights in our statistical analyses, thus the amounts are not reported; however, it is worth noting that we had participants across all ranks and elements. Unless otherwise footnoted in our results section, analyses were weighted.


2.2 Procedure 2.2.1 Overview

Participants were randomly selected to receive an invitation to participate in our online survey via email, which included details about the study and the link to the survey conducted on the platform Qualtrics3 (Utah, USA). A reminder email was sent approximately two weeks later thanking those who had already participated for their time and requesting participation from those who had not. The survey was live for approximately two months’ time (April 20th to June 26th 2018); however, most individuals participated within the first few days of receiving the email invitation or the email reminder.

Prior to completing the self-mapping task, all participants viewed the instructions and background information on the MHCM that are presented in the mobile application (see Annex A). The instructions provide an overview of the MHCM conceptualizing mental health as existing on a continuum, rather than as a dichotomy (i.e., healthy v. ill). The session then involved the self-mapping task, a set of questionnaires and the matching task, all described in detail below. In addition, the ordering of the tasks within the study varied across groups. For instance, as indicated in Figure 1, the order of the presentation of the validated measures and the self-mapping task was counterbalanced across groups. Moreover, all participants viewed each one of the six functional domains of the MHCM separately and in a randomized order. In addition, for Groups 1 and 2, the self-mapping task had two orders of presentation (described in detail in the next section). Once the self-mapping task and the validated measures were completed, all participants then completed the matching task, which as described later, also had two types of presentations.

3 Qualtrics complies with Government of Canada Regulations regarding the protection of information in that its server is located in Canada and our data was stored on that server.


Figure 1: Visual representation of the ordering of the experimental design based on groups.

2.2.2 MHCM Self-Mapping Task

Participants were randomly assigned to one of four MHCM self-mapping task conditions, varying whether 1) the MHCM colour spectrum was annotated with mental health anchors (e.g., healthy, reacting) and 2) whether the mental health descriptors were presented with the MHCM colour spectrum. Table 3 presents examples of each group’s presentation (see also Annex A for full size versions). Accordingly, Group 1 participants viewed the unannotated MHCM colour spectrum and the mental health

Group 1

MeasuresOrder 1

Order 2

Self-mapping

Measures Self-mappingOrder 1

Order 2

Matching task

Matching task

Group 2

MeasuresOrder 1

Order 2

Self-mapping (with anchors)

Measures Self-mapping (with anchors)

Order 1

Order 2

Matching task

Matching task

Group 3

MeasuresSelf-mapping (as in R2MR)

Measures Self-mapping (as in R2MR)

Matching task

Matching task

Group 4

MeasuresSelf-mapping (as in R2MR with anchors)

MeasuresSelf-mapping (as in R2MR with anchors)

Matching task

Matching task


descriptors were presented separately, while participants in Group 2 were also presented with the mental health descriptors separately from the MHCM colour spectrum which in this case was annotated with the MHCM anchors. As well for Groups 1 and 2, presentation of the MHCM and the descriptors were counterbalanced, so half of Groups 1 and 2 participants saw the MHCM colour spectrum before the mental health descriptors; half of the Groups 1 and 2 participants saw the descriptors before the colour spectrum. As Table 3 also indicates, participants assigned to Group 3 were presented with the unannotated MHCM colour spectrum, but the descriptors were presented next to its corresponding colour on the visual spectrum, which is most similar to how the continuum is presented in the app. Group 4 participants viewed the annotated colour spectrum and the descriptors were presented at the same time as the colour spectrum.

Table 3: Visual representation of the experimental design.

MHCM UNANNOTATED ANNOTATED (with MHCM Anchors)

DES

CR

IPTO

RS

DES

CR

IPTO

R &

SPE

CTR

UM

PR

ESEN

TED

SEP

AR

ATE

LY

(a &

b c

ount

erba

lanc

ed)

a Group 1

Group 2

b

DES

CR

IPTO

R &

SP

ECTR

UM

PR

ESEN

TED

TO

GET

HER

Group 3

Group 4

2.2.3 Matching Task

Figure 2 provides an example of the matching task for the Mood domain (see also Annex B). In this case participants were presented with each one of the descriptors from the MHCM and they were instructed to select which anchor each one corresponded to. In addition, participants either completed the task with each functional domain presented separately with its four descriptors one at a time or combined in which all of the descriptors were presented in one questionnaire in a randomized order without distinguishing between functional domains. As noted earlier, the inclusion of two versions of the task was done in an effort to determine if the descriptors can stand alone or if they should be presented alongside their functional domain in order to improve accuracy.


Figure 2: Example of the matching task for the Mood domain.

2.3 Measures 2.3.1 Derived Variables to Assess Self-Mapping Consistency

In order to assess consistency between descriptor and spectrum self-mapping of the participants assigned to Group 1 and Group 2, a new variable was created such that participants who selected the descriptor (e.g., “Angry outbursts / aggression and/or Excessive anxiety / panic and/or Depressed / suicidal thoughts”) and colour on the spectrum that aligns (i.e., red) received a scored that indicated that they were consistent. Participants who selected a descriptor that did not align with the associated colour on the spectrum received a score of 0 that indicated that they were inconsistent. The spectrum displays four colours (green, yellow, orange, and red), however, the raw spectrum scores ranged from 0 to 100. The spectrum scores were divided into quartiles, such that scores ranging from 0 to 24 were categorized as healthy (i.e., green), 25 to 49 were categorized as reacting (i.e., yellow), 50 to 74 were categorized as injured (i.e., orange), and 75 to 100 were categorized as ill (i.e., red).

Additionally, we examined the frequencies of the degree of inconsistency such that for healthy and ill participants were assigned to low (e.g., participant selected healthy and reacting descriptor), medium (e.g., participant selected healthy and injured descriptor), or high (e.g., participant selected healthy and ill descriptor) inconsistency; for injured and reacting, participants were assigned to either low or medium inconsistency. We also examined the direction of self-mapping inconsistency, such that negative values indicated self-mapping was higher on the visual spectrum and lower for the descriptor (e.g., self-mapping as injured on the spectrum but selecting the healthy descriptor). Positive values indicated the opposite pattern (e.g., self-mapping as healthy on the spectrum but selecting the injured descriptor).


2.3.2 Validity Measures

The selected validated scales differ in their timeframe (e.g., in the past seven days or over the past year); thus, we adjusted the timeframe of each functional domain self-mapping task so that it corresponded with the timeframe of its associated validated scale (e.g., the K10 is based on the past month, so for the MHCM Mood task participants were asked to indicate their mood based on the past 30 days). Healthy is located on the low end of the spectrum, thus all validated scales were scored such that lower scores reflect the “positive” end of the scale (e.g., high work engagement, high social support) and/or low levels of symptomatology (e.g., little to no anxiety or depression symptoms).

2.3.2.1 Mood Domain: Kessler Psychological Distress Scale & The Suicide Behaviors Questionnaire-Revised

To assess the Mood domain, we used the Kessler Psychological Distress Scale (K10; Kessler et al., 2002), a 10-item scale that assesses anxiety and depression. The scale is measured on a 5-point scale ranging, from 1 (none of the time) to 5 (all of the time). Respondents are asked to indicate “over the past 30 days” how often they have experienced various symptoms. Scores of 20 and above are indicative of a depressive and/or an anxiety disorder (Donker et al., 2010). Alternatively, the Australian Bureau of Statistics (2001) recommends using categories: 10–19 (likely well), 20–24 (mild mental disorder), 25–29 (moderate mental disorder), and 30–50 (severe mental disorder). The K10 has been validated with military samples, with a recommended cut-off score between 16 to 18 points (Searle et al., 2015) and it has been validated in a sample of CAF personnel (Blanc, Zamorski, Ivey, McCuaig Edge, & Hill, 2014). In the current study, we ran and describe analyses using both cut-off scores. Based on data from the Canadian Forces Mental Health Survey, the scale demonstrates good predictive value and is considered to be a good screening measure in military populations (Sampasa-Kanyinga, Zamorski, & Colman, 2018). The scale demonstrated good internal consistency in our study (Cronbach’s α = .92).

Suicidal thoughts appear under the ill descriptor on the MHCM within the mood domain, so a brief 4-item measure of suicide ideation was included. The Suicide Behaviors Questionnaire-Revised (SBQ-R; Osman et al., 2001) assesses lifetime suicide ideation and attempt, suicide ideation over the past 12 months, suicide attempt threat, and likelihood of suicidal behaviour in the future. The scale response differs for each item and demonstrates good psychometric properties (Osman et al., 2001). The scale demonstrated adequate internal consistency in our sample (Cronbach’s α = .75).

2.3.2.2 Attitude & Performance Domain: Utrecht Work Engagement Scale

To assess the functional domain of Attitude and Performance, we used the shortened 9-item version of the Utrecht Work Engagement Scale (UWES-9; Schaufeli, Bakker, & Salanova, 2006). The scale assesses three components of work engagement: (a) vigor, (b) dedication, and (c) absorption; however, a total summing of the scale for one overall score is commonly used. The scale is measured on a 7-point scale, ranging from 0 (never) to 6 (always / every day). The scale demonstrates good reliability and construct validity (Seppälä et al., 2009). Additionally, the scale has been used in research with CAF members and demonstrates good internal consistency (Ivey, Blanc, & Mantler, 2015); it demonstrated good internal consistency in our sample as well (Cronbach’s α = .94).


2.3.2.3 Sleep Domain: Sleep Disturbance Subscale, Patient-Reported Outcomes Measurement Information System (PROMIS)

To assess the Sleep functional domain, we used the 8-item sleep disturbance measure developed by the Patient-Reported Outcomes Measurement Information System (PROMIS; for detailed information, see http://www.healthmeasures.net/explore-measurement-systems/promis). The scale assesses sleep quality over the past week on a 5-point scale, ranging from 5 (very poor) to 1 (very good). Additionally, it assesses overall sleep disturbance over the past week on a 5-point scale ranging from 1 (not at all) to 5 (very much). The scale demonstrates good psychometric properties (Yu et al., 2011) and demonstrated good internal consistency in our study (Cronbach’s α = .93).

2.3.2.4 Physical Symptoms Domain: Global Health Scale, Patient-Reported Outcomes Measurement Information System (PROMIS)

For assessing the Physical Symptoms functional domain, we employed the 10-item global health scale from the Patient-Reported Outcomes Measurement Information System (PROMIS). The measure assesses five domains: (a) physical function, (b) fatigue, (c) pain, (d) emotional distress, and (e) social health. The scale uses a 5-point scale, with the final item using a 10-point rating scale to assess the respondents’ “pain on average.” The scale has been validated in community and clinical samples (Hays, Bjorner, Revicki, Spritzer, & Cella, 2009; Cella et al., 2010). The scale demonstrated adequate internal consistency in our sample (Cronbach’s α = .80).

2.3.2.5 Social Behaviour Domain: Multidimensional Scale of Perceived Social Support (MSPSS)

To assess the Social Behaviour functional domain, we included the Multidimensional Scale of Perceived Social Support (MSPSS; Zimet, Dahlem, Zimet, & Farley, 1988), a 12-item scale that assesses perceptions of social support from family, friends, and significant others, using 7-point ratings ranging from 1 (very strongly disagree) to 7 (very strongly agree). The scale has good reliability and adequate construct validity (Zimet et al., 1988), and has been used in military/veteran samples (e.g., Wilcox, 2010). The scale demonstrated good internal consistency in the current sample (Cronbach’s α = .94).

2.3.2.6 Alcohol & Gambling Domain: Alcohol Use Disorders Identification Test (AUDIT) & Problem Gambling Severity Index

To assess the Alcohol and Gambling Domain, we used the Alcohol Use Disorders Identification Test (AUDIT; Allen, Reinert, & Volk, 2001), a 10-item measure that assesses alcohol consumption, dependence, and concerns. Scale scores range between 0 and 40. Scores of 8 or higher are indicative of hazardous and harmful alcohol use for males under the age of 65; scores of 7 or higher are used for females and males over the age of 65 (Reinert & Allen, 2002). The scale has good reliability and criterion validity (Allen, Litten, Fertig, & Barbor, 1997; Degenhardt, Conigrave, Wutze, & Saunders, 2001; Reinert & Allen, 2002). Furthermore, the scale has been tested and validated with a military population (Searle et al., 2015). The scale demonstrated adequate internal consistency in the current study (Cronbach’s α = .77).

Additionally, we used the Problem Gambling Severity Index (PGSI; Ferris & Wynne, 2001), a 9-item subset of items from the Canadian Problem Gambling Inventory (CPGI). The PGSI assesses problem gambling in the general population. The scale uses a 4-point scale ranging from 0 (never) to 3 (almost

http://www.healthmeasures.net/explore-measurement-systems/promis


always) and includes a don’t know response. The scale demonstrates adequate psychometric properties (Ferris & Wynne, 2001). The scale demonstrated excellent internal consistency in the current sample (Cronbach’s α = .91); however, 97.2% of the sample were classified as non-problem gamblers, thus the PGSI was not included in our analyses.

2.4 Secondary Measures 2.4.1 Positive Mental Health

In order to assess if participants who map onto the healthy category of the MHCM also reported positive mental health, we included the Mental Health Continuum (MHC-SF; Keyes, 2005). It is a 14-item measure that assesses well-being over the past month, on a 6-point scale ranging from 1 (never) to 6 (every day). The scale consists of three subscales: emotional well-being, social well-being, and psychological well-being. The internal consistency for each of the subscales is high (Keyes, 2005) and was for each of the subscales in the current sample: emotional (Cronbach’s α = .92), social (Cronbach’s α = .85), and psychological (Cronbach’s α = .87).

2.4.2 Social Desirability & Self-Concealment

Two scales were included in the current study to assess the tendency to alter responses when assessing sensitive information such as mental health. First, the Marlowe-Crowne Social Desirability Scale (MCSDS; Crowne & Marlowe, 1960) assesses truthful responding. The scale consists of 33 items; respondents select true or false for each item and demonstrated adequate internal consistency (Cronbach’s α = .79). Second, the Self-Concealment Scale (SCS; Larson & Chastain, 1990) assesses the tendency to conceal distressing and negative personal information. The SCS is a 10-item scale ranging from 1 (strongly disagree) to 5 (strongly agree). The scale demonstrates good psychometric properties (Cramer & Barry, 1999; Larson & Chastain, 1990) and had good internal consistency in our sample (Cronbach’s α = .91). Although we included the relationship between these two measures in our correlation matrices, additional analyses focused on these traits is outside of the scope of the current report and future work will address it separately.

2.4.3 Demographic, Mental Health & Training Questions

In addition to descriptive demographic information on age, sex, highest level of education, and length of service in years, we included a binary (yes/no) item assessing mental health knowledge or understanding through one’s trade or education. Additionally, we assessed the presence of prior mental health treatment-seeking as those individuals may be more aware of their own mental health symptomatology. The mobile app is intended to be a complimentary tool to R2MR training; therefore, we also included a binary item (yes/no) to assess if the participant received R2MR training in the past. Participants with R2MR training were also asked to indicate how many months ago was their most recent exposure to R2MR (M = 16.63, SE = 1.07).

2.5 Statistical Analysis

All analyses were conducted using IBM’s Statistical Package for the Social Sciences (SPSS) Version 25, with the exception of the polyserial correlations which were conducted using R’s statistical package “polycor.” Whenever possible, analyses were conducted using the Complex Samples feature in SPSS so that our analyses were weighted in order to allow us to generalize our findings to the CAF. Some of our


weighted analyses were conducted using sub-samples. For instance, self-mapping consistency analyses could only be conducted on participants who had been assigned to Groups 1 or 2. Although this may be a limitation of our analyses, it is worth noting that our unweighted results are very similar to our weighted results. Moreover, we chose not to reweight in these cases as this would have introduced additional challenges in conducting our analyses.

We examined bivariate correlations among (i) the validated measures, including the secondary measures, and, (ii) the validated measures and the spectrum scores. We also conducted polyserial correlations between the MHCM ordinal descriptor categories and their associated validated measure. We conducted a series of discriminant function analyses to examine whether the validated measures predicted MHCM self-mapping to the four anchors.4 The rates for correctly classifying participants into the reacting and injured anchors were low, thus we also conducted discriminant function analyses with three categories (i.e., collapsing reacting and injured into one category) to examine if classification rates improved.

A series of two-way contingency table analyses were conducted to examine if participants (i) who have received R2MR training, and/or (ii) had a history of Mental Health Service Use (MHSU), were more consistent in the self-mapping task. We also conducted a series of one-way Analysis of Variance (ANOVAs) to evaluate (i) the relation between spectrum scores and anchors, and (ii) if having received R2MR training compared to not having received training was associated with higher accuracy rates in the matching task for the separated version. The Wald F statistic was reported for weighted analyses based on availability in the Complex Samples feature in SPSS.

To assess matching task accuracy we collapsed across all of the functional domains and calculated mean scores for the four anchors. Additionally, we examined the proportions associated with each of the anchor/descriptor matching responses.

4 Polyserial correlations and discriminant function analyses are unweighted due to software data analysis restrictions.


3 Results

3.1 A Comparison of Subjective Self-Mapping in the MHCM to Validated Measures

3.1.1 Correlation Analyses

Correlations of .75 or higher are believed to indicate the presence of adequate concurrent validity (Kline, 2013). However, it is worth noting that some of the validated measures included in the current study did not necessarily assess all of the constructs in each of the functional domains, thus, we expect that some of the correlations might be lower than .75, but still indicative of the potential presence of concurrent validity. As expected, most of the validated measures were significantly correlated with one another (see Annex C) and all of the validated measures were significantly and positively correlated with their associated functional domain (see Table 4 and Annex C). Results of our main hypotheses were supported, as reflected in the polyserial correlations between the MHCM ordinal descriptor categories and their associated validated measure that were significant and large (see Table 5; see D’Agata et al., 2018 for a more in-depth discussion). 5

Table 4: Bivariate correlations among validated measures and MHCM spectrum scores (n = 364).

Functional Domain Variable r Mood Depression/Anxiety (K10) .716*

Suicide Ideation (SBQ-R) .370* Attitude & Performance Work Engagement (UWES-9) .547* Sleep Sleep Disturbance .751* Physical Symptoms Global Health .690* Social Behaviour Social Support (MSPSS) .469* Alcohol & Gambling Alcohol Use (AUDIT) .576*

Note. K10 = Kessler Psychological Distress Scale; SBQ-R = Suicide Behaviors Questionnaire-Revised; UWES-9 = Utrecht Work Engagement Scale; MSPSS = Multidimensional Scale of Perceived Social Support; AUDIT = Alcohol Use Disorders Identification Test. * p < .01.

Table 5: Polyserial correlations of participants’ functional domain anchors and its corresponding validated scale.

Functional Domain | Variable n Correlation Mood | Depression/Anxiety (K10) 384 .630* Attitude & Performance | Work Engagement (UWES-9) 386 .526* Sleep | Sleep Disturbance 386 .672* Physical Symptoms | Global Health 387 .532* Social Behaviour | Social Support (MSPSS) 388 .433* Alcohol & Gambling | Alcohol Use (AUDIT) 388 .572*

5 All correlation analyses are unweighted due to software data analysis restrictions.


Figure 6: Mean scores on the Sleep Disturbance Short Form by MHCM

Sleep anchors. [Note. Error bars represent standard error.]

Figure 7: Mean scores on the Global Health Short Form by MHCM Physical

Symptoms anchors. [Note. Error bars represent standard error.]

0

8

16

24

32

40

Healthy Reacting Injured Ill

n = 190 n = 78 n = 41 n = 77

Mea

n

0

2

4

6

8

10

12

14

16

18

20


n = 183 n = 117 n = 70 n = 17

Mea

n


Figure 8: Mean scores on the MSPSS by MHCM Social Behaviour

anchors. [Note. Error bars represent standard error.]

3.1.3 Mental Health Screening Scales with Established Cut-Off Scores

Following the recommended cut-off scores for the K10, SBQ-R, and the AUDIT, we examined the proportion of our sample that screened positive for depression/anxiety, suicide ideation, and hazardous and harmful alcohol use, respectively (see Table 6). The remaining validated measures did not have recommended cut-off scores, thus we examined those measures using alternative analyses, presented in the next section.

Table 6: Percentage of participants who screened positive on the K10, SBQ-R, and AUDIT.

Variable Unweighted

n Weighted %/n

a (95% Confidence

Interval) Depression/Anxiety 389 36.8/23202 (31.5 to 42.3) Suicide Ideation 390 15.0/9519 (11.4 to 19.7) Hazardous & Harmful Alcohol Use 391 18.3/11596 (14.3 to 23.2) a Weighted cell counts were rounded to the nearest whole number.

Similar to the percentage of participants who screened negative for depression and/or anxiety, 59.4% of participants self-assessed as healthy on the Mood domain for the MHCM task. However, for the participants who screened positive for depression and/or anxiety,7 23.1% self-assessed as healthy, 48.5% as reacting, 20.9% as injured, and 7.5% as ill (see Figure 9). Additionally, the K10 provides four categories using cut-off scores: likely well (10–19), mild (20–24), moderate (25–29), and severe (30–50), thus we also examined the percentage of participants who scored within each cut-off category by the

7 The weighted analyses could not be accurately interpreted; thus in this case, frequencies could only be interpreted at the sample level.

7

18

29

40

51

62

73

84


n = 218 n = 119 n = 46 n = 5

Mea

n


MHCM self-mapping for the Mood domain (see Figure 10). Most participants who fell into the moderate and severe categories self-mapped as reacting, as opposed to injured and ill, respectively.

Of the participants who screened positive for suicide ideation in the past year, 38.9% self-assessed as healthy, 35.2% as reacting, 11.1% and 14.8% as injured and ill, respectively (see Figure 11).

Fewer participants (72.9%) self-mapped as healthy on the Alcohol and Gambling domain compared to the proportion of participants who screened negative on the AUDIT (81.7%). Of the participants who screened positive on the AUDIT, 18.8% self-assessed as healthy, 73.4% as reacting, 6.3% as injured, and 1.6% as ill (see Figure 12).

Figure 9: MHCM Mood anchor self-mapping among participants

who screened positive for depression/anxiety (n = 138).

23.1%

48.5%

20.9%

7.5%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Per

cen

tage


Figure 10: MHCM Mood anchor self-mapping by K10 categories (n = 389).

Figure 11: MHCM Mood anchor self-mapping among participants

who screened positive for suicide ideation (n = 54).

51.3%

6.8%

0.5% 0.8%

11.5%

7.6% 4.9% 4.4%

2.1% 2.9% 1.6% 2.9% 0.3% 0.5% 0.8% 1.3%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Likely well Mild Moderate Severe

Per

cen

tage

K10 Categories

Healthy

Reacting

Injured

Ill

38.9% 35.2%

11.1% 14.8%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Per

cen

tage


Figure 12: MHCM Alcohol & Gambling anchor self-mapping among participants

who screened positive for hazardous and harmful alcohol use (n = 64).

3.1.4 Predicted Group Membership of Validated Measures

The validated measures significantly predicted MHCM self-mapping to the four anchors (see Table 7).8 The rates for correctly classifying participants’ functional domain anchors based on its corresponding validated scale(s) ranged between 58.1% to 79.9%. The rates for correctly classifying participants into the reacting (14.1% to 58.1%) and injured (0.0% to 17.4%) anchors were low, thus we conducted discriminant function analyses with three categories (i.e., collapsing reacting and injured into one category) to examine if classification rates improved (see Table 8). The correct classification rates improved, with a range of 66.3% to 82.2%.

Table 7: Correct classification of participants’ functional domain anchors based on its corresponding validated scale(s) for 4 anchors.

Functional Domain | Variable Wilks’ Λ df, N χ2 Classification % Mood | Depression/Anxiety & Suicide Ideation (K10 & SBQ-R)

.57 6, 383 216.12* 67.6

Attitude & Performance | Work Engagement (UWES-9) .77 3, 386 101.03* 71.8 Sleep | Sleep Disturbance .53 3, 386 240.09* 58.5 Physical Symptoms | Global Health .71 3, 387 133.65* 58.1 Social Behaviour | Social Support (MSPSS) .83 3, 388 70.13* 58.2 Alcohol & Gambling | Alcohol Use (AUDIT) .68 3, 388 148.90* 79.9

8 Discriminant function analyses are unweighted due to software data analysis restrictions.

18.8%

73.4%

6.3% 1.6%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Per

cen

tage



Table 8: Correct classification of participants’ functional domain anchors based on its corresponding validated scale(s) for 3 anchors.

Functional Domain | Variable Wilks’ Λ df, N χ2 Classification % Mood | Depression/Anxiety & Suicide Ideation (K10 & SBQ-R)

.58 4, 383 208.13* 74.2

Attitude & Performance | Work Engagement (UWES-9) .78 2, 386 97.47* 73.6 Sleep | Sleep Disturbance .53 2, 386 240.15* 66.3 Physical Symptoms | Global Health .71 2, 387 133.65* 69.3 Social Behaviour | Social Support (MSPSS) .87 2, 388 54.40* 66.0 Alcohol & Gambling | Alcohol Use (AUDIT) .70 2, 388 138.11* 82.2


3.2 The Role of Subjective Self-Mapping in the MHCM 3.2.1 Self-Mapping Consistency

Having provided overall support for the validity of the MHCM self-mapping approach relative to validated measures, we now provide an overview of the general state of our participants by examining the proportion of participants who self-mapped onto each of the four mental health categories (see Table 9). In general, approximately half of the participants self-mapped onto healthy for each domain, with Alcohol and Gambling having the highest proportion (72.9%). Approximately 25–30% of participants self-mapped as reacting, ~10% as injured, and very few participants (less than 5%, with the exception of the Sleep domain) self-mapped as ill.

Table 9: Percentage of participants who self-mapped onto each of the four mental health categories.

Functional Domain n Healthy % Reacting % Injured % Ill % Mood 384 59.4 28.4 9.4 2.9 Attitude & Performance 386 67.4 25.4 6.5 0.8 Sleep 386 49.2 20.2 10.6 19.9 Physical Symptoms 387 47.3 30.2 18.1 4.4 Social Behaviour 388 56.2 30.7 11.9 1.2 Alcohol & Gambling 388 72.9 25.3 1.5 0.3

We then examined the proportion of participants whose visual spectrum anchor was consistent with the mental health descriptor they selected to describe themselves for each of the functional domains (see Table 10). As Table 10 indicates, consistency rates ranged between 54.7% (Sleep) to 83.1% (Alcohol and Gambling).


Table 10: Percentage of participants with consistent descriptor and visual spectrum self-mapping.

Functional Domain Unweighted n Weighted %/na (95% Confidence Interval)

Mood 257 68.1/27882 (61.2 to 74.2)* Attitude & Performance 258 77.2/31914 (70.8 to 82.5)* Sleep 259 54.7/22654 (47.7 to 61.4) Physical Symptoms 257 62.4/25634 (55.6 to 68.7)* Social Behaviour 257 68.0/27878 (61.2 to 74.1)* Alcohol & Gambling 254 83.1/33725 (77.6 to 87.4)* a Weighted cell counts were rounded to the nearest whole number. * p value for χ2 < .001

Nonetheless, some inconsistencies between MHCM anchor and descriptor selections were evident. For the Mood domain,9 of the participants who were inconsistent, 88.6% were classified as low (e.g., participant selected healthy anchor and reacting descriptor), 11.4% as medium (e.g., participant selected healthy anchor and injured descriptor), and 0.0% as high (e.g., participant selected healthy anchor and ill descriptor). For Attitude and Performance, 91.2% were classified as low, 8.8% as medium, and 0.0% as high. For the Sleep domain, 54.8% were classified as low, 36.5% as medium, and 8.7% as high. For the Physical Symptoms domain, 65.0% were classified as low, 34.0% were classified as medium, and 1.0% as high. For the Social Behaviour domain, 88.8% were classified as low, 11.2% as medium, and 0.0% as high. For the Alcohol and Gambling domain, 95.8% were classified as low, 0.0% as medium, and 4.2% as high. We then assessed the direction of the overall inconsistencies for each functional domain to better determine the magnitude and the specific nature of the inconsistencies between descriptors and anchors. For instance, to examine if particular patterns of inconsistencies arise, such as whether or not participants typically rate themselves higher on the spectrum or the descriptors. As Table 11 indicates, the degree of inconsistency between the selected anchor and descriptor was low. For Mood and Attitude and Performance, overall participants self-mapped higher on the visual spectrum and lower on the descriptors. The opposite pattern emerged for the four remaining functional domains.

Table 11: Mean scores for inconsistent self-mapping by functional domain.

Functional Domain n M S.D. Mood 79 -0.56 1.02 Attitude & Performance 57 -1.07 0.26 Sleep 115 0.72 1.51 Physical Symptoms 100 0.70 1.28 Social Behaviour 80 0.22 1.14 Alcohol & Gambling 47 0.83 0.70

3.2.1.1 R2MR Training

As noted in the introduction, one of the assumed benefits of R2MR training, is greater self-awareness with respect to mental health symptoms. Thus, we expected that prior R2MR training would be associated with greater awareness with the model itself and thus less self-mapping inconsistency. However, our results revealed that prior R2MR training was generally unrelated to inconsistency. Specifically, self-mapping

9 Reported proportions are unweighted.


inconsistency and prior R2MR training were unrelated in the Mood (Pearson χ2(1, N = 257) = .28, p = .6), Attitude and Performance (Pearson χ2(1, N = 258) = 1.76, p = .2), Sleep (Pearson χ2(1, N = 259) = .12, p = .8), Physical Symptoms (Pearson χ2(1, N = 257) = 1.41, p = .3), and Social Behaviour (Pearson χ2(1, N = 257) = 2.42, p = .157) domains. However, for the Alcohol and Gambling domain, self-mapping inconsistency and R2MR training were significantly related, Pearson χ2(1, N = 254) = 4.84, p = .036; participants who had previously received R2MR training were more consistent in their self-mapping compared to those without the training (87.9% vs 77.5%).

3.2.1.2 Mental Health Service Use

Similar to the rationale stated above, we expected that mental health service use would be associated with greater consistency between anchors and descriptors because individuals who accessed mental health services recognized that they were in need of assistance and/or they had developed greater self-awareness of their symptoms (i.e., akin to the descriptors) and their overall mental health status. However, our results did not support this hypothesis. Instead, individuals who indicated no history of MHSU were more consistent in their self-mapping in comparison to those with MHSU for the Mood (Pearson χ2(1, N = 257) = 6.61, p = .030, (77.1% vs 61.9%)), Attitude and Performance, (Pearson χ2(1, N = 258) = 12.11, p = .004, (88.3% vs 69.7%)), Sleep (Pearson χ2(1, N = 259) = 6.67, p = .025, (64.4% vs 48.1%)), and Physical Symptoms (Pearson χ2(1, N = 257) = 21.63, p < .001 (79.7% vs 51.0%)) domains. MHSU was unrelated to self-mapping inconsistency in the Social Behaviour (Pearson χ2(1, N = 257) = 1.48, p = .3) or Alcohol and Gambling (Pearson χ2(1, N = 254) = .000, p = .9) domains.

3.2.2 MHCM Visual Spectrum Scores as a Function of Embedding Text Anchors

To see whether or not the anchors play a role in calibrating how participants’ self-mapped, we compared self-mapping between participants who had anchors to help them make a judgment with those who did not. Approximately half of the participants (48.2%) were presented with the MHCM visual spectrum containing anchors (healthy, reacting, injured, and ill); the remaining participants were presented with a visual spectrum without the anchors. Those who mapped themselves onto the spectrum containing the anchors were more likely to rate themselves higher on the spectrum for Mood (Wald F(1, 375) = 6.16, p = .014), Attitude and Performance (Wald F(1, 378) = 21.15, p < .001), Social Behaviour (Wald F(1, 378) = 12.41, p < .001), and Alcohol and Gambling (Wald F(1, 374) = 8.01, p = .005) (see Figure 13) compared to those who did not see anchors. Interestingly, no statistically significant differences were found for the Sleep (Wald F(1, 378) = .002, p = .9) and Physical Symptoms (Wald F(1, 376) = 2.50, p = .12) domains—the only two domains that are not primarily psychological in nature.

Spec

trum

Sco

res


Figure 14: Mean scores for accuracy on the matching task (N = 391).

[Note. Error bars represent standard error]

The proportion of participants who correctly matched each descriptor with their healthy, reacting, injured, and ill anchors was higher for participants who completed the separated version of the matching task compared to participants who completed the combined version, providing support for our hypothesis (see Tables 12 and 13). There were a few notable exceptions to this pattern that emerged in the Sleep and Physical Symptoms domains. First, accuracy proportions were higher for the Sleep domain for reacting and injured for participants who completed the combined version of the matching task. Second, accuracy proportions for both versions were the same for Physical Symptoms for reacting. Next we provide an overview of the range of accuracy proportions across domains and anchors. For healthy, accuracy proportions ranged from 70.8% to 87.9% (separated version) and 62.4% to 82.7% (combined version) across functional domains; Sleep and Attitude and Performance were the least and most accurate domains, respectively in both versions. For reacting, accuracy proportions ranged from 32.8% to 57.8% (separated version) and 31.1% to 52.3% (combined version); the least and most accurate domains varied across versions. For injured, accuracy proportions ranged from 23.3% to 39.6% (separated version) and 26.1% to 35.8% (combined version); again, the least accurate domain varied across versions, however the highest accuracy occurred for the Mood domain in both versions. For ill, accuracy proportions ranged from 22.2% to 45.3% (separated version) and 14.2% to 38.0%; accuracy proportions for the Sleep and Mood domains were the lowest and highest, respectively in both versions.

Table 12: Percentages of frequencies for the separated version of the matching task.

Healthy % Reacting % Injured % Ill % Mood 80.7 54.2 38.5 44.8 Attitude & Performance 86.5 55.2 40.6 43.8 Sleep 70.8 29.7 24.6 21.0

0

1

2

3

4

5

6


Mea

n


Healthy % Reacting % Injured % Ill % Physical Symptoms 75.1 52.3 35.2 40.4 Social Behaviour 75.3 57.2 32.5 32.0 Alcohol & Gambling 85.5 44.6 39.4 44.6

Table 13: Percentages of frequencies for the combined version of the matching task.

Healthy % Reacting % Injured % Ill % Mood 73.7 32.4 35.7 39.5 Attitude & Performance 83.9 47.6 31.4 31.2 Sleep 64.3 34.4 30.8 14.5 Physical Symptoms 74.6 50.8 28.0 26.9 Social Behaviour 75.8 51.1 23.1 19.4 Alcohol & Gambling 81.2 41.4 30.8 31.9

For completeness, we also assessed the distribution of the anchor/descriptor pairings in the matching task for each functional domain separately.10 Here we present the results for the separated version of the matching task only—this is reflective of how the MHCM is presented in R2MR training as well as the app, thus these findings are more informative than the combined version of the task. For the Mood domain, accuracy was highest for healthy (80.7%) and decreased by over 25% for reacting, injured and ill (see Figure 15). Notably, 28.1% of participants selected healthy for the ill descriptor. Finally, despite having an equal number of descriptor items associated with each anchor in the matching task, the frequency of response rates was not equal across the four anchors (χ

2(3, N = 192) = 14.32, p < .001); healthy was selected most frequently (39.1%), followed by reacting (28.0%), injured (17.4%), and ill (15.4%).11

For the Attitude and Performance domain, accuracy was the highest for healthy (86.5%) and decreased by over 30% for reacting, injured and ill (see Figure 16). A similar pattern to the Mood domain emerged such that 32.8% of participants selected healthy for the ill descriptor. Finally, the frequency of response rates was not equal across the four anchors (χ

2(3, N = 192) = 24.13, p < .001); healthy was selected most frequently (45.5%), followed by reacting (23.8%), injured (15.9%), and ill (15.0%).

For the Sleep domain, accuracy was the highest for healthy (70.8%) and decreased by over 40% for reacting, injured and ill (see Figure 17). Error rates were high such that similar proportions of descriptors emerged for reacting and injured. Additionally, the most frequently selected anchor for the ill descriptor was reacting. Finally, the frequency of response rates for each anchor varied (χ

2(3, N = 195) = 9.26, p = .026); healthy was selected most frequently (36.1%), followed by reacting (30.5%), injured (20.1%), and ill (16.9%).

For the Physical Symptoms domain, accuracy was the highest for healthy (75.1%) and decreased by over 20% for reacting, injured and ill (see Figure 18). The frequency of response rates for each anchor varied (χ


For the Social Behaviour domain, accuracy was highest for healthy (75.3%) and decreased by over 15% for reacting, injured, and ill (see Figure 19). Similar to some of the other functional domains, 27.3% of

10 Coefficient of variation was high for some cells, thus unweighted proportions are presented here. 11 Percentages exclude ‘none of the options’ in the matching task.


participants selected healthy for the ill descriptor. Finally, the frequency of response rates for each anchor varied (χ


For the Alcohol and Gambling domain, accuracy was the highest for healthy (85.5%) and decreased by over 40% for reacting, injured and ill (see Figure 20). For the reacting, injured, and ill descriptors, 33.7%, 28.5%, and 29.5%, respectively, were classified as healthy. Finally, the frequency of response rates for each anchor varied (χ2(3, N = 193) = 32.79, p < .001); healthy was selected most frequently (48.4%), followed by reacting (14.9%), injured (14.7%), and ill (18.7%).

Figure 15: Separated matching task (n = 192) for MHCM Mood descriptors by anchors. [Note.

Percentages outlined with a box represent the proportion of the accurately selected anchor for its corresponding descriptor.]

80.7%

17.2% 17.7%

28.1%

13.0%

54.2%

27.1%

8.9%

2.6%

16.1%

38.5%

6.8%

0.5%

4.7% 6.8%

44.8%

3.1%

7.8% 9.9%

11.5%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Healthy descriptor Reacting descriptor Injured descriptor Ill descriptor

Per

cent

age Healthy

Reacting

Injured

Ill

None


Figure 16: Separated matching task (n = 192) for MHCM Attitude & Performance descriptors by

anchors. [Note. Percentages outlined with a box represent the proportion of the accurately selected anchor for its corresponding descriptor.]

Figure 17: Separated matching task (n = 195) for MHCM Sleep descriptors by anchors. [Note.

Percentages outlined with a box represent the proportion of the accurately selected anchor for its corresponding descriptor.]

86.5%

22.9% 25.0%

32.8%

8.3%

55.2%

18.8%

5.2%

1.6%

8.3%

40.6%

7.8%

0.5%

4.7% 6.3%

43.8%

3.1%

8.9% 9.4% 10.4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Per

cent

age

Healthy

Reacting

Injured

Ill

None

70.8%

21.0% 21.0% 17.9%

20.0%

29.7%

23.6%

36.9%

2.6%

28.2%

24.6%

17.4%

3.6%

13.8%

22.6% 21.0%

3.1%

7.2% 8.2% 6.7%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Per

cent

age

Healthy

Reacting

Injured

Ill

None


Figure 18: Separated matching task (n = 193) for MHCM Physical Symptoms descriptors by


Figure 19: Separated matching task (n = 194) for MHCM Social Behaviour descriptors by


75.1%

17.1% 16.6%

23.8%

15.5%

52.3%

36.8%

15.0%

2.6%

20.7%

35.2%

13.5%

1.6% 4.1%

5.7%

40.4%

5.2% 5.7% 5.7% 7.3%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Per

cent

age

Healthy

Reacting

Injured

Ill

None

75.3%

20.1% 21.6%

27.3%

16.0%

57.2%

18.6%

11.9%

2.1%

10.8%

32.5%

15.5%

1.0% 3.6%

16.5%

32.0%

5.7% 8.2%

10.8% 13.4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Per

cent

age

Healthy

Reacting

Injured

Ill

None


Figure 20: Separated matching task (n = 193) for MHCM Alcohol & Gambling descriptors

by anchors. [Note. Percentages outlined with a box represent the proportion of the accurately selected anchor for its corresponding descriptor.]

A substantial proportion of the sample inaccurately selected healthy for the ill descriptor across the functional domains (17.9% to 32.8%). Thus, in order to ascertain whether this response pattern was driven by the same participants across functional domains, a total score for the matching task was calculated for each participant (aggregating across functional domains). The variation associated with the healthy-ill descriptor inaccuracy indicates it was not the same group of participants who continually mismatched (see Table 14). Additionally, 46.8% of the sample mismatched at least one time.

Table 14: Percentage of participants (n = 188) who selected “healthy” for the ill descriptor 0 to 6 times.

Number of Errors % 0 53.2 1 10.1 2 6.4 3 4.3 4 11.2 5 5.9 6 9.0

We calculated an overall accuracy score for the matching task across functional domains, yielding a possible range of scores between 0 and 24 (6 functional domains by 4 descriptors). We then explored the degree to which R2MR training exposure affected matching accuracy. Indeed, participants who had received R2MR training had significantly higher accuracy scores (M = 13.04, SE = .68) compared to participants who had not received R2MR training (M = 10.35, SE = .81; Wald F(1, 183) = 6.52, p = .011).

85.5%

33.7%

28.5% 29.5%

2.6%

44.6%

6.2%

1.0% 1.0%

6.7%

39.4%

6.7%

1.0% 0.0%

7.8%

44.6%

9.8%

15.0% 18.1% 18.1%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Per

cent

age

Healthy

Reacting

Injured

Ill

None


4 Discussion

This research constitutes the first assessment of the validity of the MHCM, specifically as it is presented in the recently developed R2MR app. We sought to determine the validity of the MHCM assessment tool, and its presentation in the R2MR app in several ways. First, in the self-mapping task, participants were asked to select the descriptor and place themselves on the visual spectrum for each functional domain. Here we examined how the MHCM descriptors and the visual spectrum (with and without anchors) self-mapping on each functional domain related to corresponding validated measures to establish its initial concurrent validity. Second, we assessed the extent to which the current wording of the MHCM descriptors in each of the six functional domains corresponded to the four anchors as well as the in-app visual representation of the continuum. Third, after the self-mapping task, participants completed a matching task in which participants were presented with the MHCM descriptors and had to indicate which anchor corresponded to each descriptor. The goal here was to examine the extent to which participants agree with the current way in which each descriptor has been paired to its corresponding anchor. As discussed below, overall, our results provide some reasonable initial support for the validity of the MHCM self-assessment tool. First, there was a fair amount of agreement between the MHCM self-mapping task and the validated measures. Second, for most of the functional domains, a large proportion of participants’ descriptor and the MHCM visual continuum spectrum self-mapping selections were consistent. For those participants who were inconsistent in this respect, they typically self-mapped onto the adjacent descriptor. Third, the results of our matching task indicated that accuracy was high for correctly identifying the descriptors associated with healthy. However, it is worth noting that some of our findings indicated some inconsistencies in self-mapping, the matching task, and the validated measures suggesting that modifications to the content of the MHCM may improve individuals’ ability to accurately self-assess their mental health status.

4.1 A Comparison of Subjective Self-Mapping in the MHCM to Validated Measures

4.1.1 Correlation Analyses

With respect to our first goal of assessing the concurrent validity of the MHCM domains, overall the correlational analyses revealed that all of the validated measures were significantly associated with their corresponding functional domain and spectrum scores. As mentioned prior, correlations of .75 or higher are believed to indicate the presence of adequate concurrent validity (Kline, 2013). Given that some of the validated measures included in the current study did not necessarily assess all of the constructs in each of the functional domains we expected that correlations below this threshold may still be indicative of the presence of concurrent validity. The strong connections between the functional domain anchors and the corresponding validated measures also indicated the potential presence of adequate concurrent validity of the MHCM anchors. However, these results should be interpreted with caution due to the use of polyserial correlations to assess concurrent validity (which do not have clear threshold guidelines for the adequacy of concurrent validity) and the reliance of using only one scale to assess each corresponding functional domain. Further research is required to establish strong evidence for the degree of concurrent validity of the MHCM self-mapping tool. Moreover, establishing the validity of a measure also involves assessing divergent and predictive validity, which should be examined in future research.


4.1.2 Mean Scores & MHCM Anchors

Across functional domains, most participants self-mapped as healthy and, with the exception of the Sleep domain, few participants self-mapped as ill. Additionally, the mean scores of the validated measures for each MHCM anchor, as expected, generally increased at each progressively symptomatic anchor across domains. This parallels the findings of the correlation analyses, suggesting that self-reported symptomatology on the validated measures corresponded to the MHCM anchors accordingly.

4.1.3 Mental Health Screening Scales with Established Cut-Off Scores

For the validated scales with recommended cut-off scores, we examined the proportion of our sample that screened positive for depression/anxiety, suicide ideation, and hazardous and harmful alcohol use. Over one-third of participants screened positive for depression/anxiety according to the recommended cut-off scores associated with the K10. However, of those participants, nearly one-fourth self-mapped as healthy on the Mood domain of the MHCM app. Moreover, approximately 15% of participants screened positive for suicide ideation on the SBQ-R, and nearly 40% of those participants self-mapped as healthy on the MHCM Mood domain. Finally, nearly one-fifth of participants screened positive for hazardous and harmful alcohol use according to the cut-off scores of the AUDIT, and of those, nearly one-fifth self-mapped as healthy on the Alcohol and Gambling domain. Although most participants who screened positive on these validated measures are self-mapping outside of healthy on the MHCM, that there is still a rate of false negatives in some cases warrants concern regarding the MHCM tool, and at a minimum additional studies need to confirm these findings.

4.1.4 Predicted Group Membership

The validated measures were able to differentiate among the four MHCM anchors across functional domains. Our results indicated that, in general, the validated measures correctly classified group membership to the four anchors fairly well. Overall, predicted group membership was high across domains for healthy. However, when examining the predicted group membership proportions by each anchor, it was evident that correct classification was lower for reacting, injured, as well as ill. Thus, we examined predicted group membership collapsing the two anchors into one, and classification rates improved across domains, such that reacting and injured may have some overlap in perceived symptom severity. These results suggest that participants may have difficulty distinguishing between symptom severity on the MHCM, and this is particularly true for distinguishing between reacting and injured. Refinements that better distinguish between symptom severity across the reacting and injured descriptors may improve upon the accuracy of the self-mapping process. Alternatively, it may be that as a self-assessment tool (rather than a clinical diagnostic tool), it may be sufficient to collapse across the reacting and injured categories—although at this point this remains a question for future research to pursue.

4.2 The Role of Subjective Self-Mapping in the MHCM 4.2.1 Self-Mapping Consistency

Overall, the degree to which participants were inconsistent was generally low, such that for most domains, participants were generally self-mapping on the descriptor parallel to its associated anchor. However, it is also worth noting that our results suggest that certain descriptors did not align with the visual spectrum and this may be especially true for certain domains. For instance, participants displayed higher inconsistency for certain domains (e.g., Sleep) compared to others (e.g., Alcohol & Gambling),


suggesting that distinguishing symptom severity for some domains is more challenging compared to others. In addition, our results indicate that participants, overall, self-mapped higher on the visual spectrum and lower on the descriptors for the Mood and Attitude and Performance domains. However, the opposite pattern emerged for Sleep, Physical Symptoms, Social Behaviour, and Alcohol and Gambling, such that participants, overall, self-mapped higher on the descriptors and lower on the visual spectrum.

4.2.1.1 R2MR Training & Mental Health Service Use

We also sought to determine the relationship between self-mapping inconsistency and prior R2MR training and past history of mental health service use. Results of these analyses showed that overall, having received R2MR training did not reduce self-mapping inconsistency. Specifically, inconsistency in self-mapping onto anchors and associated descriptors was not associated with prior R2MR training, with the exception of the Alcohol and Gambling domain where participants who had received training were less inconsistent in their self-mapping compared to those who had not received training. Interestingly, our results indicated that the proportion of participants who were inconsistent was higher for those who had a history of MHSU, compared to those who did not, for the Mood, Attitude and Performance, Sleep, and Physical Symptom domains. We expected the opposite pattern to emerge; thus, perhaps future research should examine if time since MHSU impacts consistency.

4.2.2 MHCM Visual Spectrum Scores as a Function of Embedding Text Anchors

Participants assigned to the groups where the visual spectrum was presented with the four MHCM text anchors had higher spectrum scores on four of the functional domains (Mood, Attitude & Performance, Social Behaviour, and Alcohol & Gambling) compared to participants who were presented with the visual spectrum without the anchors. In the current research it was not possible to ascertain whether anchors are associated with an underestimation or overestimation, and ultimately, which version of the spectrum is associated with more accurate self-mapping. Thus, our results suggest that future research should assess the direction of the relation in order to determine the most appropriate visual presentation of the spectrum.

4.3 Matching Task Accuracy

We also had participants complete a separate matching task in order to assess the percentage of participants who accurately selected the corresponding anchor for each descriptor, as is the case in the current MHCM. Several notable findings emerged from the matching task. First, accuracy was higher for participants who completed the separated rather than the combined version of the matching task. This suggests that the task was easier when each functional domain was presented on its own. This aligns with the current presentation in the mobile app, such that participants view each functional domain and its descriptor and spectrum individually; our findings lend support to this approach. Second, matching accuracy was high for the healthy descriptors, but decreased noticeably for reacting, injured, and ill, suggesting participants’ ability to distinguish among symptom severity was weakened on the matching task. This aligns with the findings that emerged in the self-mapping task. Moreover, this finding suggests some of the inconsistencies may be a function of the descriptors themselves, particularly because even though the task involved simply matching the descriptors to their anchor (i.e., being accurate) and did not involve revealing their own sympotomatology, accuracy did not dominate the task. Third, the healthy anchor was selected most often across functional domains. This suggests that participants were resistant to select the more severe anchors, such as injured or ill. Alternatively, it may be that many participants do not perceive the injured or ill descriptors as being sufficiently severe to warrant being labelled as such.


Fourth, approximately one-third of the participants selected the healthy anchor for the ill descriptor across most of the functional domains. Moreover, follow-up analyses indicated that it was not a specific set of participants who were consistently making the error, rather, nearly half of the sample made the inaccuracy at least one time. Altogether, this finding suggests that future research aimed at modifications to some of the descriptors may improve upon the ability to distinguish symptom severity across the descriptors associated with each functional domain. Finally, accuracy on the matching task was higher for participants who had received R2MR training compared to their counterparts who had not yet received the training. This finding indicates that previous exposure to the R2MR content (including the MHCM) may have aided in the matching task. However, it is worth noting, that R2MR training was not associated with higher consistency in self-mapping. This suggests that although the familiarity with the continuum may have aided in the matching task—which did not involve considering their own current state, when the task involved their own current self-assessment, it did not aid in the process. Considerations of how to improve the understanding of the MHCM so that it also improves self-mapping as well is warranted.

4.4 Limitations & Future Directions

Although our study provided a useful approach and assessment of the current state and the initial tests of the validity of the MHCM, there were some limitations associated with this research. First, our mental health screening results should be interpreted with caution because our validated measures are not replacements for clinical assessments, thus, not all of the participants who screened positive would necessarily meet clinical diagnoses for depression/anxiety, suicide ideation, or hazardous and harmful alcohol use. Moreover, due to the self-report nature of the mental health screeners, some of the participants who did not screen positive on the established measures may meet the criteria for a clinical diagnosis. Furthermore, the comparison of the suicide ideation and the Mood domain self-mapping should be interpreted with caution because responses to the suicide ideation measure referred to the past year, whereas the depression/anxiety measure and the Mood domain referred to the past 30 days. Unfortunately, it was not possible to modify the timeframe of the suicide ideation measure, thus it is likely that some of the participants who screened positive for suicide ideation but self-mapped as healthy on the Mood domain were experiencing suicide ideation sometime within the past year, but have not experienced symptoms within the past 30 days. Additionally, research indicates that not all individuals who attempt suicide or die by suicide meet the criteria for a clinical diagnosis (Campos et al., 2016). Future research could include clinical interviews in order to more accurately compare MHCM self-mapping to clinical assessments, albeit this would be most pertinent for understanding the mapping for the ill anchor as opposed to the reacting or injured anchors.

Second, some of our validated measures did not have recommended cut-off scores (i.e., UWES-9, Sleep Disturbance, Global Health, MSPSS). Some of our measures were taken from the social psychology literature, for which it is uncommon to use cut-off scores. Thus, we were unable to assess the proportion of participants for each MHCM anchor who “screened positive” on each functional domain. A potential approach to conduct follow-up analyses on our current study would involve obtaining recommended cut-off score categories that align with the four MHCM anchors for each of the validated measures. Such an approach could involve contacting the authors of the measures as well as consulting with subject matter experts from each functional domain to advise on appropriate thresholds. Such cut-off scores would be used to run additional analyses with our existing data set or could be used in future research, not as a clinical assessment, but rather as a way of conducting more fine-grained analyses that compare each of the validated measures to the four MHCM anchors for each functional domain.


Third, individuals were invited to participate in our study using stratified random sampling, however, it is unknown if there are significant differences between individuals who chose to participate compared to those who did not. There may be some degree of self-selection bias to participate in a study on mental health, thus, employing different sampling techniques in future research may be worthwhile.

Fourth, due to insufficient sampling as well as the difficulty in reaching Reservists for study participation, we could not include Reservists in our analyses. Our goal was to obtain a sample that was representative of the entire CAF, not only Regular Force members. Our expected participation rate for Reservists was overestimated, thus, increasing the amount of invitations in future stratified random sampling should improve representation.

4.5 Conclusions

Overall, we found reasonable initial support for the presence of adequate concurrent validity between the MHCM functional domains and associated established measures. The current research also found some support for MHCM self-mapping in its current form such that the rates of self-mapping consistency on descriptor and its associated anchor were relatively high for certain functional domains. However, our findings also indicate that some descriptors do not accurately reflect how participants self-map on the visual spectrum and that, at times, participants’ ability to distinguish symptom severity between descriptors was restricted, particularly for reacting and injured. As mentioned prior, the inclusion of results pertaining to the role of social desirability and self-concealment are outside of the scope of the current report and will be examined separately, however, it is worth noting that our preliminary work suggests that individual differences may also provide insight into some of the patterns that emerged in the current research. As promising as these results are in terms of the initial validity of the MHCM model and its use as a tool as represented in the app, future research should focus on better understanding how and why self-mapping discrepancies occur; this will contribute to improved presentation, content, and utility of the MHCM both in the mobile app as well as within the R2MR training program for the CAF.


References

Allen, J. P., Litten, R. Z., Fertig, J. B., & Babor, T. (1997). A review of research on the Alcohol Use Disorders Identification Checklist (AUDIT). Alcoholism: Clinical and Experimental Research, 21, 613–619.

Allen, J. P., Reinert, D. F., & Volk, R. J. (2001). The Alcohol Use Disorders Identification Test: An aid to recognition of alcohol problems in primary care patients. Preventive Medicine, 33, 428–433.

Australian Bureau of Statistics ABS (2001). Information paper: Use of the Kessler Psychological Distress scale in ABS health surveys. (ABS cat. No. 44817.0.55.001) Australian Bureau of Statistics, Canberra, ACT. Retrieved from http://www.abs.gov.au/ausstats/[email protected]/papersbyReleaseDate/4D5BD324FE8B415FCA2579D500161D57?OpenDocument (accessed September 8, 2018).

Bailey, S. M. (2015). The Canadian Forces Health Services Road to Mental Readiness Programme. In Medical Corps International Forum, 2, 37–48.

Blanc, S., Zamorski, M., Ivey, G., Edge, H. M., & Hill, K. (2014). How much distress is too much on deployed operations? Validation of the Kessler Psychological Distress Scale (K10) for application in military operational settings. Military Psychology, 26, 88–100.

Bouchard, S., Bernier, F., Boivin, É., Morin, B., & Robillard, G. (2012). Using biofeedback while immersed in a stressful videogame increases the effectiveness of stress management skills in soldiers. PloS One, 7, e36169.

Campos, R. C., Holden, R. R., Laranjeira, P., Troister, T., Oliveira, A. R., Costa, F., Abreu, M., & Fresca, N. (2016). Self-report depressive symptoms do not directly predict suicidality in nonclinical individuals: Contributions toward a more psychosocial approach to suicide risk. Death Studies, 40, 335–349.

Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., Amtmann, D., Bode, R., Buysse, D., Choi, S., Cook, K., DeVellis, R., DeWalt, D., Fries, J. F., Gershon, R., Hahn, E. A., Lai, J., Pilkonis, P., Revicki, D., Rose, M., Weinfurt, K., Hays, R., & PROMIS Cooperative Group (2010). The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63, 1179–1194.

Cramer, K. M., & Barry, J. E. (1999). Psychometric properties and confirmatory factor analysis of the Self-Concealment Scale. Personality and Individual Differences, 27, 629–637.

Crowne, D. P., & Marlowe D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24, 349–354.

D’Agata, M. T., Granek, J. A., Nazarov, A., Boland, H., & Hendriks, T. (2018). The role of subjective self-mapping in the Mental Health Continuum Model. Defence Research and Development Canada, Scientific Letter, DRDC-RDDC-2018-L202.

http://www.abs.gov.au/ausstats/[email protected]/papersbyReleaseDate/4D5BD324FE8B415FCA2579D500161D57?OpenDocument

http://www.abs.gov.au/ausstats/[email protected]/papersbyReleaseDate/4D5BD324FE8B415FCA2579D500161D57?OpenDocument


D’Agata, M. T., Nazarov, A., Granek, J. A., & Kwantes, P. (2018). A comparison of subjective self-mapping in the Mental Health Continuum Model to validated measures. Defence Research and Development Canada, Scientific Letter, DRDC-RDDC-2018-L219.

Degenhardt, L., Conigrave, K., Wutzke, S., & Saunders, J. (2001). The validity of an Australian modification of the AUDIT questionnaire. Drug and Alcohol Review, 20, 143–154.

Donker, T., Comijs, H., Cuijpers, P., Terluin, B., Nolen, W., Zitman, F., & Penninx, B. (2010). The validity of the Dutch K10 and extended K10 screening scales for depressive and anxiety disorders. Psychiatry Research, 176, 45–50.

Driskell, J. E., Copper, C., & Moran, A. (1994). Does mental practice enhance performance? Journal of Applied Psychology, 79, 481–492.

Ferris, J. A., & Wynne, H. J. (2001). The Canadian Problem Gambling Index. Ottawa, ON: Canadian Centre on Substance Abuse.

Fikretoglu, D., Liu, A., & Blackler, K. (2016). Testing different methods to optimize change in mental health service use attitudes. Defence Research and Development Canada, Scientific Report, DRDC-RDDC-2016-R025.

Firth, J., Torous, J., Nicholas, J., Carney, R., Pratap, A., Rosenbaum, S., & Sarris, J. (2017). The efficacy of smartphone-based mental health interventions for depressive symptoms: A meta-analysis of randomized controlled trails. World Psychiatry, 16, 287–298.

Firth, J., Torous, J., Nicholas, J., Carney, R., Rosenbaum, S., & Sarris, J. (2017). Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trials. Journal of Affective Disorders, 218, 15–22.

Granek, J. A., Jarmasz, J., Boland, H., Guest, K., & Bailey, S. (2016). Mobile applications for personalized mental health resiliency training. Interservice/Industry Training, Simulation, and Education Conference, 16120. Defence Research and Development Canada, External Literature (peer reviewed), DRDC-RDDC-2017-P008.

Hays, R. D., Bjorner, J. B., Revicki, D. A., Spritzer, K. L., & Cella, D. (2009). Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items. Quality of Life Research, 18, 873–880.

Ivey, G. W., Blanc, J.-R. S., & Mantler, J. (2015). An assessment of the overlap between morale and work engagement in a nonoperational military sample. Journal of Occupational Health Psychology, 20, 338–347.

Kauer, S. D., Reid, S. C., Crooke, A. H. D., Khor, A., Hearps, S. J. C., Jorm, A. F., & Patton, G. (2012). Self-monitoring using mobile phones in the early stages of adolescent depression: Randomized controlled trial. Journal of Medical Internet Research, 14.


Kessler, R. C., Andrews, G., Colpe, L. J., Hiripi, E., Mroczek, D. K., Normand, S. L., & Zaslavsky, A. M. (2002). Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychological Medicine, 32, 959–976.

Keyes, C. L. M. (2005). Mental illness and/or mental health? Investigating axioms of the complete state model of health. Journal of Consulting and Clinical Psychology, 73, 539–548.

Kline, R. B. (2013). Handbook of psychological testing. New York, NY: Routledge.

Königbauer, J., Letsch, J., Doebler, P., Ebert, D., & Baumeister, H. (2017). Internet-and mobile-based depression interventions for people with diagnosed depression: A systematic review and meta-analysis. Journal of Affective Disorders, 223, 28–40.

Larson, D. G., & Chastain, R. L. (1990). Self-concealment: Conceptualization, measurement, and health implications. Journal of Social and Clinical Psychology, 9, 439–455.

National Defence (2017). Strong, Secure, Engaged: Canada’s Defence Policy. Retrieved from http://dgpaapp.forces.gc.ca/en/canada-defence-policy/docs/canada-defence-policy-report.pdf (accessed September, 2017).

Osman, A., Bagge, C. L., Gutierrez, P. M., Konick, L. C., Kopper, B. A., & Barrios, F. X. (2001). The Suicidal Behaviors Questionnaire-Revised (SBQ-R): Validation with clinical and nonclinical samples. Assessment, 8, 443–454.

Reinert, D. F., & Allen, J. P. (2002). The Alcohol Disorders Use Identification Test (AUDIT): A review of recent research. Alcoholism: Clinical and Experimental Research, 26, 272–279.

Rickard, N., Arjmand, H. A., Bakker, D., & Seabrook, E. (2016). Development of a mobile phone app to support self-monitoring of emotional well-being: A mental health digital innovation. JMIR Mental Health, 3, e49.

Sampasa-Kanyinga, H., Zamorski, M. A., & Colman, I. (2018). The psychometric properties of the 10-item Kessler Psychological Distress Scale (K10) in Canadian military personnel. PloS One, 13, e0196562.

Schaufeli, W. B., Bakker, A. B., & Salanova, M. (2006). The measurement of work engagement with a short questionnaire: A cross-national study. Educational and Psychological Measurement, 66, 701–716.

Searle, A. K., Van Hooff, M., McFarlane, A. C., Davies, C. E., Fairweather-Schmidt, A. K., Hodson, S. E., Benassi, H., & Steele, N. (2015). The validity of military screening for mental health problems: Diagnostic accuracy of the PCL, K10, and AUDIT scales in an entire military population. International Journal of Methods in Psychiatric Research, 24, 32–45.

Seppälä, P., Mauno, S., Feldt, T., Hakanen, J., Kinnunen, U., Tolvanen, A., & Schaufeli, W. (2009). The construct validity of the Utrecht Work Engagement Scale: Multisample and longitudinal evidence. Journal of Happiness Studies, 10, 459–481.

Wilcox, S. (2010). Social relationships and PTSD symptomatology in combat veterans. Psychological Trauma: Theory, Research, Practice, and Policy, 2, 175–182.

http://dgpaapp.forces.gc.ca/en/canada-defence-policy/docs/canada-defence-policy-report.pdf


Yu, L., Buysse, D. J., Germain, A., Moul, D. E., Stover, A., Dodds, N. E., & Pilkonis, P. A. (2012). Development of short forms from the PROMIS™ sleep disturbance and sleep-related impairment item banks. Behavioral Sleep Medicine, 10, 6–24.

Zimet, G. D., Dahlem, N. W., Zimet, S. G., & Farley, G. K. (1988). The Multidimensional Scale of Perceived Social Support. Journal of Personality Assessment, 52, 30–41.


Annex A Visual Representations of the Experiment

A.1 MHCM Instructions Presented Prior to Self-Mapping Task

Mental health exists on a continuum, from healthy, adaptive coping (green), through mild and reversible distress (yellow), to more persistent functional impairment (orange), and finally, to severe functional impairment (red).

The mental health continuum model identifies indicators along these different phases to assist you in monitoring your own performance and well-being.

Review the indicators in each phase regularly. Regular monitoring helps to increase your self-awareness which promotes early identification of problems and solutions.

Green categories suggest normal functioning and positive coping strategies are encouraged. Yellow categories suggest mild distress and employing healthy coping strategies with rest and adequate recovery time are encouraged. Orange and red categories suggest more persistent functioning impairment that may be impacting your life and seeking further resources is encouraged.

Figure A.1: Visual representation of the MHCM that was presented alongside the instructions.


A.2 Group 1

OR

Figure A.2: Visual representation of Group 1 participants who either completed the spectrum

first followed by the descriptors, or completed the descriptors followed by the spectrum for each functional domain.


A.3 Group 2

OR

Figure A.3: Visual representation of Group 2 participants who either completed the spectrum

with anchors first followed by the descriptors, or completed the descriptors followed by the spectrum with anchors for each functional domain.


A.4 Group 3

Figure A.4: Visual representation of Group 3 participants who

completed the spectrum and descriptors together.

A.5 Group 4

Figure A.5: Visual representation of Group 4 participants who completed

the spectrum with anchors and descriptors together.


Annex B Visual Representations of the Matching Task

B.1 Separated Version of the Matching Task

Figure B.1: Visual representation of matching task for participants assigned to

the separated version (i.e., each domain was presented separately).


B.2 Combined Version of the Matching Task

Figure B.2: Visual representation of matching task for participants assigned to

the combined version (i.e., all domains were presented together).


Annex C Correlation Matrices

Table C.15: Bivariate correlations among all validated measures (n = 351). 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 1. Depression/Anxiety (K10) .451** .428** .604** .599** .390** .119* .399** -.648** -.538** -.594** -.286** 2. Suicide Ideation (SBQ-R) .313** .289** .290** .306** .218** .353** -.399** -.337** -.374** -.298* 3. Work Engagement (UWES-9) .471** .512** .378** .065 .241** -.546** -.481** -.582** -.294** 4. Sleep Disturbance .588** .330** .177** .333** -.541** -.460** -.494** -.281** 5. Global Health .412** .070 .276** -.571** -.476** -.539** -.208** 6. Social Support (MSPSS) .041 .348** -.509** -.451** -.546** -.120* 7. Alcohol Use (AUDIT) .229** -.071 -.078 -.125* -.157** 8. Self-Concealment (SCS) -.446** -.407** -.445** -.341** 9. Emotional Well-Being (MHC-SF) .725** .804** .310** 10. Social Well-Being (MHC-SF) .737** .333** 11. Psychological Well-Being (MHC-SF) .361** 12. Social Desirability (MCSDS)

Note. For variables 1–7 higher scores reflect the “negative” end of the scale (e.g., low social support) and/or higher levels of symptomatology. K10 = Kessler Psychological Distress Scale; SBQ-R = Suicide Behaviors Questionnaire-Revised; UWES-9 = Utrecht Work Engagement Scale; MSPSS = Multidimensional Scale of Perceived Social Support; AUDIT = Alcohol Use Disorders Identification Test; SCS = Self-Concealment Scale; MHC-SF = Mental Health Continuum—Short Form; MCSDS = Marlowe-Crowne Social Desirability Scale. ** p < .01; * p < .05.


Table C.16: Bivariate correlations among validated measures and MHCM spectrum scores (n = 364). 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 1. Mood Spectrum .716** .370** .752** .472** .615** .527** .645** .553** .710** .417** .220** .074 2. Depression/Anxiety (K10) .460** .598** .450** .607** .612** .616** .596** .630** .372** .136** .125* 3. Suicide Ideation (SBQ-R) .413** .318** .318** .278** .233** .279** .380** .291** .093 .210** 4. Attitude & Performance Spectrum .547** .530** .448** .577** .484** .704** .347* .200** .072 5. Work Engagement (UWES-9) .417** .467** .457** .502** .495** .362** .184** .063 6. Sleep Spectrum .751** .635** .581** .579** .372** .174** .082 7. Sleep Disturbance .580** .593** .527** .314** .178** .169** 8. Physical Symptoms Spectrum .690** .601** .341** .184** .060 9. Global Health .559** .399** .142** .078 10. Social Behaviour Spectrum .469** .224** .061 11. Social Support (MSPSS) .133* .031 12. Alcohol & Gambling Spectrum .576** 13. Alcohol Use (AUDIT)

Note. Correlations are colour-coded to highlight correlations between each functional domain and its associated validated measure(s). K10 = Kessler Psychological Distress Scale; SBQ-R = Suicide Behaviors Questionnaire-Revised; UWES-9 = Utrecht Work Engagement Scale; MSPSS = Multidimensional Scale of Perceived Social Support; AUDIT = Alcohol Use Disorders Identification Test. ** p < .01; * p < .0

DOCUMENT CONTROL DATA *Security markings for the title, authors, abstract and keywords must be entered when the document is sensitive

1. ORIGINATOR (Name and address of the organization preparing the document. A DRDC Centre sponsoring a contractor's report, or tasking agency, is entered in Section 8.) DRDC – Toronto Research Centre Defence Research and Development Canada 1133 Sheppard Avenue West Toronto, Ontario M3K 2C9 Canada

2a. SECURITY MARKING (Overall security marking of the document including special supplemental markings if applicable.)

CAN UNCLASSIFIED

2b. CONTROLLED GOODS

NON-CONTROLLED GOODS DMC A

3. TITLE (The document title and sub-title as indicated on the title page.) Subjective self-mapping in the Mental Health Continuum Model (MHCM): Examining mental health self-mapping and its relation to validated measures

4. AUTHORS (Last name, followed by initials – ranks, titles, etc., not to be used) D'Agata, M.; Nazarov, A.; Granek, J. A.

5. DATE OF PUBLICATION (Month and year of publication of document.) June 2019

6a. NO. OF PAGES (Total pages, including Annexes, excluding DCD, covering and verso pages.)

58

6b. NO. OF REFS (Total references cited.)

39 7. DOCUMENT CATEGORY (e.g., Scientific Report, Contract Report, Scientific Letter.)

Scientific Report

8. SPONSORING CENTRE (The name and address of the department project office or laboratory sponsoring the research and development.) DRDC – Toronto Research Centre Defence Research and Development Canada 1133 Sheppard Avenue West Toronto, Ontario M3K 2C9 Canada

9a. PROJECT OR GRANT NO. (If appropriate, the applicable research and development project or grant number under which the document was written. Please specify whether project or grant.)

04g

9b. CONTRACT NO. (If appropriate, the applicable number under which the document was written.)

10a. DRDC PUBLICATION NUMBER (The official document number by which the document is identified by the originating activity. This number must be unique to this document.) DRDC-RDDC-2018-R294

10b. OTHER DOCUMENT NO(s). (Any other numbers which may be assigned this document either by the originator or by the sponsor.)

11a. FUTURE DISTRIBUTION WITHIN CANADA (Approval for further dissemination of the document. Security classification must also be considered.)

Public release

11b. FUTURE DISTRIBUTION OUTSIDE CANADA (Approval for further dissemination of the document. Security classification must also be considered.)

12. KEYWORDS, DESCRIPTORS or IDENTIFIERS (Use semi-colon as a delimiter.)

Mental Health; mental health continuum model; Validation

13. ABSTRACT (When available in the document, the French version of the abstract must be included here.)

Background: To complement the Canadian Armed Forces’ (CAF) Road to Mental Readiness (R2MR) training program, a mobile application (i.e., “app”) was developed to provide users with an online opportunity to review and practice strategies to cope with stress and increase resilience. One of the components of the mobile app is the Mental Health Continuum Model (MHCM), designed to allow users to self-assess on six mental health and well-being domains (i.e., Mood, Attitude & Performance, Sleep, Physical Symptoms, Social Behaviour, Alcohol & Gambling) using a visual spectrum of the continuum that includes four anchors: healthy, reacting, injured, and ill. More specifically, for each functional domain the app involves self-rating on scales guided by different descriptors, which have been grouped to represent the four anchors of the MHCM. The current research was undertaken in order to inform and guide future R2MR mobile app development and training content. Specifically, we sought to understand 1) whether MHCM self-mapping aligns with established validated physical and mental health scales; and 2) how CAF members perceive the MHCM content (i.e., the degree of concurrence or consistency between endorsed anchors and their descriptors). Thus, the current research aimed to examine the concurrent validity of the MHCM compared to validated, well-established measures. Additionally, using a separate matching task, our research assessed if the current wording of the descriptors from the MHCM correspond to the four anchors as well as the in-app visual representation of the continuum.

Methods: Data were collected online from 392 Regular Force CAF members. Participants self-mapped onto each of the functional domains of the MHCM, and were randomly assigned to be presented with one of four versions of the MHCM. Participants assigned to Group 1 were presented with the descriptors separately (i.e., without the visual spectrum); they were also presented with the visual spectrum separately (i.e., without the descriptors) and were instructed to self-map onto each of the functional domains of the MHCM. Group 2 were also presented with the descriptors and visual spectrum separately; however, the visual spectrum included the four anchors. Group 3 was presented with the MHCM as is in the mobile app (i.e., the descriptors and spectrum were presented together), and Group 4 was presented with the same version as Group 3, however, the spectrum included the four anchors. All of the participants completed demographic information and validated measures of physical and mental health that assessed similar or the same constructs of each of the functional domains. Finally, all participants completed a matching task to assess the accuracy rates of matching each descriptor to its corresponding anchor.

Results: Overall, there was an adequate amount of agreement between MHCM self-mapping and the validated measures providing some support for the MHCM self-mapping approach, although the degree of agreement differed across functional domains. Importantly, of those participants screening positive for clinically significant symptoms of depression/anxiety, suicide ideation, or hazardous and harmful alcohol use, at least one-fifth self-assessed as healthy on the MHCM. Additionally, the rate at which each validated scale predicted group membership to the four MHCM anchors was high for healthy, however, the rates decreased for reacting, injured, and ill. Approximately 15% (Alcohol and Gambling domain) to 45% (Sleep domain) of participants were inconsistent in self-mapping across the functional domains. Similarly, the results of the matching task indicated that accuracy was high for identifying the descriptors that are healthy, but rates decreased noticeably for reacting, injured, and ill.

Discussion: The results provide initial support for the MHCM self-mapping approach, such that there was a fair amount of agreement between the MHCM self-mapping task and the validated measures. Moreover, for most of the functional domains, a large proportion of participants’ descriptor and spectrum self-mapping were consistent. Nonetheless, our results indicate that there is a noticeable rate of false-negatives on the MHCM self-assessment tool, indicating some participants are underestimating their symptom severity on the continuum. The results of the predicted group membership analyses on the self-mapping task indicate a potential discrepancy between some of the descriptors, particularly the descriptors for the reacting, injured, and ill anchors, and the validated measures. Finally, for most of the functional domains, a large proportion of participants’ descriptor and spectrum self-mapping were consistent. However, our findings from the self-mapping task indicate that some of the MHCM descriptors do not accurately reflect how participants self-assess on the visual spectrum and that participants’ ability to distinguish and assess symptom severity between descriptors was restricted,

especially for reacting, injured, and ill. A similar pattern emerged for the matching task such that accuracy was high for identifying the descriptors that are healthy, but rates decreased noticeably for reacting, injured, and ill. Future research should focus on further assessing the source of inconsistency and discrepancy between self-mapping and between the MHCM and well-established, validated measures.

Contexte : On a élaboré une application mobile comme complément au programme de formation En route vers la préparation mentale (RVPM) des Forces armées canadiennes (FAC). Cette application permet aux utilisateurs de passer en revue et de mettre en pratique virtuellement des stratégies pour composer avec le stress et augmenter leur résilience. L’une des composantes de l’application est le modèle de continuum de la santé mentale (MCSM). Cette composante permet aux utilisateurs d’évaluer leur bien-être et leur santé mentale en fonction de six critères (l’humeur, l’attitude et le rendement; la qualité du sommeil; les symptômes physiques; le comportement en société; la dépendance à l’alcool et au jeu) selon un spectre de couleurs comportant quatre catégories – en santé, en réaction, blessé et malade. Plus précisément, pour chaque domaine fonctionnel, l’application s’appuie sur une autoévaluation sur des échelles guidées par différents descripteurs, qui ont été regroupés pour représenter les quatre critères du MCSM. Les travaux de recherche actuels ont été entrepris pour faire connaître et orienter le développement futur de l’application mobile RVPM et du contenu de la formation. En particulier, on a cherché à comprendre 1) si la méthode d’autoévaluation du MCSM cadrait bien avec les échelons de santé mentale et physique déjà validés et reconnus; et 2) la façon dont les membres des FAC perçoivent le contenu du MCSM (pour évaluer le degré de concordance ou de cohérence entre les caractéristiques et les critères avalisées). En somme, on a voulu vérifier la validité du MCSM en le comparant à des mesures reconnues et validées. Un exercice de jumelage distinct a également été réalisé pour évaluer la qualité de la formulation existante des caractéristiques du MCSM et déterminer dans quelle mesure celles-ci correspondaient bien aux quatre critères et à la représentation visuelle du continuum dans l’application mobile.

Méthodes : Les données ont été recueillies en ligne auprès de 392 membres de la Force régulière des FAC. Les participants ont procédé à une autoévaluation en fonction des domaines fonctionnels du MCSM et de la version du modèle leur ayant été attribuée de façon aléatoire parmi les quatre versions possibles. Les participants du groupe 1 ont reçu les descripteurs séparément (c.-à-d. sans le spectre de couleurs). Les participants du groupe 2 ont aussi reçu les descripteurs et le spectre de couleurs séparément, mais ce dernier comportait les quatre critères. Le groupe 3 a reçu la version du MCSM qui figure actuellement dans l’application mobile (c.-à-d. les descripteurs et le spectre de couleurs sont présentés ensemble). Le groupe 4 a reçu la même version que celle du groupe 3, sauf que le spectre comportait les quatre critères. Tous les participants ont fourni des renseignements démographiques et validé les mesures de la santé physique et mentale qui visaient à évaluer des concepts semblables ou identiques à ceux associés à chaque domaine fonctionnel du MCSM. Tous les participants ont également effectué un exercice de jumelage afin de déterminer le degré de précision du taux de correspondance entre les descripteurs et les critères.

Résultats : Dans l’ensemble, on a observé un degré de concordance adéquat entre l’autoévaluation du MCSM et les mesures validées et ayant servi de point de comparaison, ce qui, dans une certaine mesure, appuie le MCSM. On a toutefois constaté que le degré de concordance était très variable d’un domaine fonctionnel à un autre. Par ailleurs, l’un des principaux points qui est ressorti de l’étude a été que, parmi les participants présentant des symptômes cliniques importants associés à la dépression ou à l’anxiété, à des idées suicidaires ou à une consommation dangereuse ou nocive d’alcool, au moins un cinquième d’entre eux ont déclaré être en santé dans leur autoévaluation. En outre, pour chacun des échelons validés qui prévoyait des membres du groupe dans les quatre critères du MCSM, les taux étaient élevés pour en santé, mais ils diminuaient pour en réaction, blessé et malade. Approximativement, de 15 % (dépendance à l’alcool et au jeu) à 45 % (qualité du sommeil) des participants ont été incohérents dans leur autoévaluation en fonction des différents domaines fonctionnels. Dans le même ordre d’idée, dans le cadre de l’exercice de jumelage, le taux de réussite a été très bon pour l’association des descripteurs avec la catégorie en santé, alors qu’il a été considérablement moins élevé pour les autres catégories (en réaction, blessé et malade).

Analyse : Les résultats appuient la méthode d’autoévaluation associée au MCSM, car le degré de concordance observé entre l’autoévaluation et les mesures validées et reconnues était relativement élevé. De plus, pour la plupart des domaines fonctionnels, une grande proportion des descripteurs des participants et de l’autoévaluation à l’aide du spectre étaient cohérents. Néanmoins, on a observé un taux évident de faux négatifs parmi les autoévaluations, ce qui signifie que certains participants sous-estimaient la gravité de leurs symptômes dans le continuum. Les résultats des analyses des membres du groupe prévu pour l’exercice d’autoévaluation indiquent un écart potentiel entre certains descripteurs – notamment ceux associés aux catégories en réaction, blessé et malade – et les mesures validées. Enfin, pour la plupart des domaines fonctionnels, une grande proportion des descripteurs et des autoévaluations à l’aide du spectre de couleurs étaient cohérentes. Toutefois, les résultats de l’exercice d’autoévaluation ont révélé que certains descripteurs du MCSM ne reflétaient pas précisément la façon dont les participants s’étaient autoévalués selon le spectre de couleurs et que leur capacité à discerner et à évaluer la gravité de leurs symptômes à l’aide des caractéristiques s’en est trouvée limitée, surtout dans les catégories en réaction, blessé et malade. On a observé une tendance similaire dans l’exercice de jumelage. Le taux de réussite du jumelage des descripteurs avec la catégorie en santé s’est avéré élevé, mais il était beaucoup plus modeste dans les catégories en réaction, blessé et malade. De prochains projets de recherche devraient porter sur l’évaluation de la source des incohérences et des divergences entre les autoévaluations, le MCSM et les mesures reconnues et validées.

Subjective self-mapping in the Mental Health Continuum ... · Les participants du groupe 1 ont...

Documents

Transcript of Subjective self-mapping in the Mental Health Continuum ... · Les participants du groupe 1 ont...