RESEARCH ARTICLE Open Access

Patient understanding of two commonlyused patient reported outcome measuresfor primary care: a cognitive interviewstudyMairead Murphy* , Sandra Hollinghurst and Chris Salisbury

Abstract

Background: Standardised generic patient-reported outcome measures (PROMs) which measure health status areoften unresponsive to change in primary care. Alternative formats, which have been used to increase responsiveness,include individualised PROMs (in which respondents specify the outcomes of interest in their own words) andtransitional PROMs (in which respondents directly rate change over a period). The objective of this study wasto test qualitatively, through cognitive interviews, two PROMs, one using each respective format.

Methods: The individualised PROM selected was the Measure Yourself Medical Outcomes Profile (MYMOP). Thetransitional PROM was the Patient Enablement Instrument (PEI). Twenty patients who had recently attended the GPwere interviewed while completing the questionnaires. Interview data was analysed using a modification of Tourangeau’smodel of cognitive processing: comprehension, response, recall and face validity.

Results: Patients found the PEI simple to complete, but for some it lacked face validity. The transitional scale wassometimes confused with a status scale and was problematic in situations when the relevant GP appointmentwas part of a longer episode of care. Some patients reported a high enablement score despite verbally reporting lowenablement but high regard for their GP, which suggested hypothesis-guessing. The interpretation of the PEI itemswas inconsistent between patients.MYMOP was more difficult for patients to complete, but had greater face validity than the PEI. The scale used wasopen to response-shift: some patients suggested they would recalibrate their definition of the scale endpointsas their illness and expectations changed.

Conclusions: The study provides information for both users of PEI/MYMOP and developers of individualisedand transitional questionnaires.Users should heed the recommendation that MYMOP should be interview-administered, and this is likely toapply to other individualised scales. The PEI is open to hypothesis-guessing and may lack face-validity for alonger episode of care (e.g. in patients with chronic conditions). Developers should be cognisant that transitional scalescan be inconsistently completed: some patients forget during completion that they are measuring change frombaseline. Although generic questionnaires require the content to be more general than do disease-specificquestionnaires, developers should avoid questions which allow broad and varied interpretations.

Keywords: Questionnaires, Primary care, Cognitive interviews, Patient-reported outcomes

* Correspondence: mairead.murphy@bristol.ac.ukCentre for Academic Primary Care, Bristol Medical School, University ofBristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK

© The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Murphy et al. BMC Family Practice (2018) 19:162 https://doi.org/10.1186/s12875-018-0850-2

BackgroundAssessing the effectiveness of healthcare interventionsfrom a patient perspective involves the use of patient-re-ported outcome measures (PROMs). Many PROMs aredisease-specific, that is, tailored to the symptoms andimpacts on function of a particular condition [1]. How-ever, primary care services are first contact, comprehen-sive and co-ordinating [2] which means that patients canhave a range of conditions. Many primary care studiesthus require a generic PROM: one which can be admin-istered regardless of condition [3]. A key problem withmany generic PROMs is that they are limited to symp-toms and function. Primary care patients frequentlypresent with other problems, [4] and many havelong-term conditions [5] whereby improvement in func-tion may be unlikely. Leading generic PROMs such asthe SF-36 [6] and EQ-5D [7], therefore often show nochange following interventions in primary care [8–10].In this context, the authors of this paper set out to de-

velop a new PROM for primary care which would bemore responsive to change than other PROMs. A24-item PROM was developed, the Primary Care Out-comes Questionnaire (PCOQ). The qualitative develop-ment work [11–13] and psychometric testing [14] of thishave been published elsewhere. This article reports on astudy which was carried out as part of the qualitative de-velopment work. Some PROMs have been designed toincrease responsiveness when used in primary care[10, 15, 16] and the researchers decided to carry out pri-mary research to assess two of these, before embarking ondevelopment of the PCOQ. Patients who were interviewedfor the qualitative study were asked to complete these twoexisting questionnaires designed to measure outcomes inprimary care and the process of completion was assessedusing cognitive interviews.Cognitive interviews are based on the theory that

responding to a questionnaire involves a number of cog-nitive tasks [17]. The most common theory to explainthis process originates with Tourangeau [18] and wasfurther developed by Willis [19]. It describes the cogni-tive tasks which people go through when responding toquestionnaires as consisting of four components: com-prehension, retrieval, decision and response. Question-naire respondents must firstly comprehend the questionand secondly retrieve information from memory, or forma judgement on the spot. They then decide what theywish to share with the researcher before fitting theirown mental response to the categories provided by thequestionnaire [17].The purpose of cognitive interviews is not just to

understand these cognitive processes, but to use this todetect potential sources of response error. Through useof such a model, cognitive interviews can successfullyidentify problems with questionnaires which can then be

corrected before the data are collected from a largersample. Despite the fact that this is an essential elementof all PROM development, many PROM developers missthis important stage out, or provide only a cursory re-port on it [20].Two formats, which are often more responsive than

standardised generic formats, are firstly PROMs withtransitional scales and secondly individualised PROMs.Transitional scales capture outcome without the needfor a baseline, relying on the patient remembering theirhealth status before the intervention and directly assessingtheir level of change. For example, a common generictransitional item is “thinking about the main problem youconsulted your doctor with, is this problem…”, with re-sponse options given on a five-point Likert scale from verymuch better to very much worse [21].Individualised PROMs allow patients to specify their

problems themselves. They are therefore focussed on theissues particular to the patient in question and thusshow change when other PROMs do not [10].The objective of this study was to test qualitatively,

through cognitive interviews, two PROMs designed spe-cifically for primary care: a PROM which uses a transi-tional scale and an individualised PROM.

MethodsResearch settingThe research setting was the National Health Service(NHS) in the UK. As described above, this study wascarried out alongside a larger qualitative study, whichhad been designed to explore patients’ and practitioners’views on the most important outcomes arising from pri-mary care consultations [11]. The larger qualitative studywas itself carried out to inform development of a PROM,which has since been quantitively tested [22]. Thecurrent study received ethical approval from Nottingham1 National Research Ethics Service (NRES) [23]. Patientswere recruited from the waiting rooms of three healthcentres in areas of varying deprivation and one walk-incentre in Bristol. Patients were purposefully sampled toinclude both men and women, and a range of ages,conditions and ethnicities. Immediately following theirsemi-structured interview for the main qualitative study,patients who were willing to do so also completed a cogni-tive interview to test the two PROMs.

Selection of instrumentsThe selected PROMs were the patient enablement in-strument (PEI) and the measure yourself medical out-come profile version 2 (MYMOP). These were selectedbecause they were both designed to address the limita-tions of generic PROMs in relation to measuring out-come in primary care.

Murphy et al. BMC Family Practice (2018) 19:162 Page 2 of 12

PEI instrument overviewThe PEI was the first PROM to be developed specificallyfor primary-care and is based on the principle that thekey purpose of primary care consultations is to ‘enable’patients to better live their lives and understand andmanage any conditions they may have. It was originallydesigned as a broader instrument, to measure the input,processes and outcomes of high-quality primary care.The PEI retained six items from the original instrument,which capture the construct “enablement”, comprisingelements of empowerment, understanding and coping[15]. It is a transitional instrument which aims to meas-ure increase in enablement resulting from a single con-sultation. The scoring precludes decreased enablement.The PEI is shown in Fig. 1.The PEI has undergone extensive psychometric testing

and has been widely translated and used in different coun-tries [24]. It has shown good test-retest reliability [25] andinternal consistency [15]. Construct validity has beendemonstrated through correlation of the PEI with patientempowerment in patients with multiple long-term condi-tions [26]. The PEI also correlates with measures of patientexperience, such as the doctor’s communication skills, [27]knowing the doctor well, [15, 24] consultation length [24,28] and receiving a prescription when desired [24, 29].The PEI is normally completed straight after a GP

consultation. A modified version of the PEI was used inthis study which used the words “as a result of yourrecent visit to the doctor/nurse.”

MYMOP instrument overviewMYMOP was the first individualised instrument devel-oped to measure outcomes in primary care. Develop-ment was influenced by the Patient-Generated Index,[30] driven by the fact that primary care patients have

different conditions, symptoms and priorities, so out-comes measurement in primary care needs to be simi-larly individualised. MYMOP measures symptoms,activity and well-being at a point in time. The symptomsand activity section is individualised and related to a single(as opposed to multiple) health problem; respondentsidentify the problem-related symptoms and activity whichare most important to them. Individualised tools in gen-eral have weaker psychometric properties than do standar-dised [31–33]. At its initial development, cogent criticismswere made of the derivation and mathematical principlesunderlying MYMOP [34, 35]. Since then, MYMOP hasbeen widely used and further validated, in particular withpatients accessing alternative and complementary therap-ies, such as acupuncture or homeopathy [36–38].Although it has been used as a self-completed instru-

ment in a limited number of trials, [39, 40] the developerof the MYMOP has recommended that it is completedduring a consultation with a patient through a structuredinterview [41]. A follow-up questionnaire is completed atan agreed time in the future (also by interview) and thescore calculated as the change from baseline status.A modified version of MYMOP was used in this study,

whereby the patient name and address was removedfrom the top of the questionnaire and questions onmedication were removed from the bottom. Thesechanges are specified in the MYMOP website as allow-able without compromising validity [41].The section of MYMOP given to respondents is shown

Fig. 2.

Data collectionData were collected through cognitive interviews: amethod whereby people are interviewed as they completequestionnaires and asked to explain the cognitive

Fig. 1 Patient enablement instrument

Murphy et al. BMC Family Practice (2018) 19:162 Page 3 of 12

processes used when arriving at an answer. Cognitiveinterviewing is often used to improve questionnaire design[19]. In this study, it was used to understand the benefitsand limitations of these two PROMs, in particular the re-spective use of individualised items and a transitionalscale. There are two main methods for conducting a cog-nitive interview: verbal probing (in which the interviewerprobes respondents as they complete a questionnaire) andthink-aloud (in which respondents describe their cognitiveprocesses as they complete a questionnaire without inter-vention from the interviewer) [42].To minimise burden on patients, [19] this study primar-

ily used the verbal probing method. If patients began tonaturally think-aloud, this was adopted in addition to ver-bal probes. The topic guide is shown in Additional file 1.

Data analysisThe cognitive interviews were audio-recorded and tran-scribed verbatim. One researcher [MM] read and re-readthe interview transcripts in order to gain an overall viewof the accounts given, to identify themes and develop aninitial coding framework based on the Torangeau model.Tourangeau’s model [18] was adjusted because, in com-mon with most other studies, we found few problems withthe decision process [43, 44]. We therefore focussed onthe other three processes, as follows:

1. Comprehension Process: Does the respondentunderstand what is intended by the question?

2. Retrieval Process: Is the respondent able to retrievethe information from memory correctly from thecorrect time-period?

3. Response process: Does the respondent manage tomap their desired response onto the scale withoutintroduction of error? For example, do theyunderstand the scale and are the scale responsesavailable appropriate?

To finalise the coding frame, the two co-investigators[CS&SH] independently reviewed four interview tran-scripts and identified themes within the above three cat-egories. CS, SH and MM then discussed these themes andagreed on the coding frame. MM then electronicallycoded all the interviews according to this coding frameusing a spreadsheet-based tabular format. If a problemwas identified, this was mapped to the coding frameworkusing memos and the verbatim quotes to justify the deci-sion. We also assessed face validity: whether the question-naire appeared to respondents “on the face of it” tocapture what it is purported to capture, and whether theirresponses gave a fair reflection of their situation.COREQ guidelines were followed in carrying out the

data collection and analysis [45].

ResultsSample and respondentsTwenty people completed a cognitive interview for thePEI. Three of these did not complete MYMOP because

Fig. 2 MYMOP

Murphy et al. BMC Family Practice (2018) 19:162 Page 4 of 12

they thought it was not relevant to them as they had nothad any symptoms when they attended for a consult-ation. The characteristics of the 20 participants is shownin Table 1. Comprehension, recall and response processproblems are shown in Table 2.

Patient enablement instrumentComprehensionThe PEI interviews were characterised by very different in-terpretations for each of the six questions. There is nodocumentation available on how the PEI items should beinterpreted, so these different readings were not countedas “problems” in Table 2, as it is not clear which interpret-ation is correct. However, such ambiguity clearly doesdemonstrate a problem with the questionnaire; and thedifferent interpretations are listed by item in Table 3. Fourpeople commented that they found the questions “vague”or “open”. Others hesitated before responding andhighlighted possible different interpretations. Dual inter-pretations were found for most questions, generally withone broad interpretation and one more narrowly focussedon the consultation. For example, 13 patients interpretedan impaired ability to “cope with life” as being depressed,or having lost the motivation or ability to function. Oneparticipant explained: “If you’re not [able to cope with life]you’ve had it, haven’t you?” Five considered any alleviationof minor concern or symptoms as improving their abilityto cope with life. One participant became frustrated decid-ing how to interpret this saying: “it’s a real ‘nothing’ sen-tence. Isn’t it?”(Patient 9) Similarly, the question “able to

help yourself” was interpreted by some as able to take anyaction to improve symptoms or problems, and others asan absence of helplessness: i.e. ability to function.The use of the word “illness” was the subject of some

discussion. Eight people felt they were not “ill”. This in-cluded both people with long-term conditions (epilepsy,polycystic ovaries, heart condition with valve fitted), andthose with short term conditions (allergic rash, bruisedleg, baker’s cyst) as well as a woman who was pregnant.

“I don’t really think of my heart as an illness funnyenough. It's not like suddenly it can be cured. Isuppose you think your illness is something whichyou'll hopefully get over it. Whereas the heart, guessI'm stuck with that.” (Patient 5)

Most people who raised this issue were, however, stillcomfortable with responding to the item as if their prob-lem were an illness.

Recall processThe recall process is how patients retrieve the necessaryinformation from memory [19]. Some patients found thetransitional nature of the scale difficult. These mostlyconsisted of people with on-going issues which were notfully resolved by their last doctor’s appointment. Six re-spondents found it difficult to identify whether improve-ments resulted from primary care or their ownself-management. Patient 16 dealt with this by givingtwo sets of responses: one for how much she had im-proved because of her own actions (unrelated to theconsultation) and one for how much she had improvedbecause of the consultation.Six patients also had difficulty rating a single consult-

ation during an episode of care, rather than the improve-ment delivered through all primary care consultationssince the start of the episode. One respondent caveatedthe answer she had given as follows:

Patient 18: “Yeah. I suppose these things are all sort of,like rather than specifically today, or yesterday … it’sas a whole I suppose.”

INT: “Yeah. Do you find it difficult to separate outyesterday because it’s a recurrent appointment?”

Patient 18: “Yeah. [INT: Yeah.] Yeah because …. Ialways see the same lady and it’s always about thesame thing.”

Other respondents similarly referred to their improve-ment since the start of an episode, or repeatedly soughtconfirmation from the interviewer that the item referredto improvements only related to the last appointment.

Table 1 Patient characteristics

Characteristic Number

Gender

Female 13

Male 7

Age Bracket

18–34 5

35–54 5

55–64 3

65–74 4

75+ 3

Ethnicity

Asian 1

Black 1

Mixed race 1

White 17

Number LTCs

No long-term conditions 4

One long-term condition 7

> One long-term condition 9

Murphy et al. BMC Family Practice (2018) 19:162 Page 5 of 12

Table 2 PEI and MYMOP Comprehension, Response and Recall problems

PROM Cognitive Process Problem Area Total patientsa

PEI (n = 20) Compre-hension Patient expressed uncertainly on how to interpret item 5

Patient left an applicable item blank or ticked n/a 1

Total patients with at least one problem 6

Recall Scored overall improvement since start of illness, not because of doctor. 6

Difficulty keeping the score to a single appointment, within an episode of care 6

Total patients with at least one problem 9

Response Suggested same or less category should be split into two 5

Difficulty choosing between “same or less” / “not applicable” 4

Data entry error made as a result of response scale 5

Total patients with at least one problem 9

MYMOP (n = 17) Compre-hension Difficulty with choosing, or sticking to a single “problem” 4

Symptoms misunderstood as conditions 8

Activity misunderstood as sporting / paid work 3

Total people with at least one problem 8

Recall Difficulty averaging over a week for something that is cyclical or has changed over the week. 3

Total people with at least one problem 3

Response Patient expressed confusion on how to interpret scale 4

Patient interpreted scale inconsistently between questions 2

Total people with at least one problem 6asome patients reported more than one type of problem in each processPEI items = 1) Able to cope with life, 2) Able to understand your Illness, 3) Able to cope with your illness, 4) Able to keep yourself healthy, 5) Confident about yourhealth, 6) Able to help yourselfMYMOP items = 1) Two symptoms (individualised) 2) One activity (individualised) 3) Overall well-being

Table 3 PEI dual interpretation of items

PEI item Interpretation of item Total

Able to cope with life Not “coping” means depressed or unable to go about day to day tasks 13

Improved “coping” can be any reduction in minor concern. 5

Able to understand illness My long-term condition is not an “illness” 3

My short-term condition is not an “illness” 5

I do / did have an illness when I consulted the doctor / nurse 12

Able to cope with your illness Taking practical action to make the illness less problematic 9

Includes reduction of concern even if no practical action taken 7

Able to keep yourself healthy Refers to general diet, exercise, well-being 7

Refers to the particular problem consulted for 8

Confident about your health Increased understanding / confidence in managing condition 7

Confidence in diagnosis and management plan 3

Confident that condition can be dealt with / is not serious 7

Confident that you an overall healthy person 2

Able to help yourself Able to manage in daily life (not helpless) 8

Any actions taken to improve or alleviate symptoms / condition 11

Murphy et al. BMC Family Practice (2018) 19:162 Page 6 of 12

Response processNine people had problems with mapping their response tothe scale. Five respondents made errors with the “same orless” category. One respondent scored out the word “less”on the questionnaire. Three respondents ticked “better”when the correct response was “same” but they did notwant to create confusion that their response might be “less”:

INT: “As a result of your visit to the doctor two weeksago do you feel you are able to cope with life ‘muchbetter’, ‘better’, ‘same or less’, or ‘not applicable’?”

Patient 10: “I have a comment here. [INT: Mm.]Wouldn’t it have been better to have ‘much better’,‘better’, ‘same’, then ‘less’?”

INT: “Mm, some people have said that.”

Patient 10: “Yes. Because it is … they’re not the samething. And … because of that I’m going to put ‘better’rather than ‘the same’ […] it WAS the same. You see.So I can’t … but I can’t say ‘the same’ […] because it …it might be interpreted as less. And that would make itwrong.”

Four people commented that they had similar difficul-ties in choosing between “not applicable” and “same orless” and one respondent, following the heuristic of aLikert scale, completed the questionnaire believing “notapplicable” read “much less.”

Face validityFace validity is the extent to which a questionnaire ap-pears to be measuring what it is in fact measuring [46].A number of people questioned the relevance of the PEI.One respondent, who said she thought the questionnairelooked like a “waste of time”, justified this by the ambi-guity of the questions:

I might have ticked different boxes actually if I’d havedone it straight away after my appointment. […] Inthat situation, I probably would have just (mimes selfticking without thought) and because they’re so almostvague and open […] I just wouldn’t have sat andfocused on it. (Patient 4)

Another respondent, who said “for me that is a uselessquestionnaire”, explained that she felt unable to givemeaningful responses because of the grouping of “Sameor Less” into a single response option.Face validity is not always desirable [31]. In some

cases, the PEI seemed to possess too much face validity,leading to hypothesis-guessing, with patients seeing PEIas an assessment of the GP and responding based on

this assessment rather than a change in enablement. Forexample, one woman, who held her GP in extremelyhigh regard, recounted how she attended her GP with acough and was given medication which had not yetworked and she was still slightly worried about. Althoughthe consultation sound only moderately enabling, shescored 11 out of a possible 12 on enablement.

MYMOPThis section reports on the same cognitive processes forthe 17 patients who completed MYMOP.

Comprehension processParticipants generally found MYMOP more difficult tocomplete than PEI. Despite an initial explanation that thisshould be for a single condition/problem, four people com-pleted MYMOP for more than one condition. Eight re-spondents wrote the name of the condition instead of thesymptom, although most of these could clearly define whatsymptom meant when asked. One respondent, who wrote“arthritis” as his symptom gave the following responsewhen asked to explain what symptom meant to him:

Patient 14: “Well symptom is, is well what’s happeningwith you, symptom is, is, is the problem you’ve got atthe, at the moment. […], you know, you’ve got a rash.You know, that’s a symptom of something, it might bea stinging nettle, might be a drug you’ve taken, but tome that’s a symptom. […] So to me the symptom is theaches, the pains. […]”

INT: “Yeah, and is there a reason you would writearthritis rather than aches and pains?”

Patient 14: “Yeah because that’s what I’ve been told.That’s what it is.”

Two respondents associated the word activity with sportand one initially felt that, as he was retired, he did nothave any important activities, because nothing he did wasessential. However, most understood activity as doing any-thing they enjoyed doing or had a responsibility to do.There was a very strong convergence of well-being as re-lating to the whole person, mind and body.

Recall processThe recall period for MYMOP is 1 week. Some respon-dents found difficulty in averaging a variable symptomover a week. One woman, with an allergic rash, hesitatedand then circled response 2, explaining:

Patient 8: “A week is quite a long time sometimes forlittle conditions like this … if you’d asked me the daythat it erupted and the following day, I’d have said

Murphy et al. BMC Family Practice (2018) 19:162 Page 7 of 12

well it stops me doing it completely. But then … afterthat, it … no, it sort of gradually … you know, therewere times in the day when no problem whatsoever.And then now, you know, it’s just spot there and asmall spot there.”

INT: “So is what you’re saying, the two doesn’t reallygive a fair picture […] of something that […] has gonefrom six […] to zero?”

Patient 8: “To zero, yes.”

Response processMYMOP has a seven-point Likert scale from 0 to 6 la-belled “as good as it could be” at one end and “as bad asit could be” at the other. Patients differed in their inter-pretation of the top endpoint, with some interpreting itas asymptomatic and others meaning as good as pos-sible, given their knowledge of their own health condi-tion. Other patients used the bottom of the scale as ananchor for as bad as their problem had ever been, andone used the middle of the scale as an anchor for hisaverage symptom level.

Face validityIn general, despite initially finding it more difficult, mostpeople seemed to find MYMOP more face valid thanPEI. Nine respondents thought the questionnaire wasrelevant to them, as opposed to three who thought itwas not and five who were neutral. Those who found itapplicable liked that it measured status directly and didnot require an assessment of change in status due to adoctor’s appointment. As one respondent put it “Thisone was about the doctor (PEI). This one was about me(MYMOP)” (Patient 16). Some participants appreciatedthat it was individualised and measured well-being, al-though some questioned whether well-being was likelyto be changed by a single GP appointment.

DiscussionKey findingsThe PEI questions were open to many different interpre-tations. The results are in line with previous researchfindings [47] that transitional instruments like the PEIare more difficult than status questionnaires for respon-dents to complete, because they require a greater num-ber of internal calculations. The format of the PEI,which explicitly asks about the role of the doctor, maymake it particularly prone to hypothesis-guessing: somepatients seemed to give their response based on theirown assessment of GP performance rather than their en-ablement. This is consistent with research that showspatients will give high satisfaction scores for negative

experiences, if they perceive any failures were not thedoctor’s fault [48].Lastly, the merging of two response categories into

one (same or less) on the PEI scale caused unnecessaryconfusion, which is an important lesson for future ques-tionnaire development.The individualised nature of MYMOP appealed to

people completing it. However, the difficulty people hadin adhering to the instructions of sticking to one condi-tion and naming a symptom, not a condition, suggeststhat MYMOP, or similar instruments, would be verydifficult to administer outside an interview. The scaleof MYMOP, anchored with “as good as it can be”may lend itself to response shift, [49] as patients re-calibrate their expectation of illness, particularly thosewith long-term conditions. Figure 3 shows a summaryof what this research adds to what is already knownon this subject.

Strengths and limitationsThe key strength of this study is that it has providedvaluable findings which can inform users of these twoPROMs and PROM developers more generally. Thesefindings are based on an established model of cognitivetheory [18, 19]. There were some weaknesses with thisstudy. The majority of interviews were with female,white participants. Although the proportion of ethnicminorities was representative of the UK population out-side London, it would have been preferable to have agreater ethnic variation in the sample. Most of the cod-ing was carried out by a single researcher. The twoco-researchers did independently review four transcriptsto inform the coding framework and reviewed the finalcoded data against this framework; nonetheless, inde-pendent coding of all transcripts may have provided amore rigorous analysis.The use of the same patients as the previous qualita-

tive study is a weakness, because it is likely that the cog-nitive interviews were reactive [47] to the qualitativeinterviews carried out immediately prior: i.e. patients’ re-sponse to the two questionnaires could have been af-fected, as they had already reflected on the outcomesfrom the consultation.Lastly, neither questionnaire was used exactly as rec-

ommended. PEI is designed to be completed straightafter a consultation but, in this study, patients completedit 1 day to 3 weeks after their consultation. A key prob-lem with such reterospective self-reporting is that re-spondents often do not remember their baseline healthstate; [47] many participants who do not accurately re-call a prior health state will attempt to construct orguess a response [50]. MYMOP is designed to be com-pleted through interview, but this study tested self-com-pletion. However, PEI and MYMOP have both been

Murphy et al. BMC Family Practice (2018) 19:162 Page 8 of 12

adapted for use in this way, [39, 51] because self-com-pletion straight after a consultation or completionthrough interview is not always possible in research.

Comparison with literatureSome of the issues found have also been reflected inearlier studies. The problem of retrospective survey ac-curacy has been widely recognised [47] and this is beingaddressed by increasing use of digital methods to cap-ture information about patient’s current health status inreal-time [52, 53]. As in the current study, Patersonfound that, when completing PEI sometime after anintervention, patients had difficulty attributing change tothe intervention; that the PEI had lower face validitythan MYMOP when used for chronic conditions andthat such patients did not see their problems as an “ill-ness” [54], a finding which has been observed in cogni-tive interviews of other related questionnaires [55, 56].In a large study of GPAQ data, Mead et al. found 16% ofPEI scores had at least one “not applicable” response[27]. Mead and Bower suggested this might be due toconsultations in which enablement is not an explicit fea-ture, such as those for repeat prescriptions, and that fur-ther research was required to potentially ascertain otherreasons. The current study has found that the ambiguityof the double-barrelled “same or less” is at least partlycontributory to respondent’s overuse of the “not applic-able” category.Despite some problems with face validity, the PEI has

been widely translated [24] and validated [15, 24, 27–29].

Testing shows that enablement and satisfaction are relatedbut distinct constructs [28]. In investigating this, Merceret al. found that there could be empathy without enable-ment, but there was no enablement without empathy [57].Although this is consistent with the importance of em-pathy in the therapeutic relationship, it is feasible to thinkof consultations where patients were not satisfied, yetwere enabled. Mercer’s findings are also consistent withthe finding in the current study, of hypothesis-guessing onthe part of patients. Two patients in the current study de-scribed consultations which did not sound particularly en-abling, yet had high enablement scores. Both patients hadhigh regard for their GP. They justified these scoresthrough verbal report and appeared to understand thequestions. In contrast, three patients described consulta-tions which did sound enabling, but had low enablementscores (1–2). One of these had low regard for her GP, andthe other two felt that the questions (e.g. “cope with life”)were at odds with their reasons for seeking care. Thesecomplications may partly explain some non-intuitive re-sults in quantitative testing of the PEI [58, 59].As found in other studies [54] the format of MYMOP

appealed to patients more than the PEI and carriedgreater face validity. MYMOP, and nearly all other indi-vidualised questionnaires, are recommended to be ad-ministered in interview [30, 33, 60] and this research hasconfirmed this recommendation. The scale of MYMOP,anchored with “as good as it can be” may lend itself toresponse shift, [49] as patients recalibrate their expect-ation of illness, particularly those with long-term

Fig. 3 What this study adds

Murphy et al. BMC Family Practice (2018) 19:162 Page 9 of 12

conditions. Patients also experienced problems scoringminor conditions with rapidly changing symptoms, giventhe requirement to score how the condition has beenover the last week. This also serves to highlight the issueof whether it is relevant to focus only on symptoms insomething short-term or self-limiting, as any changemeasured is likely to be positive. The domains of under-standing, reduction of concern and ability to help self inthe future may be more important to patients with acuteor minor illnesses than current symptoms (which mayhave resolved within a week). Furthermore, primary carepatients frequently present with problems unrelated tosymptoms or function, [4] and many primary care pa-tients have multiple long-term conditions [5, 61, 62]. Astheir function may not improve, experts have suggestedthe need to measure wider outcomes in such patients,such as a sense of control and the ability to self-care[63].

ConclusionsThe current study was originally designed to inform de-velopment of a new PROM for primary care. The in-sights which informed the development of this PROMare transferable to both users and developers of indivi-dualised and transitional questionnaires.Users should heed the developer’s recommendation

that MYMOP should be administered through interview[64]. The scale of MYMOP may lend itself to responseshift, which should be taken into account if using it tomeasure change over time from a point baseline. ThePEI, although widely used and translated in differentcountries for primary care, lacks face validity for somepatients with chronic conditions and may be open tohypothesis-guessing. When administered days or weeksafter a consultation, it may be more difficult for patients tocomplete than the immediate post-consultation version.For developers, this study has confirmed that cognitive

interviews can uncover problems that cannot be uncov-ered through quantitative psychometric testing, evenwith existing questionnaires that are in widespread use.Many such questionnaires were developed withoutcomprehensive cognitive testing and lack clear docu-mentation about the intended meaning of questions,which makes post-hoc cognitive testing difficult.Cognitive interviews should be an essential part ofnew measure development, to ensure the questionsare understood consistently and are measuring thedesired concept.

Additional file

Additional file 1: Verbal Probing Schedule for PEI and MYMOP. List ofverbal probes used for the cognitive interviews with PEI and MYMOPrespectively. (DOCX 23 kb)

AbbreviationsEQ-5D: European Quality of Life-5 Dimensions; MYMOP: Measure YourselfMedical Outcomes Profile; PEI: Patient Enablement Instrument; PROM: Patient-reported outcome measure; SF-36: Short Form 36

AcknowledgementsThe authors would like to thank all the participants in this study, the BristolPrimary Care Research Network for assisting with recruiting the participants,the NIHR SPCR for funding the research and the Avon Primary Care ResearchCollaborative for funding time to write the paper.

FundingThis study was funded by a capacity building grant from the NIHR School forPrimary Care Research (SPCR). Service support costs for data collection inNHS health centres was provided by the NIHR, through Avon Primary CareResearch Collaborative. The Avon Primary Care Research Collaborative fundedtime to write the paper. The funders had no role in either the design of thestudy; data collection, analysis and interpretation or writing of the manuscript.The NIHR SPCR is a partnership between the Universities of Bristol,Cambridge, Keele, Manchester, Newcastle, Nottingham, Oxford, Southamptonand University College London. The views expressed are those of the author(s)and not necessarily those of the NHS, the NIHR or the Department of Health.

Availability of data and materialsThe ethical approval does not allow data sharing.

Authors’ contributionsMM designed the study and collected the data. MM carried out the dataanalysis, and CS/SH reviewed and validated it. MM drafted the manuscript.CS/SH reviewed and revised the manuscript. All authors approved the finalmanuscript.

Authors’ informationMairead Murphy: University of Bristol, MM is the primary investigator andcorresponding author for this study, which was done as a phase in her PhDsubmitted in 2016: Developing a patient-reported outcome measure forprimary care.Chris Salisbury: University of Bristol, CS is a professor of primary care, priorhead of the Centre for Academic Primary Care, University of Bristol and wasa supervisor of MM’s PhD.Sandra Hollinghurst: University of Bristol, SH is a senior lecturer in healtheconomics at CAPC University of Bristol and was supervisor of MM’s PhD.

Ethics approval and consent to participateThe study received ethical approval from the committee of Nottingham 1National Research Ethics Service, ref. 13/EM/0197. Patients gave writteninformed consent prior to interview.

Consent for publicationThe signed patient consent forms included a statement that the patients agreedto their interviews to be used in published work, including anonymised quotations.

Competing interestsThe authors declare that they have no competing interests.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.

Received: 22 June 2017 Accepted: 17 September 2018

References1. Black N. Patient reported outcome measures could help transform healthcare.

Br Med J. 2013;346:f167.2. Heath I, et al. Quality in primary health care: a multidimensional approach

to complexity. Br Med J. 2009;338:b1242.3. Mokkink LB, et al. The COSMIN study reached international consensus on

taxonomy, terminology, and definitions of measurement properties forhealth-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737–45.

Murphy et al. BMC Family Practice (2018) 19:162 Page 10 of 12

4. Salisbury C, et al. The content of general practice consultations: cross-sectionalstudy based on video recordings. Br J Gen Pract. 2013;63(616):751–9.

5. Salisbury C, et al. Epidemiology and impact of multimorbidity in primarycare: a retrospective cohort study. Br J Gen Pract. 2011;61(582):e12–21.

6. Ware JE Jr, Sherbourne CD. The MOS 36-item short-form health survey (SF-36).I. Conceptual framework and item selection. Med Care. 1992;30(6):473–83.

7. Brooks R. EuroQol: the current state of play. Health Policy. 1996;37(1):53–72.8. Venning P, et al. Randomised controlled trial comparing cost effectiveness

of general practitioners and nurse practitioners in primary care. Br Med J.2000;320(7241):1048–53.

9. McKinley RK, et al. Comparison of out of hours care provided by patients'own general practitioners and commercial deputising services: a randomisedcontrolled trial. II: the outcome of care. Br Med J. 1997;314(7075):190–3.

10. Paterson C. Measuring outcomes in primary care: a patient generatedmeasure, MYMOP, compared with the SF-36 health survey. Br Med J.1996;312(7037):1016–20.

11. Murphy M, et al. Patient and practitioners' views on the most importantoutcomes arising from primary care consultations: a qualitative study. BMCFam Pract. 2015;16:108.

12. Murphy M, Hollinghurst S, Salisbury C. Agreeing the content of a patient-reported outcome measure for primary care: a Delphi consensus study.Health Expect. 2017;20(2):335–48. Published online 2016 Apr 28. https://doi.org/10.1111/hex.12462.

13. Murphy M, Hollinghurst S, Salisbury C. Qualitative assessment of the primarycare outcomes questionnaire: a cognitive interview study. BMC Health ServRes. 2018;18(1):79.

14. Murphy M, et al. Primary care outcomes questionnaire: psychometric testingof a new instrument. Br J gen Pract. 2018;68(671):e433–40.

15. Howie JG, et al. Quality at general practice consultations: cross sectionalsurvey. Br Med J. 1999;319(7212):738–43.

16. Haddad S, et al. Patient perception of quality following a visit to a doctor ina primary care unit. Fam Pract. 2000;17(1):21–9.

17. Schwarz N. Cognitive aspects of survey methodology. Appl Cogn Psychol.2007;21(2):277–87.

18. Tourangeau, R., Cognitive sciences and survey methods., in Cognitive Aspectsof Survey Methodology: Building a Bridge Between Disciplines, T. Jabine, et al.,editors. 1984: Washington, DC, National Academy Press. p. 73–100.

19. Willis, G., Cognitive Interviewing - A How To Guide. 1999, Research TriangleInstitute.

20. Patrick DL, et al. Content validity—establishing and reporting the evidencein newly developed patient-reported outcomes (PRO) instruments formedical product evaluation: ISPOR PRO good research practices task forcereport: part 2—assessing respondent understanding. Value Health. 2011;14(8):978–88.

21. Kamper SJ, Maher CG, Mackay G. Global rating of change scales: a review ofstrengths and weaknesses and considerations for design. J Man Manip Ther.2009;17(3):163–70.

22. Mairead M, Hollinghurst S, Cowlishaw S, Salisbury C. Primary Care OutcomesQuestionnaire: psychometric testing of a new instrument. Br J Gen Pract.2018;68 (671):e433-40. https://doi.org/10.3399/bjgp18X695765.

23. NIHR, UK clinical research network : portfolio database: primary careoutcomes study. 2013.

24. Pawlikowska TR, et al. Patient involvement in assessing consultation quality:a quantitative study of the patient enablement instrument in Poland. HealthExpect. 2010;13(1):13–23.

25. Lam CL, et al. A pilot study on the validity and reliability of the patientenablement instrument (PEI) in a Chinese population. Fam Pract. 2010;27(4):395–403.

26. Small N, et al. Patient empowerment in long-term conditions: developmentand preliminary testing of a new measure. BMC Health Serv Res. 2013;13:263.

27. Mead N, Bower P, Roland M. Factors associated with enablement in generalpractice: cross-sectional study using routinely-collected data. Br J Gen Pract.2008;58(550):346–52.

28. Howie JG, et al. A comparison of a patient enablement instrument (PEI)against two established satisfaction scales as an outcome measure ofprimary care consultations. Fam Pract. 1998;15(2):165–71.

29. Dowell J, et al. A randomised controlled trial of delayed antibiotic prescribingas a strategy for managing uncomplicated respiratory tract infection in primarycare. Br J Gen Pract. 2001;51(464):200–5.

30. Ruta DA, et al. A new approach to the measurement of quality-of-life - thepatient-generated index. Med Care. 1994;32(11):1109–26.

31. Bowling A. Measuring Health: A review of quality of life measurementscales, vol. 1. 3rd ed. Maidenhead, Berkshire: Open University Press; 2004.

32. MacDuff C, Russell EM. The problem of measuring change in individualhealth-related quality of life by postal questionnaire: use of the patient-generated index in a disabled population. Qual Life Res. 1998;7(8):761–9.

33. Patel KK, Veenstra DL, Patrick DL. A review of selected patient-generatedoutcome measures and their application in clinical trials. Value Health. 2003;6(5):595–603.

34. Jenkinson C. MYMOP, a patient generated measure of outcomes. Researchinto outcomes has moved away from symptom based assessments. Br MedJ. 1996;313(7057):626 author reply 627.

35. Ruta D, Garratt A. MYMOP, a patient generated measure of outcomes.Reliability of such instruments needs to be proved. Br Med J. 1996;313(7057):626–7.

36. McClean S, Brilleman S, Wye L. What is the perceived impact of Alexandertechnique lessons on health status, costs and pain management in the reallife setting of an English hospital? The results of a mixed methods evaluationof an Alexander technique service for those with chronic back pain.BMC Health Serv Res. 2015;15:293.

37. Thompson E, Viksveen P, Barron S. A patient reported outcome measure inhomeopathic clinical practice for long-term conditions. Homeopathy. 2016;105(4):309–17.

38. Krug K, et al. Complementary and alternative medicine (CAM) as part ofprimary health care in Germany-comparison of patients consulting generalpractitioners and CAM practitioners: a cross-sectional study. BMC ComplementAltern Med. 2016;16(1):409.

39. Salisbury C, et al. Effectiveness of PhysioDirect telephone assessment andadvice services for patients with musculoskeletal problems: pragmaticrandomised controlled trial. Br Med J. 2013;346(jan29 3):f43.

40. Flower A, Lewith GT, Little P. A feasibility study exploring the role of Chineseherbal medicine in the treatment of endometriosis. J Altern Complement Med.2011;17(8):691–9.

41. Paterson, C. University of Bristol website, PHC section, MYMOP. 2012 [cited2014 25/04/2014]; Available from: http://www.bristol.ac.uk/primaryhealthcare/resources/mymop/strengthsandweaknesses/.

42. Beatty PC, Willis GB. Research synthesis: the practice of cognitive interviewing.Public Opinion Quarterly. 2007;71(2):287–311.

43. Horwood J, et al. Listening to patients: using verbal data in the validation ofthe Aberdeen measures of impairment, activity limitation and participationrestriction (ab-IAP). BMC Musculoskelet Disord. 2010;11:182.

44. Horwood J, Sutton E, Coast J. Evaluating the face validity of the ICECAP-Ocapabilities measure: a “think aloud” study with hip and knee arthroplastypatients. Appl Res Qual Life. 2014;9:667–82. https://doi.org/10.1007/s11482-013-9264-4.

45. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitativeresearch (COREQ): a 32-item checklist for interviews and focus groups. Int JQual Health Care. 2007;19(6):349–57.

46. Fitzpatrick R, et al. Evaluating patient-based outcome measures for use inclinical trials. Health Technol Assess. 1998;2(14): p. i-iv):1–74.

47. Streiner DL, Norman GR. Health measurement scales: a practical guide totheir development and use. New York: Oxford University Press; 2008.

48. Haggerty JL. Are measures of patient satisfaction hopelessly flawed? Br MedJ. 2010;341:c4783.

49. Osborne RH, Hawkins M, Sprangers MA. Change of perspective: a measurableand desired outcome of chronic disease self-management interventionprograms that violates the premise of preintervention/postinterventionassessment. Arthritis & Rheumatology. 2006;55(3):458–65.

50. Herrmann D. Reporting current, past, and changed health status. What weknow about distortion. Med Care. 1995;33(4 Suppl):AS89–94.

51. Haughney J, et al. The use of a modification of the patient enablementinstrument in asthma. Prim Care Respir J. 2007;16(2):89–92.

52. Reade S, et al. Cloudy with a chance of pain: engagement and subsequentattrition of daily data entry in a smartphone pilot study tracking weather,disease severity, and physical activity in patients with rheumatoid arthritis.JMIR Mhealth Uhealth. 2017;5(3):e37.

53. Veer, S.V.D., et al., FRI0175 Using smartphones to improve remote monitoringof rheumatoid arthritis: completeness of patients' symptom reports, in Annalsof the Rheumatic Diseases 2017. p. 547.1–54547.

54. Paterson C. Measuring changes in self-concept: a qualitative evaluation ofoutcome questionnaires in people having acupuncture for their chronichealth problems. BMC Complement Altern Med. 2006;6:7.

Murphy et al. BMC Family Practice (2018) 19:162 Page 11 of 12

55. Mallinson S. Listening to respondents: a qualitative assessment of the short-form 36 health status questionnaire. Soc Sci Med. 2002;54(1):11–21.

56. de Jong M, et al. The quality of working life questionnaire for Cancer survivors(QWLQ-CS): a pre-test study. BMC Health Serv Res. 2016;16:194.

57. Mercer SW, et al. Patient enablement requires physician empathy: a cross-sectional study of general practice consultations in areas of high and lowsocioeconomic deprivation in Scotland. BMC Fam Pract. 2012;13:6.

58. Brusse CJ, Yen LE. Preferences, predictions and patient enablement: apreliminary study. BMC Fam Pract. 2013;14:116.

59. Wensing M, et al. The patients assessment chronic illness care (PACIC)questionnaire in the Netherlands: a validation study in rural generalpractice. BMC Health Serv Res. 2008;8:182.

60. O'Boyle CA, et al. Individual quality of life in patients undergoing hipreplacement. Lancet. 1992;339(8801):1088–91.

61. Fortin M, et al. Prevalence of multimorbidity among adults seen in familypractice. Ann Fam Med. 2005;3(3):223–8.

62. Barnett K, et al. Epidemiology of multimorbidity and implications for healthcare, research, and medical education: a cross-sectional study. Lancet. 2012;380(9836):37–43.

63. Peters M, et al. Pilot study of patient reported outcome measures (PROMs)in primary care: report to the Department of Health. Oxford: University ofOxford, Department of Public Health; 2013.

64. Paterson, C. University of Bristol, CAPC Wesite: MYMOP. 2012 [cited 2014 05/01/2014].

Murphy et al. BMC Family Practice (2018) 19:162 Page 12 of 12