Reproducibilidad mapping

ASSESSMENT OF GLOBAL AND INDIVIDUAL REPRODUCIBILITYOF PROJECTIVE MAPPING WITH CONSUMERSLETICIA VIDAL1,5, RAFAEL SILVA CADENA1, SILVANA CORREA2, ROSA A. ÁBALOS2, BEATRIZ GÓMEZ2,ANA GIMÉNEZ1, PAULA VARELA3,4 and GASTÓN ARES1

1Departamento de Ciencia y Tecnología de Alimentos, Facultad de Química, Universidad de la República, General Flores 2124, CP 11800,Montevideo, Uruguay2Facultad de Bromatología, Universidad Nacional de Entre Ríos, Gualeguaychú, Entre Ríos, Argentina3Instituto de Agroquímica and Tecnología de Alimentos (CSIC), Paterna, Valencia, Spain4Nofima AS, PO Box 210, 1431 Ås, Norway

5Corresponding author.TEL: +598-29248003;Fax: +598-292419906;EMAIL: [email protected]

Accepted for Publication January 7, 2014

doi:10.1111/joss.12083

ABSTRACT

The popularity of projective mapping with consumers for sensory characteriza-tion has markedly increased in the last 5 years. To have confidence in this method-ology, it is necessary to ensure that a similar product profile would emerge if thetest was repeated. Also, deciding whether the study should be replicated or not is akey issue in test implementation. In this context, the aim of the present work wasto evaluate global and individual reproducibility of projective mapping forsensory characterization with consumers and to evaluate the influence of the sizeof difference among samples. Six consumer studies were conducted using a test–retest paradigm. In each study, responses from the same group of consumers tothe same sample set in two different sessions were compared. Across the sixstudies, individual reproducibility tended to be low. However, the RV coefficientsof consensus sample configurations between sessions were higher than 0.75,suggesting that test–retest reproducibility of projective mapping with consumersproved to be relatively high.

PRACTICAL APPLICATIONS

The present work provides evidence of the reproducibility of projective mappingfor sensory characterization with consumers. Although sample configurationswere stable, some differences in conclusions regarding similarities and differencesamong samples were identified between sessions. This indicates that care must betaken when relying on results of projective mapping with consumers obtainedwithout the use of replicates, particularly when working with sample sets withsmall differences. Results from the present work showed that stability indices ofsample configurations based on bootstrapping resampling approaches wererelated to global reproducibility. These indices could be useful to decide whetheror not it is necessary to replicate projective mapping in order to ensure thatconclusions regarding similarities and differences among samples would berepeatedly identified. This is of particular interest, considering the difficulty ofasking consumers to attend separate sessions.

INTRODUCTION

Sensory characterization is one of the most powerful andextensively used tools in sensory science (Lawless andHeymann 2010). Descriptive analysis with highly trainedassessors has been the most popular method for sensorycharacterization in the last decades (Stone et al. 1974;

Meilgaard et al. 1999; Murray et al. 2001). Although thismethodology provides detailed, consistent, reproducibleand stable in time results, it is time-consuming and can bequite expensive and difficult to apply in many situations(Murray et al. 2001; Varela and Ares 2012). Therefore, thedevelopment of simpler and faster methods that use con-sumers to describe products are becoming more accepted

bs_bs_banner

Journal of Sensory Studies ISSN 0887-8250

74 Journal of Sensory Studies 29 (2014) 74–87 © 2014 Wiley Periodicals, Inc.

mailto:[email protected]

within the sensory science community and are increasinglyconsidered a valid alternative to obtain the sensory profileof a set of products (Valentin et al. 2012; Varela and Ares2012).

Projective mapping or Napping is one of the novel meth-odologies for sensory characterization, which has beenincreasingly used in the last 5 years (Varela and Ares 2012).It is a projective type method which collects bidimensionalperceptual maps for each assessor in a single sensory session(Risvik et al. 1994). Samples are simultaneously pre-sented and have to be positioned by each assessor on abidimensional space according to the global differences andsimilarities among them, in such a way that the moresimilar they are, the closer they should be on the providedspace (Risvik et al. 1994; Risvik et al. 1997).

Projective mapping has been reported to be a simplemethodology, which can be performed by trained assessorsor consumers (Valentin et al. 2012; Varela and Ares 2012). Ithas been applied to a wide range of food products such aschocolate (Risvik et al. 1994), ewe milk cheeses (Barcenaset al. 2004), wine (Pagès 2005; Perrin et al. 2008; Ross et al.2012), apples (Nestrud and Lawless 2010), milk desserts(Ares et al. 2010a), fish nuggets (Albert et al. 2011) andpowdered drinks (Ares et al. 2011).

It is necessary to ensure that both valid and reproducibleinformation is provided by projective mapping before it canbe established as a standard methodology for sensorycharacterization with consumers. If validity is taken tomean that projective mapping provides sensory character-izations similar to those from descriptive analysis withtrained assessors, then it has been already established byseveral authors (Risvik et al. 1997; Pagès 2005; Perrin et al.2008; Louw et al. 2013).

Reproducibility of projective mapping has been lessexplored in the literature, and one of the questions thatarises when implementing projective mapping for sensorycharacterization is whether the task should be replicated ornot (Hopfer and Heymann 2013). Projective mapping canbe regarded as a reproducible methodology if it providessimilar results when executed under identical conditions indifferent sessions separated in time (Yu 2005). In the greatmajority of studies using projective mapping, assessorscomplete the task only once (King et al. 1998; Pagès 2005;Nestrud and Lawless 2008, 2010; Perrin et al. 2008; Kennedyand Heymann 2009; Ares et al. 2010a; Pagès et al. 2010;Albert et al. 2011; Ares et al. 2011; Dehlholm et al. 2012a).In some studies, the reproducibility of projective mappinghas been evaluated using a blind duplicate sample withinthe same session (Nestrud and Lawless 2008, 2010;Moussaoui and Varela 2010; Veinand et al. 2011). Only fewstudies have reported repeated evaluations of projectivemapping (Risvik et al. 1994, 1997; Barcenas et al. 2004;Perrin and Pagès 2009; Kennedy 2010; Hopfer and

Heymann 2013). At the individual level, Kennedy (2010)and Risvik et al. (1994, 1997) have reported low reproduc-ibility, which have been attributed to changes in consumerarrangement criteria. In particular, Kennedy (2010)reported that most consumers showed an RV coefficientlower than 0.5 for three replicated sample configurations ofgranola bars. However, at the aggregate level, most studieshave shown that consensus sample configurations and con-clusions regarding overall similarities and dissimilaritiesamong the samples are very similar across replicates (Risviket al. 1994, 1997; Perrin and Pagès 2009; Kennedy 2010;Hopfer and Heymann 2013). Barcenas et al. (2004) reportedsome changes in sample configurations from triplicateevaluations of ewe milk cheeses. However, the authors couldnot explain if these differences were due to changes in asses-sors’ perception or to changes in processing conditions,which modified the sensory characteristics of the samples.

Considering that in many situations it is not practical torecruit consumers for replicate evaluations, the reproduc-ibility of projective mapping in consumer studies deservesfurther exploration to ensure that reliable information canbe gathered without the use of replicates. In this context, theaim of the present work was to evaluate global andindividual reproducibility of projective mapping withconsumers and to assess how they would be affected by thedegree of differences among samples.

MATERIALS AND METHODS

Six consumer studies were conducted using a test–retestparadigm to assess individual and global reproducibility ofprojective mapping. In each study, responses from the samegroup of respondents to the same sample set in twodifferent sessions were compared. Studies 1 and 2 requiredconsumers to evaluate crackers in two sessions separated48 h, while in studies 3–6 consumers evaluated vanilla milkdesserts in two sessions held 2 weeks apart. In both cases,the time between replicates was enough to assure that par-ticipants would not remember their responses from theprevious session. Different times between replicates wereconsidered to provide greater robustness to the findings.

Studies 1 and 2

Samples. Sixteen commercial brands of plain crackers(named A–P) available in the Argentinean market wereevaluated. Two sets of eight plain crackers were consideredwith varying degree of difference among samples: one setwith large differences among four salted – I to L – and fourunsalted – M to P – crackers (study 1), and a second onewith smaller differences among samples, using salted plaincrackers only – A to H – (study 2).

L. VIDAL ET AL. GLOBAL AND INDIVIDUAL REPRODUCIBILITY OF PROJECTIVE MAPPING

75Journal of Sensory Studies 29 (2014) 74–87 © 2014 Wiley Periodicals, Inc.

Participants. One hundred and eighty participants wererecruited among students and workers of the Facultad deBromatología of Universidad Nacional de Entre Ríos(Gualeguaychú, Argentina). Their ages ranged from 16 to 63years and 73% were female. Consumers were randomlydivided into two groups: 89 consumers participated in study1, while 91 consumers participated in study 2. Consumersevaluated the sample set of each study in two separatesessions, 48 h apart. They signed an informed consentagreement.

Data Collection. For each study, consumers evaluatedeight samples using a projective mapping task followed by adescription phase in each session. Consumers were asked totry the eight samples and to place them on an A3 whitesheet (42 × 30 cm), according to their similarities or dis-similarities (similar samples should be located close, whiledifferent samples should be located far from each other). Itwas explained to them that they had to complete the taskaccording to their own criteria and that there were no rightor wrong answers. After positioning the samples, consumerswere asked to provide a description of the samples. Testingtook place in a sensory laboratory in individual sensorybooths, designed in accordance with ISO (1988) 8589.Artificial daylight, constant temperature (22C) and aircirculation were controlled. Still mineral water wasavailable for rinsing.

Studies 3 to 6

Samples. Eight samples of vanilla milk desserts were pre-pared for each study, varying in degree and type of differ-ences among samples. Samples in study 3 (named A1–A8)and study 5 (named C1–C8) only differed in flavor, whilesamples of study 4 (named B1–B8) and study 6 (namedD1–D8) presented both flavor and texture differences. Addi-tionally, based on sample formulations, studies 3 and 4involved the evaluation of samples with large differencesamong them, while in studies 5 and 6, differences amongsamples can be regarded as small. The formulation of themilk desserts is shown in Table S1.

Desserts were prepared by mixing the solid ingredientswith water and poured into a Thermomix TM 31 (VorwerkMexico S. de R.L. de C.V., México D.F., México). The disper-sion was heated at 90C for 5 min under strong agitation(1,100 rpm). The desserts were placed in closed glass con-tainers, cooled to room temperature (25C) and then storedin the refrigerator (4–5C) for 24 h prior to their evaluation.

Participants. Four different groups of consumers wererecruited among students and workers of the Facultad deQuimica of the Universidad de la República (Montevideo,

Uruguay). Participants ranged in age from 20 to 50 years oldand approximately 60% were female. Two groups of 48 con-sumers participated in studies 3 and 4, while studies 5 and 6were carried out with two groups of 42 consumers. In eachstudy, consumers participated in two separate sessions 14days apart. They signed an informed consent agreement andwere given a small present for their participation.

Data Collection. For each of the four studies (Studies3–6), consumers evaluated eight samples of each set using aprojective mapping task followed by a description phase ineach session. Consumers received 15 g of each vanilla milkdessert coded with 3-digit random numbers at 10C inplastic containers and a spoon. Still mineral water was avail-able for rinsing between samples. Participants were asked totry the samples and to place them on an A3 white sheet(42 × 30 cm), according to their similarities or dissimilari-ties. Testing took place in a sensory laboratory in standardsensory booths that were designed in accordance with ISO(1988) 8589, under artificial daylight, temperature control(22C) and air circulation was controlled.

Data Analysis

For each consumer map, the X and Y coordinates of eachsample were determined, considering the left bottom cornerof the sheet as the origin of coordinates. The X and Y coor-dinates for each session and sample set were analyzed usingmultiple factor analysis (MFA) (Pagès 2005). Confidenceellipses were constructed as suggested by Dehlholm et al.(2012b). The stability of sample configurations from eachsession was evaluated using a bootstrapping resamplingapproach. According to Blancher et al. (2012), sample con-figurations can be regarded as stable if simulated repeatedexperiments provide similar results as those obtained withthe original data set. In the present work, the bootstrappingprocess consisted of obtaining 1,000 subsets of size equal tothe total number of consumers using random samplingwith replacement. For each subset, sample configurationswere obtained using MFA, and agreement between eachof these configurations and the reference configuration(obtained with all the consumers who participated in thestudy) was evaluated by computing the RV coefficient (Abdi2010). Average values and standard deviations over the RVcoefficients were calculated. The RV coefficient has beenused as a tool to assess the global similarity between twofactorial configurations of the same products (Faye et al.2004; de Saldamando et al. 2013). This coefficient takes thevalue of 0 if the configurations are uncorrelated and thevalue of 1 if the configurations are homothetic. It dependson the relative position of the points in the configurationand therefore is independent of rotation and translation(Robert and Escoufier 1976).

GLOBAL AND INDIVIDUAL REPRODUCIBILITY OF PROJECTIVE MAPPING L. VIDAL ET AL.


The similarities among the sample configurations over allassessors and sessions were evaluated with the RV coeffi-cient. Also, RV coefficients of individual sample configura-tions between sessions were calculated as a measure ofindividual reproducibility. The significance of the RV coeffi-cient was tested using a permutation test, as suggested byJosse et al. (2008). If the RV coefficient between two sampleconfigurations is significant, it can be concluded that theyare correlated and therefore, information about thesimilarities and differences among samples is similar.

The words elicited by consumers in the description phasewere qualitatively analyzed. Words with similar meaningwere grouped into categories, and their frequency ofmention was determined by counting the number ofconsumers who elicited words within each category. Termsmentioned by at least 5% of the consumers were retainedfor further analysis (Symoneaux et al. 2012). In eachsession, consensual terms were identified using the method-ology proposed by Kostov et al. (2014). Consensual termswere identified as those for which the P value, computed asthe proportion of random subsets, selected followinga bootstrap methodology, having a within-inertia smaller orequal to the observed inertia, was smaller than 0.10. Mul-tiple factor analysis for contingency tables (MFACT) wasapplied on the frequency table of each session to obtaina representation of terms (Bécue-Bertaut and Pagès 2004).In this analysis, only the terms used by consumers in bothsessions were considered.

All statistical analyses were performed with R language(R Development Core Team 2007) using FactoMineR (Lêand Husson 2008; Lê et al. 2008) and SensoMineR(Lê and Husson 2008).

RESULTS

Global Reproducibility

No differences were observed in the percentage of inertiaexplained by the first and second dimensions of the MFAbetween sessions (Figs. 1 and 2). Average RV coefficientacross simulations from the bootstrapping resamplingapproach did not vary between sessions, suggesting thatduplicate evaluation did not increase the stability of sampleconfigurations (Table 1). As expected, average RV coefficientincreased with the size of difference among samples, i.e., itwas higher for the studies with large differences amongsamples than for studies with small differences amongsamples. Besides, the stability of sample configurations forthe studies that included samples with flavor and texturedifferences was higher than that of the studies that onlyincluded flavor differences (Table 1).

At the aggregate level, the RV coefficient of sample con-figurations from different sessions was higher than 0.75

(Table 2), providing evidence for the global reproducibilityof projective mapping. As expected, global reproducibilityincreased with the size of differences among samples, asdenoted by the increase in RV coefficient of sample configu-rations between sessions. Besides, when small differencesamong samples were considered, consumers were morereproducible when evaluating samples with texture andflavor differences. As shown in Table 2, the RV coefficientof sample configurations was higher for study 6 than forstudy 5.

Despite the high similarity in sample configurationsbetween sessions, some differences in conclusions regardingsimilarities and differences among samples were identifiedin some of the studies. Although the RV coefficient ofsample configurations between sessions for study 1 was0.96, the position of sample I clearly differed (Fig. 1a). Inthe first session, sample I was located in a distinct positionin the first and second dimensions of the MFA, whereas inthe second session it was regarded as largely similar tosamples L and J (their confidence ellipses overlapped).A similar difference was observed in the position of sampleH in study 2 (Fig. 1b). Studies 4 and 6 showed highlysimilar sample configurations in both sessions (Figs. 2b,d),with no differences in relation to the confidence ellipses thatoverlapped. The fact that samples differed in texture couldhave helped consumers to locate samples more easily andmore reproducibly. In studies 3 and 5, several differencescan be identified in the relative positioning of the samplesand consequently, in the conclusions regarding similaritiesand differences among samples (Figs. 2a,c).

Consumer Descriptions

As shown in Table 3, for the six studies, the number ofterms used for describing samples in the description phaseof projective mapping was similar for session 1 and 2, andthe majority of the terms were used in both sessions. This

TABLE 1. AVERAGE RV COEFFICIENT OF SAMPLE CONFIGURATIONACROSS SIMULATIONS OBTAINED VIA A BOOTSTRAPPINGRESAMPLING APPROACH FOR THE SIX CONSUMER STUDIES

Study

Average RV coefficient across simulations

Session 1 Session 2 Average

1* 0.967 0.970 0.9692† 0.812 0.826 0.8193*‡ 0.980 0.980 0.9804*§ 0.983 0.987 0.9855†‡ 0.946 0.942 0.9446†§ 0.958 0.973 0.966

* Large differences among samples.† Small differences among samples.‡ Samples with flavor differences.§ Samples with texture and flavor differences.



provides preliminary evidence of the stability of consumerdescriptions. The terms used in both sessions of the sixstudies for describing samples are shown in Table S2.

For each study, consensual terms for a significance levelof P ≤ 0.10 were determined following the methodologyproposed by Kostov et al. (2014). For all the studies, thenumber of consensual terms was markedly lower than thetotal number of terms used for describing samples(Table 3). It is interesting to note that for studies 3–6, thenumber of consensual terms was higher for the secondsession than for the first session. Besides, the number ofconsensual words tended to increase with the size of differ-ence among samples.

The majority of the consensual terms identified in thefirst session were also consensual in the second session. For

example, six of the eight consensual terms identified in thefirst session of study 3 were also consensual in the secondsession (Caramel flavor, Consistent, Not much flavor inten-sity, Not very sweet, Vanilla flavor and Very sweet) (Table S2).Moreover, none of the consensual terms identified in thefirst session of study 5 were consensual in the secondsession, which could be related to the fact that samples hadsmall flavor differences.

MFACT allows the visualization of the descriptors usedby consumers to describe samples in the two sessions of thesix studies (Fig. 3). Identical terms are connected with a lineto indicate the size of the difference in how the term wasused between the sessions. The terms used for describingsamples differed in their reproducibility. On the one hand,some of the terms were used in a markedly similar way in

FIG. 1. SAMPLE REPRESENTATION ON THE FIRST AND SECOND DIMENSIONS OF MULTIPLE FACTOR ANALYSIS PERFORMED ON DATA FROM THETWO SESSIONS CONSIDERED IN: (a) STUDY 1 (SALTED – I TO L – AND UNSALTED PLAIN CRACKERS – M TO P) AND (b) STUDY 2 (SALTEDCRACKERS – A TO H). CONFIDENCE ELLIPSES AROUND SAMPLES WERE CREATED USING PARAMETRIC BOOTSTRAPPING



FIG. 2. SAMPLE REPRESENTATION ON THE FIRST AND SECOND DIMENSIONS OF MULTIPLE FACTOR ANALYSIS PERFORMED ON DATA FROM THETWO SESSIONS CONSIDERED IN: (a) STUDY 3 (LARGE FLAVOR DIFFERENCES), (b) STUDY 4 (LARGE FLAVOR AND TEXTURE DIFFERENCES),(c) STUDY 5 (SMALL FLAVOR DIFFERENCES), AND (d) STUDY 6 (SMALL FLAVOR AND TEXTURE DIFFERENCES). CONFIDENCE ELLIPSES AROUNDSAMPLES WERE CREATED USING PARAMETRIC BOOTSTRAPPING



both sessions, being located close to each other in the firstand second session. In general, the most stable terms werethose which described the main sensory differences amongsamples. For example, in study 1, the terms Salty, No salt,Toasted, Burnt, Not toasted and Crunchy were highly repro-ducible (Fig. 3a). Something similar was observed in study 6with the terms Liquid, Runny, Consistent, Thick, Viscous,Creamy, Sweet and Very sweet (Fig. 3f).

On the other hand, terms describing complex sensoryproperties or characteristics of the desserts that did not varyamong samples tended to be less stable. For example, instudy 3, which included samples with flavor differences butwith the same texture, the terms Consistent, Creamy andSmooth were unstable, together with complex flavor attri-butes as Aftertaste, Cookie and Milky flavor (Fig. 3c) The restof the terms, particularly those related to flavor differences(e.g., Caramel flavor, Vanilla flavor, Very Sweet, Sweet, Notsweet and Not very sweet), were located close to each other,suggesting high reproducibility in how consumers describedsamples across sessions. Similarly, the least reproducibleterms in study 5 were mainly related to texture characteris-tics, which did not differ across samples (Smooth, Thick)and complex flavor terms (Artificial flavor, Tasty) (Fig. 3e).The reproducibility of the terms depended on the size ofdifference among samples. Consumers tended to be morereproducible when describing samples with large differences(Fig. 3a,c,d) than when describing sample sets with smalldifferences (Fig. 3b,e,f). Besides, in the milk dessert experi-ments (studies 3–6), consumers were more reproducible indescribing samples with texture and flavor differences thansamples that only differed in their flavor characteristics (cf.Fig. 3c–f).

The terms that were consensual in both sessions tendedto be highly reproducible between sessions (Fig. 3), suggest-

ing that the terms that were used similarly by consumerswere also used in the same way over sessions. However, it isinteresting to note that the most reproducible terms werenot necessarily consensual in both sessions. Many terms thatwere used in a highly reliable way in both sessions were notconsensual in any of the sessions. For example, as shown inFig. 3a, the term No salt was reliably used in study 1 but wasnot consensual in any of the sessions. On the contrary, theterms Toasted flavor and Bitter were among the least repro-ducible while they were consensual in one of the sessions.

The RV coefficients between the frequency tables of bothsessions tended to be high, reaching values higher than 0.80(Table 3). These results suggest that although some of theterms were not reliably used between sessions, descriptionsobtained in both sessions provided similar informationregarding similarities and differences among samples. Asexpected, RV coefficient between the frequency tables ofconsumer descriptions increased with the size of differencesamong samples, reaching values higher than 0.94 for thestudies that included large differences among samples(Table 3).

Consumer Individual Reproducibility

Although global reproducibility was high, consumer indi-vidual reproducibility tended to be low (Table 2). The RVcoefficients of individual sample configurations betweensessions ranged from 0.001 to 0.975, indicating large differ-ences among consumers’ performance. However, averageconsumer reproducibility was low, as well as the percentageof consumers whose configurations were significantly corre-lated. For four out of the six studies, less than 50% ofthe consumer sample configurations were significantlycorrelated.

FIG. 2. CONTINUED.



TAB

LE2.

ESTI

MA

TIO

NO

FG

LOBA

LA

ND

IND

IVID

UA

LRE

PRO

DU

CIB

ILIT

YO

FPR

OJE

CTI

VE

MA

PPIN

GIN

THE

SIX

CO

NSU

MER

STU

DIE

S,U

SIN

GTH

ERV

CO

EFFI

CIE

NT

BETW

EEN

SAM

PLE

CO

NFI

GU

RATI

ON

SO

FTH

ETW

OEV

ALU

ATI

ON

SESS

ION

S

Stud

yIn

ters

essi

onin

terv

alN

umbe

rof

cons

umer

sPr

oduc

tN

umbe

rof

sam

ples

Glo

balR

Vco

effic

ient

betw

een

sess

ions

Con

sum

erin

divi

dual

repr

oduc

ibili

ty¶

Min

imum

indi

vidu

alRV

coef

ficie

nt

Max

imum

indi

vidu

alRV

coef

ficie

nt

Ave

rage

indi

vidu

alRV

coef

ficie

nt

Perc

enta

geof

cons

umer

sw

ithsi

gnifi

cant

RVco

effic

ient

(P≤

0.05

)(%

)

1*2

days

91Pl

ain

crac

kers

80.

960

0.00

10.

958

0.42

234

2†2

days

89Pl

ain

crac

kers

80.

770

0.00

10.

746

0.25

115

3*‡

14da

ys48

Vani

llam

ilkde

sser

ts8

0.98

00.

009

0.97

50.

520

544*

§14

days

48Va

nilla

milk

dess

erts

80.

960

0.01

50.

951

0.51

650

5†‡

14da

ys42

Vani

llam

ilkde

sser

ts8

0.84

00.

004

0.97

20.

256

186†

§14

days

42Va

nilla

milk

dess

erts

80.

920

0.00

30.

968

0.32

115

*La

rge

diff

eren

ces

amon

gsa

mpl

es.

†Sm

alld

iffer

ence

sam

ong

sam

ples

.‡

Sam

ples

with

flavo

rdi

ffer

ence

s.§

Sam

ples

with

text

ure

and

flavo

rdi

ffer

ence

s.¶

Indi

vidu

alre

prod

ucib

ility

was

estim

ated

usin

gth

eRV

coef

ficie

ntbe

twee

nin

divi

dual

sam

ple

confi

gura

tions

betw

een

the

two

sess

ions

.

TAB

LE3.

TOTA

LN

UM

BER

OF

TERM

SA

ND

CO

NSE

NSU

AL

TERM

SFO

RTH

ED

ESC

RIPT

ION

PHA

SEO

FPR

OJE

CTI

VE

MA

PPIN

GFO

RTH

ETW

OSE

SSIO

NS

OF

THE

SIX

CO

NSU

MER

STU

DIE

S

Stud

ySe

ssio

nTo

taln

umbe

rof

term

s

Num

ber

ofco

mm

onte

rms

betw

een

sess

ions

Num

ber

ofco

nsen

sual

term

sat

P≤

0.10

Num

ber

ofco

mm

onco

nsen

sual

term

sbe

twee

nse

ssio

ns

RVco

effic

ient

betw

een

sess

ions

from

MFA

CT

1*1

3024

136

0.98

226

122†

135

276

20.

802

284

3*‡

129

258

60.

982

3717

4*§

131

2716

120.

942

3518

5†‡

120

184

00.

812

275

6†§

127

2210

80.

942

2611

*La

rge

diff

eren

ces

amon

gsa

mpl

es.

†Sm

alld

iffer

ence

sam

ong

sam

ples

.‡

Sam

ples

with

flavo

rdi

ffer

ence

s.§

Sam

ples

with

text

ure

and

flavo

rdi

ffer

ence

s.



FIG. 3. REPRESENTATION OF THE TERMS USED BY CONSUMERS TO DESCRIBE THE SAMPLES, ON THE FIRST AND SECOND DIMENSIONS OFTHE MULTIPLE FACTOR ANALYSIS FOR THE CONTINGENCY TABLES PERFORMED ON DATA FROM THE TWO SESSIONS CONSIDERED IN:(a) STUDY 1 (PLAIN CRACKERS, LARGE DIFFERENCES), (b) STUDY 2 (PLAIN CRACKERS, SMALL DIFFERENCES), (c) STUDY 3 (MILK DESSERTS,LARGE FLAVOR DIFFERENCES), (d) STUDY 4 (MILK DESSERTS, LARGE FLAVOR AND TEXTURE DIFFERENCES), (e) STUDY 5 (MILK DESSERTS,SMALL FLAVOR DIFFERENCES), AND (f) STUDY 6 (MILK DESSERTS, SMALL FLAVOR AND TEXTURE DIFFERENCES). TERMS USED IN THE FIRSTSESSION ARE INDICATED USING GREY DIAMONDS AND ITALIC LETTERS, WHILE TERMS USED IN THE SECOND SESSION ARE INDICATED USINGBLACK DIAMONDS AND REGULAR LETTERS. TERMS HIGHLIGHTED IN BLACK WERE CONSENSUAL FOR P ≤ 0.10 (KOSTOV ET AL. 2014).IDENTICAL TERMS ARE CONNECTED WITH A LINE TO INDICATE THE SIZE OF THE DIFFERENCE IN HOW THE TERM WAS USED BETWEEN THESESSIONS



As expected, consumer individual reproducibility mark-edly increased with the size of the differences amongsamples. For example, average RV coefficient of individualconfigurations was 0.52 for milk dessert samples with largeflavor differences (study 3) and 0.26 for samples with smallflavor differences (study 5). Additionally, in these studies,the percentages of consumers whose configurations weresignificantly correlated between sessions were 54% and18%, respectively (Table 2).

DISCUSSION

The present work evaluated global and individual reproduc-ibility of projective mapping for sensory characterizationwith consumers using sample sets that differed in the size ofdifference among samples. Across the six studies, the RVcoefficients of sample configurations between sessions werehigher than 0.75. The minimum RV value that has beenconsidered as indicator of good agreement between sampleconfigurations ranges from 0.65 to 0.85 (Lawless andGlatter 1990; Faye et al. 2004; Abdi et al. 2007; Lelièvre et al.2008; Kennedy 2010). Considering these values, it can beconcluded that in the present study, sample configurationswere relatively stable across sessions, and that in the sixstudies, test–retest reproducibility of projective mappingwith consumers proved to be relatively high. These resultsare in agreement with several authors that reported thatconsensus sample configurations from projective mappingwith trained and untrained assessors were stable across ses-sions (Risvik et al. 1994, 1997; Perrin and Pagès 2009;Kennedy 2010; Hopfer and Heymann 2013). High repro-ducibility of consumer-based sensory characterization hasalso been reported for other methodologies like sortingtasks (Lawless and Glatter 1990; Cartier et al. 2006; Cholletet al. 2011) and check-all-that-apply (CATA) questions(Jaeger et al. 2013).

Despite the fact that RV coefficients were higher than0.75, some differences in conclusions regarding similaritiesand differences among samples were identified between rep-licates, particularly for studies which involved samples withsmall differences. A similar result has been reported byBarcenas et al. (2004) when working with ewes’ milkcheeses. These authors reported that the relative position oftwo samples changed across replicates, modifying conclu-sions regarding their similarities and differences with therest of the sample set. On the contrary, Kennedy (2010) andHopfer and Heymann (2013) reported that overall similari-ties and dissimilarities among the samples were stable overthe triplicate evaluation. Results from the present worksuggest that for sample sets with small differences, caremust be taken when drawing conclusions from sampleconfigurations obtained using projective mapping withconsumers without the use of replicates. Further research is

necessary to determine if replicated projective mapping isnecessary prior to the design of the study.

In the present work, the majority of the terms elicited todescribe samples in the description phase of projectivemapping were used in a similar way in both sessions(Fig. 3). Overall, the terms responsible for the main differ-ences in the sensory characteristics of the samples werehighly reproducible, while terms related to complex sensoryattributes or characteristics that did not differ amongsamples tended to be not reproducible. This suggests thatconsumer descriptions in projective mapping tasks shouldbe taken with care, particularly when evaluating sampleswith small differences. Although open-ended questions havebeen considered as an alternative method for sensory char-acterization with consumers (Ares et al. 2010b; Symoneauxet al. 2012), results from the present work show that con-sumers are not reproducible when using many terms. Thiswould suggest the need to check the reliability of the termsfor concluding on the main sensory characteristics respon-sible for similarities and differences among samples.

Methodologies that enable the selection of reliable termswould be useful to improve the interpretation of sensoryspaces obtained from the application of holistic methodolo-gies with consumers. Kostov et al. (2014) proposed theidentification of consensual terms for selecting the mostreliable terms elicited in free description tasks. In thepresent work, this methodology was not able to predict thereproducibility of the terms. Although consensual words inboth sessions were used in a reproducible way, there weremany terms that were not consensual but reproducible, aswell as terms that were consensual in one of the sessions butwere not reproducible. Thus, further research is needed toimprove the interpretation of consumer responses to free-description tasks.

Although global reproducibility was high, consumer indi-vidual reproducibility tended to be low in the six studies(Table 2). The average RV coefficients between sample con-figurations of the two sessions were lower than 0.55, whilethe percentage of consumers with significant RV coefficientbetween sessions was lower than 54%. This result is inagreement with Risvik et al. (1994, 1997), Barcenas et al.(2004), Hopfer and Heymann (2013) and Kennedy (2010).In particular, this last author reported that 10 out of 15 con-sumers had RV coefficient between replications lower than0.5. Similar results have been reported for CATA questionsfor sensory characterization. Jaeger et al. (2013) reportedthat despite the fact that global reproducibility of CATAquestions was high, consumer individual reproducibilitytended to be low. This suggests that differences in individualperformances between sessions tend to compensate amongconsumers, yielding stable consensus configurations.

The low RV coefficients between individual sample con-figurations can be attributed to differences in consumers’



criteria for placing the samples, particularly due to trainingand familiarization with projective mapping and thesample set. In this sense, Kennedy (2010) reported that theinternal consistency and agreement of untrained consum-ers when using projective mapping increased over triplicateevaluations. In the present work, the percentage of vari-ance explained by the first and second dimensions of theMFA and the stability of sample configurations (as evalu-ated through a bootstrapping resampling approach) didnot increase with duplicate evaluation. However, thenumber of consensual terms tended to be larger in thesecond session than in the first one, which suggests thatfamiliarization with the sensory space can improve con-sumer performance in descriptive tasks. Therefore, consid-ering these results, it would be interesting to study iffamiliarization with projective mapping and/or with thesample set increases assessor reproducibility when usingprojective mapping for sensory characterization, particu-larly considering that some consumers can find this meth-odology difficult to apply (Nestrud and Lawless 2008;Veinand et al. 2011). Several authors have included a shortintroduction or training prior to the projective mappingtask (Risvik et al. 1994, 1997; Barcenas et al. 2004; Veinandet al. 2011; Carrillo et al. 2012; Hopfer and Heymann2013), which can contribute to improve consumers’performance.

Global and individual reproducibility of projectivemapping increased with the size of differences amongsamples. This observation, together with the fact that con-clusions regarding similarities and differences amongsamples were not stable in some cases, indicates the needto define stability indices for sample configurations. Theseindices could be useful to decide whether or not to repli-cate projective mapping in order to ensure that conclu-sions regarding similarities and differences among sampleswould be repeatedly identified. Further research is neces-sary to determine if increasing the number of consumerscan be an alternative approach to replicated evaluations forthe stabilization of sample configurations. This is aninteresting idea to explore, considering that in manysituations, it is not practical to get the same consumers torepeat the study.

Studying the stability of sample configurations bysubsampling using bootstrapping approaches could be aninteresting approach and can contribute to the developmentof guidelines for practitioners. In the present study, the sta-bility of sample configurations was studied using simulatedrepeated experiments by sampling repeatedly from thepopulation of interest, as proposed by Faye et al. (2006) andBlancher et al. (2012) for sorting tasks. As shown in Tables 1and 2, there was a good agreement between the stability andreproducibility of sample configurations. The studies thatshowed average RV coefficients across replications higher

than 0.95 (studies 1, 3, 4 and 6) were highly reproducible,reaching RV coefficients between replicates higher than0.90. These results suggest the need to further study therelationship between the stability and reproducibility ofsample configurations from projective mapping. This typeof research can contribute to the definition of threshold fordeciding if results from projective mapping are reliable andwhether or not replication is needed. When the stability ofsample configuration is found to be low, replication of thestudy would be recommended to check that similarities anddifferences among samples remain when repeating thewhole study. When replicating projective mapping tasks,conclusions should be drawn from consensus sample con-figurations across replicates from hierarchical multiplefactor analysis (HMFA) (Le Dien and Pagès 2003). Thismethodology is an extension of MFA and balances therelevance of groups of variables with different hierarchyand provides an overall result. In the context of replicatedprojective mapping tasks, HMFA provides consensussample configurations after balancing data from eachseparate session.

CONCLUSIONS

Results from the present work showed that althoughmost consumers were only slightly reproducible, globalconfigurations from projective mapping were reasonablystable across sessions. Descriptions of samples were usedin a similar way in both sessions, the terms responsiblefor the main differences were highly reproducible,while complex sensory attributes or characteristicsthat did not differ among samples tended to be notreproducible.

The degree (large or small) and type (flavor or flavor andtexture) of difference among samples had a strong influenceon both global and individual reproducibility of projectivemapping, suggesting that care must be taken when relyingon results of projective mapping with consumers obtainedwithout the use of replicates. In this sense, the use of indicesthat evaluate the stability of sample configurations can con-tribute to decide whether or not a replication is needed. Inthe present work, the stability index calculated using a boot-strapping resampling approach was strongly related toconsumer global reproducibility. Research in this area couldcontribute to the selection of criteria for evaluating the reli-ability of sensory characterization with consumers and todefine the need of using replicates with trained, semitrainedand untrained assessors. Besides, further research on thereproducibility of projective mapping when working withsample sets of different complexity can help to decide ifreplicated projective mapping is necessary prior to thedesign of the experiment.



ACKNOWLEDGMENTS

The authors are indebted to Comisión Sectorial deInvestigación Científica (Universidad de la República,Uruguay) and to CAPES-Brazil for financial support, toAgencia Nacional de Investigación e Innovación (ANII,Uruguay) for the scholarship granded to author LeticiaVidal.

The authors would also like to thank the Spanish Minis-try of Science and Innovation for the contract awarded tothe author P. Varela (Juan de la Cierva Program) and to theSpanish Ministry of Education, Culture and Sports for theJosé Castillejo grant awarded to author P. Varela.

REFERENCES

ABDI, H. 2010. Congruence: Congruence coefficient,RV coefficient, and mantel coefficient. In Encyclopedia ofResearch Design (N.J. Salkind, D.M. Dougherty andB. Frey, eds.) pp. 222–229, Sage, Thousand Oaks,CA.

ABDI, H., VALENTIN, D., CHOLLET, S. and CHREA, C. 2007.Analyzing assessors and products in sorting tasks:DISTATIS, theory and applications. Food Qual. Prefer. 18,627–640.

ALBERT, A., VARELA, P., SALVADOR, A., HOUGH, G. andFISZMAN, S. 2011. Overcoming the issues in the sensorydescription of hot served food with a complex texture.Application of QDA, flash profiling and projective mappingusing panels with different degrees of training. Food Qual.Prefer. 22, 463–473.

ARES, G., DELIZA, R., BARREIRO, C., GIMÉNEZ, A. andGÁMBARO, A. 2010a. Comparison of two sensory profilingtechniques based on consumer perception. Food Qual. Prefer.21, 417–426.

ARES, G., GIMÉNEZ, A., BARREIRO, C. and GÁMBARO, A.2010b. Use of an open-ended question to identifydrivers of liking of milk desserts. Comparison withpreference mapping techniques. Food Qual. Prefer. 21,286–294.

ARES, G., VARELA, P., RADO, G. and GIMENEZ, A. 2011. Areconsumer profiling techniques equivalent for some productcategories? The case of orange-flavored powdered drinks. Int.J. Food Sci. Technol. 46, 1600–1608.

BARCENAS, P., PÉREZ ELORTONDO, F.J. and ALBISU, M.2004. Projective mapping in sensory analysis ofewes milk cheeses: A study on consumers andtrained panel performance. Food Res. Int. 37, 723–729.

BÉCUE-BERTAUT, M. and PAGÈS, J. 2004. A principal axesmethod for comparing contingency tables: MFACT. Comput.Stat. Data Anal. 45, 481–503.

BLANCHER, G., CLAVIER, B., EGOROFF, C., DUINEVELD, K.and PARCON, J. 2012. A method to investigate the stability ofa sorting map. Food Qual. Prefer. 23, 36–43.

CARRILLO, E., VARELA, P. and FISZMAN, S. 2012.Packaging information as a modulator of consumers’perception of enriched and reduced-calorie biscuitsin tasting and non-tasting tests. Food Qual. Prefer. 25,105–115.

CARTIER, R., RYTZ, A., LECOMTE, A., POBLETE, E.,KRYSTLIK, J., BELIN, E. and MARTIN, N. 2006. Sortingprocedure as an alternative to quantitative descriptive analysisto obtain a product sensory map. Food Qual. Prefer. 17,562–571.

CHOLLET, S., LELIÈVRE, M., ABDI, H. and VALENTIN, D.2011. Sort and beer: Everything you wanted to know aboutthe sorting task but did not dare to ask. Food Qual. Prefer. 22,507–520.

DE SALDAMANDO, L., DELGADO, J., HERENCIA, P.,GIMÉNEZ, A. and ARES, G. 2013. Polarized sensorypositioning: Do conclusions depend on the poles? Food Qual.Prefer. 29, 25–32.

DEHLHOLM, C., BROCKHOFF, P.B., MEJNERT, L., AASLYNG,M.D. and BREDIE, W.L.P. 2012a. Rapid descriptive sensorymethods – comparison of free multiple sorting, partialnapping, napping, flash profiling and conventional profiling.Food Qual. Prefer. 26, 267–277.

DEHLHOLM, C., BROCKHOFF, P.B. and BREDIE, W.L.P.2012b. Confidence ellipses: A variation based on parametricbootstrapping applicable on Multiple Factor Analysis resultsfor rapid graphical evaluation. Food Qual. Prefer. 26,278–280.

FAYE, P., BRÉMAUD, D., DURAND-DAUBIN, D.,COURCOUX, P., GIBOREAU, A. and NICOD, A. 2004.Perceptive free sorting and verbalization tasks with naivesubjects: An alternative to descriptive mappings. Food Qual.Prefer. 15, 781–791.

FAYE, P., BRÉMAUD, D., TEILLET, E., COURCOUX, P.,GIBOREAU, A. and NICOD, H. 2006. An alternative toexternal preference mapping based on consumer perceptivemapping. Food Qual. Prefer. 17, 604–614.

HOPFER, H. and HEYMANN, H. 2013. A summary ofprojective mapping observations – the effect of replicates andshape, and individual performance measurements. Food Qual.Prefer. 28, 164–181.

ISO. 1988. Sensory Analysis: General Guidance for the Design ofTest Rooms, ISO 8589, International Organization forStandardization, Geneva, Switzerland.

JAEGER, S., CHHEANG, S.L., YIN, J., BAVA, C.M.,GIMENEZ, A., VIDAL, L. and ARES, G. 2013.Check-all-that-apply (CATA) responses elicited byconsumers: Within-assessor reproducibility and stability ofsensory product characterizations. Food Qual. Prefer. 30,56–67.

JOSSE, J., PAGÉS, J. and HUSSON, F. 2008. Testing thesignificance of the RV coefficient. Comput. Stat. Data Anal.53, 82–91.

KENNEDY, J. 2010. Evaluation of replicated projective mappingof granola bars. J. Sensory Studies 25, 672–684.



KENNEDY, J. and HEYMANN, H. 2009. Projective mappingand descriptive analysis of milk and dark chocolate. J. SensoryStudies 24, 220–233.

KING, M.J., CLIFF, M.A. and HALL, J.W. 1998. Comparison ofprojective mapping and sorting data collection andmultivariate methodologies for identification ofsimilarity-of-use of snack bars. J. Sensory Studies, 13,347–358.

KOSTOV, B., BÉCUE-BERTAUT, M. and HUSSON, F. 2014. Anoriginal methodology for the analysis and interpretation ofword-count based methods: Multiple factor analysis forcontingency tables complemented by consensual words. FoodQual. Prefer. 32, 35–40.

LAWLESS, H.T. and GLATTER, S. 1990. Consistency ofmultidimensional scaling models derived from odor sorting.J. Sensory Studies 5, 217–230.

LAWLESS, H.T. and HEYMANN, H. 2010. Sensory Evaluation ofFood. Principles and Practices, 2nd Ed., Springer, New York,NY.

LE DIEN, S. and PAGÈS, J. 2003. Hierarchical multiple factoranalysis: Application to the comparison of sensory profiles.Food Qual. Prefer. 14, 397–403.

LELIÈVRE, M., CHOLLET, S., ABDI, H. and VALENTIN, D.2008. What is the validity of the sorting task for describingbeers? A study using trained and untrained assessors. FoodQual. Prefer. 19, 697–703.

LÊ, S. and HUSSON, F. 2008. SensoMineR: A package forsensory data analysis. J. Sensory Studies 23, 14–25.

LÊ, S., JOSSE, J. and HUSSON, F. 2008. FactoMineR: An Rpackage for multivariate analysis. J. Stat. Soft. 25, 1–18.

LOUW, L., MALHERBE, S., NAES, T., LAMBRECHTS, M.,RENSBURG, P. and NIEUWOUDT, H. 2013. Validation oftwo Napping techniques as rapid sensory screening tools forhigh alcohol products. Food Qual. Prefer. 30, 192–201.

MEILGAARD, M.C., CIVILLE, G.V. and CARR, B.T. 1999.Sensory Evaluation Techniques, 2nd Ed., CRC Press, BocaRaton, FL.

MOUSSAOUI, K.A. and VARELA, P. 2010. Exploring consumerproduct profiling techniques and their linkage to aquantitative descriptive analysis. Food Qual. Prefer. 21,1088–1099.

MURRAY, J.M., DELAHUNTY, C.M. and BAXTER, I.A. 2001.Descriptive sensory analysis: Past, present and future. FoodRes. Int. 34, 461–471.

NESTRUD, M.A. and LAWLESS, H.T. 2008. Perceptual mappingof citrus juices using projective mapping and profiling datafrom culinary professionals and consumers. Food Qual.Prefer. 19, 431–438.

NESTRUD, M.A. and LAWLESS, H.T. 2010. Perceptual mappingof apples and chesses using projective mapping and sorting.J. Sensory Studies 25, 309–324.

PAGÈS, J. 2005. Collection and analysis of perceived productinter-distances using multiple factor analysis: Application tothe study of 10 white wines from the Loire Valley. Food Qual.Prefer. 16, 642–649.

PAGÈS, J., CADORET, M. and LÊ, S. 2010. The sorted Napping:A new holistic approach in sensory evaluation. J. SensoryStudies 25, 637–658.

PERRIN, L. and PAGÈS, J. 2009. Construction of a productspace from the ultra-flash profiling method: Application to 10red wines from the Loire Valley. J. Sensory Studies 24,372–395.

PERRIN, L., SYMONEAUX, R., MAÎTRE, I., ASSELIN, C.,JOURJON, F. and PAGÈS, J. 2008. Comparison of threesensory methods for use with the Napping procedure: Case often wines from Loire Valley. Food Qual. Prefer. 19,1–11.

R DEVELOPMENT CORE TEAM. 2007. R: A Language andEnvironment for Statistical Computing, R Foundation forStatistical Computing, Vienna. ISBN 3-900051-07-0.

RISVIK, E., MCEWAN, J.A., COLWILL, J.S., ROGERS, R. andLYON, D.H. 1994. Projective mapping: A tool for sensoryanalysis and consumer research. Food Qual. Prefer. 5,263–269.

RISVIK, E., MCEWAN, J.A. and RODBOTTEN, M. 1997.Evaluation of sensory profiling and projective mapping data.Food Qual. Prefer. 8, 63–71.

ROBERT, P. and ESCOUFIER, Y. 1976. A unifying tool for linearmultivariate statistical methods: The RV coefficient. AppliedStat. 25, 257–265.

ROSS, C.F., WELLER, K.M. and ALLDREDGE, J.R. 2012.Impact of serving temperature on sensory propertiesof red wine as evaluated using projectivemapping by a trained panel. J. Sensory Studies 27,463–470.

STONE, H., SIDEL, J.L., OLIVER, S., WOOLSEY, A. andSINGLETON, R.C. 1974. Sensory evaluationby quantitative descriptive analysis. Food Technol.28, 24–33.

SYMONEAUX, R., GALMARINI, M.V. and MEHINAGIC, E.2012. Comment analysis of consumer’s likes anddislikes as an alternative tool to preference mapping.A case study on apples. Food Qual. Prefer. 24,59–66.

VALENTIN, D., CHOLLET, S., LELIEVRE, M. and ABDI, H.2012. Quick and dirty but still pretty good: A review of newdescriptive methods in food science. Int. J. Food Sci. Technol.47, 1563–1578.

VARELA, P. and ARES, G. 2012. Sensory profiling, the blurredline between sensory and consumer science. A review of novelmethods for product characterization. Food Res. Int. 48,893–908.

VEINAND, B., GODEFROY, C., ADAM, C. and DELARUE, J.2011. Highlight of important product characteristics forconsumers. Comparison of three sensory descriptivemethods performed by consumers. Food Qual. Prefer. 22,474–485.

YU, C.H. 2005. Test-retest reliability. In Encyclopedia of SocialMeasurement, Vol. 3 (K. Kempf-Leonard, ed.) pp. 777–784,Academic Press, San Diego, CA.



SUPPORTING INFORMATION

Additional Supporting Information may be found in theonline version of this article at the publisher’s web-site:

Table S1. Formulation of the Vanilla Milk Desserts Used inStudies 3–6.

Table S2. Terms Used by Consumers for DescribingSamples in the Description Phase of Projective Mapping inthe Six Consumer Studies.



Reproducibilidad mapping

Education

Transcript of Reproducibilidad mapping