MODELING DIFFERENCES BETWEEN PANELISTS IN USE OF MEASUREMENT SCALES

CLARE WILKINSON' and DOGAN YUKSEL

Agrotechnological Research Institute Department of Agricultural Research (ATO-DLO)

The Netherlands

Received for Publication May 12, 1996

ABSTRACT

In sensory data sets, an important source of differences between panelists is in their use of the measurement scale. These differences can be summarized in differences in location, the overall level, and differences in dispersion, the range of the scale used. This paper discusses a method of correcting for these differences by jointly modeling location and dispersion using a see-saw algorithm. This ap- proach is also applicable when scores are not n o m l l y distributed and when there is a (nonlinear) relationship between the dispersion and the location. The ap- proach is illustrated with an example for flavor data offreeze-dried and hot-air dried peppers.

INTRODUCTION

Sensory experiments will typically take the form of a small number of sessions in which a group of trained panelists scores a selection of food products for the intensity of a set of attributes. The aim of an analysis is usually to compare the sensory profiles of the various products; however, even after training, panelists will differ systematically in the scores they give in identical circumstances. A number of reasons for these individual differences may be identified. Panelists may differ in what they perceive, exhibiting (1) different threshold intensities, or (2) differences in the ability to perceive (small) differences in intensity. They may also differ in how they record their perceptions. (3) They may record the same stimulus under different attribute names. (4) They may differ in their use of the measurement scale, both in location (where on the scale the scores tend to be located) and in dispersion (the range of the scale used). Thus some panelists Correspondence should be sent to: Clare Wilkinson, ATO-DLO, PO Box 17, NL-6700 AA Wagen- ingen, the Netherlands. E-mal: E.C.WILKINSON@ATO.DLO.NL Journal of Sensory Studies 12 (1997) 55-68. All Rights Reserved. 0 Copyright 1997 by Food & Nutrition Press, Inc., Trumbull, Connecticut, 55

56 C. WILKINSON and D. YUKSEL

will use the whole available range for scoring while others will concentrate on one part of the scale (differences in dispersion). Moreover, some panelists may primarily score in the lower part of the scale while others may score largely in the middle or upper region (differences in location) (Naes 1990).

Any analysis of the data matrix will need to take account of systematic dif- ferences between panelists in order to arrive at estimates for the attribute inten- sities for each product. This paper concentrates on one particular source of in- dividual differences, namely differences in the use of the measurement scale. It considers existing methods for dealing with such differences and proposes a new method using a see-saw algorithm for the joint modeling of location and disper- sion effects. Differences in the use of the measurement scale are corrected for, allowing better estimates of product effects and other systematic panelist effects. This method has the advantage that it is also applicable when the usual requirements of linearity and normality do not apply.

In what follows, it is assumed for the purposes of illustration that the session effect is purely random. V

Measurement Scale

Traditionally differences in location and dispersion are considered nuisance parameters, that is, parameters which bear no further relation to panelist perfor- mance and carry no relevant information. The scores for each attribute are cor- rected for these panelist effects to give so-called “z-scores” with mean 0 and variance 1 ; the panelist sample mean is subtracted from the score to correct for the location effect. This value is divided by the panelist sample standard devia- tion to correct for differences in dispersion (Naes 1990).

This standardization technique is easy to use and has been widely used in sen- sory analysis to correct for individual differences in the use of the measurement scale. However it has a number of limitations. Firstly, this technique assumes that the data are normally distributed: the use of z-scores derives from the z- transformation used in statistics to obtain the standard normal distribution. Use of the technique implies a symmetrical distribution, and that the dispersion for an individual panelist is constant and independent of the location effect. While robust to small deviations from normality, the standardization technique will not be appropriate when many observations are close to the bounds of the measure- ment scale, leading to a skew distribution.

A further limitation is that the correction for dispersion effects, the panelist sample standard deviation, is calculated without taking account of product ef- fects, Thus a panelist showing large between-product variation and small within- product (replication) variation may have the same total variance as another panelist showing small between-product and large within-product variation. Using the stan-

MODELING DIFFERENCES BETWEEN PANELISTS 51

dardization technique, they will both have the same dispersion correction factor, however it would seem desirable to differentiate between these two sources of variation. Differences between panelists in between-product variation may reflect differences in the use of the measurement scale, but may equally reflect differences in the perceived relative intensity of that attribute for the different products.

Other, more sophisticated, methods have been proposed which correct for dif- ferences in the use of the measurement scale. Generalized Procrustes Analysis (GPA) (Arnold and Williams 1987) scales and transforms the scores for each panelist to achieve a consensus profile. This method applies a different location correction for each attribute but uses the same scaling factor for all attributes, assuming that differences in dispersion are a characteristic of the panelist, irrespec- tive of the attribute which he/she is scoring. Naes (1990) finds scaling factors gja for each attribute a which minimize the sum of the squared differences be- tween each pair of profiles after centering and multiplication by the scaling fac- tors. As with GPA, this method focuses on the profile as a whole rather than each attribute separately, although the scaling factors are now allowed to vary per attribute. Like the standardization technique, neither of these methods dif- ferentiate between differences in between-product variation and differences in within-product variation.

Methods which do take account of product effects have been described by Prit- chett Mangan (1992) and Brockhoff and Skovgaard (1994). In both cases, as with the z-scores, the scaling is carried out separately for each individual attribute. Pritchett Mangan proposes a weighted analysis where the weights are an inverse function of the ratio between the replication variance for a panelist-product com- bination and the maximum obtainable variance given the observed range for that panelist. This penalizes panelist-product combinations with relatively high variability (low precision). In effect a different scaling factor is being applied for every panelist product combination. Brockhoff and Skovgaard propose a parametric model in which panelist effects are sub-divided into a location effect, a “stretching and shrinking” constant for between-product variation and a panelist- dependent within-product variance. Panelist and product effects are estimated alter- nately using an iterative method.

This paper discusses an approach to dealing with measurement scale effects in which panelist location and dispersion effects are modeled jointly, and con- currently with product and other effects, using a see-saw algorithm. The aim is to obtain the best estimates of these effects, taking account of differences in the use of the measurement scale. The approach deals with each attribute separately and explicitly allows for nonnormally distributed scores.

58 C. WILKINSON and D. YUKSEL

MATERIALS AND METHODS

Modeling Panelist Dispersion and Location Effects Using a See-saw Algorithm

First the case is discussed where the scores are normally distributed, although later this restriction will be relaxed. It is assumed that differences in location are a function of panelist, product and their interaction and that differences in disper- sion are a function of panelist. The panelist-product interaction for the loctaion incorporates all differences between panelists in the estimated relative intensities for the different products. This includes differences in the range used to separate the different products (sometimes, but not necessarily, a measurement scale ef- fect), and differences in the rank order of the products (the effect of differences in perception).

If the scores for an individual attribute are (approximately) normally distributed, a weighted regression is carried out on the scores in which the weights are equal to the inverse of the residual variance after fitting for location effects, and the residual variance is allowed to vary per panelist. This involves two models, a location model for the scores and a dispersion model for the variance.

The location model is developed for the mean scores, which are modeled as a function of panelist and product, as follows:

where E[yi,k) is the expected value of the ith score of panelist j on product k, PO.) is the overall mean, Cuj is the panelist main effect, Pk is the product main effect and aPjk is the panelist product interaction. The panelist location effect in the use of the measurement scale is then captured in the panelist main effect. The estimated effects are obtained by minimizing the weighted residual sum of squares

and E[d@,j] is the expected value of the dispersion parameter for panelist j , db)j. For the normal distribution, this is equal to the panelist variance, uj2.

The dispersion model takes the squared residuals (the difference between the actual scores and fitted values) from the location model as the response variable. The residuals are modeled as a function of panelist. The fitted values for the disper- sion model are the estimates of the panelist dispersion parameter.

MODELING DIFFERENCES BETWEEN PANELISTS 59

where d@)j = (Yijk - E[Yijk])’, p(d) is the overall mean effect and Xj is the panelist effect. The function f links the linear combination of mean effect and panelist effect to the expected value for the dispersion parameter. It is chosen to ensure, for example, that the expected values are always positive. Typically the log func- tion is used.

In a model such as the location model, in which the response variable follows the normal distribution, the squared residuals follow the Gamma distribution. In this case, estimates for the parameters p(d) and Xj can be obtained by fitting a Generalized Linear Model (GLM) (McCullagh and Nelder 1989). It is a generaliza- tion of the ordinary linear model, and is described in more detail below.

It will be seen that the estimation of the effects for the location model requires E[d~)j] (the expected value from the dispersion model) to be known. On the other hand, the estimation of the effects for the dispersion model requires E[yijk] (the expected values from the location model) to be known. This problem is solved by alternately fitting the location and dispersion model and in each case using instead of the expected value, the most recent fitted values from the other model. The algorithm used to do this is a see-saw algorithm and works as follows:

(1) Initial weights Wj are defined for panelist j. For example, all Wj = 1 . (2) The location model is fitted using weighted regression or ANOVA with

(3) The dispersion model is fitted to the squared residuals and the fitted values

(4) New weights Wj = l/d(,.)j are calculated. (5) Steps 2 to 4 are repeated until subsequent results are sufficiently close. For the location and dispersion models described here, a panelist with higher

dispersion (variance) receives less weight. This dispersion is calculated with respect to the fitted scores given by the location regression model for each panelist pro- duct combination.

Use of the normal distribution for the location model assumes that the variance is independent of the mean score, that the distribution is not skew and that the score is a linear function of the predictor variables (panelist, product effect, etc.). Clearly these assumptions will not always hold. Taking the example of the an- chored line scale, theoretically the relationship cannot be strictly linear as this would allow for predicted values outside the bounds of the line. Also a smaller variance and a skew distribution is often observed for mean scores which are in the region of the bounds of the line.

To accommodate these types of deviation from the assumption of normality, it is suggested that the location models should be drawn from the class of general-

weights Wj. Ths residuals are saved.

doj are saved.

60 C . WILKINSON and D. YUKSEL

ized linear models (McCullagh and Nelder 1989). These models are a generaliza- tion of the classic.ANOVA and linear regression models. The expected value is related to a linear combination of the predictor variables (the linear predictor) via a given function, known as the linkfunction.

f(E[y]) = linear predictor (5 )

In the case of the ANOVA and linear regression, the function f is simply the identity function. Furthermore the variance is set equal to a given function of the expected value, known as the variancefunction, multiplied by a dispersion parameter.

Var[y] = d*V(E[y]) (6)

where V(E[y]) is the variance function and d the dispersion parameter. For ANOVA and linear regression, the variance function is 1 and the variance is therefore equal to the dispersion parameter. Estimates for the effects in the linear predictor can be obtained using an iterative method, available in most major statistical packages.

In applying a GLM for the location model, the user needs to decide on an ap- propriate relationship between the score expected value and the linear predictor (the link function) and between the score expected value and the variance (the variance function). Taking the case of the anchored line scale, a logistic link func- tion will ensure that the expected values for the mean score are constrained to be within the bounds of the line. Thus

In E[yl =linear predictor (n-E[yl )

( 7 )

where n is the maximum possible score. If the variance for the anchored line scale is thought to vary according to proximity to the line bounds, then the variance function associated with the binomial distribution may be used.

This variance function is largest when E[y] = n/2 (the middle of the line) and decreases as E[y] approaches the bounds of the line.

As mentioned above, the dispersion model should always be fitted using a GLM. As the squared residuals of a GLM always approximately follow a Gamma distribution (exactly in the case of the normal distribution), the variance function associated with the Gamma distribution,

MODELING DIFFERENCES BETWEEN PANELISTS 61

should be used. Furthermore, a log link function is recommended to link the ex- pected value of the dispersion parameter to the linear predictor, as this ensures that the dispersion parameter is always positive.

The see-saw algorithm can be applied using any statistical package which allows the fitting of GLM’s. The code used for the statistical package Genstat (Genstat 5 comittee, Rothemstead, U.K., 1993) to fit the example dataset is shown in the Appendix.

Example Data Set

The application of the see-saw algorithm will be illustrated using flavor data for hot-air dried and freeze-dried bell peppers.

The flavor data set of hot-air dried and freeze-dried bell peppers (Capsicum Annuum) is part of a study on the flavor characterization of fresh and dried vegetables (Luning et al. 1994, Luning et al. 1995). Fruits of several Dutch bell pepper cultivars of different colors (two green, two red, one yellow and one white) which had either been hot-air dried or freeze-dried, were tasted by a panel of 12 trained panelists using a consensus vocabulary with 25 flavor descriptors. The samples were scored on a line scale (length 100 units) that was anchored at the ends by the terms “not present” (score of 0) and “present intensely” (score of 100). The hot-air dried and freeze-dried samples were presented after rehydra- tion (approximately 60% of initial fresh weight was recovered). Three replicates of each sample were tasted by the panel and the experiment was completed within one week.

The total flavor intensities of the hot air-dried and freeze-dried peppers were low and this was reflected in the values for the individual descriptors. The final data matrix had a relatively high proportion of low and zero scores. As a result, the data for all attributes was distinctly skewed to the left. Moreover, a lower mean score was associated with a low spread in scores (Fig. 1). The assumptions of normality clearly do not hold.

The flavor data was analyzed using the see-saw algorithm. For the location model, the variance function for the binomial distribution was used. Furthermore the logistic link function was used to ensure that predicted values were constrained to be within the limits of the line. The location model incorporated a panelist effect, a product effect (split into drying method and cultivar) and their interac- tion. For the dispersion model, the variance function associated with the Gamma distribution was used, together with a log link function. The see-saw algorithm was implemented using the Genstat statistical package.

62 C. WILKINSON and D. YUKSEL

frequency 3 00

250

200

150

100

50

0 0 I 0 10-10

1 . . . . . . . . . . . - - . - . .

Ir grassy

n

20 10 90-40 40.50 50-60 60-70 10-80 80-90 90-100

score ranges FIG. 1. HISTOGRAM OF SCORES ON ANCHORED LINE SCALE (0 = NOT PRESENT, 100

BINED DATA FOR ALL PANELISTS AND PRODUCTS = PRESENT INTENSELY) FOR TWO FLAVOR ATTRIBUTES OF DRIED PEPPERS - COM-

RESULTS

Figure 2 gives an example of the predicted location mean scores, g.j., for five individual panelists and four of the attributes. Figure 3 shows the predicted disper- sion parameters, a(,,lj. These examples show clearly that there are considerable differences between panelists in their use of the measurement scale, both in loca- tion and in dispersion. However, panelists also vary in their use of the scale for the different attributes. Thus while panelist P3 has consistently low location ef- fects, panelist P5 varies considerably depending on the attribute to be measured. Similarly, the dispersion parameters of panelist P4 vary within a narrow range while panelist PI shows much more variation in the dispersion parameter from attribute to attribute.

MODELING DIFFERENCES BETWEEN PANELISTS

predicted mean location score

""?

63

___

attributes

j =grassy

' bitter

~ m s o u r 1 m s w e e t

I

, 60

40

20

n P1 P2 P3 P4 P5

Panelists FIG. 2. PREDICTED MEAN SCORE ON ANCHORED LINE SCALE (0 = NOT PRESENT, 1M) = PRESENT INTENSELY) FOR FIVE PANELISTS AND FOUR FLAVOR ATTRIBUTES OF

DRIED PEPPERS

Estimation of the product effects and their significance are now made taking into account panelist location and dispersion effects. As an example, the attribute sweet is taken. The predicted mean scores for each cultivar and drying method are shown in Fig. 4. The significance of these effects can be ascertained in the standard way for a weighted GLM, by considering the analysis of deviance (where data are normally distributed, this is the analysis of variance). Table 1 shows the analysis of deviance for the attribute sweet.

In the analysis of the pepper data, no assumptions are made about the sources of differences between panelists in between-product variation. All differences are assigned to the (fixed) panelist product interaction, and no differentiation is made between scaling effects and differences in perception. Indeed, this analysis should

C. WILKINSON and D. YUKSEL

predicted mean dispersion parameter

I 1

20 1

I 5 1 1

15

attribute grassy bitter

[%sour 0 sweet

n v , v PI P2 P3 P4 P5

Panelists FIG. 3 . PREDICTED MEAN DISPERSION PARAMETER FOR FIVE PANELISTS AND FOUR

FLAVOR ATTRIBUTES OF DRIED PEPPERS

be seen as a method to obtain the best estimates for the panelist-product interac- tion; these estimates can then be submitted to a further analysis to check for out- lying panelists with regard to perception, for example using the Eggshell Plot of Naes er al. (1994).

DISCUSSION

The joint modeling of the mean and dispersion with a see-saw algorithm car- ries out corrections for panelist location and dispersion effects at the same time as the estimation of the other fixed effects. This means, for example, that infor- mation about product effects can be taken into account when estimating disper- sion effects. The technique is a rather general and flexible method which easily incorporates deviations from normality and linearity. The analysis can be changed to fit the data set rather than transforming the data set to fit the analysis. Also alternative specifications of the location and dispersion models are easily im- plemented. For example, a session effect can be included in the location model and/or the dispersion model.

MODELING DIFFERENCES BETWEEN PANELISTS 65

predicted mean score. attribute "sweet'

drying method freeze-dried hot-air dried

Cultivars FIG. 4. PREDICTED MEAN SCORE ON ANCHORED LINE SCALE (0 = NOT PRESENT,

BINATION OF CULTIVAR AND DRYING METHOD 100 = PRESENT INTENSELY) FOR THE TASTE ATTRIBUTE SWEET, AND EACH COM-

The method described here is closest in spirit to the methods proposed by Prit- chett Mangan and Brockhoff and Skovgaard. All three methods use weighted re- gression. Also the analysis is applied to each attribute separately, rather than the profile as a whole. Although information from correlated attributes is then not used, it does give the freedom for differences in individual panelists' use of attri- butes, such as is observed for the dried pepper data. Finally all three methods take into account product effects when calculating weights. The differences lie primarily in the different treatment of between-product variation, replication variation and the ratio between the two. The weights used by Pritchett Mangan explicitly take the form of a penalty for a high replication variation to between-product varia- tion ratio. This assumes in advance that there are real differences between pro- ducts for that attribute. This may not always be a reasonable assumption, for ex- ample if the sensory experiment is one of a series for different products, and the same attribute list is required throughout. Brockhoff and Skovgaard weight according to the replication variation, however they also apply stretching and

C. WILKINSON and D. YUKSEL

Deviance

443 62

17 96

451 52

42 87

23 76

TABLE 1 .

ANALYSIS OF DEVWNCE FOR ATTRIBUTE SWEET ~~

Deviance mho F value

40 33 co.001

17 96 <o 001

90 30 <0 001

8 57 <o 001

2 16 0 016

Effect I d t.

panelibt

method

cultivilr

method.culrivar S I I panelist method I I I

I pan.meth.cult I 55

I Residual I 288

I

170.07 I 3.09 I <0.001 I 78.97 I 1.44 I 0.031 I 28799 I I I

shrinking to bring the between-product variation of the different panelists closer together. This approach brings with it the danger of losing information about real differences in perception. The technique proposed in this paper is rather conser- vative. It makes no a priori assumptions about the existence of real differences between products, nor about the relative importance of scaling and differences in perception in explaining panelist product interaction. With regard to the latter aspect, it is suggested that the predicted mean scores for each panelist product interaction are subjected to a subsequent analysis.

As regards the product effects, the method described in this paper essentially carries out a weighted regression for the location model, The product effects in the location model can therefore be tested in the usual way for a weighted GLM, using an analysis of deviance (analysis of variance for normally distributed data). As the weights are not predetermined but arise from the fitting of the location model, it is however, important that the location model should be the full mdoel so that there is no danger of “real” effects being assigned to the residual.

The decision as to which method to use to correct for individual differences in the use of the measurement scale will depend on many factors, including the characteristics of the data set to be analyzed and the assumptions which can be made about the kinds of individual differences likely to be found. It is suggested that the joint modeling technique presented in this paper is particularly valuable in the analysis of data sets which seriously violate the assumption of normality. These include data sets where descriptor intensities are low and close to panelist threshold intensities, such as many aroma data sets, as well as data sets where the scores are measured on an ordinal scale with few categories.

MODELING DIFFERENCES BETWEEN PANELISTS 67

APPENDIX

The Genstat code (Genstat 5 version 3.1) used for fitting the example dataset is shown below.

“The vector of weights, w, is initialized, together with the vector previousdisp which is used in the convergence criterium. The value N is the total number of objects (rows) in the dataset. ”

VARL4TE “VALUE = N] w, previousdisp CALCULATE w, previousdisp = 1 “The length of the line in the anchored line scale, n, is set. ” CALCULATE n = 100 “Iterate until subsequent results are suficiently close ’’ FOR [NTIME=999]

“Fit location model using weighted regression ” MODEL [WEIGHT = w; DISP = 1; DISTRIBUTION = binomial; LINK =

FIT panelist*method*cultivar “Save the standardized residuals and convert to deviance residuals. The residuals are augmented by a factor N/(N-p), p being the number ofj t ted parameters in the location model, to allow for loss of degrees of freedom in jtting the location model. ” RKEEP RESIDUALS = standres; LEVERAGE = h; DF = N-min-p CALC devres = N*(standres**2)*(1-h)/w/(N-min-p) “Fit dispersion model” MODEL [DISTFUBUTION = Gamma; LINK = log] devres FIT panelist “Save the estimated dispersion parameters ’’ RKEEP FITTED VALUES = fitdisp ‘ ‘Check whether convergence criterium has been reached” CALCULATE difference = SUM((fitdisp - previousdisp)**2) EXIT difference < 0.0001 *SUM (previousdisp) ‘‘Calculate new weights as a function of the estimated dispersion parameters ’ ’ CALCULATE w = l/fitdisp CALCULATE previousdisp = fitdisp

logit] descriptor; NBINOMIAL = n

ENDFOR

68 C. WILKINSON and D. YUKSEL

REFERENCES

ARNOLD, G.M. and WILLIAMS, A.A. 1987. The use of Generalized Procrustes techniques in sensory analysis. In Statistical procedures in food research (J.R. Piggott, ed.) p. 233-253, Elsevier Appl. Sci. Publ, London.

BROCKHOFF, P.M. and SKOVGAARD, I.M. 1994. Modeling individual dif- ferences between assessors in sensory evaluations. Food Qual. Pref. 5 ,

LUNING, P.A., VAN DER VUURST DE VFUES, R., YUKSEL, D., EBBENHORST-SELLER, T., WICHERS, H.J. and ROOZEN, J.P. 1994. Combined instrumental and sensory evaluation of flavor of fresh bell peppers (Capsicum annuum) harvested at three maturation stages. J. Agr. Food Chem.

LUNNG, P.A., YUKSEL, D., VAN DER VUURST DE VRIES and ROOZEN, J.P. 1995. Aroma changes in fresh bell peppers (Capsicum Annuum) after hot-air drying. Food Sci. 60, 1269-1276.

McCULLAGH, P. and NELDER, J.A. 1989. Generalized Linear Models, 2nd Ed., Chapman and Hall, London.

NAES, T. 1990. Handling individual differences between assessors in sensory profiling. Food Qual. Pref. 2, 187-199.

NAES, T., HIRST, D. and BAARDSETH, P. 1994. Using cumulative ranks to detect individual differences in sensory profiling. J. Sensory Studies 9, 87-99.

PRITCHETT MANGAN, P.A. 1992. Performance assessment of sensory panelists. J. Sensory Studies 7, 229-252.

215-224.

42, 2855-2861.