Reverse correlation tutorial - Ron Dotsch · Reverse correlation tutorial using R and E-Prime...

Reverse correlation tutorial using R and E-Prime (version 4)

Reverse correlation tutorialUsing R and E-PrimeWritten by: Ron Dotsch ([email protected], http://ron.dotsch.org)

Introduction

This tutorial consists of three parts:

1. Generating stimuli (starting at page 1)2. Programming the reverse correlation task (starting at page 6)3. Analyzing data (starting at page 16)

In this tutorial, I’ll assume no prior programming knowledge. We will make use of the free and open source statistical software R (http://www.r-project.org) and Rstudio (http://www.rstudio.com) to generate the stimuli and analyze the data. I developed a package for R that makes all of this really easy, called rcicr (http://ron.dotsch.org/rcicr/).

We will use E-Prime 2 (http://www.pstnet.com/eprime.cfm) for programming the task and collecting data. This is not a prerequisite for reverse correlation, you can use any experiment programming language you like. In the Research Seminar in Social Neuroscience all other teachers use E-Prime, so it made sense to me to stick to one programming language for the whole course. Some of the programming languages that you could use for this, ranked from easy to difficult are: Inquisit, E-prime, OpenSesame, Python with PsychoPy, Matlab with psych toolbox, or Presentation.

For those participating in the Research Seminar in Social Neuroscience course: All of the necessary programs should be installed already on the computer you are using at Unnik 108. If not, please let me know immediately.

1. Generating Stimuli

The first order of business is generating stimuli. We will work with noise based reverse correlation as described in Dotsch & Todorov (2012). The stimuli consist of a single base face and random noise patterns, that are combined (see Figure 1 on the next page).

The noise consists of 4092 sinusoid patches that have random contrasts (see Figure 2 on the next page). A contrast close to 1 means that the contrast is at its maximum (clear black and white distinctions). A contrast close to 0 means that the dark and light parts of the sinusoids are barely visible. Basically, everything is gray when the contrast is 0. A negative contrast means that light and dark is reversed: what was dark is now light and vice versa.

(c) 2015 - Ron Dotsch ([email protected]) Page � of �1 24

mailto:[email protected]

http://www.r-project.org

http://www.rstudio.com

http://ron.dotsch.org/rcicr/

http://www.pstnet.com/eprime.cfm



It’s good to understand the idea of the contrasts, but the software package that I developed does all of this stuff under the hood, so you never actually get to see these contrasts if you don’t want to.

Let’s get started. First, we need a base image. We will use the same base image as in Dotsch & Todorov (2012), which is the male average of the Karolinska face database. Download it here (http://materials.dotsch.org/SNS/MNES.jpg). Create a folder for your experiment and save it there. Save it somewhere that you will be able to find it back after you log out of the computer. Remember the folder’s location, as this will be the so-called ‘working directory’ that you need to enter in some of the other programs we’re using.


faces. Sampling error could have constrained face representa-tion in ways that affect the final models. Second, stimulus facesgenerated using the face space did not have hair, an importantfeature affecting person perception (e.g., Macrae & Martin,2007). Third, only shape information was used to construct themodels, although reflectance information (pigmentation andtexture) might play an important role in social perception(Todorov & Oosterhof, 2011). Last, although Oosterhof andTodorov included visualizations of the constructed socialdimensions, they did not report quantitative analyses withrespect to diagnostic face regions.

Here, we use a different class of RC techniques aimed atextending the findings of Oosterhof and Todorov (2008) byassessing whether their findings are method invariant andby quantitatively identifying which specific facial regions areinvolved in social perception. This class of RC techniques wasdeveloped in parallel by Kontsevich and Tyler (2004) andMangini and Biederman (2004, also see Gosselin & Schyns,2003). This technique enables researchers to generate imagesthat reflect participants’ internal representations of faces, with-out making any assumption about what those representationsmight look like. Recently, this RC variant has become increas-ingly popular in social cognitive research (see, e.g., Dotsch,Wigboldus, Langner, & van Knippenberg, 2008; Dotsch, Wig-boldus, & van Knippenberg, 2011; Imhoff, Dotsch, Bianchi,Banse, & Wigboldus, in press; Jack, Caldara, & Schyns,2011; Karremans, Dotsch, & Corneille, in press).

A typical RC image classification task (in this example wediscuss a two images forced choice, or 2IFC, variant used byDotsch et al., 2008) employs random variations of facialimages created with a constant base face (Figure 1A) andrandomly generated noise patterns (Figure 1B) superimposedon the face. Because the noise distorts the base face image, theface looks different with each different random noise pattern.For each superimposed random noise pattern, a negative pat-tern (the mathematical opposite) is generated. Each pixel thatis dark in the original noise pattern is bright in the negativenoise pattern, much like photo negatives. In a single trial, thebase image with the original noise and the base image withthe negative noise superimposed are presented side by side(Figure 1C). Participants are then asked to select the face thatbest resembles the target category. The average of all selectednoise patterns constitutes the classification image (CI), whereas

the average of all unselected noise patterns is the anti-CI. Forinstance, Dotsch, Wigboldus, Langner, and van Knippenberg(2008) used a neutral male face as base face, and superimposedrandom noise consisting of multiple truncated sinusoids. Thesewere then used as stimuli in a Moroccan classification task:participants chose from two stimuli (Figure 1C) the stimulusthat best resembled a Moroccan face. The average of all noisepatterns that participants classified as Moroccan constituted theMoroccan CI. Superimposing this CI on top of the original baseimage resulted in approximations of what participants thoughttypical Moroccan faces looked like.

The 2IFC task described above is based on the psychophy-sical RC methodology described by Mangini and Biederman(2004), where one image was presented in each trial and parti-cipants classified the image into one of two categories. In thiscase, CIs for each category are calculated by averaging allimages classified as the respective category. Mangini and Bie-derman demonstrated that this technique can be used to modelidentities (John Travolta vs. Tom Cruise), gender categories(male vs. female), and emotional expressions (happy vs. sad).However, the extent to which this task can be applied to mod-eling social perception is somewhat limited. First, the task usedas base face a morph between two images that accurately rep-resented the two target categories (e.g., John Travolta’s andTom Cruise’s face). When modeling social dimensions,researchers do not possess images that accurately represent thetwo target categories. Instead, those images are exactly whatresearchers set out to discover. Second, participants discrimi-nated between two specific categories, making it impossibleto tap into the internal representation of just one category with-out contrasting it with another category. The 2IFC variant doesnot suffer from these problems, because participants select theimage that best fits one target category and the images can becreated using a base face unrelated to the target category.

Here, we use the 2IFC RC image classification task to modelperception of face trustworthiness and dominance. Importantly,the 2IFC task enables us to go beyond the results of Oosterhofand Todorov (2008) in two ways. First, the base image of sti-mulus faces can include hair and the superimposed noiseaffects shape as well as reflectance information. Second, withthe 2IFC task we can identify facial regions diagnostic forsocial perception. Arguably, this might be achieved using yetanother RC technique, bubbles (Gosselin & Schyns, 2001), in

Figure 1. Base face (A), random noise example (B), and example stimuli of noise superimposed on a single base image (C). The left stimulusshows the base image with original noise superimposed and the right stimulus shows the base image with the negative noise superimposed.

Dotsch and Todorov 563

at PRINCETON UNIV LIBRARY on November 5, 2012spp.sagepub.comDownloaded from

is frowning, has thin lips and sad-looking eyes. Moreover, theface delineates itself less from the background.

Participant Agreement

To assess participant agreement, we calculated Cronbach’s aon the pixel luminance values of each participants’ CI noisepattern (masked by an oval shape to only include pixels of theface), for each social judgment. a, in this analysis, reflects theextent to which the pixels of individual participants’ CIs withina cell covary with the pixels of all other participants’ CIs in thatcell. It was not possible to assess agreement based on the deci-sion data, because two different sets of stimuli were used. Ascan be seen in Table 1, there is fair agreement with a rangingbetween .56 and .76.

Objective Metrics

Because the CIs represent psychologically meaningful con-structs that have well-defined mutual relations (e.g., trust-worthiness is the opposite of untrustworthiness, and relatively

unrelated to dominance or submissiveness), the resulting CIsshould have the same mutual relations (e.g., the trustworthyCI should be similar to the opposite of the untrustworthy CI,and not too similar to the dominance or submissive CIs). Theserelations can be quantified by calculating the correlationbetween the pixel luminance values of one classification pat-tern and the pixel luminance values of another classificationpattern. These correlations can be best understood as measuresof similarity. A high-positive correlation means that the CIs arephysically similar, a high-negative correlation means that theCIs are physically opposite, and a zero correlation means thatthe CIs have little in common. These correlations (based on CIsmasked with an oval shape to only include pixels of the face)are summarized in Table 2.1

The correlations in Table 2 show that (1) CIs for a trait onone side of the dimension (e.g., trustworthy) are physically dif-ferent from (i.e., correlate negatively with) CIs for a trait on theother side of the dimension (e.g., untrustworthy); (2) CIs for atrait on one side of the dimension (e.g., trustworthy) arephysically similar to (i.e., correlate positively with) CIs of theantiface of a trait on the other side of the dimension (e.g., anti-

Phase 0

Phase /2

Orientations

Contrasts* .23 -.14 -.73 .40 .56 .71

.12 .61 .25 -.48 .33 .45

Combined

2

Cycles per image

4 8 16 32 Combined noise

Number of sinusoids

12 48 192 768 3072 4092

Contrasts*

}

}Figure 2. This figure outlines the process of generating the noise pattern that was superimposed on the base image. For each spatial frequency(2, 4, 8, 16, and 32 cycles per image), 12 sinusoids per cycle were superimposed (6 Orientations ! 2 Phases). Each sinusoid had its own randomparameter indicating its contrast. The first 2 lines indicate how 12 sinusoids form the combined noise for 1 spatial frequency (2 cycles per image).The last line indicates how sinusoids across all spatial frequencies are combined to create the resulting noise. Note. *Contrasts are randomexample values to illustrate how contrast values relate to the resulting sinusoid patches.

Dotsch and Todorov 565

at PRINCETON UNIV LIBRARY on November 5, 2012spp.sagepub.comDownloaded from


http://materials.dotsch.org/SNS/MNES.jpg

http://materials.dotsch.org/SNS/MNES.jpg


Next, load up Rstudio (should be somewhere in the Windows Start Menu). The screen you get should look something like this:

Basically, there are three panes right now, but we’ll add a fourth, which will be the script window. This is where you will type the code that you need to generate the stimuli. Let’s open up that fourth pane right now. Click File > New File > R Script…

Your screen should look like this now:


A

B

C

D



I labeled the four panes A through D in the figure above. A is where you will write your code. B is where the code will be executed (it’s called the console). Pane C has two tabs, the one showing by default is your environment. It contains all the stuff that’s loaded in the computer’s working memory and that you’ll be able to use in R. The other tab in Pane C is the History tab, which contains all the code that you have executed. Pane D has multiple useful tabs. We will use one of them, the Files tab, but feel free to explore the other tabs. Especially the Help tab might be useful if you get stuck.

As I said, you write your code in Pane A most of the time. Whatever you write there, does not get executed right away. You need to click run to execute the (selected) code. Once you click run, your code will be pasted in Pane B, the console, where it will run one line at a time, starting from the first line, going all the way down. You can also directly type commands in the console, which will be executed as soon as you hit enter.

We’re going to point Rstudio’s working directory to the folder you created that contains the base image. Navigate in Pane D to your folder and make sure you see MNES.jpg (the base image) in it. It should look something like this:

Then click More > Set as working directory. From now on, R will know to find everything it needs in that folder, and it will also know to put the stimuli it creates into that folder. You will need to repeat this every time you start Rstudio, to make sure you are still in the correct working directory.

We’re almost good to go, but we need to take care of one more thing. Out of the box, RStudio doesn’t know anything about reverse correlation. We need to install the rcicr package that I developed in order for R to understand the commands we will use. To do this, enter the following line in the console (Pane B) and press enter:

install.packages("rcicr")




Wait for the installation to finish. If you see red text, don’t worry, it’s usually not an error. Now we’ll start writing our stimulus generation script in Pane A.

The first line loads the rcicr package into working memory. Enter it into Pane A on the first line:

library(rcicr)

See if the code runs without errors or warnings. Do this by selecting the line you want to run and click Run at the top right of Pane A. Later on, when we have a script with more lines, you can select multiple lines and run all of them in sequence. You don’t have to do this line by line. An easy shortcut is using CTRL + A to select all lines in Pane A and then click Run (or press CTRL + ENTER to run the selected lines).

The only thing left to do is to actually generate the stimuli. This is just one line of code as well, which you can add to the script:

generateStimuli2IFC(base_face_files=list('mnes'='MNES.jpg'), n_trials=300)

Set n_trials to the number of trials you want your task to have. The minimum number of trials of reverse correlation tasks is usually 300. Let’s stick to this number for now.

Select all lines in your script and run them. A progress bar pops up in Panel B which will also provide you with an estimation of the remaining time. Your computer is working hard to generate the reverse correlation stimuli. This may take a while, depending on the speed and amount of memory of your computer, but read on in the meantime.

While you’re waiting for the progress bar to complete, let me explain what you will end up with. The command will have created a new folder in your working directory, called ‘stimuli’, which contains lots of files after its finished. These files are your stimuli in jpg format. Each filename contains the name of the base image you used, a serial number, and ends with _ori or _inv. Filenames with the same serial number go together in a single trial. You present the _ori version and the _inv version of that serial number in a trial. The _ori version is merely the base image + the generated noise and the _inv version is the base image - the generated noise. Subtracting an image from another image is the same thing as adding the inverse of that image. That means that pixels of the noise in the _ori version that make the base image darker, make the base image lighter in the _inv version, and vice versa.

Moreover, you will end up with an .Rdata file. This file is extremely important, do not lose it! This file contains everything that is needed to analyze your data. For each stimulus serial number, it contains all the information necessary to build the stimulus, specifically the base image and the random contrasts for the sinusoids that make up a particular stimulus. Without this file, it will not be possible to analyze your data. We’ll get back to this file when we talk about data analysis, but in any case, do not delete it.

Hopefully by now your stimuli are all generated and we can go on to the next step: programming the experiment. If not, grab some coffee and let your computer complete its number crunching, or move on to programming while RStudio is still going at it. In that case, revisit the next paragraph when RStudio is done to check whether all went fine.




When the progress bar is compelete, there should be 601 files in your stimuli folder in total. The stimuli folder is located inside your working directory, which you pointed RStudio to earlier. It is the folder that also contains the base image you downloaded. The stimuli folder contains 300 _ori.jpg files, 300 _inv.jpg files, and 1 .Rdata file. Please check this now and let me know if this is not the case. Save your R script (for instance: generate_stimuli_group1.R), you’ll need to hand it in for the assignment (see below). You may quit Rstudio at this point.

2. Programming the reverse correlation task

Before we actually start programming, let’s consider what our task will look like. Participants will get 300 trials. On each trial, two stimuli are presented (from a given serial number the _ori and the _inv version). Participants should select one of the two stimuli, and this should be saved to a data file. We should probably also include a welcome screen, instructions for the participant and a screen at the end thanking participants for their participation. Sounds good?

Now it’s time to start E-Prime. E-Prime is actually a collection of programs and each performs different functions. To design our experiment, you start the E-Studio program (from the Start Menu). When it finished starting up, you should have a screen that looks like this:

Let’s just stick to basic. A lazy programmer is a good programmer. Open up a new Basic (Standard) project from this screen. This will save us time, because it already includes the basic framework for an experiment that we need to edit, instead of starting from scratch. By the way, it doesn’t hurt to pick Professional, it will open up more advanced options for you. You can always convert your Standard project to a Professional project later.

Let’s take a quick look at the E-Studio interface. Located to the far left of the interface, the Toolbox contains the E-Objects which you will use to create your experiment. The objects can




be added to the experiment by dragging and dropping them onto the Structure window. The Structure window shows the layout of your experiment and works similarly to a Windows Explorer window. When you highlight an object in the Structure window, information about the object will be displayed in the Properties window. Note that you may resize the windows to reveal more of the Workspace.

Let’s just run the damn thing and see what happens

At this point, we can take two approaches: we can painstakingly explain every little detail of E-Prime, or we can just learn by trial and error. It is my experience that the first is more efficient, but the second approach will make you better programmers.

So let’s just dive into the deep and run this basic framework without changing it. Go to E-Run in the menubar and click Run. You’ll be asked to save your experiment as an .es2 file. Go ahead and navigate to the folder you created for your experiment (where the stimuli folder is also located, but not inside your stimuli folder) and save it there. Give it a name that includes your group number, for instance: rc_experiment_group4.es2.

After this, the experiment will start. You’ll see it asks for a subject number (any other number than 0 will automatically save data). How convenient! That’s one thing you don’t have to program yourself. Enter a subject number and press Ok.

Then, it asks for a session number. We don’t care about sessions now, so just enter 1. Click Ok. Click Yes.

Now try out the experiment. Can you figure out which keys you need to press in order to give a correct answer? After a couple of trials the experiment will terminate. Note how many trials there were in total. As you could see, the program already gave feedback to the participants during the experiment. We don’t need that, that will have to go. Moreover, the stimuli are text, we want to present images. We also want to present two stimuli instead of one per trial. We probably want to change even more, but this gives us a nice head start.

While we’re at it, let’s inspect the data files that were generated in this test run. After the script ran, you’ll see in the E-Studio interface, in the bottom, an output window. There, in the last line, the location of the data file is stated. It’s an .edat2 file, and when you click the link in the Output window, the data file is opened in another program, called E-DataAid (seriously, who comes up with these names?). From here we’ll be able to export the data file to a format that we can later use for analysis. But let’s not worry about that for now. Inspect all the stuff that was saved by this basic experiment. Can you locate the column that contains the stimuli that were presented? And the column containing the responses and the response latencies? These are probably the most important ones. Count the number of rows. Does this correspond to the number of trials?

Close the data window and go back to E-Studio.




Understanding the experiment structure

We’re going to walk through the experiment structure that is already present and edit wherever we need to. Click the Experiment Object in the structure pane, way at the top. Nothing interesting happens. The world is still spinning around its axis and you have not been magically transported outside of the Matrix (it’s a movie reference; if you have not seen the Matrix, go watch it tonight after class). This Experiment Object will always be present in any new experiment you create.

Click Unreferenced E-Prime objects. Again, nothing happens. This is where all the objects you create in the course of programming your experiment and do not attach to the Experiment Object are located. Any objects attached to the Unreferenced E-Prime objects won’t be executed when you run your experiment, only those that are attached to the Experiment Object. There is one exception, and that is if the Unreferenced E-Prime objects are referenced dynamically while the program runs. This last sentence may have hurt a little bit. Ignore it for now, we’ll get to it later.

We clicked the first object in the structure, and the last object. Now all that’s left is what is in the middle. If you look closely at the structure, you can already notice a few things. We have some mysterious SessionProc object which does who knows what. Then we have something familiar: Instructions. And look, at the end we have GoodBye. Those are easy, I bet you if we click those, we’re going to see the instruction and goodbye screens.

TextDisplay objects

One-click Instructions. You’ll see that the properties window below changes. At the top it shows that Instructions is a TextDisplay object. This object class is also located in the Toolbox (the side bar on the left). The properties window displays all kinds of settings for the Instructions TextDisplay. Scroll through it, just to see what kind of settings there are. You’ll notice the Text setting. This is where you can change the text that will be displayed on this screen. But that’s a bit lame, because it’s cooler if you can actually change the text in a WYSIWYG (What You See Is What You Get) way. To do so, double click the Instructions object in the Instructions. A window opens, and you can change the text to whatever you like. In this case, type in the instructions for the participant for the reverse correlation task. At this point you guys should also decide what you’ll reverse correlate. Attractiveness? Trustworthiness? Arrogance? Barack Obama? Yoda? Tyrion Lannister? Be creative.

You don’t want to put too much instructions on one screen (most people don’t like reading screens filled with text). So let’s add a second instruction screen immediately after this Instructions screen. To do so, first double-click SessionProc. You’ll see the timeline going from the beginning of your experiment (green dot) to the end of your experiment (red dot). Adding the extra instruction page is nothing more than dragging the TextDisplay from Toolbox at the left into the timeline of SessionProc. Do so now. It should look something like this:




Let’s give the new object a more appropriate name. Click it once to make the properties window on the left display the properties of the TextDisplay1 object. At the top you see the property (Name), and its value: ‘TextDisplay1’. Change this to something more appropriate, like Instructions2. You’ll see that once you changed this, the change will be reflected both in the Structure and in the SessionProc time line.

Now add some text to the second instruction page. Just double click it in the SessionProc time line or in the structure window. Then start typing.

For some instant gratification run your experiment again and witness your second instruction screen appear after the first. You programmer you! A job well done! Time for coffee. But first, quit the experiment. You’re saying you don’t know how, without going through the whole experiment again? That’s nuts, we don’t have time for doing the whole experiment every time we try it out. Luckily, you can break out of the experiment before it is finished with this very important key combination: Shift + Ctrl + Alt (don’t tell your participants!).

Change the GoodBye TextDisplay object too if you want and then: onwards with the actual experiment!

All the other objects

Our next object in the structure, below the instruction pages is BlockList. Basically, an experiment contains sessions (a session is defined in a SessionProc), each session contains blocks (each block is defined in a BlockProc), and each block contains trials (each trial is defined a a TrialProc). All of these Proc objects define how a session, a block, and a trial is carried out. Why do we need BlockList and TrialList objects then? Well, those objects contain lists of things that change in each block (BlockList) and each trial (TrialList). We have only one block, so we’re not going to change anything there. But we have multiple trials and we want each trial to display different stimuli. That’s where TrialList comes into play.

So we skip BlockList and BlockProc for now, and move on to TrialList. Double click it. Before we change anything, let’s first understand what is happening. We see four lines (which corresponds with the fact that we had four trials in our test run and in our data file). The first four columns are predetermined: ID, Weight, Nested, and Procedure. Ignore those. To the right, there are two columns, and the names of these columns are not predetermined. The programmer of this basic framework decided to call these Text and Correct, but we can decide to call these ImageLeft and ImageRight if we want to (we want to, but not yet. Patience you must have my young padawan). The names of these columns are important because the objects that follow in the TrialProc refer to these. On each run of TrialProc (so, on each trial), the next row of TrialList will be read by E-Prime, and the information that is contained on that specific row is available to the objects in TrialProc.

Where might there be any references to the columns Text and Correct? Can you guess? It must be somewhere below TrialProc. Double click on TrialProc to see its timeline. Fixation just displays a fixation cross (check it out). That’s constant for each trial, it does not change. Compare this with the Stimulus TextDisplay. Double-click it. You’ll see immediately that the contents of the TextDisplay is [Text]. Now E-Prime does not actually displays [Text] when you run the experiment. Instead, [Text] is a place holder for the value in the Text column in TrialList that is selected for the current trial. If we would have chosen a different name for the column in




TrialList, such as SocialNeuroscienceRules, we would have to put [SocialNeuroscienceRules] in the Stimulus TextDisplay instead of [Text], so it maps onto the right values in TrialList.

Now go back to Fixation. You didn’t need to respond with a key to proceed to the stimulus from the fixation cross. How does that work? Scroll through the properties of Fixation and you’ll see the Duration property. It’s set to 1000 (milliseconds), which means that after 1 second it will automatically proceed to the next object. For Stimulus this property is set to 3000 ms, which means that there is a response window of 3 seconds.

The third object in the TrialProc is the FeedbackDisplay object, which displayed the feedback about correctness and response latency to the participant. We won’t discuss it here, because we don’t need it for our task, but just double-click it to see how it works. Notice the tabs below the screen for the different cases (Correct, Incorrect, NoResponse, and Pending) and click the icon in the top left (� )to get more information about the various properties. You’ll see that there is a setting that tells the Feedback page to get its information from the response to the Stimulus TextDisplay, find out where. Then delete the whole Feedback object from the TrialProc timeline. Select it and press the delete key on your keyboard. It will now be placed in the Unreferenced E-Objects part of the Structure. We need to remove objects from there again before they are really gone. So select the deleted object from the Unreferenced E-Objects collection and press the red cross icon at the top menu bar.

While you’re at it, delete the fixation TextDisplay too, we don’t need it. Check out the Stimulus TextDisplay object one more time by double clicking it, click the icon in the top left to go through the more advanced properties and select the Duration/Input tab to see some useful settings on how Eprime should behave depending on what the participant does. This window allows you to add additional input hardware (button boxes, joy sticks, etc.), and more importantly, it allows you to specify the correct response for the stimulus. Here you can see that the Correct setting is set to [Correct], which means E-prime can find the correct response from the Correct column for this particular trial in the TrialList. And for your information, there is much more you can set here, like when and how long Eprime should poll for responses, but this is all not relevant for us now.

Changing the script to a reverse correlation task

Time to delete the Stimulus object, we don’t need it either. We want to display two images and some text at the same time, not just one piece of text. Can we use ImageDisplay for that? No. It only displays a single image. We need two. And text. We can accomplish this by dragging a Slide Object into the TrialProc timeline. Change its name to Stimuli. Your TrialProc timeline should look like this right now:

Double click the Stimuli Object and let us work our magic. With your mouse pointer, hoover over the icons on the top row. Lots of goodies to choose from. Add three SlideTexts and two




SlideImages to the slide and arrange them in such a way that it looks something like the screenshot below (and edit the instruction to whatever you want to reverse correlate):

Make sure the SlideImages are square (the actual jpgs we generated are square and we don’t want them to look weird). The best way is to just create one, and then copy paste it, and reposition the second one. Important: set the width of both SlideImages to 40% and the height to 53%. Set the Stretch property of both to Yes. This will match with the stimuli we generated, when we set the screen resolution later.

Then rename the SlideImages, such that the left one is called ImageLeft and the right one is called ImageRight. Select the left image object and click the green properties icon in the top row (one icon to the left of the SlideText icon). If you would want to show the same image on each trial, you would specify the filename of the image here. However, we want to have a different image on each trial, which we’ll specify in TrialList. All this specific SlideImage object needs to know is in what column we’ll specify that. Let’s say we’ll specify this in the ImageLeft column, so enter [ImageLeft] for Filename. Then enter [ImageRight] for Filename for the other SlideImage object.




We want to edit some additional settings. First, what data should be saved? Click the properties icon on the top left in the Stimuli window and click the Logging tab. It should look like the following screenshot:

Check the boxes of the data we want to save (in this case RESP, and maybe RT if we want to be able to look at response times).

Then go to the Duration/Input tab and set the duration to a response window that allows participants enough time to respond. For instance, 10 seconds, which is 10000 milliseconds. Allow participants to even give a response by click the Add button under Input Masks > Device(s). Select Keyboard and click OK. Then click OK again to apply all changes.

We’re almost done with programming the task, but we have to get back to our TrialList to create our ImageLeft and ImageRight columns. Double-click TrialList in the Structure window. Use the buttons at the top to remove the Text and Correct columns, and then add the ImageLeft and ImageRight columns (I’m sure you’ll figure out how).




The next step would be to type all the 300 _ori and 300 _inv images in this TrialList. Do you feel like doing that? Me neither, so let’s be smart and lazy programmers. Open up the TrialList property window (sixth icon from the left in the TrialList window) and select the General tab. Then select File as Load Method. Now type in a file name, say stimuli.txt. This file doesn’t exist yet, so let’s make that file.

We could use R to make that file, but that requires some programming skills you don’t have yet. Luckily, Microsoft Excel offers wonderful functionality that we can use to create this file. Save your changes to the script so far and fire up Excel.

There is a system in the naming scheme of the stimulus files we discussed earlier. We can use that system in excel to create a nice text file. In the first line, first cell (cell A1), type:

stimuli\rcic_mnes_1_

In the cell next to it (cell B1) type 1.

In cell C1 type:

=TEXT(B1;"00000")

In cell D1 type:

_ori.jpg

In cell E1 type:

_inv.jpg

In cell F1 type:

=CONCATENATE(A1;C1;D1)

In cell G1 type:

=CONCATENATE(A1;C1;E1)

Your first line should look like this:

Now select cells A1 though G1 like this:




Move your mouse pointer to the bottom right corner of the selection (there is a little black demarkation there), hold down the left mouse button and scroll down 300 rows, dragging down the selection. After deselecting, the end of the file should look this:

Columns F and G contain the actual file names we want to have in our text file. We need to get rid of the rest. The way to do that is to select columns F and G, and use CTRL+C to copy the contents of the columns. Then open up a new excel file, and click the arrow below the Paste button to bring up the Paste Special menu, like this:

Select the the first option under Paste Values. The result should look something like this:




Now we need to edit this file a bit to conform to what Eprime is expecting. It will want to have all column names in there that you normally have in the TrialList, including ID, weight and everything. So edit the file until it looks like this (Hint: if you fill out the first two lines as they should be and then drag the new columns ID, Weight, Nested and Procedure down like we did with the image names, you’ll be done a lot faster):

The final step is exporting this file to a text file. Go to File > Save As, select for Save as type “Text (Tab delimited) (*.txt)”, navigate to the folder you created for the experiment and save it as stimuli.txt. You’ll get some warnings, click Ok and Yes without reading the messages like you would do with any license agreement you ever read (Facebook, iTunes, etc.).

One more thing. Open up the stimuli.txt file with Notepad (Notepad can be found in the start menu, it’s a default Windows program) and remove the last empty line that Excel put it. If you don’t it will confuse the hell out of E-Prime, and you will get an error after the last trial of the experiment, instead of the GoodBye screen.

Back to E-prime. E-prime will read the TrialList from the stimuli.txt file, so you can ignore what’s written in the TrialList. When you remove all rows from TrialList, TrialProc will move to the Unreferenced E-Objects. Don’t worry, leave it there, it’ll still run. But keep the rows in if you don’t mind them: it will prevent TrialProc from moving to Unreferenced E-Objects. Feels nice.

Before we run the experiment, we want to change the resolution in which the experiment is presented, such that the stimuli will be presented in high quality. To do this, double click Experiment in the structure. Go to the Devices tab. Select Display and click the Edit… button. You will see the following screen:




You have two options. If you want absolute control, set the width and height to something that you know the display and computer you work with support. 1280 width and 960 height will be perfect (the stimuli you generated have a resolution of 512x512, and you want to fit at least two next to each other). For typical resolutions see http://en.wikipedia.org/wiki/Display_resolution. Alternatively set Match desktop resolution at runtime to Yes. But then, whenever you run an experiment, be sure you check the desktop resolution to be what you want before running each participant. You don’t want to have different resolutions for different participants.

And now… You’re done! You have a reverse correlation task. Go ahead. Try it out. If you do the task seriously on this run, you can even analyze your data and see your personalized classification images (visualization).

When you go back to your experiment folder, you’ll see that E-prime will have generated more files than just your .es2 file. The .es2 file is the one you need if you ever want to edit your program. Additionally there will be a .esb2 file, which you can just double-click to start your experiment immediately. However, you won’t be able to edit this file using the graphical interface in E-studio, so it’s a file you’d distribute, but not the most important file to save, because you can always re-generate it from the .es2 file.

If you have free time left, poke around and try some the more advanced things in this list (using your own curiosity, the help function in E-prime, this getting started pdf, and the internet as your guide). Important: Save your file before you change anything, I want you to run the file as it is now when you collect the data that we’ll analyse in the next sections. Save any changes to your E-Prime script as a new .es2 file.

- Make sure that participants can only press the A or the L key instead of any key- Divide the single block into multiple blocks, with breaks in between- randomizing the order in which stimuli are presented (hint: look at the properties of TrialList)- counter-balancing or randomizing the side on which the _ori and _inv noise are presented (be

sure that your data file records on which side which image was presented, otherwise your data will be unusable).

- use a four-point scale for responses, that includes certainty measurements: probably X, possibly X, possibly Y and probably Y (where X and Y are your categories, such as unattractive and attractive).

3. Analyzing data

Collect data first. If you read this in my Research Seminar in Social Neuroscience class, you can stop here. We’ll collect data in our fourth and last meeting.

If your experiment ran fine, your data will be saved in .edat2 files. If for some reason your experiment crashed, you might still be able to recover your data, based on the .txt files that E-Prime saves and using the E-Recovery program. This program attempts to convert those .txt files to .edat2 files, which we can use for later analysis.

After data collection, continue with this tutorial to analyze the reverse correlation data and visualize your first classification images. We will need to go through two steps, one using E-



http://en.wikipedia.org/wiki/Display_resolution

http://lyrawww.uvt.nl/~cenv/dci-lab/e-prime/GettingStartedGuide.pdf


Primes E-Merge program to merge all data files together and a next step in R(studio) to actually analyze the data.

Merging data filesE-Prime saves the data for every participant in a separate data file. We need to merge them together. Let’s assume you have three data files, each of a separate participant who completed your task. If you called your experiment script rc.eb2 (like I did), the three files will be named rc-1-1.edat2 (for participant 1, session 1), rc-2-1.edat2, and rc-3-1.edat2. We need to combine the data from all three files. So close the text file, and load up E-Merge from the start menu.

When you start E-Merge, you’ll get a screen looking something like the one below. If you’re lucky, you get a Quick Reference message box that already tells you what we will do:

On the left is the Folder Tree. Navigate to your experiment folder, where your data files are located. On the right, the files contained in that folder will be located. Select those that contain your data (rc-1-1.edat2, etc.). Then in the top menu bar, click Merge. The Standard Merge will do fine, so click Next. You’ll get to option to give the merged data file a new name. I’ll call it rc-data-merged.emrg2 (yet another file extension, yay!).

You’re done merging, so you can exit E-Merge, but we’re not moving to Rstudio yet. R doesn’t know how to read these .emrg2 files, so we need to convert it to something more convenient. Navigate to your experiment folder in Windows Explorer and double-click the .emrg2 file you just generated. It will open up in E-DataAid, looking something like the image below.




We need to export this to a new file type. Click File in the menu bar and click Export. From the Export to dropdown box, select Excel en click Ok. Give it an identifiable name (e.g., rc-data-merged.txt) and save it in your experiment folder.

If you still have your data file open, you can see that the third column is called Clock.Information. Its contents contain weird characters and R is almost certain to choke on them. We have to get rid of them by opening the exported .txt file in Excel. So, start Excel, and open the exported .txt file. By default the Open file dialog box in Excel doesn’t show .txt files, so look for the dropdown box that says “All Excel Files (*.xl*; *.xlsx, …)”, and select instead “All Files (*.*)”. Find your exported .txt file and go through what is called the Text Import Wizard.

In Step 1 of 3, select Delimited, and set Start import at row to 2. Click Ok.

In Step 2 of 3 keep “Tab” checked and all the others unchecked, and press Finish.

Your Excel file should look something like the image below. If on the first line you see some text that is not in the screenshot below, remove that line now.




Select the column with Clock.Information (in this example. column D), right click it, and click Delete. Now save the remaining file again as a Text (tab-delimited file). You can overwrite the rc-data-merged.txt file that we were editing.

Now, finally, we can move on to RStudio.

Preparing your data in RFirst, fire up RStudio. Open up a new script like we did when we created stimuli. This will be our analysis script. All of the code we enter next, we’ll enter in the script (Panel A) and not directly in the console (Panel B). This will make it easy to fix any mistakes we make. We can simply change the script and re-run the whole thing instead of having to enter each separate line again. Moreover, it will make our analysis reproducible, something most scientist value highly.

Our first line of code (the first in our script) loads the necessary packages for this analysis in your computer’s working memory:

library(rcicr)

See if this line executes without errors. You remember how to do that right? It was explained in the first part of the tutorial, when we generated stimuli (on page 5). If you see red lines appearing in the code, don’t worry: it doesn’t necessarily mean that you have any errors.




If the rcicr library doesn’t load, it might not be installed on your system. Type the following line in your console (you don’t have to do this if the previous lines executed fine):

install.packages('rcicr')

Important: Now is a good time to set the working directory to your experiment folder (explained on page 4).

Next, type in the code to load our exported excel file. This (and everything else), you’ll type in the script window:

rcdata <- read.csv('rc-data-merged.txt', sep='\t')

When you run this line of code, in the Environment pane you should see this:

If you click on rcdata in the environment pane, you’ll get to see the contents as a spreadsheet in a special tab where your script was displayed previously, like this:

Click on the Untitled1 tab to go back to the script you were writing. Perhaps it’s a good time now to save it and give it a name. Go ahead and save it in your experiment folder as rc_analysis_group4.R, assuming you are group 4 in my class.

Let’s take a look at the response distribution by participant. Use the following command (it’s case sensitive, so get the upper case and lower case characters correct) and execute it:

table(rcdata$Subject, rcdata$Stimuli.RESP)

The data file I use here ended up including only one participant, participant number 4. If I run the line of code above, I get the following output:




As you can see, participant 4 gave used the a response key 100 times and the l response key 199 times. He missed 1 trial (because he was to slow to respond; I know this, because I was that participant), which shows up here as an empty value (just a blank, to the left of the a). Compare this with your output. Is your output plausible?

We have to remove the trials where no responses are recorded because participants missed the response-window. Do this with the following line of code:

rcdata <- subset(rcdata, Stimuli.RESP != "")

!= means “does not equal”, and "" (two double quotes) means empty. Run the previous table command again if you want to check whether the missing values are gone now. For me, this was the case:

At this point the responses are coded as a and l, which is arbitrary, because it represents whatever response keys you chose to use when you programmed the experiment in E-Prime. So we have to recode the responses to something the rcicr package will understand, which is 1 when the original noise was selected and -1 when the inverted noise was selected.

Assuming that ImageLeft always contained the original stimulus and ImageRight always the inverted stimulus, we can recode this variable quite easily: a was the left face, which always was the original face, so should be recoded to 1. l was the right face, which always was the inverted face, so should be recoded to -1.1

Our recoded variable is going to be called response. Our old values are stored in Stimuli.RESP. Both of these variables will be stored in the dataset (or, what R calls: the data frame) rcdata. This is the code you need to enter to accomplish the recoding:

rcdata$response[rcdata$Stimuli.RESP == 'a'] <- 1rcdata$response[rcdata$Stimuli.RESP == 'l'] <- -1

Let’s see whether we get the same distributions of responses we had previously for the original values, as a security checks (it’s easy to make a small mistake):

If you chose to counter-balance or randomize the location of where you presented the original 1

and the inverted noise during the programming phase, you will need to account for that. This will mean you need a slightly more complex recoding scheme. Basically, you’ll need to set the new response variable to 1 (original noise selected) when participants selected the left key when the left image contained the original noise and when participants selected the right key when the right image contained the original noise. In all other cases, the inverted noise was selected, and -1 should be assigned. I will not be going into more detail, but if it takes you more than three lines of R code, e-mail me.




table(rcdata$Subject, rcdata$response)

For me everything is OK, because I get as many -1 responses now as I got l responses earlier:

We also need to have a variable coding for what the serial number was of the two images that were presented in any given trial. If you just presented the stimuli in sequence (starting from 00000, then 00001, all the way up to 00300), you’re in luck, because we actually already have a column that codes for that, namely ID. Check it out by viewing the rcdata data frame (click on it in the Environment pane). You can skip the next block of code.

If you had E-prime randomize the presentation order, you’re in luck too, because that does not affect the ID variable. You can still use it. If, on the other hand, you changed the order in your stimuli.txt file, such that ID no longer corresponds with the serial number of the presented images, you need to extract it with the following piece of code. Skip this specific block of code if you’re, as I said, in luck.

library(stringr)rcdata$ID <- as.numeric(str_extract(rcdata$ImageLeft, '[0-9]{5}'))

Performing the reverse correlation analysisWe will perform to kind of analyses. One is the analysis by subject, computing a visualization (a so-called classification image) for each participant. And we will do a group-level analysis, where the data of all participants is lumped together to compute a single powerful analysis. In my case, this will yield the same results, because I only have one single participant in my dataset. But if you have more, it will of course look different, to the extent to which participants differed in their responses in the task.

In all cases we will need to tell R what the name of our .Rdata file is. It was generated when we generated our stimuli, and contains all the values of the random contrast parameters of the sinusoids. Without that information, we can’t perform the reverse correlation analysis (I hope you kept it safe like I told you ). So, locate your .Rdata file. By default, it’s located in the stimuli 2

subfolder inside your experiment folder. Mine was called: rcic_seed_1_time_feb_05_2015_12_58.Rdata. It’s not enough to just enter this name if the file was located in a subfolder of the working directory. You’ll need to add the relative path to the file, stated from your current working directory. And you’ll need to use forward slashes (/)

If you don’t have it any more, there is a way to recreate it, and that’s by simply re-running your 2

stimulus generation script. The reason is that the random number generator is seeded by default with the number one, every time the stimuli are generated. This means that it will produce the same random numbers every time you re run the script. However, do not rely on this, because when R versions, or rcicr versions, or computer hardware, or operating systems are updated, the random number generator may or may not suddenly spit out different numbers. It’s always better to just keep the .Rdata file, it is a perfect record of the parameters of your stimuli.




instead of back-slashes (\) if you’re used to windows. For me this results in the following line of code, which you should adapt to your own location and name:

rdatafile <- 'stimuli/rcic_seed_1_time_feb_05_2015_12_58.Rdata'

First, the by-participants analysis (it may take a few moments when you run it):

cis <- batchGenerateCI2IFC(rcdata, by='Subject', stimuli='ID', responses='response', baseimage='mnes', rdata=rdatafile)

In your working directory, you’ll see a new cis folder, that contains an image for each participant. Every image file name ends with _autoscaled. Under the hood, the rcicr package rescaled the noise values in a way that optimizes it’s visibility when superimposed on the face, but taking into account the ranges of all other noise patterns that were generated for these participants. Go ahead, navigate to your working directory in Windows Explorer and take a look at the classification images in the cis folder.

I ended up selecting on 300 trials the face that looked the most angry to me. This yielded the following classification image to the left. Looks plausible, no?

We can generate the anti-classification images too (basically averaging the noise patterns of the faces that were not selected). Just repeat the line above, but add as argument to the command antiCI=TRUE. Add this within the parentheses, and don’t forget to add a comma to separate this argument from the other arguments in the command. If you’d run it now, the old CIs would get overwritten. To make sure that doesn’t happen, add a label to this files, using label='anti', for instance.

More on the next page…




Next up, the group-wise analysis, where we explicitly state the autoscaling, and leave out the names of the individual arguments for the commands:

groupci <- generateCI2IFC(rcdata$ID, rcdata$response, 'mnes', rdatafile, scaled='matched')

Again, you can generate the antiCI by adding the antiCI=TRUE argument to the first line in this block of code, and label='anti' to make sure your original CIs don’t get overwritten. Navigate again to your working directory in Windows Explorer and then the cis folder to take a look at the classification images.

You have now completed your first empirical reverse correlation cycle. Congratulations!

If you have any comments, thoughts, feedback on this tutorial, please e-mail me at [email protected].

References Dotsch, R., & Todorov, A. (2012). Reverse correlating social face perception. Social Psychological and Personality Science, 3 (5), 562-571.

Additional information for students participating in Research Seminar in Social Neuroscience

What to do when you reached this point?

Send your group-wise classification images and anti classification images to [email protected] now, so we can take a look at the end results in public. See if we can guess what social dimension you were trying to visualize.

What do you need to hand in for this assignment?

A zip file containing:- Your research proposal for the social perception workshop- The R script used to generate stimuli (.R file)- The e-prime script of the experiment (.es2 file)- The data file of the collected data (.edat2 file), or the merged data file- The R script used to analyze the data (.R file)- The final classification images (.jpg files)- Note: don’t include all the stimulus files.

Put your group number in the zip file.



http://www.apple.com



Reverse correlation tutorial - Ron Dotsch · Reverse correlation tutorial using R and E-Prime...

Documents

Transcript of Reverse correlation tutorial - Ron Dotsch · Reverse correlation tutorial using R and E-Prime...