It’s Not You, it’s Me: Detecting Flirting and its Misperception inSpeed-Dates

Rajesh RanganathComputer Science Department

Stanford Universityrajeshr@cs.stanford.edu

Dan JurafskyLinguistics Department

Stanford Universityjurafsky@stanford.edu

Dan McFarlandSchool of EducationStanford University

dmcfarla@stanford.edu

Abstract

Automatically detecting human social in-tentions from spoken conversation is animportant task for dialogue understand-ing. Since the social intentions of thespeaker may differ from what is perceivedby the hearer, systems that analyze humanconversations need to be able to extractboth the perceived and the intended socialmeaning. We investigate this differencebetween intention and perception by usinga spoken corpus of speed-dates in whichboth the speaker and the listener rated thespeaker on flirtatiousness. Our flirtation-detection system uses prosodic, dialogue,and lexical features to detect a speaker’sintent to flirt with up to 71.5% accuracy,significantly outperforming the baseline,but also outperforming the human inter-locuters. Our system addresses lexical fea-ture sparsity given the small amount oftraining data by using an autoencoder net-work to map sparse lexical feature vectorsinto 30 compressed features. Our analy-sis shows that humans are very poor per-ceivers of intended flirtatiousness, insteadoften projecting their own intended behav-ior onto their interlocutors.

1 Introduction

Detecting human social meaning is a difficult taskfor automatic conversational understanding sys-tems. One cause of this difficulty is the pervasivedifference between intended social signals and theuptake by the perceiver. The cues that a speakermay use to attempt to signal a particular socialmeaning may not be the cues that the hearer fo-cuses on, leading to misperception.

In order to understand the impact of this dif-ference between perception and intention, in this

paper we describe machine learning models thatcan detect both the social meaning intended by thespeaker and the social meaning perceived by thehearer. Automated systems that detect and modelthese differences can lead both to richer sociallyaware systems for conversational understandingand more sophisticated analyses of conversationalinteractions like meetings and interviews.

This task thus extends the wide literature onsocial meaning and its detection, including thedetection of emotions such as annoyance, anger,sadness, or boredom (Ang et al., 2002; Lee andNarayanan, 2002; Liscombe et al., 2003), speakercharacteristics such as charisma (Rosenberg andHirschberg, 2005), personality features like ex-troversion or agreeability (Mairesse et al., 2007;Mairesse and Walker, 2008), speaker depressionor stress (Rude et al., 2004; Pennebaker and Lay,2002; Cohn et al., 2004), and dating willingnessor liking (Madan et al., 2005; Pentland, 2005).

We chose to work on the domain of flirtationin speed-dating. Our earlier work on this cor-pus showed that it is possible to detect whetherspeakers are perceived as flirtatious, awkward, orfriendly with reasonable accuracy (Jurafsky et al.,2009). In this paper we extend that work to de-tect whether speakers themselves intended to flirt,explore the differences in these variables, and ex-plore the ability and inability of humans to cor-rectly perceive the flirtation cues.

While many of the features that we use to buildthese detectors are drawn from the previous liter-ature, we also explore new features. Conventionalmethods for lexical feature extraction, for exam-ple, generally consist of hand coded classes ofwords related to concepts like sex or eating (Pen-nebaker et al., 2007). The classes tend to per-form well in their specific domains, but may notbe robust across domains, suggesting the need forunsupervised domain-specific lexical feature ex-traction. The naive answer to extracting domain-

specific lexical features would just be to throwcounts for every word into a huge feature vector,but the curse of dimensionality rules this methodout in small training set situations. We proposea new solution to this problem, using an unsuper-vised deep autoencoder to automatically compressand extract complex high level lexical features.

2 Dataset

Our experiments make use of the SpeedDate Cor-pus collected by the third author, and describedin Jurafsky et al. (2009). The corpus is basedon three speed-dating sessions run at an Ameri-can university in 2005, inspired by prior speed-dating research (Madan et al., 2005). The grad-uate student participants volunteered to be in thestudy and were promised emails of persons withwhom they reported mutual liking. All partici-pants wore audio recorders on a shoulder sash,thus resulting in two audio recordings of the ap-proximately 1100 4-minute dates. Each date wasconducted in an open setting where there was sub-stantial background noise. This noisy audio wasthus hand-transcribed and turn start and end werehand-aligned with the audio. In addition to the au-dio, the corpus includes various attitude and de-mographic questions answered by the participants.

Each speaker was also asked to report how of-ten their date’s speech reflected different conver-sational styles (awkward, flirtatious, funny, as-sertive) on a scale of 1-10 (1=never, 10=con-stantly): “How often did the other person behavein the following ways on this ‘date’?”. In additionthey were also asked to rate their own intentions:”How often did you behave in the following wayson this ‘date’?” on a scale of 1-10.

In this study, we focus on the flirtation ratings,examining how often each participant said theywere flirting, as well as how often each participantwas judged by the interlocutor as flirting.

Of the original 1100 dates only 991 total datesare in the SpeedDate corpus due to various lossesduring recording or processing. The current studyfocuses on 946 of these, for which we have com-plete audio, transcript, and survey information.

3 Experiment

To understand how the perception of flirting dif-fers from the intention of flirting, we trained bi-nary classifiers to predict both perception and in-tention. In each date, the speaker and the inter-

locutor both labeled the speaker’s behavioral traitson a Likert scale from 1-10. To generate binaryresponses we took the top ten percent of Likertratings in each task and labeled those as positiveexamples. We similarly took the bottom ten per-cent of Likert ratings and labeled those as negativeexamples. We ran our binary classification exper-iments to predict this output variable. Our experi-ments were split by gender. For the female exper-iment the speaker was female and the interlocu-tor was male, while for the male experiment thespeaker was male and the interlocutor was female.

For each speaker side of each 4-minute conver-sation, we extracted features from wavefiles andtranscripts, as described in the next section. Wethen trained four separate binary classifiers (foreach gender for both perception and intention).

4 Feature Descriptions

We used the features reported by Jurafsky etal. (2009), which are briefly summarized here.The features for a conversation side thus indicatewhether a speaker who talks a lot, laughs, is moredisfluent, has higher F0, etc., is more or less likelyto consider themselves flirtatious, or be consideredflirtatious by the interlocutor. We also computedthe same features for the alter interlocutor. Al-ter features thus indicate the conversational behav-ior of the speaker talking with an interlocutor theyconsidered to be flirtatious or not.

4.1 Prosodic Features

F0 and RMS amplitude features were extracted us-ing Praat scripts (Boersma and Weenink, 2005).Since the start and end of each turn were time-marked by hand, each feature was easily extractedover a turn, and then averages and standard devia-tions were taken over the turns in an entire conver-sation side. Thus the feature F0 MIN for a conver-sation side was computed by taking the F0 min ofeach turn in that side (not counting zero values ofF0), and then averaging these values over all turnsin the side. F0 MIN SD is the standard deviationacross turns of this same measure.

4.2 Dialogue and Disfluency Features

A number of discourse features were extracted,following Jurafsky et al. (2009) and the dialogueliterature. The dialog acts shown in Table 2were detected by hand-built regular expressions,based on analyses of the dialogue acts in the

F0 MIN minimum (non-zero) F0 per turn, averagedover turns

F0 MIN SD standard deviation from F0 minF0 MAX maximum F0 per turn, averaged over turnsF0 MAX SD standard deviation from F0 maxF0 MEAN mean F0 per turn, averaged over turnsF0 MEAN SD standard deviation (across turns) from F0

meanF0 SD standard deviation (within a turn) from F0

mean, averaged over turnsF0 SD SD standard deviation from the f0 sdPITCH RANGE f0 max - f0 min per turn, averaged over

turnsPITCH RANGE SD standard deviation from mean pitch rangeRMS MIN minimum amplitude per turn, averaged

over turnsRMS MIN SD standard deviation from RMS minRMS MAX maximum amplitude per turn, averaged

over turnsRMS MAX SD standard deviation from RMS maxRMS MEAN mean amplitude per turn, averaged over

turnsRMS MEAN SD standard deviation from RMS meanTURN DUR duration of turn in seconds, averaged over

turnsTIME total time for a speaker for a conversation

side, in secondsRATE OFSPEECH

number of words in turn divided by dura-tion of turn in seconds, averaged over turns

Table 1: Prosodic features from Jurafsky et al.(2009) for each conversation side, extracted usingPraat from the hand-segmented turns of each side.

hand-labeled Switchboard corpus of dialog acts.Collaborative completions, turns where a speakercompletes the utterance begun by the alter, weredetected by finding sentences for which the firstword of the speaker was extremely predictablefrom the last two words of the previous speaker,based on a trigram grammar trained on the Tree-bank 3 Switchboard transcripts. Laughter, disflu-encies, and overlap were all marked in the tran-scripts by the transcribers.

4.3 Lexical Features

We drew our lexical features from the LIWC lex-icons of Pennebaker et al. (2007), the standardfor social psychological analysis of lexical fea-tures. We chose ten LIWC categories that haveproven useful in detecting personality-related fea-tures (Mairesse et al., 2007): Anger, Assent, In-gest, Insight, Negemotion, Sexual, Swear, I, We,and You. We also added two new lexical features:“past tense auxiliary”, a heuristic for automati-cally detecting narrative or story-telling behavior,and Metadate, for discussion about the speed-dateitself. The features are summarized in Table 3.

4.4 Inducing New Lexical Features

In Jurafsky et al. (2009) we found the LIWC lex-ical features less useful in detecting social mean-ing than the dialogue and prosodic features, per-haps because lexical cues to flirtation lie in differ-ent classes of words than previously investigated.We therefore investigated the induction of lexicalfeatures from the speed-date corpus, using a prob-abilisitic graphical model.

We began with a pilot investigation to seewhether lexical cues were likely to be useful; witha small corpus, it is possible that lexical fea-tures are simply too sparse to play a role giventhe limited data. The pilot was based on us-ing Naive Bayes with word existence features (bi-nomial Naive Bayes). Naive Bayes assumes allfeatures are conditionally independent given theclass, and is known to perform well with smallamounts of data (Rish, 2001). Our Naive Bayespilot system performed above chance, suggestingthat lexical cues are indeed informative.

A simple approach to including lexical fea-tures in our more general classification systemwould be to include the word counts in a high di-mensional feature vector with our other features.This method, unfortunately, would suffer fromthe well-known high dimensionality/small train-ing set problem. We propose a method for build-ing a much smaller number of features that wouldnonetheless capture lexical information. Our ap-proach is based on using autoencoders to con-struct high level lower dimension features from thewords in a nonlinear manner.

A deep autoencoder is a hierarchichal graphicalmodel with multiple layers. Each layer consists ofa number of units. The input layer has the samenumber of units as the output layer, where the out-put layer is the model’s reconstruction of the inputlayer. The number of units in the intermediate lay-ers tends to get progressively smaller to produce acompact representation.

We defined our autoencoder with visible unitsmodeling the probabilities of the 1000 most com-mon words in the conversation for the speakerand the probabilities of the 1000 most commonwords for the interlocutor (after first removinga stop list of the most common words). Wetrain a deep autoencoder with stochastic nonlin-ear feature detectors and linear feature detectorsin the final layer. As shown in Figure 1, we useda 2000-1000-500-250-30 autoencoder. Autoen-

BACKCHANNELS number of backchannel utterances in side (Uh-huh., Yeah., Right., Oh, okay.)APPRECIATIONS number of appreciations in side (Wow, That’s true, Oh, great)QUESTIONS number of questions in sideNTRI repair question (Next Turn Repair Indicator) (Wait, Excuse me)COMPLETION (an approximation to) utterances that were ‘collaborative completions’LAUGH number of instances of laughter in sideTURNS total number of turns in sideDISPREFERRED (approximation to) dispreferred responses, beginning with discourse marker wellUH/UM total number of filled pauses (uh or um) in conversation sideRESTART total number of disfluent restarts in conversation sideOVERLAP number of turns in side which the two speakers overlapped

Table 2: Dialog act and disfluency features from Jurafsky et al. (2009).

TOTAL WORDS total number of wordsPAST TENSE uses of past tense auxiliaries was, were, hadMETADATE horn, date, bell, survey, speed, form, questionnaire, rushed, study, researchYOU you, you’d, you’ll, your, you’re, yours, you’ve (not counting you know)WE lets, let’s, our, ours, ourselves, us, we, we’d, we’ll, we’re, we’veI I’d, I’ll, I’m, I’ve, me, mine, my, myself (not counting I mean)ASSENT yeah, okay, cool, yes, awesome, absolutely, agreeSWEAR hell, sucks, damn, crap, shit, screw, heck, fuck*INSIGHT think*/thought, feel*/felt, find/found, understand*, figure*, idea*, imagine, wonderANGER hate/hated, hell, ridiculous*, stupid, kill*, screwed, blame, sucks, mad, bother, shitNEGEMOTION bad, weird, hate, crazy, problem*, difficult, tough, awkward, boring, wrong, sad, worry,SEXUAL love*, passion*, virgin, sex, screwINGEST food, eat*, water, bar/bars, drink*, cook*, dinner, coffee, wine, beer, restaurant, lunch, dish

Table 3: Lexical features from Jurafsky et al. (2009). Each feature value is a total count of the words inthat class for each conversation side; asterisks indicate including suffixed forms (e.g., love, loves, loving).All except the first three are from LIWC (Pennebaker et al., 2007) (modified slightly, e.g., by removingyou know and I mean). The last five classes include more words in addition to those shown.

coders tend to perform poorly if they are initializedincorrectly, so we use the Restricted BoltzmannMachine (RBM) pretraining procedure describedin Hinton and Salakhutdinov (2006) to initializethe encoder. Each individual RBM is trained usingcontrastive divergence as an update rule which hasbeen shown to produce reasonable results quickly(Hinton et al., 2006). Finally, we use backpropa-gation to fine tune the weights of our encoder byminimizing the cross entropy error. To extract fea-tures from each conversation, we sample the codelayer (30 unit layer in our encoder) with the visi-ble units corresponding to the most common wordprobabilities from that document, creating 30 newfeatures that we can use for classification. Theconditional distributions of the first layer featurescan be given by the softmax of the activations foreach gender:

p(vi|h) =

exp(biasi +∑j

hj ∗ wij)∑k∈K

exp(biask +∑j

vj ∗ wkj)(1)

p(hj |v) =1

1 + exp(bias(j) +∑i

vi ∗ wij)(2)

where K is the set of all the units representing thesame speaker as i 1, vi is the ith visible unit, hj isthe jth hidden unit, wij is the weight between visi-ble unit i and hidden unit j, and biasm is the offsetof unit m. Intuitively, this means that the proba-bility that a hidden unit is activated by the visiblelayer is sigmoid of the weighted sum of all the vis-ible units plus the unit’s bias term. Similarly, thevisible units are activated through a weighted sumof the hidden units, but they undergo an additionalnormalization (softmax) over all the other visibleunits from the speaker to effectively model themultinomial distribution from each speaker. Sincein a RBM hidden units are conditionally indepen-dent given the visible units, and visible units are

1The visible unit i models word probabilities of either thespeaker or the interlocutor, so the softmax is done over thedistribution of words for the speaker that unit i is modeling.

conditionally independent given hidden layer, theabove equations completely specify the first layerof the model.

To account for the fact that each visible unit inthe first layer contained 1000 observations fromthe underlying distribution we upweighted our fea-tures by that factor. During pretraining the “train-ing data” for the higher layers is the activationprobabilities of the hidden units of layer directlybelow when driven by that layer’s input data. Theintermediate layers in the model are symmetricwhere the activation probabilities for both the vis-ible and hidden units are of the same form asp(hj |v) in layer 1. To produce real valued featuresin the code layer we used linear hidden units. Inaddition to the likelihood portion of the objectivewe penalized large weights by using l2 regulariza-tion and penalize all weights by applying a smallconstant weight cost that gets applied at every up-date. After training to find a good initial pointfor the autoencoder we unroll the weights and usebackpropogation to fine tune our autoencoder.

While interpreting high level nonlinear featurescan be challenging, we did a pilot analysis of oneof the 30 features fixing a large (positive or neg-ative) weight on the feature unit (code layer) andsampling the output units.

The top weighted words for a positive weightare: O did, O live, S did, S friends, S went,O live, S lot, S wait, O two, and O wasn’t (S forspeaker and O for interlocutor). The top weightedwords for a negative weight are: S long, O school,S school, S phd, O years, S years, O stanford,S lot, O research, O interesting and O education.At least for this one feature, a large positive valueseemed to indicate the prevalence of questions(wait, did) or storytelling (live, wasn’t). A largenegative weight indicates the conversation focusedon the mundane details of grad student life.

5 Classification

Before performing the classification task, we pre-processed the data in two ways. First, we stan-dardized all the variables to have zero mean andunit variance. We did this to avoid imposing aprior on any of the features based on their numer-ical values. Consider a feature A with mean 100and a feature B with mean .1 where A and B arecorrelated with the output. Since the SVM prob-lem minimizes the norm of the weight vector, thereis a bias to put weight on feature A because intu-

2000

2000

1000

500

500

250

250

30

1000

2000

1000

500

500

250

250

30

1000

2000

1000

500

500

250

250

30

1000

Dialogue:F: ...M: ...F: ...

Dialogue: F: ...M: ...F: ...

Diaogue:F: ...M: ...F: ...

ReconstructDialogue:

F: ...M: ...

ReconstructDialogue:

F: ...M: ...

W1

W2

W3

W4

W5

W1

W2

W3

W4

WT5

WT4

W5

2000 2000

WT3

WT2

RBM

RBM

RBM

RBM

Code layer

Decoder

EncoderPretraining Unrolling Fine-tuning

WT1

W1+ε1

W2+ε2

W3+ε3

W4+ε4

W5+ε5

WT5+ε6

WT4+ε7

WT3+ε8

WT2+ε9

WT1+ε10

Figure 1: Pretraining is a fully unsupervised pro-cedure that trains an RBM at each layer. Once thepretraining of one layer is complete, the top layerunits are used as input to the next layer. We thenfine-tune our weights using backprop. The 30 fea-tures are extracted from the code layer.

itively the weight on feature B would need to be1000 times larger to carry the same effect. Thisargument holds similarly for the reduction to unitvariance. Second, we removed features correlatedgreater than .7. One goal of removing correlatedfeatures was to remove as much colinearity as pos-sible from the regression so that the regressionweights could be ranked for their importance in theclassification. In addition, we hoped to improveclassification because a large number of featuresrequire more training examples (Ng, 2004). Forexample for perception of female flirt we removedthe number of turns by the alter (O turns) and thenumber of sentence from the ego (S sentences) be-cause they were highly correlated with S turns.

To ensure comparisons (see Section 7) betweenthe interlocutors’ ratings and our classifier (andbecause of our small dataset) we use k-fold crossvalidation to learn our model and evaluate ourmodel. We train our binary model with the topten percent of ratings labeled as positive class ex-amples and bottom ten percent of ratings as thenegative class examples. We used five-fold crossvalidation in which the data is split into five equalfolds of size 40. We used four of the folds fortraining and one for test. K-fold cross validationdoes this in a round robin manner so every exam-ple ends up in the test set. This yields a datasplit

of 160 training examples and 40 test examples. Toensure that we were not learning something spe-cific to our data split, we randomized our data or-dering.

For classification we used a support vector ma-chine (SVM). SVMs generally do not produce ex-plicit feature weights for analysis because they area kernelized classifier. We solved the linear C-SVM problem. Normally the problem is solvedin the dual form, but to facilitate feature analysiswe expand back to the primal form to retrieve w,the weight vector. Our goal in the C-SVM is tosolve, in primal form,

minγ,w,b12||w||2 + C

m∑i=1

ξi

s.t. y(i)(wTx(i) + b) ≥ 1− ξi, i = 1, . . . ,mξi ≥ 0, i = 1, . . . ,m (3)

where m is the number of training examples, x(i)

is the ith training examples, and y(i) is the ith class(1 for the positive class, -1 for the negative class).The ξi are the slack variables that allow this algo-rithm to work for non linearly separable datasets.

A test example is classified by looking at thesign of y(x) = wTx(test) + b. To explore mod-els that captured interactions, but do not allow fordirect feature analysis we solved the C-SVM prob-lem using a radial basis function (RBF) as a kernel(Scholkopf et al., 1997). Our RBF kernel is basedon a Gaussian with unit variance.

K(x(i), x(j)) = exp(−||x(i) − x(j)||2

2σ) (4)

In this case predictions can be made by lookingat y(x(test)) =

∑mi=1 α

(i)y(i)rbf(x(i), t(test)) + b,where each α(i), for i = 1, . . . ,m is a memberof the set of dual variables that comes from trans-forming the primal form into the dual form. TheSVM kernel trick allows us to explore higher di-mensions while limiting the curse of dimensional-ity that plagues small datasets like ours.

We evaluated both our linear C-SVM and ourradial basis function C-SVM using parameterslearned on the training sets by computing the ac-curacy on the test set. Accuracy is the number ofcorrect examples / total number of test examples.We found that the RBM classifier that handled in-teraction terms outperformed linear methods likelogistic regression.

For feature weight extraction we aggregated thefeature weights calculated from each of the testfolds by taking the mean between them.2

6 Results

We report in Table 4 the results for detecting flirtintention (whether a speaker said they were flirt-ing) as well as flirt perception (whether the listenersaid the speaker was flirting).

Flirt Intention Flirt Perceptionby M by F of M of F

RBM SVM 61.5% 70.0% 77.0% 59.5%+autoencoder 69.0% 71.5% 79.5% 68.0%features

Table 4: Accuracy of binary classification of eachconversation side, where chance is 50%. The firstrow uses all the Jurafsky et al. (2009) features forboth the speaker and interlocutor. The second rowadds the new autoencoder features.

In our earlier study of flirt perception, weachieved 71% accuracy for men and 60% forwomen (Jurafsky et al., 2009). Our current num-bers for flirt perception are much better for bothmen (79.5%), and women (68.0%). The improve-ment is due both to the new autoencoder featuresand the RBF kernel that considers feature inter-actions (feature interactions were not included inthe logistic regression classifiers of Jurafsky et al.(2009)).

Our number for flirt intention are 69.0% for menand 71.5% for women. Note that our accuraciesare better for detecting women’s intentions as wellas women’s perceptions (of men) than men’s in-tentions and perceptions.

7 Feature Analysis

We first considered the features that helped clas-sification of flirt intention. Table 5 shows featureweights for the features (features were normed soweights are comparable), and is summarized in thefollowing paragraphs:• Men who say they are flirting ask more ques-

tions, and use more you and we. They laugh more,and use more sexual, anger, and negative emo-tional words. Prosodically they speak faster, withhigher pitch, but quieter (lower intensity min).

2We could not use the zero median criteria used in Juraf-sky et al. (2009) because C-SVMs under the l-2 metric pro-vide no sparse weight guarantees.

FEMALE FLIRT MALE FLIRTO backchannel -0.0369 S you 0.0279S appreciation -0.0327 S negemotion 0.0249O appreciation -0.0281 S we 0.0236O question 0.0265 S anger 0.0190O avimin -0.0249 S sexual 0.0184S turns -0.0247 O negemotion 0.0180S backchannel -0.0245 O avpmax 0.0174O you 0.0239 O swear 0.0172S avtndur 0.0229 O laugh 0.0164S avpmin -0.0227 O wordcount 0.0151O rate 0.0212 S laugh 0.0144S laugh 0.0204 S rate 0.0143S wordcount 0.0192 S well 0.0131S well 0.0192 S question 0.0131O negemotion 0.019 O sexual 0.0128S repair q 0.0188 S completion 0.0128O sexual 0.0176 S avpmax 0.011O overlap -0.0176 O completion 0.010O sdpmean 0.0171 O sdimin 0.010O avimax -0.0151 O metatalk -0.012S avpmean -0.015 S sdpsd -0.015S question -0.0146 S avimin -0.015O sdimin 0.0136 S backchannel -0.022S avpmax 0.0131S we -0.013S I 0.0117S assent 0.0114S metatalk -0.0107S sexual 0.0105S avimin -0.0104O uh -0.0102

Table 5: Feature weights (mean weights of the ran-domized runs) for the predictors with |weight| >0.01 for the male and female classifiers. An S pre-fix indicates features of the speaker (the candidateflirter) while an O prefix indicates features of theother. Weights for autoencoder features were alsosignificant but are omitted for compactness.

Features of the alter (the woman) that helpedour system detect men who say they are flirtinginclude the woman’s laughing, sexual words orswear words, talking more, and having a higherf0 (max).•Women who say they are flirting have a much

expanded pitch range (lower pitch min, higherpitch max), laugh more, use more I and well, userepair questions but not other kinds of questions,use more sexual terms, use far less appreciationsand backchannels, and use fewer, longer turns,with more words in general. Features of the alter(the man) that helped our system detect womenwho say they are flirting include the male use ofyou, questions, and faster and quieter speech.

We also summarize here the features for the per-ception classification task; predicting which peo-ple will be labeled by their dates as flirting. Herethe task is the same as for Jurafsky et al. (2009)

and the values are similar.• Men who are labeled by their female date as

flirting present many of the same linguistic behav-iors as when they express their intention to flirt.Some of the largest differences are that men areperceived to flirt when they use less appreciationsand overlap less, while these features were not sig-nificant for men who said they were flirting. Wealso found that fast speech and more questions aremore important features for flirtation perceptionthan intention.• Women who are labeled by their male date

as flirting also present much of the same linguis-tic behavior as women who intend to flirt. Laugh-ter, repair questions, and taking fewer, longer turnswere not predictors of women labeled as flirting,although these were strong predictors of womenintending to flirt.

Both genders convey intended flirtation bylaughing more, speaking faster, and using higherpitch. However, we do find gender differences;men ask more questions when they say they areflirting, women ask fewer, although they do usemore repair questions, which men do not. Womenuse more “I” and less “we”; men use more “we”and “you”. Men labeled as flirting are softer, butwomen labeled as flirting are not. Women flirtinguse much fewer appreciations; appreciations werenot a significant factor in men flirting.

8 Human Performance on this task

To evaluate the performance of our classifiers wecompare against human labeled data.

We used the same test set as for our machineclassifier; recall that this was created by taking thetop ten percent of Likert ratings of the speaker’sintention ratings by gender and called those posi-tive for flirtation intention. We constructed nega-tive examples by taking the bottom ten percent ofintention Likert ratings. We called the interlocu-tor correct on the positive examples if the inter-locutor’s rating was greater than 5. Symmetricallyfor the negative examples, we said the interlocutorwas correct if their rating was less than or equalto 5. Note that this metric is biased somewhat to-ward the humans and against our systems, becausewe do not penalize for intermediate values, whilethe system is trained to make binary predictionsonly on extremes. The results of the human per-ceivers on classifying flirtation intent are shown inTable 6.

Male speaker Female speaker(Female perceiver) (Male perceiver)62.2% 56.2%

Table 6: Accuracy of human listeners at labelingspeakers as flirting or not.

We were quite surprised by the poor quality ofthe human results. Our system outperforms bothmen’s performance in detecting women flirters(system 71.5% versus human 56.2%) and alsowomen’s performance in detecting male flirters(system 69.0% versus human 62.2%).

Why are humans worse than machines at detect-ing flirtation? We found a key insight by examin-ing how the participants in a date label themselvesand each other. Table 7 shows the 1-10 Likert val-ues for the two participants in one of the dates,between Male 101 and Female 127. The two par-ticipants clearly had very different perspectives onthe date. More important, however, we see thateach participant labels their own flirting (almost)identically with their partner’s flirting.

I am flirting Other is flirtingMale 101 says: 8 7Female 127 says: 1 1

Table 7: Likert scores for the date between Female127 and Male 101.

We therefore asked whether speakers in generaltend to assign similar values to their own flirtingand their partner’s flirting. The Pearson correla-tion coefficient between these two variables (myperception of my own flirting, and my perceptionof other’s flirting) is .73. By contrast, the poor per-formance of subjects at detecting flirting in theirpartners is coherent with the lower (.15) correla-tion coefficient between those two variables (myperception of the other’s flirting, and the other’sperception of their own flirting). This discrepancyis summarized in boldface in Table 8.

Since the speed-date data was also labeled forthree other variables, we then asked the samequestion about these variables. As Table 8 shows,for all four styles, speakers’ perception of othersis strongly correlated with the speakers’ percep-tion of themselves, far more so than with what theothers actually think they are doing.3

3This was true no matter how the correlations were run,whether with raw Likert values, with ego-centered (trans-formed) values and with self ego-centered but other raw.

Variable Self-perceive-Other

& Self-perceive-Self

Self-perceive-Other &

Other-perceive-Other

Flirting .73 .15Friendly .77 .05Awkward .58 .07Assertive .58 .09

Table 8: Correlations between speaker intentionsand perception for all four styles.

Note that although perception of the other doesnot correlate highly with the other’s intent for anyof the styles, the correlations are somewhat bet-ter (.15) for flirting, perhaps because in the speed-date setting speakers are focusing more on detect-ing this behavior (Higgins and Bargh, 1987). It isalso possible that for styles with positive valence(friendliness and flirting) speakers see more simi-larity between the self and the other than for nega-tive styles (awkward and assertive) (Krahe, 1983).

Why should this strong bias exist to link self-flirting with perception of the other? One pos-sibility is that speakers are just not very good atcapturing the intentions of others in four minutes.Speakers instead base their judgments on theirown behavior or intentions, perhaps because of abias to maintain consistency in attitudes and rela-tions (Festinger, 1957; Taylor, 1970) or to assumethere is reciprocation in interpersonal perceptions(Kenny, 1998).

9 Conclusion

We have presented a new system that is able topredict flirtation intention better than humans can,despite humans having access to vastly richer in-formation (visual features, gesture, etc.). This sys-tem facilitates the analysis of human perceptionand human interaction and provides a frameworkfor understanding why humans perform so poorlyon intention prediction.

At the heart of our system is a core set ofprosodic, dialogue, and lexical features that al-low for accurate prediction of both flirtation inten-tion and flirtation perception. Since previous wordlists don’t capture sufficient lexical information,we used an autoencoder to automatically capturenew lexical cues. The autoencoder shows potentialfor being a promising feature extraction methodfor social tasks where cues are domain specific.

Acknowledgments: Thanks to the anonymous review-ers and to a Google Research Award for partial funding.

ReferencesJ. Ang, R. Dhillon, A. Krupski, E. Shriberg, and

A. Stolcke. 2002. Prosody-Based Automatic De-tection of Annoyance and Frustration in Human-Computer Dialog. In INTERSPEECH-02.

Paul Boersma and David Weenink. 2005. Praat: doingphonetics by computer (version 4.3.14). [Computerprogram]. Retrieved May 26, 2005, from http://www.praat.org/.

M. A. Cohn, M. R. Mehl, and J. W. Pennebaker.2004. Linguistic markers of psychological changesurrounding September 11, 2001. PsychologicalScience, 15:687–693.

Leon Festinger. 1957. A Theory of Cognitive Disso-nance. Row, Peterson, Evanston, IL.

E. Tory Higgins and John A. Bargh. 1987. Social cog-nition and social perception. Annual Review of Psy-chology, 38:369–425.

G. E. Hinton and R. R Salakhutdinov. 2006. Reduc-ing the dimensionality of data with neural networks.Science, 313(5786):504–507.

G. E. Hinton, S. Osindero, and Y. Teh. 2006. Afast learning algorithm for deep belief nets. NeuralComputation, 18:1527–1554.

Dan Jurafsky, Rajesh Ranganath, and Dan McFarland.2009. Extracting social meaning: Identifying inter-actional style in spoken conversation. In NAACLHLT 2009, Boulder, CO.

David Kenny. 1998. Interpersonal Perception: A So-cial Relations Analysis. Guilford Press, New York,NY.

B. Krahe. 1983. Self-serving biases in perceived simi-larity and causal attributions of other people’s per-formance. Social Psychology Quarterly, 46:318–329.

C. M. Lee and Shrikanth S. Narayanan. 2002. Com-bining acoustic and language information for emo-tion recognition. In ICSLP-02, pages 873–876,Denver, CO.

Jackson Liscombe, Jennifer Venditti, and JuliaHirschberg. 2003. Classifying Subject Ratingsof Emotional Speech Using Acoustic Features. InINTERSPEECH-03.

Anmol Madan, Ron Caneel, and Alex Pentland. 2005.Voices of attraction. Presented at Augmented Cog-nition, HCI 2005, Las Vegas.

Francois Mairesse and Marilyn Walker. 2008. Train-able generation of big-five personality styles throughdata-driven parameter estimation. In ACL-08,Columbus.

Francois Mairesse, Marilyn Walker, Matthias Mehl,and Roger Moore. 2007. Using linguistic cues forthe automatic recognition of personality in conver-sation and text. Journal of Artificial Intelligence Re-search (JAIR), 30:457–500.

Andrew Y. Ng. 2004. Feature selection, L1 vs. L2regularization, and rotational invariance. In ICML2004.

J. W. Pennebaker and T. C. Lay. 2002. Language useand personality during crises: Analyses of MayorRudolph Giuliani’s press conferences. Journal ofResearch in Personality, 36:271–282.

J. W. Pennebaker, R.E. Booth, and M.E. Francis. 2007.Linguistic inquiry and word count: LIWC2007 op-erator’s manual. Technical report, University ofTexas.

Alex Pentland. 2005. Socially aware computation andcommunication. Computer, pages 63–70.

Irina Rish. 2001. An empirical study of the naivebayes classifier. In IJCAI 2001 Workshop on Em-pirical Methods in Artificial Intelligence.

Andrew Rosenberg and Julia Hirschberg. 2005.Acoustic/prosodic and lexical correlates of charis-matic speech. In EUROSPEECH-05, pages 513–516, Lisbon, Portugal.

S. S. Rude, E. M. Gortner, and J. W. Pennebaker.2004. Language use of depressed and depression-vulnerable college students. Cognition and Emo-tion, 18:1121–1133.

B. Scholkopf, K.K. Sung, CJC Burges, F. Girosi,P. Niyogi, T. Poggio, and V. Vapnik. 1997. Com-paring support vector machines with Gaussian ker-nels to radialbasis function classifiers. IEEE Trans-actions on Signal Processing, 45(11):2758–2765.

Howard Taylor. 1970. Chapter 2. In Balance inSmall Groups. Von Nostrand Reinhold Company,New York, NY.