“Hello! My name is... Buffy” AutomaticNaming of Characters in TV Video

Mark Everingham, Josef Sivic and Andrew Zisserman

Arun Shyam

Objective

To label television or movie footage with the identity of the people present in each frame of the video.

Challenging problem owing to change in scale,lighting,pose,hair-style etc.

To employ readily available textual annotation for TV , in the form of subtitles and transcripts, to automatically assign the correct name to each face image.

Outline

Three main parts:

1) Processing of subtitles and script to obtain proposals for the names of the characters in the video.

2) Processing the video to extract face tracks and accompanying descriptors, and to extract descriptors for clothing.

3) Combine the textual and visual information to assign labels to detected faces in the video.

Test Data:

Two 40 minute episodes of the TV serial “Buffy the Vampire Slayer”.

Subtitle and Script Processing

Subtitles extracted using a simple OCR algorithm. Script obtained from a fan site in HTML format.

Subtitles record what is being said and when but not by whom.

Script tells who says what but not when.

What we need – Who, What and When.

Solution is to align the script and subtitles by Dynamic Time Warping algorithm.

Write the subtitle text vertically, and the script text horizontally. The task then is to find a path from top-left to bottom-right which moves only forward through either text.

The word-level alignment is then mapped back onto the original subtitle units.

Subtitle-Script Alignment

Face Detection and Tracking

A frontal face detector is run on every frame of the video. Better than multi-face and person detection.

Any individual who appears in a video for any length of time generates a face track – that is, a sequence of face instances across time.

Track provides multiple examples of the character’s appearance.

Face tracks are obtained as a set of point tracks starting at some frame in the shot and continuing until some later frame.

For a given pair of faces in different frames, the number of point tracks which pass through both faces is counted, and if this number is large relative to the number of point tracks which are not in common to both faces, a match is declared

Facial Feature Localization

Output of the face detector gives an approximate location and scale of the face. After this the facial features are located.

Nine facial features are located: the left and right corners of each eye, the two nostrils and the tip of the nose, and the left and right corners of the mouth.

To locate the feature positions a Gaussian mixture model is used, where the covariance of each component is restricted to form a tree structure with each variable dependent on a single parent variable.

Gives better performance in case of pose variation and less light.

Appearance of each facial feature is assumed independent of the other features and is modeled by feature/non-feature classifier that uses a variation of Ada-boost algorithm and 'Haar-like image features'.

Face and Feature Detection

Representing Face Appearance

Computing descriptors of the local appearance of the face around each of the located facial features. Gives robustness to pose variation, lighting, and partialocclusion compared to a global face descriptor.

Before extracting descriptors normalize face region to reduce the scale uncertainty and pose variation.

An affine transformation then transforms located facial feature points to a set of feature positions.

Two descriptors were investigated: (i) the SIFT descriptor (ii) a simple pixel-wised descriptor which is formed by taking a vector of normalized pixels to obtain local photometric invariance.

Face descriptor is formed by concatenating the descriptors for each facial feature.

Representing Clothing Appearance

Sometimes matching faces is very difficult because of different expression, pose, lighting or motion blur.

Cues to matching identity can be derived by representing the appearance of the clothing.

For each face detection a bounding box which is expected to contain the clothing of the corresponding character is predicted relative to the positionand scale of the face detection.

A color histogram is computed as a descriptor of that bounding box in YCbCr color space which de-correlates the color components better.

While similar clothing appearance suggests the same character, observing different clothing does not necessarily imply a different character.

Clothing Appearance Aid

Speaker Detection

The combined subtitle and script face detection is highly ambiguous as: (i) there might be several detected faces present in the frame and we do not

know which one is speaking;(ii) It might be a reaction shot.

This ambiguity can be removed using visual clues i.e. movement of the lips.

A rectangular mouth region within each face detection is identified using the located mouth corners and mean squared difference of the pixel values within the region is computed between the current and previous frame.

If difference above a high threshold then classify face detections into ‘speaking’, if less than a low threshold then classify face detections into ‘non-speaking'. If in between then ‘refuse to predict’.

Speaker Detection Ambiguity

Lip Movement Detection

Classification by Exemplar Sets

Tracks for which a single identity is proposed are treated as exemplars with which to label the other tracks which have no, or uncertain, proposed identity.

Each unlabeled face track F is represented as a set of face descriptors and clothing descriptors {f,c}. Exemplar sets {λi} have the same representation but are associated with a particular name.

For a given track F, the quasi-likelihood that the face corresponds to aparticular name λi :

p(F|λi) =1/Z .exp{-(df (F,λi)p(F|λi) =1/Z .exp{-(df (F,λi)²/2σf²)} .exp{-(df (F,λi)²/2σc²)}²/2σf²)} .exp{-(df (F,λi)²/2σc²)}

Classification by Exemplar Sets

Face distance df (F,λi) is defined as the minimum distance between theDescriptors in F and in the exemplar tracks λi. The clothing distance dc(F,λi) is

similarly defined.

The quasi-likelihoods for each name λi are combined to obtain a posterior probability of the name by assuming equal priors on the names and applying Bayes’ rule:

P(λi|F) = p(F|λi)/ Σj p(F|λ j)P(λi|F) = p(F|λi)/ Σj p(F|λ j)

By thresholding the posterior, a “refusal to predict” mechanism is implemented. The faces for which the certainty of naming does not reach some threshold will be left unlabeled; this decreases the recall of the method but improves the accuracy of the labeled tracks.

Results

The speaking detection labels around 25% of face tracks with around 90% accuracy. No manual annotation of any data is performed other than to evaluate the method (ground truth label for each face track).

Recall here means the proportion of tracks which are assigned a name after applying the “refusal to predict” mechanism.

Two baseline methods were compared to the proposed method: (i) “Prior” – label all tracks with the name which occurs most often in the script i.e. Buffy (accuracy -21.9 %) (ii) “Subtitles only” – label any tracks with proposed names from the script (accuracy-45%).

Using the proposed method, if forced to assign a name to all face tracks, the accuracy obtained is around 69% in both episodes. Requiring only 80% of tracks to be labeled increases the accuracy to around 80%.

Results

Conclusion

Promising results obtained without any supervision except the readily available annotation.

The detection method and appearance models used can be improved by including certain weak cues like hair or eye color and by using a specific body

tracker rather than a generic point tracker in cases where face detection is very difficult