Is Someone Speaking? Exploring Long-term Temporal Features ...

Is Someone Speaking Exploring Long-term Temporal Featuresfor Audio-visual Active Speaker Detection

Ruijie Tao Zexu Pan Rohan Kumar Das Xinyuan Qian Mike Zheng Shou Haizhou Liruijietaopan_zexuunuseduecerohangmailcomeleqianmikeshouhaizhoulinusedusg

Department of Electrical and Computer Engineering National University of SingaporeSingapore Singapore

ABSTRACTActive speaker detection (ASD) seeks to detect who is speaking in avisual scene of one or more speakers The successful ASD dependson accurate interpretation of short-term and long-term audio andvisual information as well as audio-visual interaction Unlike theprior work where systems make decision instantaneously usingshort-term features we propose a novel framework named TalkNetthat makes decision by taking both short-term and long-term fea-tures into consideration TalkNet consists of audio and visual tempo-ral encoders for feature representation audio-visual cross-attentionmechanism for inter-modality interaction and a self-attentionmechanism to capture long-term speaking evidence The exper-iments demonstrate that TalkNet achieves 35 and 22 improve-ment over the state-of-the-art systems on the AVA-ActiveSpeakerdataset and Columbia ASD dataset respectively Code has beenmade available at httpsgithubcomTaoRuijieTalkNet_ASD

CCS CONCEPTSbull Information systemsrarr Speech audio search

KEYWORDSActive Speaker Detection Long-Term Temporal Network Audio-Visual Cross-Attention

ACM Reference FormatRuijie Tao Zexu Pan Rohan Kumar Das Xinyuan Qian Mike Zheng ShouHaizhou Li 2021 Is Someone Speaking Exploring Long-term TemporalFeatures for Audio-visual Active Speaker Detection In Proceedings of the29th ACM International Conference on Multimedia (MM rsquo21) October 20ndash24 2021 Virtual Event China ACM New York NY USA 9 pages httpsdoiorg10114534740853475587

1 INTRODUCTIONActive speaker detection (ASD) seeks to detect who is speaking in avisual scene of one or more speakers [35] As the speaking circum-stances are very fluid and change dynamically ASD has to predictat a fine granularity in time ie at video frame level This task isan essential frontend for a wide range of multi-modal applicationssuch as audio-visual speech recognition [1] speech separation [29]speaker diarization [14 15] and speaker tracking [33 34]

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page Copyrights for third-party components of this work must be honoredFor all other uses contact the ownerauthor(s)MM rsquo21 October 20ndash24 2021 Virtual Event Chinacopy 2021 Copyright held by the ownerauthor(s)ACM ISBN 978-1-4503-8651-72110httpsdoiorg10114534740853475587

(b) A 2-second video segment

(a) A 200 ms video segment

Figure 1 An illustration of the speaking evidence betweena short segment and a long segment (a) Five video framesevenly sampled from a 200 ms video segment where thespeaking activity is not evident (b) Five video frames evenlysampled froma 2-second video segment where the speakingactivity becomes evident in long-term temporal context andaudio-visual synchronization

Among the factors that humans judge whether a person is speak-ing are 1) Does the audio of interest belong to human voice 2) Arethe lips of the person of interest moving 3) If the above are true isthe voice synchronized with the lip movement Based on this cog-nitive finding there have been deep learning solutions that extractaudio and visual features to make binary classification [6 17 35 44]Despite much progress the existing ASD systems have not fullybenefited from two aspects of available information the temporaldynamics of audio and visual flow and the interaction betweenaudio and visual signals that limits the scope of applications espe-cially for challenging real-world scenarios

As the short-term audio and visual features represent the salientcues for ASD most of the existing studies are focused on segment-level information eg a video segment of 200 to 600 ms Howeveras illustrated in Figure 1(a) it is hard to judge the speaking activityfrom a video segment of 200ms not tomention the audio-visual syn-chronization A longer 2-second video as displayed in Figure 1(b)would be more evident of the speaking episode When humansare detecting an active speaker we typically consider saying an

arX

iv2

107

0659

2v2

[ee

ssA

S] 2

5 Ju

l 202

1

entire sentence that spans over hundreds of video frames for adecision for example an audio-visual episode lasting 5 secondscontains 15 words on average [19 46] A short-term segment of200 ms doesnrsquot even cover a complete word Furthermore singlemodality embedding is not reliable in some challenging scenariosFor example the voice we hear might come from someone elsethan the target speaker at the same time there could be false lipmovements eg laughing eating and yawning that are not relatedto speaking To summarize we consider that the inter-modalitysynchronization such as speech-lip speech-face over the span ofan utterance provides more reliable information than short-termsegments

The systems with short-term features extract audio-visual em-bedding from a fixed-length short segment eg 200 ms [12 17 50]300 ms [44] and 440 ms segment [5 26] By simply increasing thesegment size we are getting the average properties of the segmentat the cost of the time resolution of speaking activities A betterway to capture the long-term temporal context is to encode thehistory of audio or video frame sequence In this paper we studyan audio-visual ASD framework denoted as TalkNet For videosignals the minimum unit is a video frame ie a static image Westudy a temporal network to encode the temporal context overmultiple video frames For audio signals the minimum unit is anaudio frame of tens of milliseconds We study an audio temporal en-coder to encode the temporal context over multiple audio frames Interms of backend classifier we study an audio-visual cross-attentionmechanism to capture inter-modality evidence and a self-attentionmechanism to capture long-term speaking evidence

To the best of our knowledge this paper is the first study on theuse of long-term temporal context and audio-visual inter-modalityinteraction for ASD We make the following contributions

bull We propose a feature representation network to capture thelong-term temporal context from audio and visual cues

bull We propose a backend classifier network that employs audio-visual cross-attention and self-attention to learn the audio-visual inter-modality interaction

bull We propose an effective audio augmentation technique toimprove the noise-robustness of the model

The rest of the paper is organized as follows In Section 2 wediscuss the related work In Section 3 we formulate the proposedTalkNet framework and present its training process In Section 4 andSection 5 we report the experiments and their results respectivelyFinally Section 6 concludes the study

2 RELATEDWORKThis research is built on prior studies on the detection of audio andvisual events and the modeling long-term temporal dynamics ofaudio-visual signals

21 Active Speaker DetectionThere have been prior studies on ASD using audio video and thefusion of both In voice activity detector (VAD) we study howto detect the presence of speech as opposed to other acousticnoises [20 36] However in real-world scenarios audio signalsby distant microphones are inherently ambiguous because of theoverlapping speech and the corruption from background noise

which poses challenges to the VAD task For vision the facial [30]and upper-body [9 38] movements are analyzed to detect if a visibleperson is speaking However the performance is limited due toweak correlation between the body motion and speaking activitiesBesides non-speaking activities eg licking lips eating food andgrinning may also degrade the ASD performance Despite theselimitations the audio or visual single modal solutions serve as thefoundation for ASD

Audio-visual processing has seen significant benefits throughmodality fusion [21 27] As the speech rhythm and word pronun-ciation are closely correlated with facial motion an interesting andpromising alternative is to combine both audio and vision informa-tion to perform ASD Exploring audio-visual ASD one approachis to view it as an assignment task It is assumed that the detectedspeech must belong to one of the speakers on the screen [5 26]However this assumption does not always hold because there couldbe cross talk or off-screen speech in practice Another approachis to perform ASD as a classification task to evaluate the visibleface on the screen one-by-one Some studies [4 12 17] simplyconcatenate the extracted audio and visual features as the inputand apply a multi-layer perceptron (MLP)-based binary classifierto detect the active speaker at each short video segment withoutconsidering the inter-frame temporal dependency Others furtheradopt the backend classifier with temporal structure like recurrentneural network (RNN) [44 45] gated recurrent unit (GRU) [35] andlong short-term memory (LSTM) [6 40 50] which have achievedpreliminary success Our proposed TalkNet is motivated by thisthought

22 Long-term Temporal ContextAs ASD seeks to make a fine-grained decision at audio or videoframe level most of the prior studies employ short-term featuresand make decisions at split segments of less than 600 ms Whilesmoothing method can be used to aggregate short-term decisionsfor for long-term video [5 12] the potential of long-term featureshas not been fully explored yet

It is common that ASD uses individual uni-modal frontend fea-ture extractors to learn the audio and visual embeddings that isfollowed by the backend classifier to incorporate audio-visual syn-chrony For the uni-modal representation learning the utterance-level model performs better than the frame-level model in the audiotasks such as audio classification [22 31] The recent studies invideo object detection also show that it is beneficial to leveragethe temporal context at the proposal-level by end-to-end optimiza-tion to learn the completed video presentations [41 47] As theirshort-term embeddings encode long-term temporal context suchtechniques generally provide improved performance when makingshort-term decisions The success in these studies motivates us toconsider encoding long-term audio and visual temporal context atthe utterance level for ASD task

On the other hand audio-visual ASD takes advantage of thecross-modal synchronization information In audio-visual synchro-nization studies by using convolutional neural network (CNN) [1828] LSTM [39] or attention model [11] the longer video utteranceare used the more representative features can be extracted which

Speaker Detection Backend

Cross-attention

Cross-attention

Self-attention

ASD predictions

Visual TemporalEncoder

Audio TemporalEncoder

Feature Representation Frontend



Figure 2 An overview of our TalkNet which consists of visual and audio temporal encoders followed by cross-attention andself-attention for ASD prediction

eventually boost the performance [17 18] These studies demon-strate that long-term temporal context is significantly important tolearn the audio-visual relationship in ASD As ASD aims to learn themodality feature and the audio-visual relationship we believe it willbenefit from long-term temporal context either from intra-modalsignals or inter-modal signals

3 TALKNETTalkNet is an end-to-end pipeline that takes the cropped face videoand corresponding audio as input and decide if the person is speak-ing in each video frame It consists of a feature representationfrontend and a speaker detection backend classifier as illustratedin Figure 2 The frontend contains an audio temporal encoder anda video temporal encoder They encode the frame-based input au-dio and video signals into the time sequence of audio and videoembeddings that represent temporal context The backend clas-sifier consists of an inter-modality cross-attention mechanism todynamically align audio and visual content and a self-attentionmechanism to observe speaking activities from the temporal con-text at the utterance level

31 Visual Temporal EncoderThe visual temporal encoder aims to learn the long-term represen-tation of facial expression dynamics As illustrated in Figure 3 itconsists of the visual frontend and the visual temporal network

3D Conv

ResNet18

ReLU amp BN

times5

Visual Frontend

Visual Temporal Network

DS-Conv1DV-TCN

Conv1D

Figure 3 The structure of visual temporal encoder whichcontains the visual frontend and the visual temporal net-work

oplusdenotes point-wise addition

We seek to encode the visual stream into a sequence of visual em-beddings 119865119907 that have the same time resolution

The visual frontend explores spatial information within eachvideo frame It consists of a 3D convolutional layer (3D Conv) fol-lowed by a ResNet18 block [2] This frontend encodes the videoframe stream into a sequence of frame-based embedding The vi-sual temporal network consists of a video temporal convolutionalblock (V-TCN) which has five residual connected rectified linearunit (ReLU) batch normalization (BN) and depth-wise separableconvolutional layers (DS Conv1D) [1] followed by a Conv1D layerto reduce the feature dimension It aims to represent the temporalcontent in a long-term visual spatio-temporal structure For exam-ple for a visual temporal encoder that has a receptive field of 21video frames we take a segment of up to 840 ms to encode a videoembedding when the video frame rate is 25 frame-per-second (fps)

32 Audio Temporal EncoderThe audio temporal encoder seeks to learn an audio content repre-sentation from the temporal dynamics It is a 2D ResNet34 networkwith squeeze-and-excitation (SE) module [23] introduced in [13]An audio frame is first represented by a vector of Mel-frequencycepstral coefficients (MFCCs) The audio temporal encoder takesthe sequence of audio frames as the input generate the sequenceof audio embeddings 119865119886 as the output The ResNet34 are designedwith dilated convolutions such that the time resolution of audio em-beddings 119865119886 matches that of the visual embeddings 119865119907 to facilitatesubsequent attention mechanism For example the audio temporalencoder has a receptive field of 189 audio frames In other wordswe take a segment of 1890 ms to encode an audio embedding whenthe MFCC window step is 10 ms to capture the long-term temporalcontext

33 Audio-visual Cross-Attention119865119886 and 119865119907 are expected to characterize the events that are relevantto speaking activities for audio and visual respectively We aremotivated by the fact that audio-visual synchronization is an infor-mative cue for speaking activities as well As audio and visual floweach has its own dynamics they are not exactly time aligned Theactual audio-visual alignment may depend on the instantaneous

MatMul

Scale

Softmax

MatMulMatMul

MatMul

Scale

Softmax

LinearLinear

(a)

MatMul

MatMul

Scale

Softmax

Linear

(b)

Figure 4 (a) The attention layer in the cross-attention net-work Considering the audio embeddings 119865119886 as the sourceand the visual feature 119865119907 as the target we generate audio at-tention feature 119865119886rarr119907 as the output Similarly we generatevisual attention feature 119865119907rarr119886 (b) The attention layer in theself-attention network

phonetic content and the speaking behavior of the speakers We pro-pose two cross-attention networks along the temporal dimensionto dynamically describe such audio-visual interaction

The core part of the cross-attention network is the attentionlayer which is shown in Figure 4 (a) The inputs are the vectorsof query (119876119886 119876119907 ) key (119870119886 119870119907 ) and value (119881119886119881119907 ) from audio andvisual embeddings respectively projected by a linear layer Theoutputs are the audio attention feature 119865119886rarr119907 and visual attentionfeature 119865119907rarr119886 as formulated in Eq (1) and Eq (2) where 119889 denotesthe dimension of 119876 119870 and 119881

119865119886rarr119907 = 119904119900 119891 119905119898119886119909 (119876119907119870

119879119886radic119889

)119881119886 (1)

119865119907rarr119886 = 119904119900 119891 119905119898119886119909 (119876119886119870119879119907radic119889

)119881119907 (2)

As formulated in Eq (1) and Eq (2) to learn the interacted new audiofeature 119865119886rarr119907 the attention layer applies 119865119907 as the target sequenceto generate query and 119865119886 as the source sequence to generate keyand value and to learn 119865119907rarr119886 vice versa The attention layer isfollowed by the feed-forward layer Residual connection and layernormalization are also applied after these two layers to generate thewhole cross-modal attention network The outputs are concatenatedtogether along the temporal direction

34 Self-Attention and ClassifierA self-attention network is applied after the cross-attention networkto model the audio-visual utterance-level temporal informationAs illustrated in Figure 4 (b) this network is similar to the cross-attention network except that now the query (119876119886119907 ) key (119870119886119907 ) andvalue (119881119886119907 ) in the attention layer all come from the joint audio-visualfeature 119865119886119907 With the self-attention layer we seek to distinguishthe speaking and non-speaking frames

35 Loss FunctionWe finally apply a fully connected layer followed by a softmaxoperation to project the output of the self-attention network to anASD label sequence We view ASD as a frame-level classificationtask The predicted label sequence is compared with the groundtruth label sequence by cross-entropy loss The loss function ispresented in Eq (3) where 119904119894 and119910119894 are the predicted and the groundtruth ASD labels of 119894th video frame 119894 isin [1119879 ]119879 refers to the numberof video frames

119871119900119904119904 = minus 1119879

119879sum119894=1

(119910119894 middot log 119904119894 + (1 minus 119910119894 ) middot log (1 minus 119904119894 )) (3)

36 Audio Augmentation with NegativeSampling

The noise-robustness of ASD in the presence of noise and interfer-ence speakers remains a challenging topic One traditional audioaugmentation method is to use a large noise dataset [43] to aug-ment the training data by overlaying the noise on top of the originalsound track This method involves the external data source to in-crease the diversity However it is not straightforward to find suchacoustic data that matches the video scenes

To increase the amount of samples we propose a negative sam-pling method to offer a simple yet effective solution In practicewe use one video as the input data during training and then werandomly select the audio track from another video in the samebatch as the noise to perform audio augmentation Such augmenteddata effectively have the same label eg active speaker or inactivespeaker as the original sound track This approach involves thein-domain noise and interference speakers from the training setitself It does not require data outside the training set for audioaugmentation

4 EXPERIMENTS41 Dataset411 AVA-ActiveSpeaker The AVA-ActiveSpeaker dataset1 isderived from Hollywood movies [35] It contains 29 723 8 015 and21 361 video utterances in the training validation and test setsrespectively The video utterances range from 1 to 10 seconds andare provided as face tracks We follow the official evaluation tooland report the performance in terms of mean average precision(mAP)

There are several challenges involved in the AVA-ActiveSpeakerdataset The language is diverse and the frame per second (fps) ofthe movies varies Furthermore a significant number of videos haveblurry images and noisy audio It also contains many old movieswith dubbed dialogues All these factors make it hard to accuratelysynchronize the audio-visual signals

412 Columbia Active Speaker Dataset The Columbia ASDdataset2 is a standard benchmark test dataset for ASD [10] It con-tains an 87-minute panel discussion video with 5 speakers takingturns to speak in which 2-3 speakers are visible at any given time

1httpsresearchgooglecomavadownloadhtmlava_active_speaker_download2httpwwwjaychakravartycomactive-speaker-detection

Table 1 All valid conditions of ASD videos in the wild (Noteonly when the audio is active and the lips are moving do weconsider whether audio-visual is synchronized)

Index Audio Lips Audio-visual

ASD Label

1 active moving sync speaking2 active moving not sync non-speaking3 active not moving NA non-speaking4 inactive moving NA non-speaking5 inactive not moving NA non-speaking

We follow the common protocol of this benchmark to use F1 scoreas the evaluationmetric The Columbia ASD dataset doesnrsquot providea common splitting between training and test sets

413 TalkSet Due to its limited size the Columbia ASD dataset isusually only used as a test set Furthermore the AVA-ActiveSpeakerdataset is labelled with face bounding boxes with a different algo-rithm which are incompatible with those of the Columbia ASDdataset We are motivated by the call for an audio-visual ASDdataset that covers real-world scenarios This leads to the idea of anew database We leverage two large-scale audio-visual datasets inthe wild LRS3 [3] and VoxCeleb2 [16] to form a new ASD datasetnamed ldquoTalkSetrdquo that covers all valid ASD conditions

First we consider that humans detect active speakers by exam-ining three aspects of a video 1) On audio signal is there an activevoice 2) For visual signal are the lips of someone moving 3)Whenthere is an active voice and the lips of someone are moving is thevoice synchronized with the lips movement The above three cueslead to five valid conditions in the real world which are summarizedin Table 1

We select 90000 videos with active voice from VoxCeleb23 [16]We also collect 60000 videos without an active voice at the sametime longer than one second from LRS34 [3] using the Kaldi-basedvoice activity detection system [32] In total we have got 150000videos that range from 1 to 6 seconds The total length of thesevideos is 15165 hours out of which 7145 hours are speaking and8020 hours are non-speaking We randomly split it into 135000videos for training and 15000 videos for validation Finally weadopt the Columbia ASD dataset as the test data

We understand that both LRS3 and VoxCeleb2 use the S3FD facedetection method [49] to provide ground truth face tracking of thespeakers To be consistently we also apply the same method forface tracking on unknown test videos including the Columbia ASDdataset

42 Implementation DetailsWe build the TalkNet using the PyTorch library with the Adamoptimizer The initial learning rate is 10minus4 and we decrease it by5 for every epoch The dimension of MFCC is 13 All the facesare reshaped into 112 times 112 We set the dimensions of the audioand visual feature as 128 Both cross-attention and self-attentionnetwork contain one transformer layer with eight attention heads

3httpswwwrobotsoxacuk~vggdatavoxcelebvox2html4httpswwwrobotsoxacuk~vggdatalip_readinglrs3html

We randomly flip rotate and crop the original images to performvisual augmentation As the Columbia ASD dataset is an opendataset we apply the additional sources from RIRs data [25] andthe MUSAN dataset [42] to perform audio augmentation on theTalkSet during training and evaluate the performance using Sklearnlibrary5 For the AVA-ActiveSpeaker dataset we apply the proposednegative sampling technique to add the in-domain noise from thetraining set itself Finally we evaluate the performance on the testset using the official tool6 We also evaluate the performance onthe validation set as it comes with the ground truth labels for quickexamination

5 RESULTS51 Comparison with the State-of-the-artWe now compare the proposed TalkNet with the state-of-the-artsystems on both the AVA-ActiveSpeaker and Columbia ASD datasetFirst we summarize the results on the AVA-ActiveSpeaker datasetin Table 2 We observe that TalkNet achieves 923 mAP and out-performs the best competitive system ie MAAS-TAN [26] by 35on the validation set Some studies report their results in terms ofArea under the Curve of ROC (AUC) on the same validation setFor ease of comparison we also report the comparison of AUCresults in Table 3 Without surprise the TalkNet also achieves 36improvement over the best reported AUC cf Huang et al [24]

Table 2 Comparison with the state-of-the-art on the AVA-ActiveSpeaker validation set in terms of mean average pre-cision (mAP)

Method mAP ()Roth et al [26 35] 792Zhang et al [50] 840MAAS-LAN [26] 851Alcazar et al [5] 871Chung et al [12] 878MAAS-TAN [26] 888

TalkNet (proposed) 923

Table 3 Comparison with the state-of-the-art on the AVA-ActiveSpeaker validation set in terms of area under thecurve (AUC)

Model AUC ()Sharma et al [40] 820Roth et al [35] 920Huang et al [24] 932


As the ground truth labels of the AVA-ActiveSpeaker test setare not available to the public we obtain the evaluation results inTable 4 on the test set with the assistance of the organizer Our908mAP also outperforms the best prior work by 30 cf Chunget al [12]5httpsscikit-learnorgstablemodulesgeneratedsklearnmetricsf1_scorehtml6httpsgithubcomactivitynetActivityNet

Note that some prior studies [5 26] applied additional networksto learn the relationship among the cropped face videos Others [1250] used the pre-trained model in another large-scale dataset Bycontrast TalkNet only uses the AVA-ActiveSpeaker training set totrain the single face videos from scratch without any additionalpost-processing We believe that pre-training and other advancedtechniques will further improve TalkNet which is beyond the scopeof this paper

Table 4 Comparison with the state-of-the-art on the AVA-ActiveSpeaker test set in terms of mAP

Method mAP ()Roth et al [35] 821Zhang et al [50] 835Alcazar et al [5] 867Chung et al [12] 878


We then evaluate TalkNet on the Columbia Active Speaker De-tection dataset Its performance along with comparison to otherexisting methods are shown in Table 5 We observe that the F1 scorewhich is the standard metric in this benchmark is the maximumfor proposed TalkNet which is 962 for the average result thathas an improvement over the best existing system by 22 For allthe five speakers TalkNet provides the best performance for threeof them (Bell Lieb and Sick) It is noted that Columbia ASD is anopen-training dataset so the methods in Table 5 are trained ondifferent data so we only claim that our TalkNet is efficient on theColumbia ASD dataset

Table 5 Comparison with the state-of-the-art on theColumbia ASD dataset in terms of F1 scores ()

Method SpeakerBell Boll Lieb Long Sick Avg

Brox et al [8 37] 841 723 806 600 689 732Chakravarty et al [10] 829 658 736 869 818 782Zach et al [37 48] 892 888 858 814 860 862

RGB-DI [37] 863 938 923 761 863 870SyncNet [17] 937 834 868 977 861 895LWTNet [4] 926 824 887 944 959 908RealVAD [7] 920 989 941 891 928 934S-VVAD [38] 924 972 923 955 925 940

TalkNet (proposed) 971 900 991 966 981 962

52 Ablation StudyWe further perform experiments to analyze the contributions ofindividual techniques deployed in TalkNet

521 Long-term sequence-level temporal context The priorstudies usually use short-term features of 5 to 13 video frameson the AVA-Activespeaker dataset [5 12 26] for video embeddingWe believe that long-term features are more evident of speakingepisode To study the difference between long-term and short-term

features we use a fixed number of 119873 frames instead of the entirevideo sequence during training and testing where 119873 is chosen from5102550100 that amounts to 02 04 1 2 and 4 second

We report the evaluation results of TalkNet in Table 6 and ob-serve that the system can hardly work with very short video seg-ment eg when 119873 = 5 as there is not enough temporal contextin a 02-second segment As the duration of video increases mAPimproves consistently from 752 to 894

As we increase the duration of videos there are a fewer numbervideo segments for training As a result we donrsquot observe improve-ment from 50 frames to 100 frames of video duration This studyconfirms our hypothesis that the long-term sequence-level informa-tion is a major source of contributions to the improved performance

Table 6 Performance evaluation by the length of the videoon the AVA-ActiveSpeaker validation set We use a fixednumber of video frames during both training and testing

video frames Length (APRXseconds) mAP()

5 02 75210 04 82825 1 87950 2 895100 4 894

Variable 1 - 10 923

522 Short-term vs long-term features To appreciate the con-tribution of the long-term features we further compare TalkNetwith a prior work [5] which uses the short-term audio and visualembedding and the relationship between the co-occurring speakersvia a two-step training The first step encodes the low-dimensionalrepresentation for video segments of 440 ms and fuses the audio-visual information which is similar to TalkNet except that there isneither temporal encoder to increase the receptive fields and norattention mechanism

We first reproduce the system in [5] to obtain 782 mAP for 11video frames input which is slightly lower than 795 mAP in theoriginal paper due to different batch size setting Then we extendthe segments to 25 video frames and compare the results in Table 7We observe that the TalkNet obtains a 48 improvement from thelonger input videos However the prior work [5] does not benefitfrom the longer segments with a performance drop of 21 Thisstudy suggests that longer video duration doesnrsquot help without thelong audio and visual receptive fields and an adequate attentionmechanism

Table 7 A contrastive study between two systems on effi-cient use of long video segments on the AVA-ActiveSpeakervalidation set in terms of mAP ()

Method video frames Change11 25Alcazar et al [5] 782 761 minus21

TalkNet (Proposed) 831 879 +48

(a)

(b)

(c)

Figure 5 Results of TalkNet for the real-world videos with one person on the screen The green box denotes the active speakerThe red box denotes the inactive speaker The time interval between the adjacent images is 1 second (a) Theman is speaking inthe noisy environment (b) The woman is introducing themakeup process through dubbing So the speech is not synchronizedwith her lip movement (c) The woman is eating candy Although her lips are always moving she is not speaking in thebeginning

(b)

(a)

Figure 6 Results of TalkNet for the real-world videos with multiple persons on the screen (a) Two speakers take turns speak-ing and the manrsquos lips are concealed sometimes (b) Four speakers are talking in a boisterous environment with backgroundmusic Everyonersquos mouth is moving but only the girl on the right is speaking

523 Ablation study of TalkNet attention mechanism Toshow the contribution of audio-visual relationship and interactionover a long-term video to the ASD task we conduct the ablationstudy for the audio-visual attention network in TalkNet The resultsare summarized in Table 8 We find that without cross-attention or

self-attention the performance will drop 07 or 14 mAP respec-tively on the AVA-ActiveSpeaker validation set When removingthe whole audio-visual attention network the result will decreaseto only 900 mAP by 23 The results confirm the effectivenessof the cross-attention and self-attention in learning inter-modalitycues and long-term audio-visual temporal context

Table 8 Ablation study of the cross-attention and self-attentionmechanisms inTalkNet on theAVA-ActiveSpeakervalidation set

Model mAP()wo Both 900

wo Self-attention 909wo Cross-attention 916

TalkNet 923

524 Audio augmentation We report the audio augmentationexperiments of TalkNet on the AVA-ActiveSpeaker validation setin Table 9 lsquoWith neg_samplingrsquo lsquoWith noise_augrsquo and lsquoWo au-dio_augrsquo stand for our proposed negative sampling method thetraditional audio augmentationmethodwhich involves a large noisedataset and without any audio data augmentation The proposedTalkNet without any audio data augmentation still outperformsthe state-of-the-art We also observe that there is an obvious differ-ence with and without negative sampling technique The proposednegative sampling technique outperforms the traditional audioaugmentation These results confirm the efficiency of the negativesampling method which doesnrsquot involve external data

Table 9 Evaluation of TalkNet with and without audio dataaugmentation on the AVA-ActiveSpeaker validation set

Augmentation conditions mAP()Wo audio_aug 894With noise_aug 922

With neg_sampling 923

53 Qualitative AnalysisFigure 5 and Figure 6 show some results of TalkNet in the real-world videos with one person and multiple persons on the screenrespectively

In Figure 7(a) we report the performance for different face sizeson the AVA-ActiveSpeaker validation set lsquoSmallrsquo The face widthis smaller than 64 pixels lsquoMiddlersquo The face width is between 64to 128 pixels lsquoLargersquo The face width is larger than 128 pixels Weobserve that the performance decreases as face size gets smallerIt is worth noting that our proposed TalkNet always achieves thebest results across all face sizes

In Figure 7(b) we study the effect of the number of visible facesin a video frame ie 1 2 or 3 faces which represent about 90 of allthe validation data From Figure 7(b) we observe that the ASD taskbecomes more challenging as the number of faces increases Whilethe performance drops across all methods the proposed TalkNet isclearly more robust than other competing methods

6 CONCLUSIONIn this work we address and study audio-visual ASD with long-term temporal features TalkNet utilizes the sentence-level audio-visual videos information as the input to explore the audio-visualrelationship and synchronization information TalkNet outperforms

Large Middle SmallFace size

40

50

60

70

80

90

100

mAP

()

930

794

552

922

790

562

864

683

449

953

859

637

TalkNet (porposed)MAAS-TANAlcazar et alRoth et al

(a)

1 2 3Number of faces

40

50

60

70

80

90

100m

AP(

)933

858

682

918

838

676

879

716

544

954

896

803


(b)

Figure 7 The performance of our TalkNet and other compet-ing methods for (a) various face sizes and (b) specific facesize in the same video frame

the state-of-the-art results in both two mainstream ASD benchmarkby 35 and 22

7 ACKNOWLEDGMENTThis work was supported by Grant No A18A2b0046 from the Sin-gapore Governmentrsquos Research Innovation and Enterprise 2020plan (Advanced Manufacturing and Engineering domain) GrantNo 1922500054 from the Science and Engineering Research Coun-cil Agency of Science Technology and Research Singapore theNational Robotics Program

REFERENCES[1] Triantafyllos Afouras Joon Son Chung Andrew Senior Oriol Vinyals and An-

drew Zisserman 2018 Deep audio-visual speech recognition IEEE transactionson pattern analysis and machine intelligence (TPAMI) (2018)

[2] Triantafyllos Afouras Joon Son Chung and Andrew Zisserman 2018 TheConversation deep audio-visual speech enhancement Proc Interspeech (2018)3244ndash3248

[3] Triantafyllos Afouras Joon Son Chung and Andrew Zisserman 2018 LRS3-TEDa large-scale dataset for visual speech recognition arXiv preprint arXiv180900496(2018)

[4] Triantafyllos Afouras Andrew Owens Joon Son Chung and Andrew Zisserman2020 Self-Supervised Learning of Audio-Visual Objects from Video In EuropeanConference on Computer Vision

[5] Juan Leoacuten Alcaacutezar Fabian Caba Long Mai Federico Perazzi Joon-Young LeePablo Arbelaacuteez and Bernard Ghanem 2020 Active speakers in context InProceedings of the IEEECVF Conference on Computer Vision and Pattern Recognition(CVPR) 12465ndash12474

[6] Ido Ariav and Israel Cohen 2019 An end-to-end multimodal voice activitydetection using wavenet encoder and residual networks IEEE Journal of SelectedTopics in Signal Processing 13 2 (2019) 265ndash274

[7] Cigdem Beyan Muhammad Shahid and Vittorio Murino 2020 RealVAD Areal-world dataset and a method for voice activity detection by body motionanalysis IEEE Transactions on Multimedia (2020)

[8] Thomas Brox Andreacutes Bruhn Nils Papenberg and Joachim Weickert 2004 Highaccuracy optical flow estimation based on a theory for warping In Europeanconference on computer vision (ECCV) Springer 25ndash36

[9] Punarjay Chakravarty Sayeh Mirzaei Tinne Tuytelaars and Hugo Van hamme2015 Whorsquos speaking Audio-supervised classification of active speakers invideo In Proceedings of the 2015 ACM on International Conference on MultimodalInteraction (ICMI ) 87ndash90

[10] Punarjay Chakravarty and Tinne Tuytelaars 2016 Cross-modal supervision forlearning active speaker detection in video In European Conference on ComputerVision (ECCV) Springer 285ndash301

[11] Ying Cheng Ruize Wang Zhihao Pan Rui Feng and Yuejie Zhang 2020 Looklisten and attend Co-attention network for self-supervised audio-visual repre-sentation learning In Proceedings of the 28th ACM International Conference onMultimedia (ACM MM) 3884ndash3892

[12] Joon Son Chung 2019 Naver at ActivityNet Challenge 2019ndashTask B ActiveSpeaker Detection (AVA) arXiv preprint arXiv190610555 (2019)

[13] Joon Son Chung Jaesung Huh Seongkyu Mun Minjae Lee Hee Soo Heo SoyeonChoe Chiheon Ham Sunghwan Jung Bong-Jin Lee and Icksang Han 2020 Indefence of metric learning for speaker recognition In Interspeech 2020

[14] Joon Son Chung Jaesung Huh Arsha Nagrani Triantafyllos Afouras and AndrewZisserman 2020 Spot the Conversation Speaker Diarisation in the Wild In ProcInterspeech 2020 299ndash303

[15] Joon Son Chung Bong-Jin Lee and Icksang Han 2019 Who Said That Audio-Visual Speaker Diarisation of Real-World Meetings In Proc Interspeech 2019371ndash375

[16] Joon Son Chung Arsha Nagrani and Andrew Zisserman 2018 VoxCeleb2 Deepspeaker recognition In Proc of Interspeech 2018 1086ndash1090

[17] Joon Son Chung and Andrew Zisserman 2016 Out of time automated lip syncin the wild In Asian conference on computer vision (ACCV) Springer 251ndash263

[18] Soo-Whan Chung Joon Son Chung and Hong-Goo Kang 2019 Perfect matchImproved cross-modal embeddings for audio-visual synchronisation In IEEEInternational Conference on Acoustics Speech and Signal Processing (ICASSP) 2019IEEE 3965ndash3969

[19] Martin Cutts 2020 Oxford guide to plain English Oxford University Press USA[20] Shaojin Ding Quan Wang Shuo-Yiin Chang Li Wan and Ignacio Lopez Moreno

2020 Personal VAD Speaker-Conditioned Voice Activity Detection In ProcOdyssey 2020 The Speaker and Language Recognition Workshop 433ndash439

[21] Ariel Ephrat Inbar Mosseri Oran Lang Tali Dekel Kevin Wilson AvinatanHassidim William T Freeman and Michael Rubinstein 2018 Looking to listenat the cocktail party A speaker-independent audio-visual model for speechseparation ACM Transactions on Graphics 37 4 (2018) 1121ndash11211

[22] Marcelo Schreiber Fernandes Weverton Cordeiro and Mariana Recamonde-Mendoza 2021 Detecting Aedes aegypti mosquitoes through audio classificationwith convolutional neural networks Computers in Biology and Medicine 129(2021) 104152

[23] Jie Hu Li Shen and Gang Sun 2018 Squeeze-and-excitation networks In Pro-ceedings of the IEEE conference on computer vision and pattern recognition (CVPR)7132ndash7141

[24] Chong Huang and Kazuhito Koishida 2020 Improved active speaker detectionbased on optical flow In Proceedings of the IEEECVF Conference on ComputerVision and Pattern Recognition (CVPR) Workshops 950ndash951

[25] T Ko V Peddinti D Povey M L Seltzer and S Khudanpur 2017 A study ondata augmentation of reverberant speech for robust speech recognition In IEEEInternational Conference on Acoustics Speech and Signal Processing (ICASSP) 2017

5220ndash5224[26] Juan Leoacuten-Alcaacutezar Fabian Caba Heilbron Ali Thabet and Bernard Ghanem 2021

MAAS Multi-modal assignation for active speaker detection arXiv preprintarXiv210103682 (2021)

[27] Vicente Peruffo Minotto Claudio Rosito Jung and Bowon Lee 2015 Multimodalmulti-channel on-line speaker diarization using sensor fusion through SVM IEEETransactions on Multimedia 17 10 (2015) 1694ndash1705

[28] Andrew Owens and Alexei A Efros 2018 Audio-visual scene analysis withself-supervised multisensory features In Proceedings of the European Conferenceon Computer Vision (ECCV) 631ndash648

[29] Zexu Pan Ruijie Tao Chenglin Xu and Haizhou Li 2020 Muse Multi-modaltarget speaker extraction with visual cues arXiv preprint arXiv201007775 (2020)

[30] Foteini Patrona Alexandros Iosifidis Anastasios Tefas Nikolaos Nikolaidis andIoannis Pitas 2016 Visual voice activity detection in the wild IEEE Transactionson Multimedia 18 6 (2016) 967ndash977

[31] Jordi Pons and Xavier Serra 2019 Randomly weighted cnns for (music) audioclassification In IEEE International Conference on Acoustics Speech and SignalProcessing (ICASSP) 2019 IEEE 336ndash340

[32] Daniel Povey Arnab Ghoshal Gilles Boulianne Lukas Burget Ondrej GlembekNagendra Goel Mirko Hannemann Petr Motlicek Yanmin Qian Petr Schwarzet al 2011 The Kaldi speech recognition toolkit In IEEE 2011 Workshop onAutomatic Speech Recognition and Understanding (ASRU)

[33] Xinyuan Qian Alessio Brutti Oswald Lanz Maurizio Omologo and AndreaCavallaro 2021 Audio-visual tracking of concurrent speakers IEEE Transactionson Multimedia (2021)

[34] Xinyuan Qian Maulik Madhavi Zexu Pan Jiadong Wang and Haizhou Li 2021Multi-target DoA estimation with an audio-visual fusion mechanism In IEEEInternational Conference on Acoustics Speech and Signal Processing (ICASSP) 2021IEEE 4280ndash4284

[35] Joseph Roth Sourish Chaudhuri Ondrej Klejch Radhika Marvin Andrew Gal-lagher Liat Kaver Sharadh Ramaswamy Arkadiusz Stopczynski Cordelia SchmidZhonghua Xi et al 2020 AVA active speaker An audio-visual dataset for activespeaker detection In IEEE International Conference on Acoustics Speech and SignalProcessing (ICASSP) 2020 IEEE 4492ndash4496

[36] Abhishek Sehgal and Nasser Kehtarnavaz 2018 A convolutional neural networksmartphone app for real-time voice activity detection IEEE Access 6 (2018)9017ndash9026

[37] Muhammad Shahid Cigdem Beyan and Vittorio Murino 2019 Comparisons ofvisual activity primitives for voice activity detection In International Conferenceon Image Analysis and Processing Springer 48ndash59

[38] Muhammad Shahid Cigdem Beyan and Vittorio Murino 2021 S-VVAD VisualVoice Activity Detection by Motion Segmentation In Proceedings of the IEEECVFWinter Conference on Applications of Computer Vision 2332ndash2341

[39] Yoav Shalev and Lior Wolf 2020 End to end lip synchronization with a temporalautoencoder In Proceedings of the IEEECVF Winter Conference on Applications ofComputer Vision 341ndash350

[40] Rahul Sharma Krishna Somandepalli and Shrikanth Narayanan 2020 Cross-modal learning for audio-visual speech event localization arXiv preprintarXiv200304358 (2020)

[41] Mykhailo Shvets Wei Liu and Alexander C Berg 2019 Leveraging long-rangetemporal relationships between proposals for video object detection In Pro-ceedings of the IEEECVF International Conference on Computer Vision (ICCV)9756ndash9764

[42] D Snyder G Chen and D Povey 2015 MUSAN A Music Speech and NoiseCorpus CoRR abs151008484 (2015) httparxivorgabs151008484

[43] D Snyder D Garcia-Romero G Sell D Povey and S Khudanpur 2018 X-Vectorsrobust DNN embeddings for speaker recognition In IEEE International Conferenceon Acoustics Speech and Signal Processing (ICASSP) 2018 5329ndash5333

[44] Fei Tao and Carlos Busso 2017 Bimodal recurrent neural network for audiovisualvoice activity detection In INTERSPEECH 1938ndash1942

[45] Fei Tao and Carlos Busso 2019 End-to-end audiovisual speech activity detectionwith bimodal recurrent neural models Speech Communication 113 (2019) 25ndash35

[46] Steve Tauroza and Desmond Allison 1990 Speech rates in british english Appliedlinguistics 11 1 (1990) 90ndash105

[47] Haiping Wu Yuntao Chen Naiyan Wang and Zhaoxiang Zhang 2019 Sequencelevel semantics aggregation for video object detection In Proceedings of theIEEECVF International Conference on Computer Vision 9217ndash9225

[48] Christopher Zach Thomas Pock and Horst Bischof 2007 A duality basedapproach for realtime tv-l 1 optical flow In Joint pattern recognition symposiumSpringer 214ndash223

[49] Shifeng Zhang Xiangyu Zhu Zhen Lei Hailin Shi Xiaobo Wang and Stan Z Li2017 S3FD Single shot scale-invariant face detector In Proceedings of the IEEEinternational conference on computer vision (ICCV) 192ndash201

[50] Yuan-Hang Zhang Jingyun Xiao Shuang Yang and Shiguang Shan 2019 Multi-Task Learning for Audio-Visual Active Speaker Detection (2019)

Abstract

1 Introduction

2 Related work

21 Active Speaker Detection

22 Long-term Temporal Context

3 TalkNet

31 Visual Temporal Encoder

32 Audio Temporal Encoder

33 Audio-visual Cross-Attention

34 Self-Attention and Classifier

35 Loss Function

36 Audio Augmentation with Negative Sampling

4 Experiments

41 Dataset

42 Implementation Details

5 Results

51 Comparison with the State-of-the-art

52 Ablation Study

53 Qualitative Analysis

6 Conclusion

7 Acknowledgment

References

entire sentence that spans over hundreds of video frames for adecision for example an audio-visual episode lasting 5 secondscontains 15 words on average [19 46] A short-term segment of200 ms doesnrsquot even cover a complete word Furthermore singlemodality embedding is not reliable in some challenging scenariosFor example the voice we hear might come from someone elsethan the target speaker at the same time there could be false lipmovements eg laughing eating and yawning that are not relatedto speaking To summarize we consider that the inter-modalitysynchronization such as speech-lip speech-face over the span ofan utterance provides more reliable information than short-termsegments

The systems with short-term features extract audio-visual em-bedding from a fixed-length short segment eg 200 ms [12 17 50]300 ms [44] and 440 ms segment [5 26] By simply increasing thesegment size we are getting the average properties of the segmentat the cost of the time resolution of speaking activities A betterway to capture the long-term temporal context is to encode thehistory of audio or video frame sequence In this paper we studyan audio-visual ASD framework denoted as TalkNet For videosignals the minimum unit is a video frame ie a static image Westudy a temporal network to encode the temporal context overmultiple video frames For audio signals the minimum unit is anaudio frame of tens of milliseconds We study an audio temporal en-coder to encode the temporal context over multiple audio frames Interms of backend classifier we study an audio-visual cross-attentionmechanism to capture inter-modality evidence and a self-attentionmechanism to capture long-term speaking evidence

To the best of our knowledge this paper is the first study on theuse of long-term temporal context and audio-visual inter-modalityinteraction for ASD We make the following contributions

bull We propose a feature representation network to capture thelong-term temporal context from audio and visual cues

bull We propose a backend classifier network that employs audio-visual cross-attention and self-attention to learn the audio-visual inter-modality interaction

bull We propose an effective audio augmentation technique toimprove the noise-robustness of the model

The rest of the paper is organized as follows In Section 2 wediscuss the related work In Section 3 we formulate the proposedTalkNet framework and present its training process In Section 4 andSection 5 we report the experiments and their results respectivelyFinally Section 6 concludes the study

2 RELATEDWORKThis research is built on prior studies on the detection of audio andvisual events and the modeling long-term temporal dynamics ofaudio-visual signals

21 Active Speaker DetectionThere have been prior studies on ASD using audio video and thefusion of both In voice activity detector (VAD) we study howto detect the presence of speech as opposed to other acousticnoises [20 36] However in real-world scenarios audio signalsby distant microphones are inherently ambiguous because of theoverlapping speech and the corruption from background noise

which poses challenges to the VAD task For vision the facial [30]and upper-body [9 38] movements are analyzed to detect if a visibleperson is speaking However the performance is limited due toweak correlation between the body motion and speaking activitiesBesides non-speaking activities eg licking lips eating food andgrinning may also degrade the ASD performance Despite theselimitations the audio or visual single modal solutions serve as thefoundation for ASD

Audio-visual processing has seen significant benefits throughmodality fusion [21 27] As the speech rhythm and word pronun-ciation are closely correlated with facial motion an interesting andpromising alternative is to combine both audio and vision informa-tion to perform ASD Exploring audio-visual ASD one approachis to view it as an assignment task It is assumed that the detectedspeech must belong to one of the speakers on the screen [5 26]However this assumption does not always hold because there couldbe cross talk or off-screen speech in practice Another approachis to perform ASD as a classification task to evaluate the visibleface on the screen one-by-one Some studies [4 12 17] simplyconcatenate the extracted audio and visual features as the inputand apply a multi-layer perceptron (MLP)-based binary classifierto detect the active speaker at each short video segment withoutconsidering the inter-frame temporal dependency Others furtheradopt the backend classifier with temporal structure like recurrentneural network (RNN) [44 45] gated recurrent unit (GRU) [35] andlong short-term memory (LSTM) [6 40 50] which have achievedpreliminary success Our proposed TalkNet is motivated by thisthought

22 Long-term Temporal ContextAs ASD seeks to make a fine-grained decision at audio or videoframe level most of the prior studies employ short-term featuresand make decisions at split segments of less than 600 ms Whilesmoothing method can be used to aggregate short-term decisionsfor for long-term video [5 12] the potential of long-term featureshas not been fully explored yet

It is common that ASD uses individual uni-modal frontend fea-ture extractors to learn the audio and visual embeddings that isfollowed by the backend classifier to incorporate audio-visual syn-chrony For the uni-modal representation learning the utterance-level model performs better than the frame-level model in the audiotasks such as audio classification [22 31] The recent studies invideo object detection also show that it is beneficial to leveragethe temporal context at the proposal-level by end-to-end optimiza-tion to learn the completed video presentations [41 47] As theirshort-term embeddings encode long-term temporal context suchtechniques generally provide improved performance when makingshort-term decisions The success in these studies motivates us toconsider encoding long-term audio and visual temporal context atthe utterance level for ASD task

On the other hand audio-visual ASD takes advantage of thecross-modal synchronization information In audio-visual synchro-nization studies by using convolutional neural network (CNN) [1828] LSTM [39] or attention model [11] the longer video utteranceare used the more representative features can be extracted which


Cross-attention

Cross-attention

Self-attention

ASD predictions










3D Conv

ResNet18

ReLU amp BN

times5

Visual Frontend


DS-Conv1DV-TCN

Conv1D







MatMul

Scale

Softmax

MatMulMatMul

MatMul

Scale

Softmax

LinearLinear

(a)

MatMul

MatMul

Scale

Softmax

Linear

(b)




119865119886rarr119907 = 119904119900 119891 119905119898119886119909 (119876119907119870

119879119886radic119889

)119881119886 (1)

119865119907rarr119886 = 119904119900 119891 119905119898119886119909 (119876119886119870119879119907radic119889

)119881119907 (2)




119871119900119904119904 = minus 1119879

119879sum119894=1











ASD Label



































5 02 75210 04 82825 1 87950 2 895100 4 894

Variable 1 - 10 923






(a)

(b)

(c)


(b)

(a)







TalkNet 923










40

50

60

70

80

90

100

mAP

()

930

794

552

922

790

562

864

683

449

953

859

637


(a)


40

50

60

70

80

90

100m

AP(

)933

858

682

918

838

676

879

716

544

954

896

803


(b)
























































Abstract

1 Introduction

2 Related work



3 TalkNet





35 Loss Function


4 Experiments

41 Dataset


5 Results


52 Ablation Study


6 Conclusion

7 Acknowledgment

References


Cross-attention

Cross-attention

Self-attention

ASD predictions










3D Conv

ResNet18

ReLU amp BN

times5

Visual Frontend


DS-Conv1DV-TCN

Conv1D







MatMul

Scale

Softmax

MatMulMatMul

MatMul

Scale

Softmax

LinearLinear

(a)

MatMul

MatMul

Scale

Softmax

Linear

(b)




119865119886rarr119907 = 119904119900 119891 119905119898119886119909 (119876119907119870

119879119886radic119889

)119881119886 (1)

119865119907rarr119886 = 119904119900 119891 119905119898119886119909 (119876119886119870119879119907radic119889

)119881119907 (2)




119871119900119904119904 = minus 1119879

119879sum119894=1











ASD Label



































5 02 75210 04 82825 1 87950 2 895100 4 894

Variable 1 - 10 923






(a)

(b)

(c)


(b)

(a)







TalkNet 923










40

50

60

70

80

90

100

mAP

()

930

794

552

922

790

562

864

683

449

953

859

637


(a)


40

50

60

70

80

90

100m

AP(

)933

858

682

918

838

676

879

716

544

954

896

803


(b)
























































Abstract

1 Introduction

2 Related work



3 TalkNet





35 Loss Function


4 Experiments

41 Dataset


5 Results


52 Ablation Study


6 Conclusion

7 Acknowledgment

References

MatMul

Scale

Softmax

MatMulMatMul

MatMul

Scale

Softmax

LinearLinear

(a)

MatMul

MatMul

Scale

Softmax

Linear

(b)




119865119886rarr119907 = 119904119900 119891 119905119898119886119909 (119876119907119870

119879119886radic119889

)119881119886 (1)

119865119907rarr119886 = 119904119900 119891 119905119898119886119909 (119876119886119870119879119907radic119889

)119881119907 (2)




119871119900119904119904 = minus 1119879

119879sum119894=1











ASD Label



































5 02 75210 04 82825 1 87950 2 895100 4 894

Variable 1 - 10 923






(a)

(b)

(c)


(b)

(a)







TalkNet 923










40

50

60

70

80

90

100

mAP

()

930

794

552

922

790

562

864

683

449

953

859

637


(a)


40

50

60

70

80

90

100m

AP(

)933

858

682

918

838

676

879

716

544

954

896

803


(b)
























































Abstract

1 Introduction

2 Related work



3 TalkNet





35 Loss Function


4 Experiments

41 Dataset


5 Results


52 Ablation Study


6 Conclusion

7 Acknowledgment

References



ASD Label



































5 02 75210 04 82825 1 87950 2 895100 4 894

Variable 1 - 10 923






(a)

(b)

(c)


(b)

(a)







TalkNet 923










40

50

60

70

80

90

100

mAP

()

930

794

552

922

790

562

864

683

449

953

859

637


(a)


40

50

60

70

80

90

100m

AP(

)933

858

682

918

838

676

879

716

544

954

896

803


(b)
























































Abstract

1 Introduction

2 Related work



3 TalkNet





35 Loss Function


4 Experiments

41 Dataset


5 Results


52 Ablation Study


6 Conclusion

7 Acknowledgment

References


















5 02 75210 04 82825 1 87950 2 895100 4 894

Variable 1 - 10 923






(a)

(b)

(c)


(b)

(a)







TalkNet 923










40

50

60

70

80

90

100

mAP

()

930

794

552

922

790

562

864

683

449

953

859

637


(a)


40

50

60

70

80

90

100m

AP(

)933

858

682

918

838

676

879

716

544

954

896

803


(b)
























































Abstract

1 Introduction

2 Related work



3 TalkNet





35 Loss Function


4 Experiments

41 Dataset


5 Results


52 Ablation Study


6 Conclusion

7 Acknowledgment

References

(a)

(b)

(c)


(b)

(a)







TalkNet 923










40

50

60

70

80

90

100

mAP

()

930

794

552

922

790

562

864

683

449

953

859

637


(a)


40

50

60

70

80

90

100m

AP(

)933

858

682

918

838

676

879

716

544

954

896

803


(b)
























































Abstract

1 Introduction

2 Related work



3 TalkNet





35 Loss Function


4 Experiments

41 Dataset


5 Results


52 Ablation Study


6 Conclusion

7 Acknowledgment

References




TalkNet 923










40

50

60

70

80

90

100

mAP

()

930

794

552

922

790

562

864

683

449

953

859

637


(a)


40

50

60

70

80

90

100m

AP(

)933

858

682

918

838

676

879

716

544

954

896

803


(b)
























































Abstract

1 Introduction

2 Related work



3 TalkNet





35 Loss Function


4 Experiments

41 Dataset


5 Results


52 Ablation Study


6 Conclusion

7 Acknowledgment

References





















































Abstract

1 Introduction

2 Related work



3 TalkNet





35 Loss Function


4 Experiments

41 Dataset


5 Results


52 Ablation Study


6 Conclusion

7 Acknowledgment

References

Is Someone Speaking? Exploring Long-term Temporal Features ...

Documents

Transcript of Is Someone Speaking? Exploring Long-term Temporal Features ...