Detecting Re-captured Videos using Shot-Based Photo...

Detecting Re-captured Videos using Shot-BasedPhoto Response Non-Uniformity

Dae-Jin Jung1, Dai-Kyung Hyun1, Seung-Jin Ryu1

, Ji-Won Lee1, Hae-Yeoun Lee2, and Heung-Kyu Lee1

1Department of CS, Korea Advanced Institute of Science and Technology (KAIST),291 Daehak-ro, Yuseong-gu, Daejeon, Republic of Korea{djjung,dkhyun,sjryu,jwlee,hklee}@mmc.kaist.ac.kr

2Department of Computer Software Engineering, Kumoh National Institute ofTechnology, Sanho-ro 77, Gumi, Gyeongbuk, Republic of Korea

[email protected]

Abstract. With advances in digital camcorders, re-capturing commer-cial videos called camcorder theft is getting a big problem. In this paper,we propose an automatic detection method for re-captured videos basedon the photo response non-uniformity (PRNU). To discern a re-capturedvideo, a given video is divided into shots first. Several usable shots areselected and PRNU is estimated from each of the shots. Using peak-to-correlation energy (PCE), a connection matrix, which indicates whichshots were recorded with a specific camcorder, is constructed. Then, falsenegative connections are corrected by using Warshall’s algorithm. Withthe number of connections from connection matrix, the given video is de-termined whether it was the re-captured or not. The experimental resultsshow that the proposed method performs well even with compressed andscaled re-captured videos.

Keywords: Forensics, Photo Response Non-Uniformity (PRNU), Re-captured video

1 Introduction

With highly sophisticated IT technologies, digital camcorders that are capableof producing high quality footage with low prices and easy usage have beendeveloped. Those advantages of using digital camcorders make many people usedigital camcorders more common. Furthermore, traditional analog videos in themovie industry are also replaced by digital videos since digitally recorded moviesare cheap and easy to be edited and stored compared with the traditional ones.

Digital camcorders come into wide use due to their great benefits, however,increase in digital camcorder use brought many misuses. The most commonabuse is re-capturing the commercial videos, called camcorder theft. Approxi-mately 90% of newly released movies are re-captured in the theater with digitalcamcorders. The illegally re-captured videos are the largest source of fake DVDsand unauthorized copies distributed through the Internet [1]. As a result, the

2 Dae-Jin Jung et al.

camcorder theft causes a great loss on movie industry and becomes a big prob-lem.

(a) (b)

Fig. 1. An example of captured shots from a movie : (a) captured shot from originalvideo, (b) captured shot from re-captured video

In early days, re-captured videos had low quality so they could be easilydetected by naked eyes. However, with the highly functional optical device tech-nology, the quality of re-captured videos is improved. As shown in Fig. 1, there-captured video is now comparable to the original video. Therefore, we needan automatic technique which can detect re-captured videos.

Some studies were proposed for protecting videos using watermarking tech-niques against camcorder theft. The representative study was introduced byLee et al. [2]. Their scheme was designed to be robust to camcorder theft andshowed robustness. However, the watermark degrades the quality of videos. Also,the watermarking way requires an embedding process during movie playback.

Cao et al. proposed a method that identifies re-captured images on LCDscreens [3]. Forensic features such as local binary pattern, multi-scale waveletstatistics, and color features were extracted from image sets. By using the ex-tracted features, a probability support vector machine classifier was trained andthen tested. Their scheme could discriminate re-captured images with good qual-ities from original images with equal error rate lower than 0.5%. However, theirmethod took too much time and could not be applied to video directly.

The re-projected video detection by estimating a skew parameter was pro-posed by Wang et al. [4]. Their method could detect the re-projected video withsome frames and could have much lower false positive by extracting more fea-ture points. However, the feature points needed to be positioned in ridged bodygeometry. In this step, some feature points not on the ridged body geometryshould be removed manually since it is hard to check those points automatically.

In this study, we propose a method to discriminate the re-captured videobased on the shot-based photo response non-uniformity (PRNU). The proposedmethod can discriminate re-captured vieo without any additive information andit is designed for videos. Moreover, the entire procedure of the proposed methodperforms automatically.

Detecting Re-captured Videos 3

The rest of this paper is structured as follows. The differences between origi-nal videos and re-captured videos are analyzed in Sec. 2. Then, the detail of theproposed method is explained in Sec. 3. Experimental results are exhibited inSec. 4 and Sec. 5 concludes.

2 Differences between original and re-captured videos

In this chapter, we describe the differences between original and re-capturedvideos. These differences are caused by the following factors:

1) Different recording devices: The original videos can be recorded by analogcameras or digital camcorders. Even though digital camcorders provide sev-eral benefits such as editing efficiency, reducing film cost, easy process toinsert CGs, and etc., analog film cameras are still used because of their owncharacteristics such as high quality, soft shades of colors, and so on. Onthe other hand, the re-captured videos are mostly recorded by digital cam-corders. Compact size, light weight, and easy manipulation make easier forpirates to handle digital camcorders in theaters without being observed.

2) The number of cameras used in recording : In the original videos, multiplecameras are used to record shots. For example, two or more cameras are usedto shoot talking two actors; one for one actor, another for another actor, andthe other for both actors. It means that each shot in the original videos hashigh probability to be recorded by different cameras. On the contrary, onlya single digital camcorder is used to re-capture the original videos becausepirates do not need multiple camcorders to re-capture videos.

3) Different post-processing : Original videos are edited by huge amount of post-processing in general. As discussed above, original videos are recorded bymultiple cameras. Each camera has unique characteristics such as color tone,contrast, brightness and so on. Thus, post-processing for each shot is essentialto harmonize the whole content. Furthermore, it is usual to insert CGs andother visual effects into shots. However, re-captured videos are not edited bymuch post-processing. Only some of them are re-compressed or resized forconvenience.

Above three differences can affect PRNU of the original and re-capturedvideo. The PRNU is pixel variation under illumination. It was proposed to iden-tify the source digital camera by Lukas et al. [5]. Digital camera has a chargecoupled device or complementary metal-oxide-semiconductor sensor, and thePRNU is caused by sensor imperfection which is introduced in sensor manufac-turing process. Since the PRNU is unique for each sensor, it is considered as afingerprint of a digital camera. Also, the PRNU can be used to identify sourcedigital camcorders. Therefore, three differences between original and re-capturedvideos and the characteristics of the PRNU, we can infer some properties for there-captured video detection as follows:

• Spcifically, the shot-based PRNU has low correlation with each other if weestimate them from original shots. First, the shots from analog films do not


have their own PRNU because analog cameras do not include any digitalsensor. Therefore, the PRNU estimated from alnalog shots cannot be usedto identify source analog camera. Second, even though PRNU is estimatedfrom digitally recorded shots, their source camcorders would vary and the es-timated PRNU would be damaged from heavy post-processing. There mightbe several original shots which are recorded by digital camcorders and editedby little post-processing. It may give high correlation among shots. However,those shots are still not be correlated with other shots which are taken fromother digital camcorders. Thus, those correlated shots will be grouped, con-sequently the number of groups will be greater than one. This factor wouldbe an evidence that the given video is original.

• In contrast, the shot-based PRNU of re-captured videos is highly correlatedwith each other. All shots in the re-captured video are taken from the samedigital camcorder and they are edited by little and same post-processing foreach shot. These conditions let the PRNU from the re-captured video becorrelated each other.

By exploiting these properties, we can differentiate the re-captured videosfrom original videos.

3 Proposed method

Fig. 2. An overview of proposed re-captured videos detection

We propose a method that can discriminate re-captured videos from originalvideos. Fig. 2 depicts the proposed method. Once a suspected video is given,the shot change detection process is performed to find suitable shots for PRNUestimation. After dividing the given video into shots, we estimate PRNU fromeach shot. Then peak-to-correlation energy (PCE) values between PRNU is cal-culated as a measure to find out whether those shots are taken from the samedigital camcorder or not. With results of PCE values, we decide whether thegiven video is a re-captured video or not.

3.1 Shot change detection

We first divide a given video into numbers of shots. A shot can be defined as acontinuous strip of motion picture film recorded with a single camera. Accurate


shot change detector, which divides a given video into shots, is important sincewrong shot change declaration can affect the result of re-captured video detec-tion. If two or more shots are declared as a single shot by a shot change detector,PRNU estimated from that shot will be mixed PRNU from plural cameras sothat the false positive rate in PRNU comparison will be increased. In addition,if one shot is declared as two or more shots by a shot change detector, it canalso increase false positive rate in re-captured video detection.

A histogram comparison method is used for shot change detection because ithas good performance and it is relatively fast [6]. Let Hi(j) denote a histogramvalue for ith frame, where j is one of G possible gray levels and SDi is the sumof absolute differences between ith frame and (i+ 1)th frame. Then the sum ofabsolute differences, SDi, is given by the following formula:

SDi =

G∑j=1

|Hi(j)−Hi+1(j)| (1)

To use SDi for shot change detection with any size of video, SDi is normalizedby frame size. If the normalized SDi is larger than a given threshold, the shotchange is declared. Note that the operations in the equations appeared in thispaper are element-wise.

3.2 PRNU estimation

To find out whether the shots are taken from the same digital camcorder or not,PRNU is estimated from those shots and compared each other. PRNU estimationmethod for digital camcorders are proposed by Chen et al. [7]. The PRNU fordigital camcorders is modeled as follow:

I = gγ · [(1 + K)Y + Λ + Θs + Θr]γ + Θq (2)

where I denotes the sensor output compromised by numerous in-camcorder pro-cessing, g does the color channel gain, γ is the gamma correction factor, K isPRNU multiplicative factor which can be used as a fingerprint of digital cam-corder, Y is the light intensity, and Λ, Θs, Θr, Θq denote dark current, shotnoise, read-out noise, and quantization noise, respectively. Using first order Tay-lor expansion, simple form of this model can be obtained:

I = I(0) + γI(0)K + Θ (3)

Here, I(0) is the noise-free sensor output(frame) from one channel before demo-saicing is applied. Θ is a noise component including above noises.

We use simplified model in Eq (3) to estimate PRNU from each shot. To

suppress the influence of the noise-free frame I(0), an estimate I(0) of I(0) issubtracted from both sides of Eq (3). I(0) can be estimated by using denoisingfilter which is a wavelet based filter [8].

W = I− I(0) = IK + (I(0) − I(0)) + [(I(0) − I)K] + Θ (4)


PRNU factor K can be estimated by using Maximum Likelihood Estimation(MLE) method as

γK =

∑Nk=1 Wk I

(0)k∑N

k=1(I(0)k )2

(5)

where Wk is noise residual of kth frame.After MLE process, codec noise is removed by using denoising filter. Usually

a video undergoes DPCM-block DCT transform which causes block artifacts [7].The block artifact should be removed since it causes false correlations betweenuncorrelated PRNU. Wiener filter in frequency domain is used in our method tosuppress the codec noise [9].

Then, PCE is calculated for a pair of PRNU to decide whether two shots aretaken from same digital camcorder. To calculate PCE, we calculate normalizedcorrelation first:

NCC[X,Y] =(X−X) ∗ (Y −Y)

‖X−X‖‖Y −Y‖(6)

where X, Y are estimated PRNU, X is mean of X, X ∗Y is dot product and‖X‖ is the norm of X. Then PCE is calculated as follow [10]:

PCE[X,Y] =|NCC[X,Y](u0, v0)|2

ENCC[X,Y](7)

where (u0, v0) denotes the center location of correlation plane and ENCC[X,Y]

is the correlation plane energy of NCC[X,Y]. If PCE of given two PRNU fromtwo different shots is higher than certain threshold, then we decide that thosetwo shots are taken from same source digital camcorder.

3.3 Detecting re-captured videos

To decide whether a given video is re-captured or not, we investigate every PRNUfrom the video is related with each other. For this purpose, we use Warshallsalgorithm which calculates the connectivity of a given graph [11]. Let the Xi

be the PRNU of selected shot, when i = 1, . . . , N . And we can consider theXi as a vertex. Then, a connection between two vertexes (Xi, Xj) is decidedby the value of PCE[Xi, Xj ]. If the PCE[Xi, Xj ] has greater value than pre-defined threshold T , Xi and Xj have connection to each other. In contrast, lowerPCE[Xi, Xj ] value than threshold T implies Xi and Xj have no connection toeach other. As a consequence a symmetric N ×N connection matrix is createdafter calculating the connectivity for every possible pair of PRNU.

By using Warshalls algorithm, we can correct false negative connections. Fig.3 depicts a simple case of the false negative connection correction by Warshall’salgorithm. After processing Warshall’s algorithm, we can decide the origin ofa given video from the connection matrix. If the N × N connection matrixhas N2 connections, we decide the given video is re-captured video because N2

connections from N PRNU mean that the entire shots have same source digital


(a) (b) (c)

Fig. 3. A simple example of false negative connections correction : (a) Matrix withfalse negative connections, (b) Correcting false negative connections (c) Corrected falsenegative connections

Resolution Main Camera(Digital/Analog)

#1 1280 x 720 Sony PMW-F3(Digital)



#4 1920 x 1080 Red One(Digital)

#5 1920 x 1080 Red One(Digital)

#6 1280 x 720 unknown(unknown)


#8 1280 x 720 Panavision camera(Analog)

#9 1280 x 720 Panavision Panaflex Platinum(Analog)


Table 1. Information about original videos used in experiments(resolutions and theirmain cameras).

camcorder. Otherwise, the given video is decided as original video since less thanN2 connections from N PRNU mean the given video has two or more sourcedigital camcorders.

4 Experimental Results

In this section, we examine the proposed re-captured video detection method.We used 4 digital camcorders (Samsung HMX-H205BD, Sony HDR-CX500, SonyHDR-CX550, and Sony HDR-SR10) to re-capture the original videos. We used10 original videos and 5 of them were fully or partially recorded by digital videocamcorders [12]. 40 videos were created by re-capturing 10 original videos with4 digital camcorders. The resolution of the original videos varied from 1280x720to 1920x1080 and the resolution of re-captured video was set as 1920x1080.Specific information about original videos used in experiments is in Table 1.To estimate PRNU, we divided each video into shots using the shot change


detector. In shot change detection, frames are converted into gray frames tocalculate the histogram differences. Before estimating PRNU from divided shots,some shots unsuitable for PRNU estimation are excluded. More specifically, shotsconstructed with small number of frames or dark frames are excluded since thoseshots can increase false negative rate in PRNU comparison. We extracted 200successive frames from a shot in the PRNU estimation step.

Fig. 4. PCE values in log scale of shots from the same camcorders shots from thedifferent camcorders.

Before testing proposed method, the threshold T for PRNU comparison needsto be set. To decide the adequate threshold T , 2400 pairs of PRNU from samecamcorders and 2400 pairs of PRNU from different camcorders are prepared.Using PCE measurement, a scatter plot of PCE values in log scale for thosepairs was obtained. As shown in Fig. 4, PCE values can be divided by simplestraight line whose values is 80. Thus, we set 80 as a threshold T .

4.1 Re-captured video detection Experiment

We tested re-captured video detection for 10 original videos and 10 videos foreach digital camcorder, totally 40 re-captured videos. 20 shots are collected fromeach video(N = 20). Table 2 shows the result of re-captured video detection.Items in the table is the ratio of connections in N ×N connection matrix. Everyvideo had at least 20 connections in diagonal line in connection matrix since eachshot was correlated with itself. Original videos had lower number of connectionsthan N2 since it was not recorded by single digital camcorder. On the contrary,


#1 #2 #3 #4 #5 #6 #7 #8 #9 #10

Original movie 0.05 0.38 0.33 0.16 0.13 0.05 0.07 0.05 0.05 0.05

Samsung HMX-H205BD 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

Sony HDR-CX500 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

Sony HDR-CX550 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

Sony HDR-SR10 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

Table 2. Connection ratio for original videos and re-captured video (20 shots wereused).

every re-captured video had N2 connections because it had only single sourcedigital camcorder. In this experiment, the detection ratio of re-captured videoswas 100% even before applying Warshall’s algorithm.

4.2 Compression Experiment

Fig. 5. Detection ratio for compressed videos with different quality factors.

We also tested the robustness to compression. Re-captured videos were com-pressed with different quality factors (QFs) while the resolution was not changed.MPEG4 (AVC/H.264) was used in re-encoding. Fig. 5 shows the result for com-pressed re-captured videos. For QF 100∼70, the proposed method showed 100%detection ratio. A few false negative connections appeared in QF 70, but allof false negative connections are corrected by using Warshall’s algorithm. ForQF 60, the detection ratio dropped to 35% because some shots which had noconnection to other shots had appeared. Those shots were not able to be cor-rected by Warshall’s algorithm. However, QF 60 is not commonly used in videocompression due to severe quality degradation such as block artifacts.


4.3 Scaling Experiment

Re-captured videos were scaled with various scale factors (SFs) while QF wasset as 100. Since up-scaling is rare for videos, we only tested for SFs lower than1. MPEG4 (AVC/H.264) was used for re-encoding. Fig. 6 shows the result forscaled re-captured videos. The proposed method showed low detection ratio for

Fig. 6. Detection ratio for scaled videos with different scaling factors.

SF 0.3 which is not parameter for common video resizing. However, the proposedmethod detected most of re-captured videos which were scaled with SF 0.9∼0.4.

4.4 Combinational Experiment

Combinational experiment was also conducted. Usually, re-captured videos arere-encoded before being redistributed. The common options for re-encoding areQFs higher than 80% and SFs higher than 0.5. Thus, we tested proposed methodfor re-captured videos which were re-encoded with parameters of QF 80 and SF0.5. And the proposed method detected them 100%. This result is meaningfulsince those parameters are common for re-encoding videos.

We did not conduct further geometric distortions such as affine transformbecause they are not common for videos. Even if any geometric distortion isproceeded for a re-captured video, every PRNU estimated from shots will besynchronized since all frames in the video are manipulated by the same dis-tortion. Eventually, the re-captured video which has undergone any geometricdistortion will be detected by the proposed method if the distortion does notruin PRNU information severely.

5 Conclusion

In this paper, we have investigated to detect the re-captured videos. The pro-posed method operates automatically for a given video and does not use any


additive information such as watermarks. This proposed method is based on thephoto-response non-uniformity (PRNU), which is unique fingerprint of digitalimage sensors. The proposed method consists of 3 steps. First, a given video isdivided into shots. Then, PRNU is estimated from collected N shots. At last,an N ×N connection matrix is created by evaluating PCEs for each pair of Nshots. Finally, we can decide the given video is re-captured or not with the resultof the connection matrix. Experimental results show that proposed method per-forms excellent in detecting re-captured videos. The proposed method performswell even a given video is re-compressed and re-scaled. However, the proposedmethod is still weak against severe attacks. Therefore, our future work is to de-tect re-captured videos even they are re-compressed with low quality and scalingfactors.

Acknowledgments. This research project was supported by Ministry of Cul-ture, Sports and Tourism(MCST) and from Korea Copyright Commission in2011.

References

1. Motion Picture Association Of America, http://www.mpaa.org2. Lee, M.J., Kim, K.S., Lee, H.K.: Digital Cinema Watermarking for Estimating the

position of the Pirate. In: Multimedia, IEEE transactions, pp. 605–621. (2010)3. Cao,H., Kot, A.C.: Identification of recaptured photographs on LCD screens. In:

Acoustics Speech and Signal Processing (ICASSP), pp. 1790–1793. IEEE press,Texas (2010)

4. Wang, W., Farid, H.: Detecting Re-Projected Video. In: International Workshop onInformation Hiding, 72–86 (2008)

5. Lukas, J., Fridrich, J., Goljan, M.: Digital camera identification from sensor patternnoise. In: IEEE Trans. Information Forensics and Security, pp. 205–214. (2006)

6. Zhang, H.J., Kankanhalli, A., Smoliar, S.W.:Automatic partitioning of full-motionvideo. Multimedia Syst. vol. 1, pp. 10–28. (1993)

7. Chen, M., Fridrich, J., Goljan, M., Lukas, J.: Source Digital Camcorder Identifica-tion Using Sensor Photo Response Non-Uniformity. In: IThe International Societyfor Optical Engineering (SPIE), (2008)

8. Mzhqak, M.K., Kozintsev, I., Ramchandran, K.: Spatially Adaptive Statistical Mod-eling of Wavelet Image Coefficients and its Application to Denoising. In: Acoustics,Speech, and Signal Processing (ICASSP), Vol. 6, pp. 3253–3256. IEE press, Arizona,(1999)

9. Chen, M., Fridrich, J., Goljan, M., Lukas, J.: Digital Imaging Sensor Identifica-tion (Further Study). In: The International Society for Optical Engineering (SPIE),(2007)

10. Kumar, B.V.K.V., Hassebrook, L.: Performance measures for correlation filters.In: Applied Optics, Vol. 29, pp. 2997–3006. (1990)

11. Warshall, S.: A theorem on boolean matrices. In: Journal of the ACM (JACM),Volume 9, pp. 11–12. (1962).

12. The Internet Movie Database, http://www.imdb.com

Detecting Re-captured Videos using Shot-Based Photo...

Documents

Transcript of Detecting Re-captured Videos using Shot-Based Photo...