Video Analysis Tool Box for Digital Video Forensics
BySusinda PereraDepartment of Computer Science and Engineering,University of Moratuwa,
Supervised byDr. Chathura De Silva PhD (NUS-Singapore), MEng (NTU-Singapore), BSc Eng.(Hons) (Moratuwa)Senior LecturerDepartment of Computer Science and Engineering,University of Moratuwa,
02-02-2014
Agenda
• Project Intro• Implementation
– Video Stream Analyzer– Video Stabilizer– Compressed domain video analysis
• Demo– Video Stream Analyzer– Video Stabilizer
Problem Statement
• Can we trust digital videos?– Are they real, computer generated or tampered
• Extract some wanted Information from video– Difficult due to unclearness of video
Can we trust digital videos?
Figure 1‑1 : A still from controversial video aired on Channel 4Source – YouTube http://www.youtube.com/watch?v=rjVxyG_7mTk
Can we trust digital videos?
Figure 1‑2 : A still from eagle catching kidSource – http://www.cbc.ca/news/canada/montreal/child-snatching-eagle-video-created-as-student-project-1.1268125
Unclear Videos
Solution
• Video Stream analysis tool– To ensure the authenticity and integrity
• Video Enhancement tool – To improve the visual quality
Market Analysis
• Video Stream Analyzing Tools– Elecard StreamEye– Tektronix MPEG Software Tools– MPEG-2 Transport Stream packet analyser– TSReader– MTS4EA Elementary Stream Analyzer
Screenshots from Elecard StreamEye
Some Features Of Video Stream Analyzing Tools
• Navigation and display of media stream picture-by-picture (I, P, B).• Display of the current frame.• Display of the time, type, size and number of a current frame in a stream,
decoding order and offset from the file beginning.• Display of the bit rate (declared in the sequence header) and a calculated
bit rate.• Display of detailed information about macroblocks in MPEG-1 (ISO/IEC
11172-2), MPEG 2 (ISO/IEC 13818-2), MPEG-4 (ISO/IEC 14496-2) and AVC/H.264 (ISO/IEC 14496-10) video streams.
• Information about motion vectors• Frame-accurate positioning.• Display of the stream and gathering of statistics relating to the entire file.
What is missing in existing tools
• All high end tools are commercial ones, Expensive
• Almost all tools are designed for DVB purposes• Lack of detailed analyzing features for video forensic
• Lack of textual representation
Video Forecsic /Enhancement Software
• Cognitech• Ocean Systems dTective• Salient Stills VideoFOCUS• StarWitness• Avid Technology, Inc.• Intergraph Video Analyst• TREC, Inc.• Forevid• MotionDSP Ikena• Amped FIVE• Kinesense
Features Of Video Enhancement Tools
• Video Stabilization• Denoising• Deblur Filters• Detection Filters• Enhancement• Histogram Editor• Segmentation• Tracking• Transform• Zoom• Velocity• Reconstruction
Implementation of Video Stream Analyzer
Literature Review
• Market Analysis– Most of tools are for DVB, not for forensic purposes
– Lack of textual output• Possible resources [31- 35]
– Video Player libraries / source codes– Decoder Libraries/Source Codes– MPEG2Event[33]is a possible candidate
Video Stream Analyzer
• Implementation– Based on MPEG2Event[33] library– Mpeg demuxer– ffmpeg for stream information– A simple thread sync model to get motion vectors– Quartztype library for video player implementation
– C# inbuilt xml libraries for xml manipulation
What is MPEG stream
MPEG Bit-Stream Structure (Source [1] )
GOP Structure and Display order
GOP Structure and Transmission Order. (Source: [2])
Video Stream Analyzer
https://code.google.com/p/mymscproject/source/browse/trunk/Experiments/Mpeg2Event/Downloads/VideoAnalyzerSetup.msi?r=174
Video Stream Analyzer
• Features– Free and open source– Textual output (xml) of analysis reports
• For further analysis– All most all the features in slide 10– Motion vector display– Frame navigation– Bit stream index/ Display index view– User data display– Motion vector summary – xml output– Full video summary – xml output
Video Stream Analyzer -outputs<VideoFileStrucure> <FileInfo> <FilePath>D:\vedio\sampleVideo_vs.m1v</FilePath> <Duration /> <CtreatedTime>2013-02-11T20:10:37.140625+06:00</CtreatedTime> <SizeInBytes>378904</SizeInBytes> <LastAccessed>2013-02-11T20:12:20+06:00</LastAccessed> <BitRate>777</BitRate> <VideoFormat>mpeg2video (Main)</VideoFormat> <Height>180</Height> <Width>240</Width> <VideoBitRate>777</VideoBitRate> <FrameRate>2997</FrameRate> <TotalFrames>9261</TotalFrames> <AudioBitRate>0</AudioBitRate> <AudioFormat /> <RawVideoFormat>mpeg2video (Main), yuv420p, 240x180 [SAR 1:1 DAR 4:3], 777 kb/s, 29.97 fps, 29.97 tbr, 1200k tbn, 59.94 tbc</RawVideoFormat> <RawAudioFormat /> <IsMpeg2>true</IsMpeg2> <HasAudio>false</HasAudio> </FileInfo> <StreamStructure> <SequenceHeaderX> <HSizeValue>240</HSizeValue> <VSizeValue>180</VSizeValue> <VBVBufferSize>112</VBVBufferSize> <SquareSampleProp>true</SquareSampleProp> <FrameRate>30000/1001</FrameRate> </SequenceHeaderX> <SequenceExtensionX> - <ProfileNLevelIndication> <Bits>72</Bits>
Motion Vectors Summary <Pictures> <MPicture> <PicType>I</PicType> <MBlocks> <MVector>
<Row>0</Row> <Col>0</Col> <ForwardX>0</ForwardX> <ForwardY>0</ForwardY> <BackwardX>0</BackwardX> <BackwardY>0</BackwardY> <IsForwardPresent>false</IsForwardPresent> <IsbackwardPresent>false</IsbackwardPresent> <IsSkipped>true</IsSkipped> <PicType>I</PicType>
</MVector> <MVector>
<Row>0</Row> <Col>1</Col> <ForwardX>0</ForwardX> <ForwardY>0</ForwardY> <BackwardX>0</BackwardX> <BackwardY>0</BackwardY> <IsForwardPresent>false</IsForwardPresent> <IsbackwardPresent>false</IsbackwardPresent> <IsSkipped>true</IsSkipped> <PicType>I</PicType>
</MVector> <MVector>
<Row>0</Row> <Col>2</Col> <ForwardX>0</ForwardX> <ForwardY>0</ForwardY> <BackwardX>0</BackwardX> <BackwardY>0</BackwardY> <IsForwardPresent>false</IsForwardPresent> <IsbackwardPresent>false</IsbackwardPresent> <IsSkipped>true</IsSkipped> <PicType>I</PicType>
</MVector>
MacroBlocks Summary
Implementation of Video Enhancement tool
Video Enhancement Tool
• Objective– Enhance the visual quality– Video Stabilization– Noise removal– Color Correction
Video Stabilization
• Removing annoying shaky motion from videos• helpful in identifying people, number plates, etc. from low-quality video cameras
• Three aspects– Inter frame motion estimation– Motion smoothing and compensation – Filling up the missing image areas.
• Main references [3], [4] , [5]
Video StabilizationInter Frame Motion Estimation
Result of video stabilization. Top row: Original input sequence, middle row: stabilized sequence which still has missing image areas, and bottom row: stabilized and completed sequence. The grid is overlaid for better visualization. (Source [3])
a) Natural scene, b) when camera is moved, c) when image is shifted to compensate the motion.
Handling camera motion
Motion Model• = frame at time t• = frame at time t -1• = affine transform
= T * How to find T ?– There papers ([3], [4], [5]) mentioned above use diffèrent mechanisms to find the
transform matrix parametrs
𝑇=(𝑚1 𝑚2
𝑚3 𝑚4
𝑚5
𝑚6)
Motion Estimation
• Computing inter frame motion– Use of object recognition – Scale Invariant Feature Transform(SIFT) features– Minimizing quadratic error function with a
proposed model [5]
Stabilization
Motion Smoothing
• A stabilized motion path is obtained by removing undesired motion fluctuation.
• Assumed that the intentional motion in the video is usually slow and smooth
• Uses Gaussian kernel in most literatures– Applies Gaussian kernel to neighboring N frames
• Gaussian kernel + curve fitting methods
Motion Smoothing
• Let Nt = {j|t-k<=j<=t+k} be the neighboring frames• And It is the frame at the origin• Calculate the position of each neighboring frame Is, relative to frame It using transform matrixes defined above ( lets say Ts
t)• Find the correcting transformation S from the original frame It to the motion-compensated frame I’t according to
• Where G is a Gaussian kernel of size k
Motion Smoothing
The global transformation chain T defined over the original video frames Ii, and the transformation from the original path to the smoothed path S. (Source [3])
Filling up missing image areas
• Not addressed here– Due to time limitations and complexity
• Some techniques used in research literature– Motion Inpainting [3]
• the local motion data in the known image areas is propagated into the missing image areas.
• The propagation starts at pixels on the boundary of the missing image area. Using motion values of neighboring known pixels
– Use of dynamic programming [3],[6]
Noise Removal (Denoising Filters)
• Denoising Filter Categories [7]– Nonmotion compensated spatiotemporal– Motion compensated spatiotemporal– Nonmotion compensated temporal– Motion compensated temporal filters
• What is first? motion compensation or denoising?– Each have their procs and cons– Depends on the video
• References– [7], [8], [9], [10], [11]
Implementation of Video Enhancement Tool
Implementation of Video Enhancement Tool
• Video Stabilization– Based on the algorithm described in [5]– + Improvements to reduce error propagation
• Makes a additional pass (slow)– + Improvements to support color video
• Noise Removal and color Correction– Based on openCV filters
A Compressed Domain Approach for Video Tamper Detection
Common Modes of Tampering
• Classification– Spatial, Temporal or Both (Spatio-temporal)
• Temporal– Frame insertion, deletion, modification and reordering
• Spatial– Similar, but operates on objects within a frame
– Based on MPEG's layer structure ([12])• GOP level, frame level and single- or multi-block level• GOP jittering, inter-GOP frame and intra-GOP frame jittering, or
their combinations
Tamper detection techniquesVideo
Authentication
Pasive
Pixel Domain
Pixel Statistics Based
Detecting resampling
Double MPEG compresion
Inconsistant Noise patterns
Duplication detection
Physics Based
Motion Analysis
Shadow Analysis
Compressed Domain
Active
Watermarking
Digital Signature
References – [12], [13], [14], [15], [16], [17]
Compressed Domain approaches
• Many researches can be found on detecting alterations by image processing means.
• But very few initiatives in Compressed Domain– Due to the lack of techniques to grab the frame data from compressed video
• Possibilities– Analyze stream structure headers w.r.t. its contents– Analyze userdata– Analyze inconsistencies in frame sizes, block sizes– Analyze motion vectors
Our Approach
• Based on motion vectors• Assumption
– For an unaltered video the motions present in the video need to be smooth and therefore should not contain sudden changes
– Tampering disturbs the motion flow in a video • Extract the motion information from video and compare them with the above assumption.
• Motion information is derived from the motion vectors [17], [18]
Concept of Motion Vector
• MPEG achieves it its high compression rate by the use of motion estimation and compensation
• Instead of encoding each block in the current frame, matching block is search in the past frame and if suitable block is found, the difference between the two macroblocks is encoded and transmitted
• The displacement between the two macroblocks is represented using motion vector
Concept Of Motion Vector cont..
• Suppose that the macroblock “x” is the macroblock we wish to encode and macroblock “y” is the counterpart in the past frame.
• A search is done around “y” (search area) to find the best match for “x”.• The displacement between the two macroblocks gives the motion vector associated with “x”.
Motion Vectors vs MacroBlocks
• As a part of MPEG encoding process motion vectors are calculated and inserted to the bitstream.
• In mpeg bitstream motion vectors are associated with macroblocks. But not every macroblock will contain a motion vector
• There are four basic types of macroblock types– Intra– Skipped– Forward predicted– Backward predicted
Picture Type
Possible Macroblock Types Motion Vectors
I Intra No motion vectors present
P Skipped, Forward predicted Only forward motion vectors may present
B Skipped, Forward predicted, Backward predicted
Both Forward and backward motion vectors may present
Forward and Backward predictions
Summary of motion vectors and macroblocks
Deriving Motion Field from Motion Vectors
• Motion vectors represent the motions ,But not directly (therefore we need to derive)– Steps
• Reorder the bitstream into display order– A simple algorithm
• Generate Motion felid from motion vectors– [18] defines set of rules. – Simplified mechanism and rule set is described in the [19].
• Filter out the outliers– Median filter (3*3 as in [18])
Simplified rule set for deriving motion field
1. Macroblocks with no motion vector have the same movement as in the previous image.
– This is required since there is a small percentage of intra-coded blocks in B- and P-frames that would otherwise have unknown motion. Also, I-frames have no motion at all so this rule will provide a smooth motion field for them. The justification for this is step is that consecutive images are strongly correlated therefore there is a high probability that the motion field has not changed much.
2. When a macroblock has two motion vectors, the one pointing back is reversed and added to the one pointing forward.
– This step accomplishes both noise reduction (due to the averaging effect) and normalizes vector magnitude (so they span three images in B-frames which is what happens in P-frames).
3. Motion vector magnitude is normalized. – If a macroblock in a B-frame only has one motion vector then it has to be multiplied by a
factor so its magnitude corresponds to a vector spanning three images.4. Skipped macroblocks in I-frames have no movement, while in P-frames, they have
movement similar to the previous block. – This is a rule obtained directly from the MPEG-2 standard.
Implementation of rules
Transition Motion Vectors Present
I to BP to BP to P
motion = mbF2.FwdMV
B to BB to P
If only forward vectors present: motion = mbF2.FwdMV – mbF1.FwdMVIf only backward vectors present: motion = mbF2.BwdMV –mbF1.BwdMVIf both forward and backward vectors are present:motion = ((mbF2.FwdMV – mbF1.FwdMV) + (mbF2.BwdMV –mbF1.BwdMV) )/ 2
P to I Not defined
Let’s consider a transition from frame F1 to F2, Lets define mbF1 and mbF2 be the corresponding macroblocks in F1 and F2.
Motion field• The derived motion field is considered as the motion for the frame to frame transition.
• Now the derived motion field contains the motion information for every macroblock
Motion vectors of the left image
Derived Motion field for the above motion vector field and next frame
Smoothed motion field of the above
Detecting Motion Inconsistencies
• If we consider a typical video, most of pixel data in two adjacent frames are same.
• Differences are present due to moving objects of the scene, but these differences are small in magnitude.
• Similar assumption is made in [13]• In simple terms, for three adjacent frames, motion between second and third frame is very much similar to motion between first and second frames
Mathematical definition of the assumption
• Consider three frames, and a block in row j and column i in frame is denoted by for every block in these frames the following inequalities should hold.
• Where would be the functions which output the motion in X and Y directions respectively and is a very small positive constant.
• Here the functions can be simply replaced by motion field since the motion field we derived has the exact meaning.
• •
How to detect tampering
• Get motion vectors for every frame• Derive motion fields for each transition• Check whether motion fields satisfies the above two equations, for every transition (i.e every motion field with its next)
• Based on our assumption if it does not satisfy above it is more likely to be a manipulated video.
Results
• For low resolution videos the assumption we made fails 90% of the time(success for large )
• It is observed that derived motion field itself has discontinuities and noise.
• For high resolution videos assumption satisfies but fails when there are multiple flows of motion is there.
• It is difficult to fully automate the tamper detection process, need to have manual intervention.
Summary• Video Stream Analysing tool
– Existing tools are for DVB and not for forensic purposes– Implemented using open stack of tools and libraries– Supports detailed view of bit-stream as well as textual outputs fro further processing
• Video enhancement tool– Two major issues in video footprints are unclearness and shakes– This tool helps to stabilize a video and reduce noise– Order of stabilize and denoise, has their own pros and cons, and also may depends on video
• Compressed domain approach for tamper detection– Deriving motion fields from motion vectors – Rule set– Consistency in motion Assumption and mathematical derivation– Detecting motion inconsistencies
• Showed that video bit-stream data (compressed domain) gathered by stream analyzing tool can be use as an aid for video forensic analysis
Future Work• Video Stabilization algorithm can be improved to Fill the missing image areas.
• Video stream analyzer need to support new formats of video.• Compressed domain tamper detection mechanisms need to be provided as tools and can be integrate with stream analyzer tool.
• Motion object segmentation methods can be used in combination with proposed motion field method – detect motions of individual objects and then apply our assumption – which would give better results since we apply our assumption on individual motion fields
References• [1] MPEG-2 White Paper. [Online]. http://
www.pinnaclesys.com/files/MainPage/Professional/TopTabItems/products/dc1000/WhitePapers/DC1000-DVD1000MPEG2whitepaper.pdf
• [2] MPEG Encoding basics, 1st ed.: Snell & Wilcox , 2007 [Online] http://www.media-matters.net/docs/resources/Digital%20Files/MPEG/MPEG%20Encoding%20Basics.pdf
• [3] Rongjie Shi, I-fan Shen, Wenbin Chen Rong Hu, "Video Stabilization Using Scale-Invariant Features," in Proc. of the 11th International Conference Information Visualization, 2007, pp. 871-877.
• [4] Eyal Ofek, Weina Ge, Xiaoou Tang, Heung-Yeung Shum Y. Matsushita, "Full-Frame Video Stabilization with Motion Inpainting," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 7, pp. 1150-1163, July 2006.
• [5] Jeffrey B. Woodward Hany Farid, "Video Stabilization and Enhancement," Department of Computer Science , Dartmouth College , TR2007- 605, 1997.
• [6] E. Shechtman, and M. Irani Y.Wexler, "Space-time video completion," in IEEE Conf. on Computer Vision and Pattern, vol. 1, 2004, pp. 120-127.
• [7] R. P. Kleihorst, S. Efsratiadis, A. K. Katsaggelos, and R. L. Lagendijk J. C. Brailean, "Noise reduction filters for dynamic image sequences: A review," Proceedings of the IEEE, vol. 83, pp. 1272-1292, 1995.
• [8] B. Coll, J. M. Morel A. Buades, "Denoising image sequences does not require motion estimation," Dpt. Matematiquesi Informatica, Universitat Illes Balears.
• [9] William T. Freeman 2 Ce Liu 1, "A High-Quality Video Denoising Algorithm based on Motion Estimation," 1Microsoft Research New England, 2 Massachusetts Institute of Technology
• [10] M.I.Sezan, A.M. Tekalp M.K. Ozkan, "Adaptive motion compensated filtering of noisy image sequences," IEEE Trans, circuits, vol. CSVT-3, pp. 277-294, Aug 1993.
• [11] B. Coll, J. M. Morel A. Buades, "A non-local algorithm for image denoising," IEEE International Conference on Computer Vision and Pattern Recognition, 2005
• [12] Hong Heather Yu Peng yj, "Classification Of Video Tampering Methods and Countermeasures Using Digital Watermarking," SPIE, Multimedia Systems and Applications, vol. IV, pp. 239-246.
• [13] Wang Weihong, "Digital Video Forensics," DARTMOUTH COLLEGE, Hanover, New Hampshire, Thesis 2009.
• [14] Farid H Wang W, "Exposing Digital Forgeries in Video by Detecting Double MPEG Compression," ACM Multimedia and security workshop, 2006.
• [15] S.K. Singh S. Upadhyay, "Video Authentication: Issues and Challenges," IJCSI International Journal of Computer Science Issues, Vol. 9, Issue 1, No 3, January 2012.
• [16] Farid H Wang W, "Exposing Digital Forgeries in Interlaced and De-Interlaced Video," EEE Transactions on Information Forensics and Security, pp. 428-449, 2007.
• [17] Farid H Wang W, "Exposing digital forgeries in video by detecting duplication," ACM Multimedia and security workshop, 2007.
• [18] M.T.Coimbra and M. Davies, "Approximating optical flow within the MPEG-2 Compressed Domain," IEEE Trans. Circuit Syst. Video Tech., vol. 15, no. 1, pp. 103-107, Jan 2005.
• [19] Joseph Gilvarry, "Extraction of Motion Vectors from an MPEG Stream," School of Electronic Engineering, Dublin City University, 1999.
• [20] MPEG2Event Class Documentation. [Online]. http://www.cs.unc.edu/~kmp/mpeg2event/classdocs/index.html
• [21] mpegdemux. [Online]. http://www.hampa.ch/mpegdemux/• [22] Information technology – Generic coding of moving pictures and associated audio information, ITU-T, Standard Recommendation ITU-T H.222.0, 2012.
• [23] Video coding for low bit rate communication , ITU-T, Standard ITU-T Recommendation H.263, 2005.
• [24] SERIES H: AUDIOVISUAL AND MULTIMEDIA SYSTEMS - Advanced video coding for generic audiovisual services," ITU-T, Recommendation ITU-T H.264, 2013.
• [25] SalientStills VFPro 4. [Online]. http://www.salientstills.com/products/videofocus/index.html
• [26] Tektronix. [Online]. http://www.tek.com/mpeg-test-video/mpeg-software• [27] Elecard Streameye Studio. [Online].
http://www.elecard.comwww.elecard.com/en/products/professional/analysis/streameye-studio.html
• [28] MediaInfo. [Online]. http://mediainfo.sourceforge.net/en• Sliq Media’s WMSnoop. [Online]. http://www.sliq.com/default.asp?view=wmsnoop• [29] MPEG-2 Transport Stream packet analyzer. [Online].
http://www.pjdaniel.org.uk/mpeg/• [30] StreamXpert TS Analysis Software. [Online].
http://www.dektec.com/products/Apps/DTC-320/• [31] Libmpeg2. [Online]. http://libmpeg2.sourceforge.net/ • [32] FFMpeg. [Online]. http://ffmpeg.org/• [33] MPEG2Event. [Online].
http://wwwx.cs.unc.edu/~kmp/mpeg2event/blosxom.cgi/overview.html• [34] ooMPEG. [Online]. http://www.cs.brown.edu/software/ooMPEG/• [35] MPEG-1/2 DirectShow Decoder Filter. [Online].
http://gplmpgdec.sourceforge.net/
Top Related