Automatic Interlace or Progressive Video Discrimination...• Telecine process is used to convert...

Automatic Interlace or

Progressive Video

Discrimination

Manish Pindoria

BBC

© 2012 SMPTE · The 2012 Annual Technical Conference & Exhibition · www.smpte2012.org

Overview

• Introduction

– Why is knowing the type of video important?

• The algorithm

– How it works

– Experimental Results

• Archive restoration of telecine phase errors

– How are they caused, and how to detect and correct them?


Introduction

• Why do we need to know if the video is interlace or

progressive?

– Can’t it just be set at the beginning of the video clip?

• Programme makers will use a variety of technologies to

create their programme – Film, Video, CGI, etc.

• The edited content is likely to contain a mix of

technologies, often changing on a scene basis


What is Interlace/Progressive?

• Interlace

Frame 0Frame 0 Frame 1Frame 1 Frame 2Frame 2 Frame 3Frame 3 Frame 4Frame 4 Frame 5Frame 5

Field 0 – Top FieldField 0 – Top Field Field 1 – Bottom FieldField 1 – Bottom Field Field 2 – Top FieldField 2 – Top Field Field 3 – Bottom FieldField 3 – Bottom Field Field 4 – Top FieldField 4 – Top Field Field 5 – Bottom FieldField 5 – Bottom Field

Video Frame 0Video Frame 0 Video Frame 1Video Frame 1 Video Frame 2Video Frame 2

50 Frames per Second 50 Frames per Second

50 Fields per Second 50 Fields per Second



What is Interlace/Progressive?

• Progressive

Frame 0Frame 0 Frame 1Frame 1 Frame 2Frame 2 Frame 3Frame 3 Frame 4Frame 4 Frame 5Frame 5

Film Frame 0Film Frame 0 Film Frame 1Film Frame 1 Film Frame 2Film Frame 2




Interlacing Artefacts

• With motion, interlacing artefacts appear

Interlace Frame

Line Crawl

Serration/Combing


The Algorithm – Basic Principle

Even Field Frame 0 = FieldEVEN0 Odd Field Frame 0 = FieldODD0 Even Field Frame 1 = FieldEVEN1 Odd Field Frame 1 = FieldODD1

Frame 1Frame 0

FieldEVEN0 & FieldODD0 FieldEVEN1 & FieldODD0 FieldEVEN1 & FieldODD1

t0 t1 t2 t3

• Interlace Video


The Algorithm – Basic Principle

• Progressive Video

Frame 1Frame 0



t0 t1 t2 t3


Interlacing Metric – Per Pixel

• Measuring the amount of interlacing for every pixel

• Very simple pixel metric – Susceptible to noise and high frequency, but the metric will be equally affected in the

following frames, therefore this is not an issue

– Of course, the metric can be interchanged with something more sophisticated if

required

bca

yxmetric

2

),(

a

b

c


Interlacing Metric – Motion

• What happens to the metric when there is only a small

amount of motion in a noisy video?

– Bias on edges?

• Weight the metric on the difference image

– Difference image is the normalised absolute difference of two re-

constructed frames

imagedifferencenormalisedD

imagedifferenceD

IinvaluepixelMinimumMIN

IinvaluepixelMaximumMAX

MINMAX

MINyxDyxD

IimageinyxDpixeleachfor

I

I

II

I

'

),(),('

),(


Interlacing Metric – Motion

difference Normalised difference

Frame 1Frame 0


Interlacing Metric – Per Frame

• The final metric is calculate as follows:

• This provide a single metric for each re-constructed frame

),('),( yxDyxmetricmetric

width

x

height

y


Analysing the video

• Progressive video should present a highly oscillating metric, with

peaks on the “in-between” frames

• Oscillations for Interlace video should be largely absent

Frame 1Frame 0



t0 t1 t2 t3


Frame 1Frame 0


t0 t1 t2 t3


Metric data for the clips

0

2000000

4000000

6000000

8000000

10000000

12000000

14000000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Interlace and Progressive Video Metric Plot

Interlace

Progressive

• Can we separate these two traces


Frequency data for the clips

• Can the DFT help?

0

1000000

2000000

3000000

4000000

5000000

6000000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Interlace and Progressive Video Metric Frequency Analysis

Interlace

Progressive


Making the Decision

• Thresholding of metric or frequency components – Not picture independent

• Ratio thresholding more robust

• Ratio of the magnitudes fs/2 and DC frequency components can be

used to separate the interlace and progressive

• This value can be thresholded, as this is a normalised measure

2/fs

DC

MAG

MAGratio


Computing the frequency components

• Using a DFT

• As we only require 2 spectral components (k=0, N/2), more efficient

techniques exist – Goertzel Algorithm

– Sliding DFT

)()()1()(/2

nxNnxenSnSNkj

kk


Analysis in scenes

• Analysis of scenes is advantageous and also necessary for this

algorithm

• Scene changes prevent this algorithm from working reliably

Frame 3

Frame 5

Frame 6

Frame 4

0

2000000

4000000

6000000

8000000

10000000

12000000

14000000

0 1 2 3 4 5 6 7 8

Frame


Partitioning into scenes

• A new scene segmentation was designed for this and future archive

work

• Robustness is key

• The algorithm is based on combining multiple metrics over multiple frames to identify abrupt cuts only, but it could be extended to detect

other transitions

• Metrics include histograms, phase correlation, absolute differences,

etc.

• Pattern match is currently used, but machine learning techniques could be used in future versions


Results of the Interlace/Progressive Algorithm

• This algorithm has produced some good results when used in practice

• It was tested on a variety of programmes taken from the BBC archive,

in an HD format (1920x1080), which contained a variety of video types – News, documentary, action, drama, chat show, concerts, etc.

• Each clips was partitioned into scenes, and each scene was classified

into interlace or progressive – Scenes that are static, variable telecine, or a mixture of interlace and progressive were

not evaluated in this study


Results of the Interlace/Progressive Algorithm

• ~30000 frames were analysed, 486 scenes in total.

• The algorithm is assessed using precision and recall

• Precision and recall for the interlace and progressive frames are

complimentary

• Ratio used for thresholding has to picked to balance the precision and recall

for both the progressive and interlace scene

frames ve)/progressi(interlace identifiedFalsely ofNumber

frames ve)/progressi(interlace identified Missed ofNumber

frames ve)/progressi(interlace identifiedCorrectly ofNumber

false

missed

correct

lsecorrect+fa

correct=ecision , Pr

ssedcorrect+mi

correct= Recall


Results of Interlace/Progressive Algorithm

• ~30000 frames were analysed, 486 scenes in total

• Each ratio was assessed for each valid re-constructed frame

• Most missed detection with the progressive sequences were due to static

scenes or only a small amount of motion

• Interlace missed detection possibly due to compression of noisy images

In terlaced fram es correct 28758 Progressive fram es correct 31720

Interlaced fram es m issed 90 Progressive fram es m issed 562

Interlaced fram es fa lse 562 Progressive fram es fa lse 90

In terlaced fram es recall 0 .996 Progressive fram es recall 0 .982

In terlaced fram es prec is ion 0.980 Progressive fram es prec is ion 0.997


Telecine Analysis

• Telecine process is used to convert film to a tape format

Film Frame 0 Film Frame 1 Film Frame 2

Field 0 Field 1 Field 0 Field 1 Field 0 Field 1

Video Frame 0 Video Frame 1 Video Frame 2

Film Frame 3

Field 0 Field 0

Video Frame 3


Telecine Analysis – Phase Error

• Older telecine machines difficult to control, leading to a phase error and a

pseudo-interlace output


Field 0Field 1


Film Frame 3

Field 0Field 1 Field 0Field 1 Field 1Field 0


Telecine Analysis – Phase Error

• Not good news

• Processing techniques, involving motion analysis or segmentation are

significantly more difficult with the pseudo-interlace material, such as

restoration or format conversion

• Unnecessary processing such as de-interlacing is also required

• Compression is also more complicated than it needs to be as compressing interlace material is more difficult and requires more

bandwidth


Telecine Analysis – Video

• Telecine phase error material requires more bandwidth to transmit for

the same quality phase corrected material.


Telecine Analysis – Detecting a Phase Error

• The Interlace/Progressive detection scheme can be extended to

automatically detect phase errors

• Consider the case of correctly transferred telecine material

Field 0 Field 1 Field 0 Field 1 Field 0 Field 1 Field 0 Field 0

Met

ric

Val

ue


Telecine Analysis – Detecting a Phase Error

• Now consider the case of incorrectly transferred telecine material

• The same results are produced, only positioned differently

Field 0 Field 1 Field 0 Field 1 Field 0 Field 1 Field 0 Field 0

Met

ric

Val

ue


Telecine Analysis

• Telecine phase error can be detected by considering the phase of the

metric signal

• Only a small adjustment is required from the original algorithm

• For a typical progressive metric signal:

– When calculating the frequency components when centred on an peak, the fs/2

components should be negative

– When on a trough, this component should be positive.


Telecine Analysis

• For correctly aligned telecine material – All the negative values should be aligned to “in-between” frames

– All positive values should be aligned to “input” frames

• For incorrectly aligned telecine material – All the negative values should be aligned to “input” frames

– All positive values should be aligned to “in-between” frames

• This can be summarised by taking note of the position of the peak

relative to the input and in-between reconstructed frames


Telecine Analysis

• On a per scene basis, averageratio1,2 are calculated

– If these values are equal, the material is correctly phased progressive material

– If the values are equal in magnitude, but opposite signs, the material is likely to be

incorrectly phased telecine material

frameinbetweenanforismetricif

frameinputanforismetricifsign

scenetheinmetricsofNumberM

signX

X

Mioaveragerat

X

X

Mioaveragerat

M

m

DC

m

f

m

M

m

DC

m

f

m

s

s

1

1

1

1

0

2/

2

0

2/

1


Telecine Analysis – Recreating the Original

• If the material is correctly identified as telecine phase error, the

restoration process is trivial – Simply re-merge the fields correctly

– No additional interpolation is required for most of the video

– Extra care is needed only at the beginning and end of scenes

Original Frames

Corrected Frames


Thank You

• Any Questions?

The authors are solely responsible for the content of this technical presentation. The technical presentation does not necessarily reflect the official position of the Society of Motion Picture and Television Engineers (SMPTE), and its printing and distribution does not constitute an endorsement of views which may be expressed. This technical presentation is subject to a formal peer-review process by the SMPTE Board of Editors, upon completion of the conference. Citation of this work should state that it is a SMPTE meeting paper. EXAMPLE: Author's Last Name, Initials. 2011. Title of Presentation, Meeting name and location.: SMPTE. For information about securing permission to reprint or reproduce a technical presentation, please contact SMPTE at [email protected] or 914-761-1100 (3 Barker Ave., White Plains, NY 10601).

SMPTE Meeting Presentation

Automatic Interlace or Progressive Video Discrimination

Manish Pindoria BBC, UK, [email protected]

Tim Borer, SMIEEE, MIET BBC, UK, [email protected]

Written for presentation at the Annual SMPTE Technical Conference

Abstract. Video content originates from a wide variety of sources. Even within one programme, several different video technologies may have been used during production. This paper discusses an algorithm that is able to reliably identify progressive and interlace frames. The algorithm is based on calculating a metric based on the degree of “interlacing artefacts” produced when adjacent fields from different frames are re-interleaved to reform a frame. The metrics are analysed over multiple frames to detect whether the material originates from a progressive or interlace source. This process has successfully been adapted to correct film-phase errors found in telecined archive material.

Keywords. Film-mode detector, telecine

Copyright © 2012 Society of Motion Picture and Television Engineers. All rights reserved.

2

1. Introduction In a television production shoot, a variety of different camera technologies may be used, for example, indoor scenes may be produced using interlaced video cameras, whereas outdoor scenes may be shot using (progressive) film cameras. Therefore the complete programme is likely to contain a mix of interlace and progressive scenes. Hence, the video type cannot be reliably determined for the whole programme, each scene within the video (or ideally each frame) must be analysed independently to assess the type of video. In some cases there may be interlace video overlay on top of progressive video, for example, caption titles may be overlaid on progressive video footage. Similarly, progressive titles on interlace video may also be observed, with the former case being more common.

This paper describes an algorithm that is able to reliably distinguish between interlaced or progressive video, with a recall/precision of 0.982/0.997 for progressive video and 0.996/0.980 for interlace video when frames are assessed individually. Perfect recall/precision is achieved when data is averaged and analysed on a per scene basis. The method described in this paper involves deconstructing the input frames into fields and then re-combining adjacent fields to produce twice the number of original frames. This process essentially attempts to introduce interlacing artefacts in progressive data rather than simply looking for interlacing artefacts in interlaced material. An interlace artefact measure is then calculated for each of these synthesised frames which are analysed to detect the type of video.

Correct identification of progressive and interlace video type is required to correctly guide various image processing algorithms such as motion analysis and automated restoration. This algorithm can also be extended to detect material with a telecine phase error, in which film frame phase is offset from the video frame phase by one field, causing an interlace like output rather the expected progressive output. Correction of this error is simple once detected and results in an increase in picture quality and also eases complexity of further processing.

2. Related Work The subject of automated interlace or progressive detection seems to have very limited prior publications, which was unexpected considering that correct identification of interlace or progressive material is crucial for the correct functionality for many algorithms, for example, with compression schemes such as H.264/AVC [1] .

The algorithm presented in [2] is one of the few published method found that attempts to automatically distinguish between interlace and progressive video. The algorithm is based on detecting interlaced frames by measuring the amount of curvature in the image. In general, interlaced frames are expected to have a higher curvature than progressive frames. To speed up processing time, edges are detected using a Canny edge detector, and only these positions are analysed for curvature. A threshold is used to classify the frames as progressive or interlace. This algorithm assesses content in “bites” of video, typically in 80-160 frame length, a recall of 0.9875 is achieved for the progressive material and 0.9958 is achieved for the interlace material.

Reference [3] describes an algorithm for detecting the presence of de-interlacing in scaled video, while reference [4] describes an algorithm to detect de-interlacing artefacts. An area of study that is popular, which shares similarities with the problem at hand is the detection of 3:2 (or other variation) pull-down used for telecine such as [5] [6]. The repeat fields make the 3:2 pull-down detection significantly simpler than the task presented in this paper, however the former is complicated with the presence of noise or compression effects. None of these references except [2] address the problem at hand.

There also seems to be very little published work regarding telecine phase errors. A white paper published by Cambridge Imaging Systems [9] describes the problem of telecine phase errors in great detail and its consequences on subsequent image compression. However, no method is provided for the detection for such errors.


3

3. Algorithm Description

3.1 Interlacing Artefacts

For moving interlace video, if two consecutive fields are merged to produce a frame very noticeable artefacts are visible, typically referred to as serration and line crawl artefacts [2]. These artefacts are also commonly known as combing or line twitter. Typical interlacing artefacts are shown in Figure 1. Due to the motion, serration patterns are observed at any points there is horizontal motion, and line crawl artefacts are observed for any vertical movement between fields.

Interlace Frame Line CrawlSerration Figure 1: Interlacing Artefacts

3.2 Basic Principle

Noting that fields in an interlace scan are captured at regular time intervals can be exploited to accurately determine the presence of interlaced or progressive scan video. Consider the interlaced sequence of the left hand side of Figure 2, showing two HD frames.


Frame 1Frame 0


t0 t1 t2 t3

Frame 1Frame 0



t0 t1 t2 t3

Figure 2: Left: Interlaced Frames, Right: Progressive Frames

In this example, the two interlaced frames have been separated into their respective fields, i.e. four (half height) fields positioned in time order. The fields are then reconstructed into (full height) frames by re-interleaving each field (t) with the preceding field (t-1). The reconstructed frames all comprise of fields which are ½ a frame period apart, and therefore, if there is any motion present, interlacing artefacts will be present in all of the reconstructed frames. Also, as the motion is expected to be similar across fields, the “amount” of interlacing artefacts will also be similar across the reconstructed frames. As seen by the three lower reconstructed frames, all the images contain interlacing artefacts of the same order of magnitude.

Now consider the case of the progressive sequence of the right hand side of Figure 2. The progressive input is treated as above and the frames are split into (pseudo) fields and re-constructed as complete frames. As each pair of fields in a frame originates from the same instant, when the correct pairs are re-constructed, there are no visible interlacing artefacts. This is observed in the first and third reconstructed frames which correspond to the original frames 0 and 1. However, when fields from different frames are recombined, as with the second (in-between) reconstructed frame, there are very noticeable interlacing artefacts wherever there is motion because the fields are sampled at different times. So, with progressive frames, the amount of interlacing will oscillate from no interlacing artefacts to a large amount of interlacing from frame to frame. This pattern forms the basis of the algorithm that is proposed in this paper.


4

3.3. Quantifying the Amount of Interlacing

Combing artefacts can be seen when a pixel is significantly different to the pixels directly above and below, whilst the pixels above and below are similar. So the presence of interlacing artefacts may be detected by summing the differences between them. If b is the pixel under evaluation at position (x, y), and a and c are the pixels directly above and below b, the amount of interlacing can be measured very simply using the following equation:

bcayxpelDiff −+

=2

),( 1

This equation is evaluated for every pixel in the frame, and the difference values accumulated to form a frame based metric. Edge repeat is used at the top and bottom of the picture as required. A large value for the metric may indicate that there are large number of interlace artefacts present, or that there is a lot of high vertical frequency content in the image. The metric value is susceptible to noise and a high value may be observed for noisy pictures, however, as the subsequent frames will also be noisy, the overall effect of noise should have little impact on the final decision.

Reference [2] weighted their metric by only calculating curvature for edge pixels. There are two potential problems with this approach, the first being that interlacing is also commonly observed in textures. Secondly, weighting the metric on edges alone will bias the measurement in the presence of limited motion. An alternate approach is to bias the measure dependent on the amount of motion between frames. The amount of motion can be measured using the absolute difference between two frames that surround the current reconstructed frame under evaluation. In order to emphasise the differences, the difference images are normalised to utilise the image depth and bias the image on areas which have the most difference. The normalisation process is indicated below.

imagedifferencenormalisedD

imagedifferenceDMINMAXMINyxDyxD

IinvaluepixelMinimumMINIinvaluepixelMaximumMAXIimageinyxDpixeleachfor

II

I

I

I

=

=−

−=

=

=

'

),(),('

:),( 2

The normalised difference image and the artefact metric are multiplied on a per pixel basis to produce the final metric. As discussed earlier, the metrics for interlace frames will not significantly change between reconstructed frames. However, when progressive frames are analysed, there should be a very strong oscillation between a frame with relatively low metrics, corresponding to an original input frame, and a relatively high metric for reconstructed frame, where the fields have originated at different instants. The left hand image of Figure 3 shows a sample of metrics formed from an interlaced and a progressive scene. A very strong oscillation is observed for the progressive scene, the frequency of which is half the sampling rate of the metric (i.e. at fs/2).

0

2000000

4000000

6000000

8000000

10000000

12000000

14000000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Interlace and Progressive Video Metric Plot

Interlace

Progressive

0

1000000

2000000

3000000

4000000

5000000

6000000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Interlace and Progressive Video Metric Frequency Analysis

Interlace

Progressive

Figure 3: Left: Interlace/Progressive Metrics, Right: DFT Spectrum of Interlace/Progressive Metrics

For the interlace video, oscillation is largely absent, as the reconstructed frames contain a similar amount of interlacing artefacts. As seen from the plot, there is some small oscillation, but there is no obvious harmonic present, and crucially, no obvious harmonic at fs/2. Analysis of the frequency spectrum for the metric signals is the basis of the algorithm presented in this paper. The Discrete Fourier Transform (DFT) is used to convert the time based signal into its constituent frequency components, represented in the


3

right hand graph of Figure 3. The magnitude of each frequency component is plotted. The component at position 1 is DC, while the component at position 17 is fs/2. The components 2-16 represent the frequencies present between these two frequencies, i.e. fs/16, fs/8, 3fs/16 etc., the mirror frequencies have been omitted for clarity. As seen in Figure 3, the magnitude of the fs/2 frequency component is very close in magnitude compared to the DC component for the progressive clip. In contrast, for the output from the DFT for the interlace sequence, the magnitude of the fs/2 frequency component is significantly smaller than the DC component.

Using the results presented above, comparison of the fs/2 frequency magnitude (MAGfs/2) with the DC component (MAGDC) can be used as a general method of separating progressive and interlace video content. In general, equation 3 can be used as the discriminating factor. The ratio is calculated as 0.951 and 0.003 for the progressive and interlace clips respectively for the data presented above. The difference observed between these two metrics is easily separable, and the progressive and interlace clips can be classified.

2/fs

DC

MAGMAGratio = 3

3.5. Ratio Calculation

A ratio value is required for every frame, therefore a sliding window of metric values is used for the frequency analysis. Value mirroring is used for the edges where values are not available for the window. The Discrete Fourier Transform (DFT) was used to represent the frequency components for the example shown above. However, as this algorithm currently only requires the DC and fs/2 frequency components, the full DFT is inefficient as many outputs are not required. An alternative to the DFT more suited for this application is the Goertzel Algorithm [7]. While the DFT computes evenly across the bandwidth of the input sequence, the Goertzel algorithm looks at specific predetermined frequencies, i.e. DC and fs/2 for this purpose. An alternate, more efficient strategy is the Sliding DFT algorithm [8]. In this process, a slightly different formulation is used (equation 4), where Sk(n) is the new spectral component and Sk(n-1) is the previous spectral component and N is the size of the window.

)()()1()( /2 nxNnxenSnS Nkjkk +−−−= − π 4

Once the ratio of the DC and fs/2 components has been calculated, a threshold is used to determine whether or not the reconstructed frame has originated from progressive or interlace material. The threshold was set to 0.15 empirically.

As stated earlier, analysing video content after scene segmentation is advantageous and crucial for robust classification of video type. Abrupt scene changes, which make up a large percentage of all scene transition in normal video, will cause a large single peak in the interlacing metric value with a reconstructed frame that is comprised of fields from different scenes, i.e. either side of the cut boundary. The large value due to the scene change is dominant and the metric waveform will become very similar to a delta function. A spectral analysis will reveal a spread of energy evenly across all frequency bands. The scene change will obfuscate the frequency spectrum, making it virtually impossible to reliably detect the underlying interlace or progressive material. To avoid this scenario, it is proposed that the video sequences should be segmented into scenes prior to analysis. A specially designed scene segmentation algorithm, based on combining multiple frame based metrics was used for this task.

4. Telecine Phase Errors In order to convert film into a tape format, a telecine machine is typically used. In a telecine process, a film frame is spilt into two fields, in which each field contains alternate lines from the source film frame. When the fields are re-combined into video frames, the resulting images should be identical to the original film frames. In the case of 2:2 pull-down, the video frames are in phase with the film frames, as shown in the left hand image of Figure 4.


4


Field 0 Field 1 Field 0 Field 1 Field 0 Field 1


Film Frame 3

Field 0 Field 0

Video Frame 3


Field 0Field 1


Film Frame 3

Field 0Field 1 Field 0Field 1 Field 1Field 0

Figure 4: Left: Telecine transfer in phase, Right: Telecine transfer with phase error

The system shown on the left is the ideal scenario from a perfectly calibrated telecine machine. However, early telecine machine lacked advanced control or may have not been configured properly by the operator. As a result, the output video frame may consist of fields from adjacent film frames, rather than the same film frame. This is illustrated in right hand diagram of Figure 4. The incorrect phasing has given rise to a pseudo-interlace video output. In original CRT based workflows, this error was of little consequence, as the output was field based and therefore would have no affect on playback, but with modern frame based systems this fault is significant. If undetected, unnecessary de-interlacing is required and also an increase in bit-rate for compression, which both result in a loss of picture quality. In addition, restoration of the original film material is significantly harder as the artefacts are more difficult to track with the interlacing, making typical film artefacts more difficult to correct.

If the phasing error can be accurately identified, the error can be easily corrected by re-merging the correct fields from the adjacent frames. Extra care is needed at the beginning and end of the scene to ensure that fields are not incorrectly combined with fields from a different scene, and also the same number of frames is maintained.

4.1 Extending the Interlace/Progressive Algorithm

A simple extension to the algorithm discussed above can be applied to robustly detect telecine phase errors. Consider a scenario in which the algorithm is analysing progressive material that is correctly aligned. When analysing progressive footage, the metric value should be low if the analysis is centred on an input frame (which contain no interlacing artefacts) and large when centred on an in-between frame (which contain a large amount of interlacing artefacts).

When a sequence that has a telecine phase error (TPE) is analysed using the algorithm an unexpected output is observed. During the analysis, the in-between reconstructed frames in the TPE source will re-form the original source film frames. Therefore when these frames are analysed for interlacing artefacts, the resultant metric value should be small. However, when the input video frames are tested, the interlacing artefact metric will be very high. This is the complete reverse of what is expected for normal progressive data. The TPE can therefore be detected by analysis of the phase of the metric signal. Both the fs/2 frequency component and the DC component will be real values (no complex component), however, when the metric analysis is centred on a peak, the fs/2 frequency component will be negative, and when centred on a trough, the value will be positive. For correctly aligned telecine material, all in-between frame ratios will be positive, while all input frame ratios will be negative. However, for incorrectly aligned telecine material, all input frame ratios will be positive, while the in-between frame ratios will be negative.

The fs/2 frequency component (Xfs/2) and DC component (XDC) for a particular metric (m) are used to form two ratios. The first ratio uses the absolute values of both components and sums the result of the division across all metrics in the scene (M). This forms the averageratio1, which can be used to indicate the presence of progressive or interlace material with no further processing. The second ratio is used to


5

detect the presence of telecine phase error. As stated, when the metric value is positioned on a peak, the fs/2 frequency component will be negative and positive when positioned on a trough. Therefore with correctly aligned telecine material, all negative values are aligned to “in-between” reconstructed frames, while all positive values are signed to “input” reconstructed frames, and the opposite is observed for telecine phase error material.

When calculating the second ratio, for each metric, the value of the fs/2 frequency component is divided by the DC component, and then multiplied by -1 if the metric value corresponds to an “in-between” frame and 1 if the metric corresponds to an “input” frame. This will result in a positive value for the ratio for correctly aligned material, however the ratio will be negative for incorrectly phased material. Summing the ratio over all the metrics produces the final averageratio2. The equations for the formation of the averageratio1,2 are shown below:

⎩⎨⎧

−=

=

⎟⎟⎠

⎞⎜⎜⎝

⎛×=

=

∑

∑

=

=

frameinbetweenanforismetricifframeinputanforismetricif

sign

scenetheinmetricsofNumberM

signXX

Mioaveragerat

XX

Mioaveragerat

M

mDCm

fm

M

mDCm

fm

s

s

11

1

1

0

2/

2

0

2/

1

5

If the metric data was oscillating at the Nyquist frequency, indicating progressive material, the magnitudes of ratios would be the same. For in-phase telecine material, averageratio1 would be positive. If however, averageratio2 was negative, and still equal in magnitude to averageratio1 then it is quite likely that the material has a telecine phase error. In essence, this test is checking the phase of the metric trace, and checking whether the peaks are aligned with the input video or the in-between frame. Comparing the magnitude of the two ratios is required to ensure that the footage is consistently out of phase. Once the material can be flagged as containing a telecine phase error, the error can be corrected using the process described earlier.

5. Interlace and Progressive Detection Results The algorithm was tested on various HD (1920x1080) content, with a mixture of both interlace and progressive content. A breakdown of the test content is provided in Table 1:

File Genre Frames Scenes Scene Type

British Music Concert/News 3990 54 51P,3I Dr Who Drama/Action 1372 40 39P,1I Grand Events Nature, Sea, Land 4547 37 26P,2I, 2M,6T Half Moon Drama 4394 34 34P Jonathan Ross Talk Show 4128 71 70I,1M Maharajas Motor Documentary 4428 51 49P,2I After You’ve Gone Comedy 4795 47 1P,46I Later w Jools Holland Concert 2479 66 5P,61I Electric Proms Concert 3215 86 3P,82I,1M TOTAL: 33348 486

Table 1: Test Material (P=Progressive, I=Interlace, M=Mixed frames (both progressive & interlace), T=Telecine)

Each sequence was partitioned into scenes and then manually assessed to determine whether it was interlaced or progressive. No assessment was made on individual frames within a scene as this process would be far too time consuming and it was therefore assumed that a scene was the same video type throughout. Scenes which contain a mixture of interlaced and progressive footage (e.g. captioning titles, telecine (excluding 2:2), etc.) were not included in the evaluation as the video cannot be clearly classed


6

as interlace or progressive. Additionally, scenes which contain no motion (therefore could be interlace or progressive) were also omitted for the purposes of evaluating this algorithm. If the scene contained N frames, 2N-2 reconstructed frames and ratios are available for analysis. An average ratio was also formed for the scene and classified. The performance of the algorithm was assessed using recall and precision:

frames ve)/progressi(interlace identifiedFalsely ofNumber frames ve)/progressi(interlace identified Missed ofNumber

frames ve)/progressi(interlace identifiedCorrectly ofNumber

=

=

=

falsemissedcorrect

lsecorrect+facorrect=ecision , Pr

ssedcorrect+micorrect= Recall

6

As there are only two possible states, interlace or progressive, the recall for the interlace frames is complimentary to the precision of the progressive frames, i.e. the greater the number of correctly identified interlace frames, the smaller the number of falsely identified progressive frames (interlace frames falsely classified as progressive). Therefore the choice of the threshold has to be balanced so that a good result is achieved for the recall and precision for both the interlaced and progressive frames. Using a threshold value of 0.15 and a window size of 32 values, the results shown in Table 2 were achieved. By averaging the ratio and thresholding on a per scene basis, a perfect recall and precision was obtained for the 266 interlaced and 196 progressive scenes.

Interlaced frames correct 28758 Progressive frames correct 31720 Interlaced frames missed 90 Progressive frames missed 562 Interlaced frames false 562 Progressive frames false 90 Interlaced frames recall 0.996 Progressive frames recall 0.982 Interlaced frames precision 0.980 Progressive frames precision 0.997

Table 2: Test Results

6. Telecine Phase Error Results Telecine phase errors are a rare occurrence with current video, however, they may frequently occur in older archive footage. Doctor Who – Ambassadors of Death (1971) was known to contain several telecine phase errors and therefore this was an ideal sequence to test the algorithm.

The source material was PAL standard definition material of ~25 minutes in length, equating to 37937 video frames of data, where each frame is 720 pixels and 576 lines in resolution. The sequence was first processed using the scene segmentation algorithm, a total of 355 scenes were correctly detected with no obvious false detections. Each scene was analysed using the modified interlace or progressive algorithm described earlier. In this analysis, 225 of the scenes were classed as interlace, while 130 of these were classed as progressive. Of the progressive sequences, 113 were classed as incorrectly phased, while the remaining 17 were classed as progressive and correctly phased.

A visual analysis of the scenes classed as interlace was made and all were correctly classified. After a visual analysis of the the progressive scenes classed as out-of-phase, all the scenes were confirmed to originate from telecine material and were also out of phase. In these cases the video can be corrected by re-merging the correct fields, resulting in the high quality film frmaes to be be restored, as shown in Figure 5. Of the 17 progressively marked scenes, only three sequences that were identified as being in the right phase were truly progressive, i.e. telecine material that is the correct phase. In this case the footage was unaltered. The other 14 scenes were falsely identified as progressive due to a missed cut. All of the missed cuts in this video were caused by rapid cutting between different shots, typically of 1-2 frames in length, which were difficult to track reliably.


7

Original Frames

Corrected Frames

Original Frames

Corrected Frames

Figure 5: Left: Phase Correction, Right: Compression of corrected and original footage

7. Discussion The results in Table 2 show that the algorithm presented in this paper is very reliable for discriminating interlace and progressive material, especially when average results are considered on a per scene basis. However, there are some occurrences of missed detection. A large majority of the missed progressive frames observed were a result of material with having only a very small amount of motion, therefore only a small amount of interlacing artefacts are produced when fields from different frames are interleaved. This results in small oscillations for the frame metrics which are difficult to track reliably, leading to the frames being classed as interlace. There are also a small number of interlaced frames misclassed as progressive. It is thought that the interlacing artefacts in the original video frames may have been softened due to low-level compression as a result of the noisy picture.

Removal of telecine phase errors can provide a significant increase in picture quality and also ease additional requirements for further processing. For example, film restoration and other de-noising algorithms are more successful if the material is correctly adjusted. In particular, material with a telecine phase error would be much harder to compress due to the high frequency interlacing artefacts. Therefore, for the same bit-rate, a significantly better picture is observed when the phase error is corrected, as demonstrated in the right hand image of Figure 5.

8. Conclusion This paper has demonstrated that robust frame based interlace or progressive identification is possible using a novel approach. The approach described has shown to produce very reliable results for the detection of interlace material. A recall/precision for the interlace frames was 0.996/0.980 and for progressive was 0.982/0.997. When averaged on a scene basis, a perfect recall/precision was achieved. Spectral analysis of the frame based metric was demonstrated to be a robust method of classifying interlace and progressive video and there are efficient, low computational complexity methods of obtaining only the required spectral components.

9. References [1] Sullivan, G. J., Topiwala, P., Luthra, A., ‘The H.264/AVC Advanced Video Coding Standard: Overview and Introduction to the Fidelity Range Extensions’, SPIE Conference on Applications of Digital Processing, Special Session on Emerging Standard: H264/AVC, August 2004.

[2] Keller, S., Pedersen, S., Lauze, F., ‘Detecting Interlaced or Progressive Source of Video’, IEEE 7th Workshop on Multimedia Signal Processing, October 2005.

[3] Znamenski, D., Cordes, C. N., ‘Detection of De-interlacing for Scaled Video’, International Conference on Consumer Electronics, Digest of Technical Papers, January 2008.


8

[4] Gao, F., ‘Local Detection of De-interlacing Artifacts’, Graduation Symposium, Eindhoven University of Technology, Netherlands, August 2009.

[5] Schutten, R. J., De. Haan, G., ‘Real-Time 3:2 Pull-Down Elimination Applying Motion Compensation/Estimation in a Programmable Device’, IEEE Trans on Consumer Electronics, Vol. 44, No. 3. p930-938, August 1998.

[6] Ku, C. C., Liang, R. K., ‘Robust Layered Film Mode Source 3:2 Pull-down Detection/Correction’, IEEE Trans. on Consumer Electronics, Vol. 50, No. 4. p1190-1193, November 2004.

[7] Goertzel. G., ‘An algorithm for the evaluation of finite trigonometric series’, American Math. Monthly., vol. 65, p. 34-35, 1958.

[8] Jacobson, E., Lyons, R., ‘Sliding DFT’, DSP Tricks and Tips, IEEE Signal Processing Magazine, p.74-80, March 2003.

[9] McConkey, P., ‘Telecine Field Dominance’, Cambridge Imaging Systems, Draft 1.0, January 2007.


Automatic Interlace or Progressive Video Discrimination...• Telecine process is used to convert...

Documents

Transcript of Automatic Interlace or Progressive Video Discrimination...• Telecine process is used to convert...