Spatiotemporal Regularity Flow (SPREF)

Mubarak ShahComputer Vision LabSchool of Electrical Engineering & Computer ScienceUniversity of Central FloridaOrlando, FL 32765

What are good features?

Color Histograms Eigen vectors Wavelet Coefficients Edges

Spatiotemporal Surfaces of edges XY, XT, YT slices Spatial/spatiotemporal tensors

SIFT Optical Flow

SPREF

New Spatiotemporal feature for VACE Generalization of Isophotes, Optical flow,… Can be computed when gradient is zero It analyzes whole region instead of a single pixel

Applications Image and Video In-painting Object removal Video Compression Tracking, Segmentation, …

Spatiotemporal Regularity Definition: A spatiotemporal volume is regular

along the directions, in which the pixels change the least.

SPatiotemporal REgularity Flow (SPREF) 3D vector field ζ models the directions of regularity

No motion (Spatial Regularity) Depends on the regularity of a single frame

Presence of motion (Temporal Regularity) Global motion

Single regularity model

Local motion Multiple regularity models

Estimating SPREF

…gives the directions, along which the sum of the gradients is minimum:

where F is the spatiotemporal volume, and H is a regularizing filter (Gaussian)

dxdydttyx

tyxHFE

2

),,(

),,)((

The SPREF Energy Functions The energy function is modified according to

the flow type: x-y Parallel: ζ(c1'[t], c2'[t],1)

y-t Parallel: ζ(1,c2'[x], c3'[x])

x-t Parallel: ζ(c1'[y],1, c3'[y])

dxdydtfxcfxcfE tyx2

32 ][']['

dxdydtfycffycE tyx2

31 ][']['

dxdydtfftcftcE tyx2

21 ][']['

Solving for the SPREF Approximate each flow component, cm'[p],

with a 1D spline Incorporates multiple frames in the solution.

i

lim ipbpc )2(]['

• Quadratic minimization of the energy functions • Solve for the spline parameters

Solving T-SPREF Equation

The original synthetic sequence (8 frames)

x-y Parallelism: ζ(c1'[t], c2'[t],1)

y-t Parallelism: ζ(1,c2'[x], c3'[x])

There are three types of planar parallelism constraints.

x-t Parallelism: ζ(c1'[y],1, c3'[y])

The SPREF Curves

… define the actual paths, along which the GOF is regular.

}3,2,1{]['][1

micpcp

imm

T-SPREF - An Overview

Demo

x-y Parallel SPREF

y-t Parallel SPREF

y

t

x

ζ(1,c2'[x], c3'[x])x

y

x-t Parallel SPREF

y

t

x

ζ(c1'[y],1, c3'[y]) x

y

T-SPREF Results (Flower Sequence)

Oblique View Top View Side View

T-SPREF Results (Alex Sequence)

x

y

t

y

t t

x

Oblique View Top View Side View

The Affine SPREF (A-SPREF)

When the directions of regularity depend on multiple axes (zooming, rotation and etc.) Precision of T-SPREF goes down Translational flow model to Affine flow model Affine (A-)SPREF

iV t

HFtyxc

y

HFtyxc

x

HFE

2

'2

'1 *],,[)(],,[)(

][][][],,[ 131211'1 taytaxtatyxc

][][][],,[ 232221'2 taytaxtatyxc

Flow energy equation:

Comparison of T- and A- SPREF

1st row: A synthetic sequence from the Lena image.

2nd row: T-SPREF approximation to the underlying directions of regularity.

3rd row: A-SPREF approximation of the directions of regularity.

More examples

T-SPREF A-SPREF

T-SPREF A-SPREF

Comparison of T- and A- SPREF

Optical Flow Vs SPREF

SPREF carries similar but not necessarily the same information as the optical flow. SPREF captures both the spatial and temporal

regularity Optical flow only cares about motion information

in temporal direction. When motion exists, the directions of xy parallel

SPREF depend on direction of motion. If the motion is globally translational, then xy-

parallel SPREF converges to the optical flow.

Optical Flow Vs SPREF

Optical Flow is not well-defined where the spatiotemporal gradients are insignificant.

Spline-based formulation of SPREF minimizes over multiple frames.

The true optical flow usually lacks plane parallelism.

Optical Flow Vs SPREF

Applications of SPREF

Inpainting

Filling in the regions of missing data

Image Inpainting Missing regions create spatial holes Inpainting the missing region in the SPREF direction

Video Inpainting Missing regions create spatiotemporal holes Inpainting these holes require using the information

from temporal neighbors.

Image Inpainting

Video Inpainting

Requires understanding the temporal behavior of the pixels.

The temporal behavior of the undamaged pixels gives clues about the behavior of the damaged pixels

Temporal behavior Modeled explicitly by x-y Parallel SPREF

Video Inpainting

The algorithm (cont’d)1. Estimate the x-y Parallel SPREF curves using the non-

missing regions. The pixels along the SPREF curves vary smoothly

2. Fit a spline to the non-missing pixels along each flow curve.

3. Interpolate the values of the missing pixels from the splines

Results

Big Bounce (Before)

Results

Big Bounce (Flow)

Results

Big Bounce (After)

Supervised Removal of Objects from Videos

Motivation

Object removal from videos Preceding step to video inpainting Manual selection of the object from each frame is

required. Time consuming

Use x-y Parallel SPREF to decrease the amount of manual work Removal along the SPREF curves

Algorithm

Given a group of frames (GOF):1) Compute the x-y Parallel SPREF, and the

SPREF curves

2) Remove the object from the first and the last frames of the GOF

3) Remove the pixels along the curves, whose first and last pixels have been removed.

Results

Golden Eye (Final)

86% reduction in manual work!

Video Compression Using SPREF

3D Wavelet Decomposition

Problem The spatiotemporal regularity of the GOF is not

taken into account

Solution Decompose the GOF along the SPREF directions Entropy along these directions is lower:

Higher compression rate

SPREF-based Video Compression

Warping the wavelet basis along the flow curves

x-y Parallel : G(x,y,t) = (x+c1[t], y+c2[t], t)

y-t Parallel : G(x,y,t) = (x,y+c2[x],t+c3[x])

x-t Parallel : G(x,y,t) = (x+c1[y], y, t+c3[t])

Choosing the correct SPREF type The correct SPREF type is the one that

minimizes the compression cost : Di + λRi

Di: Reconstruction error

λ: Lagrange multiplier Ri: Bit cost of the bandelet and flow coefficients

Segmentation for Optimal Compression Find the segmentation of the GOF (F) into

subGOFs (Fi), such that the total compression cost is minimized:

i

ii RDRD

Fi

Oct-tree Segmentation

Recursively partition the GOF (F) into rectangular prisms (cuboids), Fi.

Compute the best flow and the compression cost for each cuboid.

Use split/merge algorithm to achieve the final segmentation. Merge the child nodes if:

i

jjii RDRD

Compression results for frames 98-105 of the Alex sequence at 1000kbps

Compression results for frames 11-18 of the Akiyo sequence at 480kbps

Compression results for frames 99-106 of the Mobile sequence at 350kbps

Compression results for frames 26-33 of the Foreman sequence at 500kbps

Compression Results

The bit-rate vs PSNR plots of (a) Alex, (b) Akiyo. Both SPREF-based compression and LIMAT framework are shown in the results.

(a) (b)

LIMAT framework, Secker and Taubman, IEEE TIP, 2004.

Compression Results (cond.)

The bit-rate vs PSNR plots of (a) Foreman, (b) Mobile. Both SPREF-based compression and LIMAT framework are shown in the results.

(a) (b)

Summary

SPREF New Spatiotemporal Feature Computes direction of regularity simultaneously in

space & Time Similar to optical flow, edge direction.. SPREF is plane parallel (xy, xt, yt) SPREF is computed using region/image

information instead of a single pixel SPREF is defined even when gradient is zero

Summary

Applications Image and Video In-painting Object Removal Video Compression Tracking Segmentation

Orkun Alatas

August 16th, 1977 - September 3rd, 2005

Publications

Orkun Alatas, Omar Javed, and Mubarak Shah, “Video Compression Using Structural Flow", International Conference on Image Processing, Genova, Italy, September 11-14, 2005.

Orkun Alatas, Omar Javed, and Mubarak Shah, “Video Compression Using Spatiotemporal Regularity Flow, IEEE Transactions on Image Processing, December 2006.

Orkun Alatas, Pingkun Yan, and Mubarak Shah, “Spatiotemporal Regularity Flow, (SPREF): Its Estimation and Applications”, IEEE Transactions on Circuit & Systems Video Technology (accepted).

Computer Vision Lab

http://www.cs.ucf.edu/~vision

shah@cs.ucf.edu

Group