Sampling and Reconstruction - Computer...

Sampling and Reconstruction

COMP 575/770 Spring 2013

What is a pixel?

A little square of color?

What is a pixel?

A little rectangle of color?

What is a pixel?

A little blob of color?

What is a pixel?

Ans: None of the above.

A pixel is not a little square! Alvy Ray Smith, Microsoft Tech Memo, 1995

http://alvyray.com/Memos/CG/Microsoft/6_pixel.pdf



What is a pixel?

An image is a continuous 2D function 𝑓 𝑥, 𝑦 = color at (𝑥, 𝑦)

𝑥

𝑦

What is a pixel?

An image is a continuous 2D function 𝑓 𝑥, 𝑦 = color at 𝑥, 𝑦

A pixel is a point sample

– Value of the function at a single point

𝑥

𝑦

Examples of pixel sampling

A ray tracer samples an image along each ray

A digital camera samples an image at each sensor

j

i

i =

–.5

i =

3.5

j = 2.5

j = –.5

Why the misconception?

To understand, we need digital signal processing (DSP) theory

Origins in audio processing

– CDs, MP3s contain sampled sound signals

A signal is any function, typically of time

– Pressure (for sound)

– Voltage (for network communications)

Outline

Introduction

Sampling

Sampling

Computers can’t work with continuous data

– Unless there’s a closed-form representation

Evaluate the function at several points

Which points?

– At regular intervals?

– More on this later

Outline

Introduction

Sampling

Reconstruction

Reconstruction

What’s the function’s value between samples?

Reconstruction


Assume function is constant between samples?

Reconstruction


Assume function is linear between samples?

Reconstruction


Assume some higher-order polynomial?

Reconstruction

What is the best choice?

Ans: It depends.

– Which is easier?

– Which one is more accurate?

– Does the underlying function have special properties?

– What is the application?

To make an informed decision, need to understand properties of filters

Filters

A filter is a kind of function

– Could be continuous or discrete (sampled)

When applied to a signal, transforms it in some way

– Bass/treble controls on an equalizer

– Echo, reverb, distortion effects

– Image blurring

– Reconstruction

Applying a filter to a signal is called convolution

Outline

Introduction

Sampling

Reconstruction

Convolution

Convolution: An Example

1 2 7 4 3

𝑎[0] 𝑎[1] 𝑎[2] 𝑎[3] 𝑎[4]

Suppose we want to replace each element with the average of its neighbors

𝑏 𝑖 =𝑎 𝑖 − 1 + 𝑎 𝑖 + 𝑎[𝑖 + 1]

3


1/3 1/3 1/3

× × ×

1 2 7 4 3

𝑎[0] 𝑎[1] 𝑎[2] 𝑎[3] 𝑎[4]

↓

3.33

𝑏[0] 𝑏[1] 𝑏[2] 𝑏[3] 𝑏[4]


1/3 1/3 1/3

× × ×

1 2 7 4 3

𝑎[0] 𝑎[1] 𝑎[2] 𝑎[3] 𝑎[4]

↓

3.33 4.33

𝑏[0] 𝑏[1] 𝑏[2] 𝑏[3] 𝑏[4]


1/3 1/3 1/3

× × ×

1 2 7 4 3

𝑎[0] 𝑎[1] 𝑎[2] 𝑎[3] 𝑎[4]

↓

3.33 4.33 4.67

𝑏[0] 𝑏[1] 𝑏[2] 𝑏[3] 𝑏[4]


𝑓[1] 𝑓[0] 𝑓[−1]

1/3 1/3 1/3

× × ×

1 2 7 4 3

𝑎[0] 𝑎[1] 𝑎[2] 𝑎[3] 𝑎[4]

↓

3.33 4.33 4.67

𝑏[0] 𝑏[1] 𝑏[2] 𝑏[3] 𝑏[4]

Filter


𝑓[1] 𝑓[0] 𝑓[−1]

1/3 1/3 1/3

× × ×

1 2 7 4 3

𝑎[0] 𝑎[1] 𝑎[2] 𝑎[3] 𝑎[4]

↓

3.33 4.33 4.67

𝑏[0] 𝑏[1] 𝑏[2] 𝑏[3] 𝑏[4]

Filter

𝑏 𝑖 = 𝑎 𝑖 − 1 𝑓 1 + 𝑎 𝑖 𝑓 0 + 𝑎 𝑖 + 1 𝑓 −1

= 𝑎 𝑗 𝑓[𝑖 − 𝑗]

𝑖+1

𝑗=𝑖−1


1/3 1/3 1/3

× × ×

? 1 2 7 4 3

𝑎[−1] 𝑎[0] 𝑎[1] 𝑎[2] 𝑎[3] 𝑎[4]

↓

? 3.33 4.33 4.67

𝑏[0] 𝑏[1] 𝑏[2] 𝑏[3] 𝑏[4]

What happens at the edges of the signal?


1/3 1/3 1/3

× × ×

0 1 2 7 4 3

𝑎[−1] 𝑎[0] 𝑎[1] 𝑎[2] 𝑎[3] 𝑎[4]

↓

1 3.33 4.33 4.67

𝑏[0] 𝑏[1] 𝑏[2] 𝑏[3] 𝑏[4]

Could assume that signal is 0 beyond its edges


1/3 1/3 1/3

× × ×

3 1 2 7 4 3

𝑎[−1] 𝑎[0] 𝑎[1] 𝑎[2] 𝑎[3] 𝑎[4]

↓

2 3.33 4.33 4.67

𝑏[0] 𝑏[1] 𝑏[2] 𝑏[3] 𝑏[4]

Could assume the signal is periodic (𝑎[𝑖] = 𝑎[𝑖 + 5])

Convolution: Another Example

1 2 7 4 3

𝑎[0] 𝑎[1] 𝑎[2] 𝑎[3] 𝑎[4]

Suppose we want to replace each element with the following difference

𝑏 𝑖 = 𝑎 𝑖 − 𝑎[𝑖 − 1]

Rewriting as a convolution:

𝑏 𝑖 = 𝑎 𝑖 𝑓 0 + 𝑎 𝑖 − 1 𝑓[1]

1 -1

𝑓[0] 𝑓[1]

Properties of Filters

Region of support

– Area over which the filter is non-zero

Filters are normalized to sum to 1

– Makes them a weighted average, not just a weighted sum

– Blurring an image shouldn’t make it brighter/darker

Most filters are symmetric about 0

– Don’t want to give preferential treatment to any direction

Properties of Convolution

Commutative 𝑎 ⋆ 𝑏 = 𝑏 ⋆ 𝑎

Associative 𝑎 ⋆ 𝑏 ⋆ 𝑐 = 𝑎 ⋆ 𝑏 ⋆ 𝑐

Distributes over addition 𝑎 ⋆ 𝑏 + 𝑐 = 𝑎 ⋆ 𝑏 + 𝑎 ⋆ 𝑐

Scalars factor out 𝛼𝑎 ⋆ 𝑏 = 𝑎 ⋆ 𝛼𝑏 = 𝛼(𝑎 ⋆ 𝑏)

Identity: unit impulse e = […, 0, 0, 0, 1, 0, 0, 0, …] 𝑎 ⋆ 𝑒 = 𝑎

Properties of Convolution

This makes filtering a linear operation:

𝑓𝑖𝑙𝑡𝑒𝑟 𝑎 + 𝑏 = 𝑓𝑖𝑙𝑡𝑒𝑟 𝑎 + 𝑓𝑖𝑙𝑡𝑒𝑟 𝑏 𝑓𝑖𝑙𝑡𝑒𝑟 𝛼𝑎 = 𝛼𝑓𝑖𝑙𝑡𝑒𝑟 𝑎

Filtering is also shift invariant

– Delaying an audio signal doesn’t change the filter’s effect

– Moving an image doesn’t make blurring work differently

Convolution: Continuous Filters

Convolution with a discrete filter:

𝑎 ⋆ 𝑓 𝑖 = 𝑎 𝑗 𝑓[𝑖 − 𝑗]

𝑗

Convolution with a continuous filter:

𝑎 ⋆ 𝑓 𝑥 = 𝑎 𝑗 𝑓(𝑥 − 𝑗)

𝑗

Continuous filters are used for reconstruction

Examples of Filters

Box Filter Simple, often useful

Examples of Filters

Tent filter Performs linear interpolation

Examples of Filters

Gaussian filter Smooth filter, often used for blurring

Examples of Filters

Cubic B-spline filter Smooth filter

Examples of Filters

Catmull-Rom cubic filter Smooth interpolating filter

Examples of Filters

Mitchell-Netravali filter Good for image upsampling

2D Convolution

Straightforward extension of 1D convolution

Convolution with a 2D discrete filter:

Convolution with a 2D continuous filter:

𝑎 ⋆ 𝑓 𝑥, 𝑦 = 𝑎 𝑖, 𝑗 𝑓(𝑥 − 𝑖, 𝑦 − 𝑗)

𝑖,𝑗

2D Convolution Example

1 1 2 3 1 1 2 1

4 3 7 2 4 2 4 2

5 6 1 2 3 1 2 1

9 1 3 5 8 𝑓

1 6 1 0 2

𝑎 𝑏


1 1 2 3 1 1 2 1

4 3 7 2 4 × 2 4 2 = 57

5 6 1 2 3 1 2 1

9 1 3 5 8 𝑓

1 6 1 0 2

𝑎 𝑏

Example of 2D Filtering

Gaussian blur

2D Convolution Complexity

1 1 2 3 1 1 2 1

4 3 7 2 4 × 2 4 2 = 57

5 6 1 2 3 1 2 1

9 1 3 5 8 𝑓

1 6 1 0 2

𝑎 𝑏

𝑂 𝑟2 multiplications per output sample

Separable Filters

We can do better with special kinds of filters

A filter is separable if:

Separable Filters

Separable filters allow factoring:

2D filtering expressed as repeated 1D filtering


1 1 2 3 1 × 1 2 1 5

4 3 7 2 4 × 1 2 1 = 17

5 6 1 2 3 × 1 2 1 18

9 1 3 5 8 𝑓𝑥

1 6 1 0 2

𝑎 𝑏


5 1

17 × 2 = 57

18 1

𝑓𝑦

𝑎 𝑏

𝑂 𝑟 multiplications per output sample

Outline

Introduction

Sampling

Reconstruction

Convolution

Image Reconstruction

Why the misconception?

So, why do we mistake pixels for little squares?

Ans: Shoddy reconstruction

– Use a simple box filter

– With just one sample in its support j

i

i =

–.5

i =

3.5

j = 2.5

j = –.5

Reconstruction Example

Original image


Magnified image, box filter


Magnified image, cubic filter

Outline

Introduction

Sampling

Reconstruction

Convolution


Anti-Aliasing

Aliasing

Undesirable artifacts that occur when enough samples are not taken

Example: “Jaggies” on the boundaries of objects

Aliasing

Indistinguishable after sampling

Hence the term “aliasing”

Anti-Aliasing

Or, how to get rid of aliasing artifacts

Use information from multiple samples per pixel: 1. Generate multiple

samples of the image for each pixel

2. Apply a reconstruction filter

3. Evaluate the reconstructed function at the pixel

j

i

i =

–.5

i =

3.5

j = 2.5

j = –.5

Anti-Aliasing

Aliased Anti-Aliased

Anti-Aliasing

Outline

Introduction

Sampling

Reconstruction

Convolution


Anti-Aliasing

Sampling Strategies

Sampling Strategies

Final detail: Where to place the additional samples?

Can have a surprising impact on results

– Good sampling strategies give better images with fewer samples (i.e., less time!)

Sampling Strategies

Regular sampling Place samples at regular intervals

Sampling Strategies

Uniform sampling Sample positions chosen based on a uniform probability distribution

Sampling Strategies

Stratified sampling Divide into subdomains, use uniform sampling within each

BACKUP


Want to replace each sample with an average of nearby samples

Straightforward:


Convolution just replaces a simple average with a weighted average:

The sequence 𝑎[𝑗] is the (discrete) filter

Complexity of 2D Convolution

𝑂(𝑟2) Too slow for large filters!

Separable Filtering

first, convolve with this

second, convolve with this

Sampling and Reconstruction - Computer...

Documents

Transcript of Sampling and Reconstruction - Computer...