Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver...

40
1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC) Joachim Falk ([email protected]) Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Transcript of Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver...

Page 1: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

1

Praktikum: Entwicklung interaktiver eingebetteter Systeme

Filter-Labs (Designing video streaming applications in SystemC)

Joachim Falk ([email protected])

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Page 2: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

2

Running the Virtual Prototype Open the Eclipse workspace “workspace03”

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

…/workspace03

Page 3: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

3

Running the Virtual Prototype Open the project “hw”

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

hw

Page 4: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

4

Virtual Prototype Implementation of a SoC for video stream processing

Using a VideoSource transmitting video data from a webcam

to a custom filter module(s) - Connected via FIFO or

- RTL signal channels

finally arriving at the VideoSink used for display.

Friedrich-Alexander-Universität Erlangen-Nürnberg Andreas Weichslgartner and Joachim Falk

Video sink

module

SystemC modules

Video filter

module

Video source module

channel channel

Communication channels (signal or FIFO type)

Today we implement filters

here

And create adapters to interface between

abstraction levels

Page 5: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

5

Agenda Video Filter Image Colorer

Task 1) Port definition for RTL signal implementation

Task 2) Port definition for FIFO based implementation

Video Filter Grayscale Conversion

FIFO->Signal & Signal->FIFO adapters

Video Source Color Check Board

Video Filter Skin Color Detector

Video Filter Sliding Window

Video Filter Integral

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Page 6: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

6

Video Filter Image Colorer Task 1) Add RTL support to VideoFilterImageColorer

Modify VideoFilterImageColorer.hpp file in order to: - Define input and output ports called Video_in and Video_out,

respectively. The ports should be for RTL level signals carrying sc_dt::sc_uint<32> values.

- Define the clock input port called "Clock" to connect to a boolean signal sc_core::sc_signal<bool>.

- HINT: An example of boolean port definition for an input port is given. The reset input port called "Reset" that is connected to a boolean signal sc_core::sc_signal<bool>.

Compile and Run the program using the eclipse

“Build” menu

“Run” menu

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

5 VideoFilterImageColorer-RTL

5 hw VideoFilterImageColorer-RTL

Run As

Run Configurations…

Organize Favourites…

Page 7: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

7

Video Filter Image Colorer (cont’d) Task 2) Add FIFO support to VideoFilterImageColorer

Modify VideoFilterImageColorer.hpp in order to: - Define input and output ports called Video_in and Video_out,

respectively. The ports should be for SystemC FIFO channels carrying the user defined data type VideoSignal.

Compile and Run the program using the eclipse

“Build” menu

“Run” menu

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

4 VideoFilterImageColorer-FIFO

4 hw VideoFilterImageColorer-FIFO

Run As

Run Configurations…

Organize Favourites…

Page 8: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

8

Agenda Video Filter Image Colorer

Video Filter Grayscale Conversion

Task 3) Port definition for RTL signal implementation

Task 4) Process implementation of grayscale conversion

Task 5) FIFO based implementation and port definition

FIFO->Signal & Signal->FIFO adapters

Video Source Color Check Board

Video Filter Skin Color Detector

Video Filter Sliding Window

Video Filter Integral

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Page 9: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

9

Video Filter Image Grayscale

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Task 3) VideoFilterGrayscaleConversion RTL support

In VideoFilterGrayscaleConversion.hpp: - Define input and output ports called Video_in and Video_out,

respectively. The ports should be for RTL level signals carrying sc_dt::sc_uint<32> values.

- Define the clock input port called "Clock" to connect to a boolean signal sc_core::sc_signal<bool>.

- HINT: An example of boolean port definition for an input port is given. The reset input port called "Reset" that is connected to a boolean signal sc_core::sc_signal<bool>.

In VideoFilterGrayscaleConversion.cpp: - Name the clock input port

- Name the Video_in input port

- Name the Video_out output port

- HINT: The port naming for the Reset port is given as an example

Page 10: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

10

Video Filter Image Grayscale Task 4) Process implementation of grayscale conversion

In VideoFilterGrayscaleConversion.hpp: - Modify the void pixelThrd() thread in order to implement the gray scale

conversion filter.

Compile and Run the program using the eclipse

“Build” menu

“Run” menu

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

convertedPixel.red =

convertedPixel.blue =

convertedPixel.green =

0.299*originalPixel.red +

0.587*originalPixel.green +

0.114*originalPixel.blue

3 VideoFilterGrayscaleConversion-RTL

3 hw VideoFilterGrayscaleConversion-RTL

Run As

Run Configurations…

Organize Favourites…

Page 11: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

11

Video Filter Image Grayscale Task 5) VideoFilterGrayscaleConversion FIFO support

Modify VideoFilterGrayscaleConversion.hpp in order to: - Define input and output ports called Video_in and Video_out,

respectively. The ports should be for SystemC FIFO channels carrying the user defined data type VideoSignal.

In VideoFilterGrayscaleConversion.cpp: - Switch the void pixelThrd() process to use FIFO channels.

Guard your changes with

Compile and Run the program using the eclipse

“Build” menu

“Run” menu

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

#if VIDEOBARABSTRACTION_VIDEOFILTERGRAYSCALECONVERSION == VIDEOBARABSTRACTION_FIFO // your changes #else

// the old code #endif // VIDEOBARABSTRACTION_VIDEOFILTERGRAYSCALECONVERSION == VIDEOBARABSTRACTION_FIFO

2 VideoFilterGrayscaleConversion-FIFO

… …

2 hw VideoFilterGrayscaleConversion-FIFO

Run As

Run Configurations…

Organize Favourites…

Page 12: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

12

Agenda Video Filter Image Colorer

Video Filter Grayscale Conversion

FIFO->Signal & Signal->FIFO adapters

Interlude) VGA Timings

Task 6) FIFO to RTL signal adapter

Task 7) RTL signal to FIFO adapter

Video Source Color Check Board

Video Filter Skin Color Detector

Video Filter Sliding Window

Video Filter Integral

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Page 13: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

13

VGA Signals VGA synchronization signals:

Friedrich-Alexander-Universität Erlangen-Nürnberg Andreas Weichslgartner

visible area (visible_signal =1)

VGA_VBACK_PORCH

VGA_VFRONT_PORCH

VG

A_

HB

AC

K_

P

OR

CH

VG

A_

HF

RO

NT

_P

OR

CH

VG

A_

HS

YN

C

VGA_VSYNC 0

1

1 0

Name In Pixels Name In Lines

VGA_HFRONT_PORCH 16 VGA_VFRONT_PORCH 10

VGA_HBACK_PORCH 48 VGA_VBACK_PORCH 33

VGA_HVISIBLE 640 VGA_VVISIBLE 480

VGA_HSYNC 96 VGA_VSYNC 2

H-sync V

-syn

c

Page 14: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

14

Model of I/O-Bar

Friedrich-Alexander-Universität Erlangen-Nürnberg Andreas Weichslgartner

Page 15: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

15

Pixel Signals FPGA operates at a higher frequency than the camera

Only every 3rd or 4th clock cycle arrives a new pixel

A rising edge in the pixel clk signalizes a new pixel

Friedrich-Alexander-Universität Erlangen-Nürnberg Andreas Weichslgartner

0

0

1 1 1

1 0 1 0 1 0 1 0 1 0 1 0 1 0 1

0 0 0 0 0 0 1 1 1 1 1 1 Pixel Clk

Sys Clk

New pixel

0

0

Page 16: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

16

Pixel Signals Signals in the VideoBar

Hsync: indicates a new line

Vsync: indicates a new frame

Visible: indicates the visible area

Friedrich-Alexander-Universität Erlangen-Nürnberg Andreas Weichslgartner

Vs

yn

c

Hs

yn

c 0 1 1 1 0 0 0 0 0 0 1 1 1 1 1 1 Pixel Clk

0 1 1 1 0 0 0 0 0 0 1 1 1 1 1 1

0 1 1 1 0 0 0 0 0 0 1 1 1 1 1 1

0 1 1 1 0 0 0 0 0 0 1 1 1 1 1 1

0

0

0

0

1

1

1

1

1

x

y

0 1 Visible

0 1

0 1

0 1

Page 17: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

17

Agenda Video Filter Image Colorer

Video Filter Grayscale Conversion

FIFO->Signal & Signal->FIFO adapters

Interlude) VGA Timings

Task 6) FIFO to RTL signal adapter

Task 7) RTL signal to FIFO adapter

Video Source Color Check Board

Video Filter Skin Color Detector

Video Filter Sliding Window

Video Filter Integral

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Page 18: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

18

FIFO to RTL Signal Adapter Implement a FIFO to RTL signal adapter that

has a sc_core::sc_fifo as input and

has a sc_core::sc_signal as output

To Compile and Run the filter chain use the eclipse

“Build” menu

“Run” menu

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

sc_fifo<> sc_signal<> sc_signal<>

FIFO to RTL signal

converter

Video Source

Video Sink

Your Filter Module

13 VideoSource-FIFO-rest-RTL

13 hw VideoSource-FIFO-rest-RTL

Run As

Run Configurations…

Organize Favourites…

Page 19: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

19

FIFO to RTL Signal Adapter Task 6) FIFO to RTL signal adapter

In VideoAdapterFIFOToRTL.hpp: - Add required methods and definitions

In VideoAdapterFIFOToRTL.cpp: - Register a SC_THREAD process in the constructor to do the work

- Hint: Within the registered SC_THREAD process:

- Hint: For each value read from the sc_fifo, write

- Hint: one value with pixelClk true to the sc_signal and

- Hint: one value with pixelClk false.

Compile, build and run the project as shown in the previous slide

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

tmp <- Video_in

wait(<correct time to correspond to vga timings>)

tmp.pixelClk = 1;

Video_out <- tmp

wait(<correct time to correspond to vga timings>)

tmp.pixelClk = 0;

Video_out <- tmp

Page 20: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

20

RTL Signal to FIFO Adapter Implement a RTL to FIFO signal adapter that

has a sc_core::sc_signal as input and

has a sc_core::sc_fifo as output

To Compile and Run the filter chain use the eclipse

“Build” menu

“Run” menu

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

sc_signal<> sc_fifo<> sc_signal<>

Your Filter Module

Video Source

Video Sink

RTL to FIFO signal

converter

1 VideoFilter-RTL-rest-FIFO

1 hw VideoFilter-RTL-rest-FIFO

Run As

Run Configurations…

Organize Favourites…

Page 21: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

21

RTL Signal to FIFO Adapter Task 7) RTL signal to FIFO adapter

In VideoAdapterRTLToFIFO.hpp: - Add required methods and definitions

In VideoAdapterRTLToFIFO.cpp: - Register a SC_THREAD process in the constructor to do the work

- Hint: Within the registered SC_THREAD process:

- Hint: For each rising edge for pixelClk write a value to the sc_fifo

Compile, build and run the project as shown in the previous slide

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

wait(<for rising edge of Video_in.pixelClk>)

tmp <- Video_in

Video_output <- tmp

Page 22: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

22

Agenda Video Filter Image Colorer

Video Filter Grayscale Conversion

FIFO->Signal & Signal->FIFO adapters

Video Source Color Check Board

Task 8) Process implementation for FIFO and RTL case (Synthesize and test on the SPARTAN3 board)

Video Filter Skin Color Detector

Video Filter Sliding Window

Video Filter Integral

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Page 23: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

23

Check Board Pattern Generation Implement a color check board generator video source

The red color of the pixels appear in boxes with size of 16 pixels, in a check board pattern

The green color of the pixels appear in boxes with size of 32 pixels, in a check board pattern

The blue color of the pixels appear in boxes with size of 64 pixels, in a check board pattern

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Video Sink

Color check board Gen.

Page 24: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

24

Check Board Pattern Generation Task 8) Process implementation for FIFO and RTL case

In VideoSourceColorCheckBoard.hpp: - Add required methods and definitions

In VideoSourceColorCheckBoard.cpp: - In the constructor: Register a SC_THREAD process for the FIFO case

- In the constructor: Register a SC_CTHREAD process for the RTL case

- Hint: Do not forget to specify the reset signal for the SC_CTHREAD process!

- Implement the registered process to generate the color check board

- Hint: Use the following for differences between FIFO and RTL case

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

#if VIDEOBARABSTRACTION_VIDEOSOURCECOLORCHECKBOARD == VIDEOBARABSTRACTION_FIFO // Code for FIFO case #else // VIDEOBARABSTRACTION_VIDEOSOURCECOLORCHECKBOARD != VIDEOBARABSTRACTION_FIFO // Code for RTL case #endif // VIDEOBARABSTRACTION_VIDEOSOURCECOLORCHECKBOARD != VIDEOBARABSTRACTION_FIFO

Page 25: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

25

Check Board Pattern Generation Compile and Run the FIFO case using the eclipse

“Build” menu

“Run” menu

Compile and Run the RTL case using the eclipse

“Build” menu

“Run” menu

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

15 VideoSourceColorCheckBoard-RTL

15 VideoSourceColorCheckBoard-RTL

Run As

Run Configurations…

Organize Favourites…

14 VideoSourceColorCheckBoard-FIFO

14 VideoSourceColorCheckBoard-FIFO

Run As

Run Configurations…

Organize Favourites…

Page 26: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

26

Agenda Video Filter Image Colorer

Video Filter Grayscale Conversion

FIFO->Signal & Signal->FIFO adapters

Video Source Color Check Board

Video Filter Skin Color Detector

Task 9) Process registration for FIFO and RTL case

Interlude) Simple Skin Color Detection

Task 10) Process implementation for FIFO and RTL case (Synthesize and test on the XUPv2 board)

Video Filter Sliding Window

Video Filter Integral

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Page 27: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

27

Video Filter Skin Color Detector Task 9) Process registration for FIFO and RTL case

In VideoFilterSkinColorDetector.hpp: - Add required methods and definitions

In VideoFilterSkinColorDetector.cpp: - In the constructor: Register a SC_THREAD process for the FIFO case

- In the constructor: Register a SC_CTHREAD process for the RTL case

- Hint: Do not forget to specify the reset signal for the SC_CTHREAD process!

- Implement the registered process to generate the color check board

- Hint: Use the following for differences between FIFO and RTL case

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

#if VIDEOBARABSTRACTION_VIDEOFILTERSKINCOLORDETECTOR == VIDEOBARABSTRACTION_FIFO // Code for FIFO case #else // VIDEOBARABSTRACTION_VIDEOFILTERSKINCOLORDETECTOR != VIDEOBARABSTRACTION_FIFO // Code for RTL case #endif // VIDEOBARABSTRACTION_VIDEOFILTERSKINCOLORDETECTOR != VIDEOBARABSTRACTION_FIFO

Page 28: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

28

Simple Skin Color Detection

Skin color detection via threshold logic

A slightly more complex algorithm proposed by [VSA03]

V. Vezhnevets, V. Sazonov, A. Andreeva. A Survey on Pixel-Based Skin Color Detection Techniques. Graphicon, 2003. [VSA03]

}||

},,min{},,max{

|){(

BRGRGR

BGRBGR

BG RR,G,B

15

15

204095

}|),,{( maxminmaxminmaxmin bBbgGgrRrBGR

Friedrich-Alexander-Universität Erlangen-Nürnberg Andreas Weichslgartner and Joachim Falk

Page 29: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

29

Video Filter Skin Color Detector Task 10) Process implementation for FIFO and RTL case

In VideoFilterSkinColorDetector.cpp: - Implement the registered process for skin color detection

- Hint: Use the following for differences between FIFO and RTL case

- Experiment with the methods in the previous slide to find an adequate method to detect skin color

Compile and Run the FIFO/RTL case using the eclipse

“Build” menu

“Run” menu

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

#if VIDEOBARABSTRACTION_VIDEOFILTERSKINCOLORDETECTOR == VIDEOBARABSTRACTION_FIFO // Code for FIFO case #else // VIDEOBARABSTRACTION_VIDEOFILTERSKINCOLORDETECTOR != VIDEOBARABSTRACTION_FIFO // Code for RTL case #endif // VIDEOBARABSTRACTION_VIDEOFILTERSKINCOLORDETECTOR != VIDEOBARABSTRACTION_FIFO

8 VideoFilterSkinColorDetector-FIFO

9 VideoFilterSkinColorDetector-RTL

8 hw VideoFilterSkinColorDetector-FIFO

9 hw VideoFilterSkinColorDetector-RTL

Run As

Run Configurations…

Organize Favourites…

Page 30: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

30

Agenda Video Filter Image Colorer

Video Filter Grayscale Conversion

FIFO->Signal & Signal->FIFO adapters

Video Source Color Check Board

Video Filter Skin Color Detector

Video Filter Sliding Window

Interlude) Edge Detection with the Sobel Filter

Task 11) Implement edge detection by combining - horizontal Sobel filter and

- Vertical Sobel filter results

Video Filter Integral

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Page 31: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

31

Edge Detection with the Sobel Filter

101

202

101

xH

Vertical edges Horizontal edges

121

000

121

yH

Combined result of both sobel filters

Friedrich-Alexander-Universität Erlangen-Nürnberg Andreas Weichslgartner and Joachim Falk

Page 32: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

32

Video Filter Sliding Window Task 11) Implement edge detection by combining two

Sobel filters

In VideoFilterSlidingWindow.cpp: - Modify the pixelThrd to calculate the two Sobel Filters and return their

results in the red and green pixel values

- Hint: The Sobel filter needs a grayscale image as input!

Compile and Run the FIFO/RTL case using the eclipse

“Build” menu

“Run” menu

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

10 VideoFilterSlidingWindow-FIFO

11 VideoFilterSlidingWindow-RTL

10 hw VideoFilterSlidingWindow-FIFO

11 hw VideoFilterSlidingWindow-RTL

Run As

Run Configurations…

Organize Favourites…

Page 33: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

33

Agenda Video Filter Image Colorer

Video Filter Grayscale Conversion

FIFO->Signal & Signal->FIFO adapters

Video Source Color Check Board

Video Filter Skin Color Detector

Video Filter Sliding Window

Video Filter Integral

Task 12) Port definition for FIFO and RTL case

Task 13) Process registration for FIFO and RTL case

Interlude) Integral Image

Task 14) Process implementation for FIFO and RTL case (Synthesize and test on the XUPv2 board)

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Page 34: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

34

Video Filter Integral Task 12) VideoFilterIntegral port definitions

Modify VideoFilterIntegral.hpp in order to add RTL support: - Define input and output ports called Video_in and Video_out,

respectively. The ports should be for RTL level signals carrying sc_dt::sc_uint<32> values.

- Define the clock input port called "Clock" to connect to a boolean signal sc_core::sc_signal<bool>.

- Define the reset input port called “Reset" to connect to a boolean signal sc_core::sc_signal<bool>.

Modify VideoFilterIntegral.hpp to add FIFO support: - Define input and output ports called Video_in and Video_out,

respectively. The ports should be for SystemC FIFO channels carrying the user defined data type VideoSignal.

- Hint: Use the following for differences between FIFO and RTL case

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

#if VIDEOBARABSTRACTION_VIDEOFILTERINTEGRAL == VIDEOBARABSTRACTION_FIFO // Code for FIFO case #else // VIDEOBARABSTRACTION_VIDEOFILTERINTEGRAL != VIDEOBARABSTRACTION_FIFO // Code for RTL case #endif // VIDEOBARABSTRACTION_VIDEOFILTERINTEGRAL != VIDEOBARABSTRACTION_FIFO

Page 35: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

35

Video Filter Integral Task 13) Process registration for FIFO and RTL case

In VideoFilterIntegral.hpp: - Add required methods and definitions

In VideoFilterIntegral.cpp: - In the constructor: Register a SC_THREAD process for the FIFO case

- In the constructor: Register a SC_CTHREAD process for the RTL case

- Hint: Do not forget to specify the reset signal for the SC_CTHREAD process!

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Page 36: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

36

Integral Image Also referred to as “summed area table”

A pixel II(x’,y’,t) in the integral image contains

the sum of all the pixel values in I(x,y,t) which above and to the left of (x’, y’)

This means:

Friedrich-Alexander-Universität Erlangen-Nürnberg Andreas Weichslgartner and Joachim Falk

'y

y

'x

x

)t,y,x(I)t,'y,'x(II0 0

II(x’,y’,t)

(x’, y’)

Page 37: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

37

Calculating the Integral Image It is not necessary to calculate

for each single pixel

Instead, the calculation can be performed iteratively:

Friedrich-Alexander-Universität Erlangen-Nürnberg Andreas Weichslgartner and Joachim Falk

'y

y

'x

x

)t,y,x(I)t,'y,'x(II0 0

(x’, y’)

s(x,y-1,t) contains the sum of pixel values of column x (up to row y-1)

II(x-1,y,t) is the value of the Integral Image left to (x,y)

)t,'y,'x(s)t,'y,'x(II)t,'y,'x(II

)t,'y,'x(I)t,'y,'x(s)t,'y,'x(s

1

1

Page 38: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

38

Calculating the Integral Image It is not necessary to calculate

for each single pixel

Instead, the calculation can be performed iteratively:

Friedrich-Alexander-Universität Erlangen-Nürnberg Andreas Weichslgartner and Joachim Falk

'y

y

'x

x

)t,y,x(I)t,'y,'x(II0 0

(x’, y’)

s(x,y-1,t) contains the sum of pixel values of column x (up to row y-1)

II(x-1,y,t) is the value of the Integral Image left to (x,y)

)t,'y,'x(s)t,'y,'x(II)t,'y,'x(II

)t,'y,'x(I)t,'y,'x(s)t,'y,'x(s

1

1

Page 39: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

39

Video Filter Integral Task 14) Process implementation for FIFO and RTL case

In VideoFilterIntegral.hpp: - Specify an array for the storage of the sum of pixel values of column x

(up to row y-1)

- Note: In hardware the array will be implemented as a synchronous RAM. Hence, each read or write of an array element will cost one cycle.

In VideoFilterIntegral.cpp: - Implement the registered process to calculate the integral image

- Store the integral sum lower 8 bits in the green pixel value and the integral sum upper 8 bits in the blue pixel value.

- Hint: To ease debugging, you might want to use the remaining red pixel value to denote where skin color was detected.

- Note: For each processed pixel, you will need to update, that is read and write, one element of the storage array for the sum of pixel values of column x (up to row y-1). Hence, you will require at least two cycles per pixel. Therefore, parameterize your while(true) {...} loop to handle data each second cycle.

- Hint: Look into VideoFilterSlidingWindow::pixelThrd() for inspiration of this two cycle processing time challenge!

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

Page 40: Praktikum: Entwicklung interaktiver eingebetteter Systeme · 1 Praktikum: Entwicklung interaktiver eingebetteter Systeme Filter-Labs (Designing video streaming applications in SystemC)

40

Video Filter Integral Compile and Run the FIFO/RTL case using the eclipse

“Build” menu

“Run” menu

Friedrich-Alexander-Universität Erlangen-Nürnberg Joachim Falk

6 VideoFilterIntegral-FIFO

7 VideoFilterIntegral-RTL

6 hw VideoFilterIntegral-FIFO

7 hw VideoFilterIntegral-RTL

Run As

Run Configurations…

Organize Favourites…