1 © 2015 The MathWorks, Inc.

Embedded Vision on FPGAs

2

Enhanced Edge Detection in MATLAB Test bench

Read Image from File

Add noise

Median Filter

Edge Detect

Video Display

Frame To

Pixel

Pixel To

Frame

Design

3

Computer Vision Ø  Feature matching, and extraction Ø  Object detection and recognition Ø  Object Tracking and motion estimation Ø  Dynamic resolution scaling Ø  Focus assessment

Video Processing Ø  Video in and out Ø  Gamma correction Ø  Color balancing Ø  Noise removal Ø  Image sharpening

Video Processing and Computer Vision

4

A Workflow for Video Image Processing

Concept Development

Algorithm Development Prototyping Architecture

design Prototyping Chip design

Image/Video Engineer

Pixel based

HW Engineer

Frame based

5

Vision HDL Toolbox, R2015a New Product

§  Customer Problems –  Prototyping video and image processing algorithms on FPGA/SoC

§  Our Solution –  Pixel-streaming algorithms for the design and implementation of vision systems on

FPGAs and ASICs –  A diverse set of frame sizes and frame rates

§  Product Detail –  Price: €5000/$5500 –  Platforms: MATLAB and Simulink –  Dependency: MATLAB for modeling, HDL Coder for code generation

6

Vision HDL Toolbox, Key Benefits

§  Modeling bit-true and cycle-accurate behavior of video image processing algorithms –  Pixel-based functions and blocks –  Optimized for HDL code generation –  Conversion between frame and pixel –  Standard and custom frame sizes and frame rates

§  Prototyping video and image processing algorithms on FPGA/SoC –  (With HDL Coder) Efficient and readable HDL code –  (With HDL Verifier) FPGA-in-the-loop testing and acceleration

7

Pixel Based Video Image Algorithms

§  Analysis & Enhancement –  Edge Detection, Median Filter

§  Conversions –  Chroma Resampling, Color-Space Converter –  Demosaic Interpolator, Gamma Corrector, Look-up Table

§  Filter –  Image Filter, Median Filter

§  Morphological Operations –  Dilation, Erosion, –  Opening, Closing

§  Statistics –  Histogram –  Image Statistics

§  I/O Interfaces –  Frame to Pixels, Pixels to

Frame, FIL versions

§  Utilities –  Pixel Control Bus Creator –  Pixel Control Bus Selector

8

A broader solution for embedded vision

Product Capabilities Vision HDL Toolbox (“VHT”)

Design and simulate image processing, video, and computer vision systems for FPGAs and ASICs

HDL Coder Provide code generation capability for the functions and blocks in Vision HDL Toolbox

HDL Verifier Provide FPGA-in-the-loop capability for Vision HDL Toolbox

Computer Vision System Toolbox (“CVST”)

Provide frame based computer vision functions and blocks as well as image and video I/O capability

Image Processing Toolbox

Provide image processing and analysis functions

9

CVST and VHT, complimentary products Scopes

Vision algorithms Ø  Feature matching, and extraction Ø  Object detection and recognition Ø  Object Tracking and motion estimation Ø  Dynamic resolution scaling Ø  Focus assessment

CVST

10

CVST and VHT, complimentary products Scopes

Camera pipeline Ø  Video in and out Ø  Gamma correction Ø  Color balancing Ø  Noise removal Ø  Image sharpening

11

CVST and VHT, complimentary products Scopes

CVST

VHT 1.0 VHT future releases

Camera Pipeline

Vision Algorithm

image image information

12

CVST and VHT, complementary products Product details

VHT CVST Image Processing Pixel based Frame based Workflow System prototyping Algorithm design

Algorithms Image filtering Color space conversion Edge detection Statistics & histogram Morphological operations

(Most of the VHT algorithms, plus) Object detection & tracking Feature extraction and matching Stereo vision Camera calibration Image registration

I/O Frame-to-pixel Pixel-to-frame FIL

File I/O Video display

Code gen HDL (with HDL Coder) C (with ML Coder or SL Coder)

13

CVST and VHT, complementary products MBD workflow for embedded vision

MBD workflow 1.  Build behavior model with CVST blocks and simulate 2.  Build prototyping model with VHT blocks and simulate 3.  Generate HDL code using HDL Coder 4.  Run HDL code on FPGA and run testbench in SL (FIL)

14

CVST and VHT, complementary products MBD workflow for embedded vision

15

Enhanced Edge Detection: Simulink Demo

16

Streaming Pixel Interface Full Frame vs Pixel Stream

full-frame source

full-frame sink

à

à

pixel-stream processing

17

Streaming Pixel Interface Full Frame vs Pixel Stream

full-frame source

full-frame sink

à

à

pixel-stream processing

18

Streaming Pixel Interface Pixel control signals

pixelcontrol bus Ø  hStart, hEnd, vStart, vEnd, and valid Ø  Consistent block interface

19

Frame To Pixels and Pixels To Frame

20

Frame To Pixels and Pixels To Frame

21

Examples: Starting Points for Your Models

22

Image Filtering Keywords: Median Filter, Image Filter, PSNR

23

Gamma Correction Example Keywords: Gamma, PSNR

24

Histogram Equalization Keywords: Histogram, Linear Equalization, External Frame Delay

0 255 0 255

25

Edge Detection and Image Overlay Keywords: Sobel, Align Video, Alpha Mix, PSNR

26

Edge Detection with Impaired Frame Keywords: Sobel, Align Video, Alpha Mix, PSNR

27

Multi-Zone Metering Keywords: ROI, Image Statistics, Mean

Source Mask Selected ROI

28

Multi-Zone Metering Keywords: ROI, Image Statistics, Mean

Source Mask Selected ROI

29

Challenge Accelerate the implementation of advanced thermal imaging filters and algorithms on FPGA hardware Solution Use MATLAB to develop, simulate, and evaluate algorithms, and use HDL Coder to implement the best algorithms on FPGAs Results §  Time from concept to field-testable prototype

reduced by 60% §  Enhancements completed in hours, not weeks §  Code reuse increased from zero to 30%

“With MATLAB and HDL Coder we are much more responsive to marketplace needs. We now embrace change, because we can take a new idea to a real-time-capable hardware prototype in just a few weeks. There is more joy in engineering, so we’ve increased job satisfaction as well as customer satisfaction.”

Nicholas Hogasten

FLIR Systems Link to user story

Raw image (left) and image after applying filter developed with HDL Coder (right).

FLIR Accelerates Development of Thermal Imaging FPGA

30

From Algorithm to FPGA Implementation

Implement Design

Map

Place & Route

Synthesis

Verification

Static Timing Analysis

Timing Simulation

Functional Simulation

Back Annotation

HDL Verifier QuestaSim

HDL Co-Simulation

HDL Coder

RTL Creation

Behavioral Simulation

MATLAB® and Simulink®

Algorithm and System Design Model Refinement for Hardware

HDL Verifier

FPGA-in-the-Loop

31

A complete solution

Frame based

Concept Development

Algorithm Development Prototyping Architecture

design Prototyping Chip design

Pixel based

MATLAB

Vision HDL Toolbox

MATLAB Coder

HDL Coder

Fixed Pt Designer HDL Verifier

Image Processing

Toolbox

Computer Vision System Toolbox

32

ROI: Customer Adoption Of Model-Based Design Time spent on FPGA implementation

§  Shorter implementation time by 48% (total project 33%) §  Reduced FPGA prototype development schedule by 47% §  Shorter design iteration cycle by 80% 1st FPGA Prototype 2nd FPGA Prototype

1st FPGA Prototype