Image Processing and Computer Vision with MATLAB … · 3 Applications: Image and Video Processing...
Transcript of Image Processing and Computer Vision with MATLAB … · 3 Applications: Image and Video Processing...
1© 2015 The MathWorks, Inc.
Image Processing and Computer Vision
with MATLAB and SIMULINK
By Dr Jasmina Lazić
Application Engineering, MathWorks
2
Computer
Intelligence
Robotic Vision
Non-linear SP
Multi-variable SP
Cognitive
Vision
Statistics
Geometry
Optimization
Biological Vision
Optics
Smart
Cameras
Computer
Vision
Machine
Vision
Image
Processing
Physics
Imaging
Neurobiology
Mathematics
Machine
Learning
Control
RoboticsArtificial
IntelligenceSignal Processing
Physics
Imaging
Neurobiology
Mathematics
Machine
Learning
Control
RoboticsArtificial
IntelligenceSignal Processing
Computer Vision
System Toolbox™
Neural Network Toolbox™
Simulink™
Signal Processing
Toolbox™
SimScape™
Image Processing
Toolbox™SimBiology™
Embedded
Coder™
Optimization
Toolbox™
MATLAB™
Statistics Toolbox™
SimMechanics™
Control System
Toolbox™DSP System
Toolbox™
Image Acquisition
Toolbox™
The Computer Vision Ecosystem
3
Applications: Image and Video Processing
Medical imaging
Surveillance
Robotics
Automotive safety
Consumer electronics
Geospatial computing
Machine vision
and more…
4
Agenda
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
5
http://www.mathworks.co.uk/products/image-video-processing
6
Agenda
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
• Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition Demo: MATLAB live video with Microsoft Kinect
Demo: Image Acquisition App• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
7
Image Acquisition
Acquire images and video directly into MATLAB and Simulink
Configure device properties
8
Agenda
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition Demo: Live video in MATLAB
Demo: Image Acquisition App
Demo: Live video in SIMULINK
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze Demo: Tissue segmentation
Demo: Image analysis Apps
Demo: Digital display reader
• 3D Vision
• Image Registration • Embedded Applications
9
Standard Segmentation Workflow
10
Standard Segmentation Workflow
Explore image
11
Standard Segmentation Workflow
Explore image
Enhance image
12
Standard Segmentation Workflow
Explore image
Enhance image
Segment image
13
Standard Segmentation Workflow
Explore image
Enhance image
Segment image
Analyze image
14
Demos: Explore-Enhance-Segment-Analyze
15
Image Processing Apps
NEW!NEW!
Image Viewer
Color Thresholder
Region Analyzer
16
Optical Character Recognition
Support for English, Japanese + others
Simple interface:
>> results = ocr(I)
17
Segmentation – other techniques
Watershed and Active Contours (NEW)
18
Agenda
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze Demo: Tissue segmentation
Demo: Image analysis Apps
Demo: Digital display reader
• 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
Demo: Video Mosaicking (Simulink) • Embedded Applications
19
Image Registration
2D and 3D
Align multi-model images
20
Image Registration
NEW: Non-rigid image
registration and warping
21
Demo: Video Mosaicking
22
Agenda
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
Demo: Video Mosaicking (Simulink) • Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
• Introduction
• Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
• Introduction
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
Geometry Estimation
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
23
Feature Detection, Extraction and Matching
Corner Template SURF MSEREdge
24
Point Tracker
Simple interface:
points = detectXFeatures(I);
tracker = vision.PointTracker;
initialize(tracker, points.Location, frame);
while ~finished
frame = step(videoFileReader);
[points, validity] = step(tracker, frame);
end
25
Geometry Estimation
RANSAC – Random Sampling and Consensus
Gets geometry from a minimal sample set and checks it against all matches
26
Agenda
• Introduction • Feature Detect, Extract & Match
Geometry Estimation
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
• Introduction
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition
• Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
Demo: Pedestrian Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
27
Object Detection and Tracking
28
Face Detection, People Detection and Foreground Detection
29
Training Image Labeler App
30
Demo: Tracking multiple objects with a Kalman Filter
31
Demo: Tracking multiple objects with a Kalman Filter
Kalman Filter interface:
– Configure:
– Predict:
– Update:
predictedCentroid = predict(kalmanFilter);
kalmanFilter = configureKalmanFilter( ...
'ConstantVelocity', centroid, ...
[200, 50], [100, 25], 100);
c2 = correct(kalmanFilter, centroid);
32
Bag of Words for Category Classification
1. Create Bag of Features
2. Encode Input Images
3. Repeat for all classes
4. Use machine learning algorithm
to discriminate between classes
33
Agenda
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
Demo: Multi-object Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• 3D Vision
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision Demo: Camera Calibrator App
Demo: 3D Reconstruction
Demo: Depth Mapping
• Image Registration
• Embedded Applications
34
3D Vision
-4-3-2-10123
-8
-6
-4
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
x (meters)z (meters)
y (
mete
rs)
Pyramid with dynamic programming and sub-pixel accuracy
0
5
10
15
35
NEW Apps: Camera Calibrator and Stereo Camera Calibrator
Simplified workflow estimating
camera intrinsic and extrinsic
parameters
Removes the effects of lens
distortion from an image
Automatically detects
checkerboard patterns
37
Point Cloud Processing in Matlab
As of R2015a
Computer Vision System Toolbox (R2014b/15a)
Computational Geometry in base Matlab
Shipping Example: 3-D Point Cloud Registration and Stitching
pointCloud Object for storing a 3-D point cloud
pcdenoise Remove noise from a 3-D point cloud
pcdownsample Downsample a 3-D point cloud
pcmerge Merge two 3-D point clouds
pcregrigd Register two point clouds with ICP algorithm
pctransform Rigid transform a 3-D point cloud
showPointCloud Plot 3-D point cloud
Pcread Read a 3-D point cloud from PLY file
Pcwrite Write a 3-D point cloud to PLY file
alignColorToDepth Align Kinect color image to depth image
depthToPointCloud Convert Kinect depth image to a 3-D point cloud
39
Agenda
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision Demo: Camera Calibrator App
Demo: 3D Reconstruction
Demo: Depth Mapping
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration
• Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications Demo: Ball Tracker
Demo: Quad-copter Model
40
Embedded Applications
More than 60 Image Processing
functions support code generation
Most Computer Vision functionality
supports code generation
Many features generate platform-
independent code
bwdist
bwlookup
bwmorph
bwpack
bwselect
bwtraceboundary
bwunpack
conndef
edge
fitgeotrans
fspecial
getrangefromclass
histeq
im2double
im2int16
im2single
im2uint16
im2uint8
imadjust
imbothat
imclearborder
imclose
imcomplement
imdilate
imerode
imextendedmax
imextendedmin
imfill
imfilter
imhist
imhmax
imhmin
imlincomb
imopen
imquantize
imreconstruct
imregionalmax
imregionalmin
imtophat
imwarp
iptcheckmap
iptcheckconn
label2rgb
mean2
medfilt2
multithresh
ordfilt2
padarray
rgb2ycbcr
strel
stretchlim
ycbcr2rgb
imref2d
imref3d
affine2d
projective2d
41
Demo: Ball Tracker
42
Agenda
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
• Introduction • Feature Detect, Extract & Match
• Image Acquisition • Object Detection and Tracking
• Explore-Enhance-Segment-Analyze • 3D Vision
• Image Registration • Embedded Applications
43
Explore-Enhance-Segment-Analyze
3D Vision
Apps
Feature Detection
Object Detection and Tracking
Image Registration
Image
AcquisitionImage
Viewer
Color
Thresholder
Region
Analyzer
Camera
Calibrator
http://www.mathworks.co.uk/products/image-video-processing
46© 2015 The MathWorks, Inc.