"Digital Gimbal: Rock-steady Video Stabilization without Extra Weight!," a Presentation from...
-
Upload
embedded-vision-alliance -
Category
Technology
-
view
142 -
download
0
Transcript of "Digital Gimbal: Rock-steady Video Stabilization without Extra Weight!," a Presentation from...
Copyright © 2016 FotoNation 1
DIGITAL GIMBAL Rock-Steady Video Stabilization Without Extra Weight!
Dr. Petronel Bigioi May 3, 2016
Copyright © 2016 FotoNation 2
IMAGING
Wide Expertise
Strong Innovation
Top Quality
SOFTWARE
Always Real Time
Platform Agnostic
Full Flexibility
HARDWARE
Proven
Low Power
High Performance
QA
Advanced Data Acquisition
Multi-million Image Database
Testing Automation Tools
FotoNation in a Nutshell
Copyright © 2016 FotoNation 3
FotoNation in Numbers
550+ 1000+ 60% 2.5+ 450+
imaging patents man years of experience high end smartphone penetration billion units shipped million devices / year
understand ● enhance ● accelerate
Copyright © 2016 FotoNation 4
DIGITAL GIMBAL
A state-of-the-art, energy-efficient image stabilization solution
for high frequency vibration by FotoNation
Copyright © 2016 FotoNation 5
Problem Statement
High quality videos 4K @ 60 FPS are accepted
as norm in cameras on the fly and move
Low frequency motion is not the only problem
anymore
HIGH and VERY HIGH frequency motion due to
motors and activities need to be addressed
including motions with horizon lock
Copyright © 2016 FotoNation 6
Mechanical Gimbal Solution — Current State-of-the-Art to
Attenuate Motors Vibration Effects
ADVANTAGES
DISADVANTAGES
WHAT IS IT? • Mechanical assembly built to counteract vibration effects
• Uses gyros in close loop to control motors
• Have large range of movement
• Cost!!!
• Large & inconvenient
• Heavy
• Consume power
• Reduce flight time due to extra weight
• Slow reaction time
Copyright © 2016 FotoNation 7
Digital Gimbal Solution — FotoNation’s Technology to
Deal with Vibrations (Low and High Frequency)
ADVANTAGES
DISADVANTAGES
WHAT IS IT?
• Algorithms removing the need for mechanical assembly; uses gyro to measure high
frequency vibration and digital image processing for frame to frame registration to reverse
the effects in the image/frames
• Low cost
• No extra weight
• Low power
• Ultra fast reaction time
• Advanced image stabilisation
• Rolling shutter correction
• Limited range of movement
Copyright © 2016 FotoNation 8
Image Stabilization Requirements
Image stabilization has always been a subject of great interest especially when associated with camera modules on the move; today’s solutions are influenced by system requirements (cost) as well as jitter specific problems (usage).
Performance System requirements
good sampling quality
stable frame rate
small error estimations
low power consumption
future proof (2K, 4K, 8K)
roll (x-y rotation)
panning
out of plane rotations
rolling shutter distortions
large moving objects
Copyright © 2016 FotoNation 9
The Rolling Shutter Problem
Hand jitter vibration patterns Drone/motorbike jitter vibration patterns
Rolling shutter effects — caused by slow
camera motion during line by line exposure
and frame readout
Rolling shutter effects — caused by rapid
camera motion during line by line exposure
and frame readout
Have low frequency (up to 15Hz) Have high frequency (hundreds of Hz)
Copyright © 2016 FotoNation 10
ISP
Inertial
Sensor
Motion
Filtering
Camera
projection
model
Motion
Correction
Image
Corrected
Image
Correction
Grid
Motion
Estimation
Image
Sensor
Legend:
Memory Buffer
Digital Gimbal Components
HW Module App Processor & DDR
256 independent motion vectors; IMU samples
acquisition and sync; sensor fusion
(IMU + frame to frame)
Corrects lens distortion, roll, pan & complex
rolling shutter in 1 pass
Adaptive motion filtering & rolling
shutter effects estimation
System Architecture and Workflow
Copyright © 2016 FotoNation 11
Motion Estimation — Synchronization
• Accurate synchronization of inertial and video data is essential
• Hardware core used to read inertial data (IMU) and assign precise timestamps,
with same time source for frames and IMU samples
• Two possible implementations
• Pure HW implementation — ensures minimum latency and highest timing
accuracy, low power consumption
• Hybrid implementation — more flexible, but higher power. Requires high
sampling frequency to minimize motion measurement errors
Copyright © 2016 FotoNation 12
SOF
Interrupt SOF
Interrupt Sample with
FSYNC marker
Sample with
FSYNC marker
unknown time
periods
Maximum unknown period is equal to IMU sample interval. Very inaccurate for low sampling frequencies
Hybrid Synchronization Timing Accuracy Limit
Copyright © 2016 FotoNation 13
Bias — a constant value added to the measured signal
Motion Filtering — IMU calibration
Copyright © 2016 FotoNation 14
Sampled IMU data needs to be filtered (4x rule) in order to avoid aliasing
Motion Filtering — Data Filtering
Copyright © 2016 FotoNation 15
Use sensor fusion to track camera
orientation over time
Using gyroscope alone to track camera
orientation leads to error accumulation
Acceleration and magnetic field lack short-term accuracy but provide good long-term reference frame
Gyroscope provides short-
term accurate changes in
the camera orientation
Motion Filtering — Sensor Fusion
Copyright © 2016 FotoNation 16
• The real camera orientation is tracked using quaternion arithmetic
• The required correction is the difference between the real and the estimated
trajectory
• Our motion filters adapts to the camera motion in order to produce naturally
smooth experience and react fast to the intentional motion
Motion Filtering
Copyright © 2016 FotoNation 17
• Stabilisation margin is limited by the sensor size. Sudden camera movements can cause
optimal correction to exceed available correction margin. Excessive motion needs to be
limited to the available correction margin
• Limits imposed on 2D projection must control 3D rotation represented as 4D quaternion —
difficult task. Problem gets even more complicated for highly distorted rectilinear lenses
(left) or fisheye lenses (right)
Motion Filtering — Motion Limitation
Copyright © 2016 FotoNation 18
tim
e
reference line
• Orientation of the camera in the first line is the
reference point
• Each consecutive line is transformed to counteract
camera movement up to the given line
• Inertial samples and the video frames must be
precisely synchronized
• Camera projection must be accurately modeled
Motion Correction — Rolling Shutter Correction
Copyright © 2016 FotoNation 19
Motion Correction — Rolling Shutter Correction
Copyright © 2016 FotoNation 20
Motion Correction — Rolling Shutter Correction
Copyright © 2016 FotoNation 21
Camera frame
reference
vector
Gravity vector
from
accelerometer
Correction
amount
After correction
vectors are
aligned
The horizon locking feature keeps the horizon line in the middle of the
frame and horizontal regardless of the drone orientation.
Motion Correction — Horizon Locking Option
Copyright © 2016 FotoNation 22
Motion Correction With “Follow Me” Feature /
Owner Tracking
Copyright © 2016 FotoNation 23
FUTURE
PROOF
FAST TO
DEPLOY EFFICIENTLY
DESIGNED
In a seamless way that is simultaneously
A DEDICATED
HARDWARE — IPU
Low gate count implementation
to offload heavy image and video
computation. Most difficult tasks
completed locally and then passing
processed data to host CPU
How Do We Deliver This?
Copyright © 2016 FotoNation 24
OBJECT DETECTION ENGINE
~ 1M gates, 240 kB Sram, 54 mW
Multi-Core CPU
GPU DSP ISP
3G/4G Baseband
Memory & I/Os IPU
Video Encoder/Decoder
MOTION PROCESSING ENGINE
~ 125K gates, 9.8 kB Sram, 2 mW
DISTORTION CORRECTION ENGINE
~ 390K gates, 48 kB Sram, 18 mW
4K@60FPS, 28nm technology
Typical AP
IPU Components Overview
Copyright © 2016 FotoNation 25
• Precise lens modeling allows for lens distortion correction and re-projection.
• Any type of correction is possible (rectilinear to perspective, fisheye to
perspective, distorted fisheye to perfect fisheye, fisheye to cylindrical, arbitrary
input projection to arbitrary output projection and freeform image warping).
• All distortion corrections are done at the same time with stabilisation and high
frequency rolling shutter correction (single pass).
• Adding distortion correction has no impact on system performance.
• Frames are resampled only once to ensure maximum image quality.
• Lowest possible power consumption (18mW for 4k@60fps correction in 28n
technology).
Motion Correction — Uses FotoNation’s DCE
(Distortion Correction Engine)
Copyright © 2016 FotoNation 26
DCE GPU
Designed to work with
rectangular texture mapping
for 2D image transformations
Native support for
bicubic resampling
Scalable with increasing input
size and frame rata
Small gate count
Low power consumption
Low bandwidth thanks to
specific cache design
Designed to support generic
triangular texture mapping
for 3D scenes rendering
Bi-cubic resampling requires
extra processing power &
time
Designed to work at display
resolution
Large gate count
High power consumption
High bandwith
Motion Correction — Done DCE (Distortion Correction
Engine) Part of FotoNation’s IPU (Image Processing Unit)
Copyright © 2016 FotoNation 27
INPUT OUTPUT
Correction — DCE example
Corrected by FotoNation DCE
Copyright © 2016 FotoNation 28
Correction — DCE example
INPUT OUTPUT
Corrected by FotoNation DCE
Copyright © 2016 FotoNation 29
FotoNation EIS / High Frequency Stabilization
Copyright © 2016 FotoNation 30
• Digital Gimbal components (synchronization, motion filtering and
correction) are part of FotoNation’s IPU (Image Processing Unit) for
optimum low power high performance implementation.
• IPU incorporates additional units to enable more features such as owner
recognition and tracking combined with image stabilization (e.g. high
performance object detection and tracking, high performance face
detection and tracking and face recognition).
Conclusion
Copyright © 2016 FotoNation 31
Thank You
Copyright © 2016 FotoNation 32
FotoNation EIS / A No Trade-off User Experience
Copyright © 2016 FotoNation 33
FotoNation EIS / A No Trade-off User Experience