CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... ·...
Transcript of CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... ·...
![Page 1: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/1.jpg)
www.luxoft.com
ENGINEERING
SOFTWARE
TECHNOLOGY AUTOMOTIVE
TRAVEL AND AVIATION
ENERGY TELECOM
FINANCIAL SERVICES
ADAS COMPUTER VISION AND AUGMENTED REALITY SOLUTION
Sergii Bykov, Technical Lead
![Page 2: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/2.jpg)
www.luxoft.com
Product Vision
![Page 3: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/3.jpg)
www.luxoft.com
Road To Autonomous Driving
Source: Intel
![Page 4: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/4.jpg)
www.luxoft.com
Representation For The Driver
Output is an extendable metadata which
describes all the augmented objects/hints
and supports natural features ON/OFF
![Page 5: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/5.jpg)
www.luxoft.com
Computer Vision and Augmented Reality Applications
COMPUTER VISION
Pattern Recognition
Image Processing
Physics
Signal Processing
Arfificial Intelligence
Maths
City Driving Pattern
Augmented Navigation
Help in low visibility mode
Signs recognition Infographics
Next Gen of Adaptive Cruise
Active Park Search
![Page 6: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/6.jpg)
www.luxoft.com
Key Challenges Of Bringing AR In The Vehicle
• Usability – augmented reality subsystem should not disturb driver as it is continuously observed
• Hardware limitations – computational, power consumption, zero latency (HUD)
• Requirements for precise environmental model estimation for occlusion avoiding
• Dependency on inaccurate map and navigation data
• Distributed HW architectures, platform flexibility requirements
• High precision absolute and relative positioning requirements
• Components synchronization and latency avoidance
• Embedded memory usage limitations, different memory models
• Algorithms should be both configurable and efficient
• Specific rendering requirements, not covered by general purpose frameworks
• Variety of inputs under different platforms
• Out-of-vehicle simulation (does not support natural simulation like classical navigation)
![Page 7: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/7.jpg)
www.luxoft.com
Framework Concept
![Page 8: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/8.jpg)
www.luxoft.com
Augmented Navigation Structure
We offer a Unique Solution capable to create Augmented, mixed visual Reality for drivers and passengers based
on Computer Vision, vehicle sensors, map data, telematics, navigation guidance using Data Fusion technique.
CVNAR Solution Automotive Cameras
Sensors/CAN
Navigation System/Map Data
Vehicle displays Projection on wind shield
Smart Glasses VR devices
Telematics/V2X
![Page 9: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/9.jpg)
www.luxoft.com
CVNAR Features
Natural Augmented Reality
• Basic vehicle data
• Lanes and road boundaries
• Road signs and cautions
• Navigation data and hints
• Facades highlights
• Parking places
• Narrow street infographics
• Street names and complex junctions boards
• POI and OEM specific information
Highlights
• CPU, GPU;
• OS: Linux, QNX
• HW: Intel, NVidia, TI, Renesas
• Extrapolation engines for latency avoidance
• Machine learning and deep learning
Road scene recognition and objects tracking
• Road boundaries
• Lane detection
• Vehicle detection and tracking
• Distance and time to collision estimation
• Pedestrian detection and tracking
• Facade recognition and texture extraction
• Road signs recognition
• Parking slot recognition
Positioning
• Precise relative and absolute positioning
• Flexible data fusion and smooth map matching
• Automotive constrained SLAM
Integration and Fusion
• Sensor data
• External positioner data (optional)
• External recognition engine integration
• Pupil tracking
• Telematics: V2I and V2V
![Page 10: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/10.jpg)
www.luxoft.com
Hardware Approach: Computer Vision Box
Live data from vehicle:
- CAN data, Sensors
- Video stream CVB
CVNAR Engine
CVB Data Layer
Web Interface
SW Update
Configuration
Diagnostic
Video Stream with
augmented objects ADAS CVNAR
Engine
HUD/LCD
Head Unit
• Quick-install demonstration solution
• Platform for CVNAR (allows to be portable)
• Integration with Head Units
• Integration with vehicle networks
• Using of own sensors if needed
Navigation data,
preprocessed
sensor data, etc.
Control/Settings
![Page 11: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/11.jpg)
www.luxoft.com
Hardware Approach: Automotive Stack Own scalable and robust Automotive Stack aimed to minimize time of project start and integration
• RTOS (OSEK, Micrium, mTRON), Microcontrollers (Renesas RH850/V850)
• Hardware Abstraction Layer (HAL), Operation System Abstraction Layer (OSAL)
• Trace Server/Client, WatchDog, IPC, Drivers (SPI, I2C, UART, Timers, etc.), SW Update
• Pre-Integrated Vector CAN/Diagnostic Stack
• Vehicle Bootloader for Renesas Microcontrollers
Domains and areas:
• System development and integration with Automotive Networks and ECUs
• Drivers development and peripheral support: Video Cameras, Automotive Sensors, external HW
• HW brings-up (BSP development)
Automotive grade technologies supported by team:
• Networks: CAN, LIN, MOST, BroadReach, Ethernet
• RTOS: OSEK, mTron, Micrium, embOS, mTRON, QNX, Linux, VxWorks
• Microcontrollers: Renesas (RH850, V850), Freescale (Bolero, i.MX6), TI (OMAP, Jacinto, MSP430)
• CAN Stacks: Vector, KPIT, own tinyCAN
• Audio/Video processing
• Media Bus: LVDS, FPDLink-III, APIX2, USB, Ethernet, BroadReach
![Page 12: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/12.jpg)
www.luxoft.com
Perception Concept
![Page 13: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/13.jpg)
www.luxoft.com
Sensor Fusion: Data Inference
Optimal fusion filter parameters adjustment problem statement and solution developed to fit different car models
with different chassis geometries and steering wheel models/parameters.
Features:
• Absolute and relative positioning
• Dead reckoning
• Fusion with available automotive grade sensors – GPS, steering wheel, steering wheel rate, wheels sensors
• Fusion with navigation data
• Rear movements support
• Complex steering wheel models identification. Ability to integrate with provided models
• GPS errors correction
• Stability and robustness against complex conditions – tunnels, urban canyons
![Page 14: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/14.jpg)
www.luxoft.com
Sensor Fusion: Advanced Augmented Objects Positioning
Solving map accuracy problems
Placing:
• Road model
• Vehicles detection
• Map data
Position clarification:
•Camera motion model: •Video-based gyroscope
•Positioner Component
•Road model
•Objects tracking
![Page 15: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/15.jpg)
www.luxoft.com
Sensors Fusion: Comparing Solutions
Update frequency ~15 Hz (+extrapolation with any fps)
Update frequency ~4-5 Hz
Reference solution Our solution
![Page 16: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/16.jpg)
www.luxoft.com
Lane Detection: Adaptability and Confidence
![Page 17: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/17.jpg)
www.luxoft.com
Lane Detection: 3D-scene Recognition Pipeline
• Low level invariant features
• Single camera
• Stereo data
• Point clouds
• Structural analysis
• Probabilistic models
• Real-world features
• Physical objects
• 3D scene reconstruction
• Road situation
• 3D space scene fusion (different sensors input)
• Backward knowledge propagation from high levels
![Page 18: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/18.jpg)
www.luxoft.com
Lane Detection: Additional Information
• Features data base
• Low level screen (3D) features to refine position
• Points clouds
• Marking details, Road borders
• High level structural elements and real world objects
• Junctions, Facades, Signs, etc.
• Features Collection
• Existing map providers
• Real-time feature extraction and understanding from video sensors
• Satellite-view photos analysis
• Map database updates
• Routes offline processing and upload
![Page 19: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/19.jpg)
www.luxoft.com
Lane Detection: HD Map Potential
Content
• Simplified and advanced geometry for roads, traffic lanes,
lanes boundaries etc.
Application
• Precise on-road vehicle positioning
• Different weather, traffic situations
• Map matching and Path planning
• Maneuver suggestions
• Cable navigation
• Junction assistance
• Possible junction maneuvers
• Traffic lights position
Useful information:
• Road border geometry and type
• Traffic signs (position and type);
• Traffic lights (position and type);
• Type and quality of roadbed;
• Roadside POIs (gas station, store, café etc.);
• Any other additional features which can be useful for
vehicle positioning or driver.
![Page 20: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/20.jpg)
www.luxoft.com
Lane Detection: Robustness in Normal Conditions
Graphs show error in meters between recognized lanes (curved model) and recognized road marking (distance to detected
features + features accuracy) in different distance range and for different road conditions.
Figure 1. Regular weather, highway, slightly curved road
with lane changes Figure 2. Rainy weather, highway, straight road with lane
changes
![Page 21: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/21.jpg)
www.luxoft.com
Lane Detection: Robustness in Difficult Conditions
Figure 3. Bright sun, highway with secondary roads,
straight road with turns and lane changes Figure 4. Hard rain, highway, straight road with lane
changes
![Page 22: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/22.jpg)
www.luxoft.com
Vehicle Detection
Figure – Vehicle detection examples
• Convolutional neural network for vehicle detection
• GPU Acceleration – CUDA
• Running real-time on NVidia Jetson TK1
• Inference speedup on embedded (TK1) GPU vs CPU is ~3x
• Training speedup on desktop GPU vs CPU is ~20x
• Classifier accuracy (about 50k, 960x540, ~55-60 deg HFOV):
• Positive: 99.65%
• Negative: 99.82%
• Size of detection down to 30 pix, detection range of about 60 m
![Page 23: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/23.jpg)
www.luxoft.com
Road Scene Semantic Segmentation
Figure – Road scene segmentation examples
• Deep fully convolutional neural network for semantic
pixel-wise segmentation
• Road scene understanding use cases: model
appearance, shape, spatial-relationship between classes
• Inference speedup GPU vs CPU is ~3x
![Page 24: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/24.jpg)
www.luxoft.com
Driver Monitoring: Haar/LBP Cascades
• Object Detection using Haar or LBP feature-based cascade
classifiers is an effective method for face detection
• It is a machine learning based approach where a cascade function
is trained from a lot of positive and negative images. It is then used
to detect objects in other images.
• Applications can achieve consistent face detection in different
environments (e.g. vehicle driver, etc.)
• Can be efficiently implemented for low power embedded devices
• Speedups on mobile GPU vs CPU are
• 20x for Haar
• 3x for LBP
Figure – Haar features
Figure – LBP features
![Page 25: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/25.jpg)
www.luxoft.com
HMI Concept
![Page 26: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/26.jpg)
www.luxoft.com
Rendering Component Structure
CVNAR Rendering
Renderer
System
Window
System
Resource
Management
Frame Buffer
Frame Renderer
HMI Renderer
Augmentation Renderer
Client of
rendering
framework
(controller)
HMI/scene
update
commands
rendered
frames
Head Up
Display
LCD
![Page 27: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/27.jpg)
www.luxoft.com
Renderer Output Layers
Frame renderer
Augmentation renderer
HMI renderer
Final image
![Page 28: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/28.jpg)
www.luxoft.com
Augmented Objects Primitives. Part 1
Barrier Lane Line
Traffic Sign & POI Lane Arrow
![Page 29: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/29.jpg)
www.luxoft.com
Augmented Objects Primitives. Part 2
Fishbone Facade
Vehicle Street Name
![Page 31: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/31.jpg)
www.luxoft.com
Head Up Display Concept. HUD vs LCD
• Hardware limitation
• HUD devices are rarely
available on market
• FOV and object size
• Timings
• Zero latency
• Driver eye position
• Driver perception
• Virtual image distance
• Information balance
![Page 32: CORPORATE SLIDE DECK - NVIDIAon-demand.gputechconf.com/gtc/2017/presentation/s7312... · 2017-04-30 · • Features data base • Low level screen (3D) features to refine position](https://reader035.fdocuments.net/reader035/viewer/2022063005/5fb4164042ee3744895d2828/html5/thumbnails/32.jpg)
www.luxoft.com
Summary: Key Technology Advantages
• Proved understanding of pragmatic intersection and synergy between fundamental theoretical results and final
requirements
• Formal mathematical approaches are complemented by deep learning
• Solid GPU optimization expertise
• Automotive grade solutions integrated with all the data sources in vehicle – data fusion approaches
• High robustness in various weather and road conditions, confidence is estimated for efficient fusion
• Closed loops designed and implemented to enhance speed and robustness of each component
• Integration with V2X and Navigation system
• System architecture supports distributed HW setup and integration with existing in-vehicle components if required
(environmental model, objects detection, navigation, positioner etc.)
• Hierarchical Algorithmic Framework design highly optimizes computations on embedded platforms
• Collaboration with scientific groups to integrate cutting edge approaches