www.luxoft.com

ENGINEERING

SOFTWARE

TECHNOLOGY AUTOMOTIVE

TRAVEL AND AVIATION

ENERGY TELECOM

FINANCIAL SERVICES

ADAS COMPUTER VISION AND AUGMENTED REALITY SOLUTION

Sergii Bykov, Technical Lead

www.luxoft.com

Product Vision

www.luxoft.com

Road To Autonomous Driving

Source: Intel

www.luxoft.com

Representation For The Driver

Output is an extendable metadata which

describes all the augmented objects/hints

and supports natural features ON/OFF

www.luxoft.com

Computer Vision and Augmented Reality Applications

COMPUTER VISION

Pattern Recognition

Image Processing

Physics

Signal Processing

Arfificial Intelligence

Maths

City Driving Pattern

Augmented Navigation

Help in low visibility mode

Signs recognition Infographics

Next Gen of Adaptive Cruise

Active Park Search

www.luxoft.com

Key Challenges Of Bringing AR In The Vehicle

• Usability – augmented reality subsystem should not disturb driver as it is continuously observed

• Hardware limitations – computational, power consumption, zero latency (HUD)

• Requirements for precise environmental model estimation for occlusion avoiding

• Dependency on inaccurate map and navigation data

• Distributed HW architectures, platform flexibility requirements

• High precision absolute and relative positioning requirements

• Components synchronization and latency avoidance

• Embedded memory usage limitations, different memory models

• Algorithms should be both configurable and efficient

• Specific rendering requirements, not covered by general purpose frameworks

• Variety of inputs under different platforms

• Out-of-vehicle simulation (does not support natural simulation like classical navigation)

www.luxoft.com

Framework Concept

www.luxoft.com

Augmented Navigation Structure

We offer a Unique Solution capable to create Augmented, mixed visual Reality for drivers and passengers based

on Computer Vision, vehicle sensors, map data, telematics, navigation guidance using Data Fusion technique.

CVNAR Solution Automotive Cameras

Sensors/CAN

Navigation System/Map Data

Vehicle displays Projection on wind shield

Smart Glasses VR devices

Telematics/V2X

www.luxoft.com

CVNAR Features

Natural Augmented Reality

• Basic vehicle data

• Lanes and road boundaries

• Road signs and cautions

• Navigation data and hints

• Facades highlights

• Parking places

• Narrow street infographics

• Street names and complex junctions boards

• POI and OEM specific information

Highlights

• CPU, GPU;

• OS: Linux, QNX

• HW: Intel, NVidia, TI, Renesas

• Extrapolation engines for latency avoidance

• Machine learning and deep learning

Road scene recognition and objects tracking

• Road boundaries

• Lane detection

• Vehicle detection and tracking

• Distance and time to collision estimation

• Pedestrian detection and tracking

• Facade recognition and texture extraction

• Road signs recognition

• Parking slot recognition

Positioning

• Precise relative and absolute positioning

• Flexible data fusion and smooth map matching

• Automotive constrained SLAM

Integration and Fusion

• Sensor data

• External positioner data (optional)

• External recognition engine integration

• Pupil tracking

• Telematics: V2I and V2V

www.luxoft.com

Hardware Approach: Computer Vision Box

Live data from vehicle:

- CAN data, Sensors

- Video stream CVB

CVNAR Engine

CVB Data Layer

Web Interface

SW Update

Configuration

Diagnostic

Video Stream with

augmented objects ADAS CVNAR

Engine

HUD/LCD

Head Unit

• Quick-install demonstration solution

• Platform for CVNAR (allows to be portable)

• Integration with Head Units

• Integration with vehicle networks

• Using of own sensors if needed

Navigation data,

preprocessed

sensor data, etc.

Control/Settings

www.luxoft.com

Hardware Approach: Automotive Stack Own scalable and robust Automotive Stack aimed to minimize time of project start and integration

• RTOS (OSEK, Micrium, mTRON), Microcontrollers (Renesas RH850/V850)

• Hardware Abstraction Layer (HAL), Operation System Abstraction Layer (OSAL)

• Trace Server/Client, WatchDog, IPC, Drivers (SPI, I2C, UART, Timers, etc.), SW Update

• Pre-Integrated Vector CAN/Diagnostic Stack

• Vehicle Bootloader for Renesas Microcontrollers

Domains and areas:

• System development and integration with Automotive Networks and ECUs

• Drivers development and peripheral support: Video Cameras, Automotive Sensors, external HW

• HW brings-up (BSP development)

Automotive grade technologies supported by team:

• Networks: CAN, LIN, MOST, BroadReach, Ethernet

• RTOS: OSEK, mTron, Micrium, embOS, mTRON, QNX, Linux, VxWorks

• Microcontrollers: Renesas (RH850, V850), Freescale (Bolero, i.MX6), TI (OMAP, Jacinto, MSP430)

• CAN Stacks: Vector, KPIT, own tinyCAN

• Audio/Video processing

• Media Bus: LVDS, FPDLink-III, APIX2, USB, Ethernet, BroadReach

www.luxoft.com

Perception Concept

www.luxoft.com

Sensor Fusion: Data Inference

Optimal fusion filter parameters adjustment problem statement and solution developed to fit different car models

with different chassis geometries and steering wheel models/parameters.

Features:

• Absolute and relative positioning

• Dead reckoning

• Fusion with available automotive grade sensors – GPS, steering wheel, steering wheel rate, wheels sensors

• Fusion with navigation data

• Rear movements support

• Complex steering wheel models identification. Ability to integrate with provided models

• GPS errors correction

• Stability and robustness against complex conditions – tunnels, urban canyons

www.luxoft.com

Sensor Fusion: Advanced Augmented Objects Positioning

Solving map accuracy problems

Placing:

• Road model

• Vehicles detection

• Map data

Position clarification:

•Camera motion model: •Video-based gyroscope

•Positioner Component

•Road model

•Objects tracking

www.luxoft.com

Sensors Fusion: Comparing Solutions

Update frequency ~15 Hz (+extrapolation with any fps)

Update frequency ~4-5 Hz

Reference solution Our solution

www.luxoft.com

Lane Detection: Adaptability and Confidence

www.luxoft.com

Lane Detection: 3D-scene Recognition Pipeline

• Low level invariant features

• Single camera

• Stereo data

• Point clouds

• Structural analysis

• Probabilistic models

• Real-world features

• Physical objects

• 3D scene reconstruction

• Road situation

• 3D space scene fusion (different sensors input)

• Backward knowledge propagation from high levels

www.luxoft.com

Lane Detection: Additional Information

• Features data base

• Low level screen (3D) features to refine position

• Points clouds

• Marking details, Road borders

• High level structural elements and real world objects

• Junctions, Facades, Signs, etc.

• Features Collection

• Existing map providers

• Real-time feature extraction and understanding from video sensors

• Satellite-view photos analysis

• Map database updates

• Routes offline processing and upload

www.luxoft.com

Lane Detection: HD Map Potential

Content

• Simplified and advanced geometry for roads, traffic lanes,

lanes boundaries etc.

Application

• Precise on-road vehicle positioning

• Different weather, traffic situations

• Map matching and Path planning

• Maneuver suggestions

• Cable navigation

• Junction assistance

• Possible junction maneuvers

• Traffic lights position

Useful information:

• Road border geometry and type

• Traffic signs (position and type);

• Traffic lights (position and type);

• Type and quality of roadbed;

• Roadside POIs (gas station, store, café etc.);

• Any other additional features which can be useful for

vehicle positioning or driver.

www.luxoft.com

Lane Detection: Robustness in Normal Conditions

Graphs show error in meters between recognized lanes (curved model) and recognized road marking (distance to detected

features + features accuracy) in different distance range and for different road conditions.

Figure 1. Regular weather, highway, slightly curved road

with lane changes Figure 2. Rainy weather, highway, straight road with lane

changes

www.luxoft.com

Lane Detection: Robustness in Difficult Conditions

Figure 3. Bright sun, highway with secondary roads,

straight road with turns and lane changes Figure 4. Hard rain, highway, straight road with lane

changes

www.luxoft.com

Vehicle Detection

Figure – Vehicle detection examples

• Convolutional neural network for vehicle detection

• GPU Acceleration – CUDA

• Running real-time on NVidia Jetson TK1

• Inference speedup on embedded (TK1) GPU vs CPU is ~3x

• Training speedup on desktop GPU vs CPU is ~20x

• Classifier accuracy (about 50k, 960x540, ~55-60 deg HFOV):

• Positive: 99.65%

• Negative: 99.82%

• Size of detection down to 30 pix, detection range of about 60 m

www.luxoft.com

Road Scene Semantic Segmentation

Figure – Road scene segmentation examples

• Deep fully convolutional neural network for semantic

pixel-wise segmentation

• Road scene understanding use cases: model

appearance, shape, spatial-relationship between classes

• Inference speedup GPU vs CPU is ~3x

www.luxoft.com

Driver Monitoring: Haar/LBP Cascades

• Object Detection using Haar or LBP feature-based cascade

classifiers is an effective method for face detection

• It is a machine learning based approach where a cascade function

is trained from a lot of positive and negative images. It is then used

to detect objects in other images.

• Applications can achieve consistent face detection in different

environments (e.g. vehicle driver, etc.)

• Can be efficiently implemented for low power embedded devices

• Speedups on mobile GPU vs CPU are

• 20x for Haar

• 3x for LBP

Figure – Haar features

Figure – LBP features

www.luxoft.com

HMI Concept

www.luxoft.com

Rendering Component Structure

CVNAR Rendering

Renderer

System

Window

System

Resource

Management

Frame Buffer

Frame Renderer

HMI Renderer

Augmentation Renderer

Client of

rendering

framework

(controller)

HMI/scene

update

commands

rendered

frames

Head Up

Display

LCD

www.luxoft.com

Renderer Output Layers

Frame renderer

Augmentation renderer

HMI renderer

Final image

www.luxoft.com

Augmented Objects Primitives. Part 1

Barrier Lane Line

Traffic Sign & POI Lane Arrow

www.luxoft.com

Augmented Objects Primitives. Part 2

Fishbone Facade

Vehicle Street Name

www.luxoft.com

Demo Application Screenshot (LCD)

www.luxoft.com

Head Up Display Concept. HUD vs LCD

• Hardware limitation

• HUD devices are rarely

available on market

• FOV and object size

• Timings

• Zero latency

• Driver eye position

• Driver perception

• Virtual image distance

• Information balance

www.luxoft.com

Summary: Key Technology Advantages

• Proved understanding of pragmatic intersection and synergy between fundamental theoretical results and final

requirements

• Formal mathematical approaches are complemented by deep learning

• Solid GPU optimization expertise

• Automotive grade solutions integrated with all the data sources in vehicle – data fusion approaches

• High robustness in various weather and road conditions, confidence is estimated for efficient fusion

• Closed loops designed and implemented to enhance speed and robustness of each component

• Integration with V2X and Navigation system

• System architecture supports distributed HW setup and integration with existing in-vehicle components if required

(environmental model, objects detection, navigation, positioner etc.)

• Hierarchical Algorithmic Framework design highly optimizes computations on embedded platforms

• Collaboration with scientific groups to integrate cutting edge approaches

www.luxoft.com

THANK YOU

Sergii Bykov

Technical Lead

email: sergeybykov1@gmail.com