Presenter : Ettikan Karuppiah (Ph.D), Director Developers Ecosystem

AI Conference Sydney, Australia. 3rd September 2018

DEEPSTREAM SDK 2.0 “ACCELERATES VIDEO ANALYTICS DEVELOPMENT”

Deepstream application Intelligent Video Analytics

Traffic Engineering Self-service Store

Access control Parking Management

3NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DeepStream 2.0 SDK

NVIDIA Platforms

Linux, CUDA

DeepStream 2.0 SDK

Sample

DNNs &

plug-ins

3rd party

application

plugins

Modular,

Scalable,

Architecture

Development

Tools

Retail analytics & checkout

Intelligent traffic systems

Law

enforcemen

t

Smart parking

Access control

User Applications

Solution

adaptation

guide

* Early release and limited availability.

TensorRT

cuDNN

Multimedia

Multi-stream Encode &

Decode

Imaging

Image capture &

processing

Deep Learning

4

INTELLIGENT VIDEO ANALYTICS

Cameras Decode Detectors/Classifiers/Trackers Composite Encode Metadata

TESLA JETSON

5

END-TO-END PRODUCT FAMILY

HPC/TRAINING INFERENCE

EMBEDDED

Jetson TX1

DATA CENTER

Tesla P4

AUTOMOTIVE

Drive PX2

Tesla P100Tesla V100Titan V

DATA CENTERDESKTOP

FULLY INTERGRATED DL SUPERCOMPUTER

Tesla V100TITAN V

DESKTOP WORKSTATIONDATA

CENTER

Tesla V100DGX StationTITAN Quadro

DGX-1 DGX-2

6

DEEPSTREAM SDK

7

BUILT ON WIDELY USED SOFTWARE

JETSON / TESLA PLATFORM

DEEPSTREAM PLUGINS

LOW LEVEL API

APPLICATION

NVCUVID

TENSORRT

CUDA

CUDNN

8

WHAT’S IN DEEPSTREAM 2.0

Reference Applications

Accelerated Plugins & Utilities

RESNET Models

Sample examples

Reference Video clips

Documentation

CUDA 9.2 , TensorRT 4.0, GStreamer 1.8.3

9

GSTREAMER

PLUGINS – Basic building block for media processing pipeline.

PADS – Pads are plugin’s input and output, where you can connect other plugins. They are used to negotiate links and data flow.

BINS - A bin is a container for a collection of plugins.

PIPELINE – Top level bin. It provides a bus for the application and manage the synchronization for its child plugins.

10

GSTREAMER

Buffers – objects for passing streaming data between plugins in the pipeline.

Events – events are objects sent between plugins or from the application to plugins.

Messages – objects posted by plugins on the pipeline's message bus, where they will be held for collection by the application.

Queries - allow applications to request information from the pipeline.

11

DEEPSTREAM PIPELINE ARCHITECTURE

Jetson

GPU CPU GPU

VIC VIC VIC

GPU DMANVDEC CPU

VIC

GPU

CPU

NVDC

Tesla

NVDEC

CPU

GPU DMA GPU CPU

GPU

GPU CPU

GPU

NVDisplay

GST-uridecodebin

GST-NvStreamMux

GST-NvInferGST-

NvTracker GST-NvInferGST-

nvvidconv GST-NvOsdGST-

NvEglGlesSink

libnvtensor-rt

(TensortRT)libnvtracker

(visionworks)

Low Level APIs

DeepStream

Libnvtensor-rt

(TensortRT) libnvosdlibnvvidconv

VIC

GPU

decode

12

LIST OF PLUGINS

Plugin Type Functionality

Decoder H265 / H264 Video decoders

Format Convertor Scaling, Format conversion, rotation

Aggregator StreamMux and Batching

Inferencing Primary / Secondary detector / classifier

Tracker Reference object tracker

Tiler Video wall output

On Screen Display For drawing text, rectangles lines etc.

Template Example plugin for 3rd party IP / models

13

VIDEO AGGREGRATOR

Plugin that accepts “n” inputs streams and converts to sequential batch frames

Scaling support – Incase video input resolution differs with the model resolution or vice-a-versa

Name - nvstreammux

nvstreammux

NPP

CPU + GPU

YUV

batched YUV +

metadata

Low Level API

HW

14

VIDEO AGGREGRATOR

Contiguous buffer of

size W x H x N

allocated by

GstNvStreamMuxer

Async Cuda

Memcpy

Queue for S1 buffers

Queue for S2 buffers

Queue for SM buffers

Batching algorithm

Batch of N

buffers of size (W

x H)

Decoder / Camera 1

Decoder / Camera 2

Decoder / Camera M

GST - NvInfer

Round Robin

then FCFS

15

SCREEN TILER

Used for creating video wall effect

Arranges multiple input sources into complete video tiled output

Configurable window size

Name - nvstreamtiler

nvstreamtiler

CUDA

GPU

Low Level API

HW

YUV + All Meta YUV + All Meta

16

ON SCREEN DISPLAY

17

… AND MANY MORE PLUGINSDESCRIPTION NAME

Read from arbitrary point in a file filesrc

Receive data over the network via RTSP rtspsrc

Reads frames from a Video4Linux2 device v4l2src

X11 based videosink xvimagesink

H264 Encoder x264Enc

Decode / Encode in / from JPEG format jpegdec/enc

Dewarp fish eye video Dewarp

3rd party IP plugin (License plate recognition) ALPR

3rd party IP plugin (Face recognition) facedetect

To jump start development for adding custom IP ds-example

18

DEEPSTREAM REFERNCE APPLICATION

1) Supports N simultaneously input sources.

• File Input

• RTSP input

• Camera input

2) Supports H265 / H264 decode.

3) 1 Primary object detector + 3 Attribute classifiers + Tracker.

4) Application generates metadata in KITTI format.

19

DEEPSTREAM REFERENCE APPLICATION

20

DEEPSTREAM REFERENCE APP OUTPUT

21

DEEPSTREAM REFERENCE APP OUTPUT

22

DEEPSTREAM APPLICATIONMap to DeepStream plugins

Gst-rtspsrc

IMAGES or VIDEO

Gst-

decodebinGst-infer Gst-tracker Gst-nvosd

Gst-

nveglglessink

IP CAMERA DECODE INFER OBJECT

TRACKERDISPLAY

Analytics

plugin

(Counting)

23

(Car-LPR)

(Car-Color)

(Car-Model)

VEHICLE DETECTION

GST-NvInfer

(Car-Detect)

Gst-

uridecode

GST-

NvTracker

GST-NvInfer

(Car-Make)

GST-

NvEglglessin

k

GST-OSD GST-Tiler

Video

Decode

Primary

Detector

Object

Tracker

Secondary Classifiers On Screen

Display

Tiler Renderer

24

VEHICLE DETECTION

Inserting video: Insert/Video/Video from File.Insert video by browsing your directory and selecting OK.

File types that works best in PowerPoint are mp4 or wmv

1) Describe/Design your use case and work flow

2) Understand required plugins for the defined workflow and

associated input and output for each stage/plugins

3) Check if you can use the reference plugins provided in DS SDK2.0

4) No, create new plugin (refer to plugin implementation guide)

5) Link the plugins and test your application

6) If you are happy with the apps, review the performance

HOW DO I DEVELOP MY DS APPS?