Multi Streaming PlayerAndroid Praktikum SS 2014

Let us introduce ourselves

Dario Banfi:BSc in computer science, University of Torino

2nd semester, MSc in informatics, TUM

Guillaume Chabin: Double degree student at TUM and Telecom ParisTech

MSc in management and information systems, Telecom ParisTech, France3rd semester, MSc in informatics, TUM

AgendaI. Project presentation

II. DemoIII. Project FlowIV. Application presentation

A. Application architecture

B. Application user interface

C. From the prototype to the next version

V. Conclusion

The Project

Team: The Partner■ Weptun GmbH

■ 60 apps developed■ Munich based■ 17 Employees, founded in 2010

■ Christoph Hausmann■ CTO

The main goal

■ Realize a low latency video stream player■ Multi-streaming playback to see more than one video at

once

Starting point

■ Existing backend ■ Custom streaming server: Orchestrator

■ JSON protocol for signaling and communicating with the client

applications

■ Real-time Transport Protocol (RTP) streaming protocol

■ Existing iOS application■ Developed internally by Weptun

Before explaining how it works, demo time!

http://youtu.be/N2ppsCCCulw

The project flow

The project flow

■ Compared to our first roadmap:■ Content roughly the same ■ Workload not really well estimated for some tasks

■ 2 major surprises:■ First media library chosen not compatible for multi

streaming■ First user interface conflicting with the stream player

Preliminary tasks■ Getting in touch with the network

protocols and the backend■ RTP, SRTP, RTSP…■ Video Codecs■ How Orchestrator works

■ Researching on how to implement RTP streaming on android■ Android native MediaPlayer■ Third party library in Java■ Third party library in C/C++ with the NDK

Finding the right tools■ libVLC

■ Stable and well supported

■ Very active community

■ Used in iOS application

■ No multistream capabilities in Android !

Finding the right tools (ctd)■ FFMpeg

■ Open-source■ Good documentation■ Complete, cross-platform multimedia framework■ Base of many video players

■ Our player is based on IjkPlayer (ffplay wrapper) ■ Some modifications to enable RTP streaming ■ Different ffmpeg compilation■ Enable network streams■ Enable the correct codecs & containers

Single RTP stream video player

■ Based on android.widget.VideoView, with our custom FFMpeg MediaPlayer

■ Customized MediaController in order to show specific controls

■ Ratio options and audio tweaks

Multi-Streaming Video Player

■ Integrate the single RTP stream into a multi-streaming view

■ Optimize display area to show more content

■ Complexity of handling the multiple connections

Improvement of the QoS■ Latency issue

■ Important area for further improvements

○ RAM and CPU limiting factor for multi-streaming

○ HD fullscreen still laggy for low-end smartphones

■ A dynamic or manual selection of the stream

quality■ Set the maximal number of streams per screen■ Benchmark of several players versions

The final application

Architecture: Video player

■ Player based on IjkPlayer ■ Core of the application

■ C player built on top of ffplay,

■ JNI to create a Android

MediaPlayer-like library

■ compile with additional flags to enable RTP

■ Architecture for ARMV7, X86 and ARM V5

■ Easy to change the player in our implementation

Architecture: Video player (ctd)

Player displayed in a custom Video View

■ Extends the video SurfaceView

■ Customized MediaController overlay

■ Controls of the player + additional features■ Error overlay

Architecture: Server interaction

■ Server called Orchestrator■ Implemented by Weptun,

closed code■ Stream to a client upon

requesting■ Bandwidth depending on the

size and the definition, ■ 10 MBit/s for HD videos ■ 1.5 Mbit/s SD videos.

Architecture: Server interaction (ctd)

■ UDP Hole Punching■ Dedicated port to the server■ Port:address mapping in the NAT

table, ■ Used to stream the data directly to

the client■ 2 external libraries:

■ GSON , unserialize the JSON messages

■ Android Asynchronous Http Client , for the REST exchanges.

If the client is behind a NAT, the server isn’t able to address him the stream !

Architecture: Application logic

■ Apps contains 3 activities:■ Settings screen■ Full screen player ■ Multistreamer Viewer (Main activity)

■ Main activity ■ Streams contained in fragment■ “Page” with up to 4 streams

■ Informative fragment (connection errors / no stream)

■ Replace fragment for new stream view

Architecture: Application logic

Several level to share information within the application■ Shared preference

■ Configuration and favorite streams

■ Bundle in fragment replacement

■ Server Data: Singleton Class storing the main information■ eg the list of available streams

User interface: The Drawer Layout

■ Create a sliding menu■ Open / Close from the home button ■ Nice callbacks, dynamic controls■ Contains all features for the user

■ Often used with expandable list ■ Adapter to populate the list■ 2 levels lists with 2 levels click listeners

■ Nice solution to display several Listviews in the same activity

A suited UI in every situation

From the prototype to the POC

Single video streaming player Mandatory

Multi-streaming application Mandatory

Improvement of the multi streaming latency Nice to have

Encryption through SRTP Optional

Overlay text while playing videos Optional

List of the target features:

From the POC to the real productSeveral improvements possible■ Performance:

■ Enabling hardware accelerated video playback.

■ Optimization on the allocation of data in the ffmpeg implementation

■ Modify the user interface depending on the final use:■ Entertraining-oriented services■ Video-surveillance systems

Conclusion■ Good introduction to android

framework■ Many skills acquired

■ Linux tools and Native code integration through NDK

■ Video streaming protocols■ Async tasks and threads

■ Introduction to the startup environment

■ Final product with a good video experience

Thank you for your attention

Credits

The following resources are under CC licences■ Android robot : http://commons.wikimedia.org/wiki/File:Android_robot.svg■ Video Player designed by Nikhil Dev from the Noun Project■ Check List designed by Sabine Wollender from the Noun Project■ Server designed by Konstantin Velichko from the Noun Project■ Video Camera designed by Drew Ellis from the Noun Project

■ Soccer Field designed by Erik Wagner from the Noun Project

■ Check Box designed by Rémy Médard from the Noun Project

■ Present designed by Megan Sheehan from the Noun Project

■ Magnifying Glass designed by vijay sekhar from the Noun Project

■ http://img3.wikia.nocookie.net/__cb20130109141609/logopedia/images/6/6d/VLC-Media-Player-Metro-icon.png

■ ffmpeg logo: https://www.ffmpeg.org/ffmpeg-logo.png■ Mountain Climbing designed by Juan Pablo Bravo from the Noun Project

■ Economic Growth designed by Adriano Gazzellini from the Noun Project

■ Window designed by Marek Polakovic from the Noun Project

■ Loading designed by Mateo Zlatar from the Noun Project

■ Beaker designed by Rohan Gupta from the Noun Project