HOW DID I CREATE A HTML5 VIDEO STREAMING PLAYER

and what Ive learned

KKBOXVideo jessechen

AGENDA Some beforehand knowhow

Why developing this player

HTML5 extensions

Introduce our player

Future of this player

SOME BEFOREHAND KNOWHOW

analog-to-digital adapter

digital raw data

PROCESS OF PLAYING VIDEO1. Make it digital

analog signal

H.264 / H.265Encode

PROCESS OF PLAYING VIDEO2. Encode video file

Raw data .mp4 / .ts file

Easier for storage and transference

PROCESS OF PLAYING VIDEO3. Play it !

.mp4 / .ts filea/v sync

codec decode and

display

extract contentfrom container

WAIT, WHERE IS PLAYER?

WHAT PLAYER DOES

.mp4 / .ts filea/v sync

codec decode and

display

HERE

extract contentfrom container

THEY ARE ALREADY HANDLED

.mp4 / .ts file

HTML5

a/v sync

codec decode and

display

extract contentfrom container

SO WHY DEVELOPING THIS PLAYER?

OUR PLAYER FOCUS ON (1)

extract content and a/v info

from container

.mp4 / .ts file

get filesin smart way

a/v sync

codec decode and

display

HOW PLAYER GET FILE ?

server holds .mp4

content

VINTAGE WAY

Player get it and play

single file

slow and inflexible

server holds .mp4

content

EFFICIENT WAY

Player get part of video and

start play

progressive download

Keep requesting rest part of video during

playback

still inflexible

prepare different qualities

of contenton server

MODERN WAYadaptive streaming

Player playsbest quality depends on

network status

Keep requesting rest part of video during

playback

HOW ABOUT WE SPLIT FILE split file into segments, described by manifest

manifest tells

- available bitrates

- how file is fragmented

- other information, e.g encryption

+

+

knows how toget fragments

ADAPTIVE STREAMING IN SHORT

fragments

manifestfile

ADAPTIVE STREAMING TYPES

SSSmooth Streaming

HLSHTTP Live Streaming

HDS

INTRODUCE DASHDynamic Adaptive Streaming through HTTP

PROS OF DASH

open source, with industry standard, which means more universal usage

support 2 types of containers, the MPEG-4 file format or the MPEG-2 Transport Stream

GET FILES IN SMART WAY

manifest parser - able to read different type of manifest

adaptive bitrate - able to decide which quality to load

OUR PLAYER FOCUS ON (2)

.mp4 / .ts file

get filesin smart way

protectionlogic

extract contentfrom container

a/v sync

codec decode and

display

DRM ?(DIGITAL RIGHTS MANAGEMENT)

Content provider / DRM provider likes it

For end user

CLIENT SIDE PROTECTION Client get a license instead of a key from DRM

service

A blackbox or sandbox get the key by processing license

That blackbox / sandbox decrypt content and directly output it to display

H.264 / H.265Encode

HOW TO PROTECT CONTENT

Raw data .mp4 / .ts file

DRM implemented

here

PROTECTION LOGIC

give what DRM server needs and retrieve license

negotiate with browser in order to implement protection on client side

deal with different browsers

knows how to

NOW, LETS START TALK ABOUT BROWSER

, MSE and EME

HTML5 VIDEO ELEMENT support progressive download

For adaptive streaming, MSE is required

extract content from container

WHAT ROLE MSE PLAYS

.mp4 / .ts file

MSE handles

a/v sync

codec decode and

display

MSE handles

INTRODUCE MSE

Given different bitrate segments and it helpshtml5 video element to play smoothly

SourceBuffer provide video and audio buffer condition

Media Source Extension

EME IS ABOUT PROTECTION

HOW BROWSER ADOPT DRM

For browser, that blackbox called CDM (Content Decrypt Module)

Each browser support different DRM

context - a blackbox or sandbox get the key by processing license

DRM ON BROWSER

Widevine FairplayPlayready Primetime

INTRODUCE EME

Even though browser support its own DRM, W3C defines a EME spec, in order to expose same api for client

prefixed api was implemented on earlier version of chrome(smart tv)

Encrypted Media Extension

EMECDM

provide context from encrypted content

get challenge

DRM license server

request with challenge

get license

provide license for CDM to decrypt content

player

EMECDM

PROTECTION LOGIC FLOW

WHAT ROLE EME PLAYS

extract contentfrom container

.mp4 / .ts file

a/v synccodec decode

anddisplay

decrypt content in blackbox

INTRODUCE OUR PLAYER

OUR GOAL

play not single file but sequence of segments, with different bitrate, a.k.a adaptive streaming

play protected stuffs, for content providers goods

a player able to

ya pi

yapi.js

DEVELOP PROCESS

make it work stable refactor

MAKE IT WORK FIRST

only 2 files in the very beginning

BE STABLE

well structured

modularized

dependency management (dijon.js)

consistent code style

MODULARIZED

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

UTILS Capabilities

Debug

ErrorHandler

EventBus

IntervalBus

UrlModel

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

EXTENSION

MediaSourceExtension

SourceBufferExtension

VideoModel

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

MANIFEST

ManifestLoader

ManifestExtension

ManifestModel

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

STREAM

StreamCtrl

BufferCtrlapi

adaptive

extension

fragment

manifest

protection

stats

stream

utils

FRAGMENT

FragmentCtrl

FragmentLoader

SegmentTemplate

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

ADAPTIVE

AbrCtrl

BandwidthRecorderapi

adaptive

extension

fragment

manifest

protection

stats

stream

utils

PROTECTION

ProtectionCtrl

ProtectionModel

ProtectionRequest

Playready / Widevine

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

STATS

MetricsCtrl

Metrics

StatsCtrl

BitrateHistory

api

adaptive

extension

fragment

manifest

protection

stats

stream

utils

REFACTOR

review flow

redefine api

APPLICATION FLOW

REDEFINE API

use jsdoc to generate spec document

define api, event and vo(value object)

spec

result

http://usejsdoc.org/http://localhost:9900/doc/spec/KKPlayer.MediaPlayer.htmlhttp://localhost:9900/doc/dist/

DEMOSample player

http://localhost:9900

HOW TO MANAGE THOSE MODULES

DEPENDENCY INJECTION

not necessary in the beginning, but became very important after it went much more complex

use dijon.js as di framework

INTRODUCE DIJON.JS

a system holds all dependencies, after mapping modules to it

an object get dependencies after injection

DEFINE DEPENDENCIES

// A depfunction A() {}

// B requires Afunction B() { this.a = undefined; // how dijon knows where to inject}

// instantiate dijonvar system = new dijon.System();

3 KINDS OF MAPPING

var a = new A();system.mapValue(aValue, a);// system.getObject(a) would return an a instance

system.mapSingleton(aSingleton, A);// system.getObject(aSingleton) would return a singleton a

system.mapClass(bClass, B);// system.getObject(bClass) would return a new inited B

OUTLET MAPPING

// map outlet to make dependency work// @param sourceKey {String} - the key mapped to system would be injected// @param targetKey {String|'global'} - the key outlet is assigned to// @param outletName {String|sourceKey} - the name of property used as an outletsystem.mapOutlet('aSingleton', 'bClass', a');

var b = system.getObject(bClass);// b.a is aSingleton

DIRECTLY INJECTION

var b = new B();system.injectInto(b);

// b WOULDNT have dependency singleton A// b/c we only mapOutlet to key bClass of system

// map outlet to globalsystem.mapOutlet(aSingleton, global, a); system.injectInto(b); // this would work

AUTO MAP OUTLET

function B(){this.aSingleton = undefined;this.c = undefined;

}function C(){}

// auto mapped outlet would assign to global// and outlet name is the same as m