Kurento cpmx

Real-time Multimedia Stream Processing Developing rich multimedia applications with Kurento

The Stream oriented GE: Developing rich multimedia applications with Kurento

[email protected] h0p://www.kurento.org

Open APIs for Open Minds

h"ps://join.me/kurentoCPMX

Mul8media infrastructures for the Future Internet

Media is here Media got

there

Media got there

Analyze Transform

Store

Transport

Enrich Augment Adapt

Sensors Context

Events Media is here

2

Enrich, augment, adapt, analyze, transform, store: what’s the problem?

Complexity

3

Future Internet

Mul8media Infrastructure

Simple Development

APIs

Kurento: the equa8on

4

• Interoperable media exchange (mul8plaUorm/mul8protocol) • WebRTC, RTP, HTTP (video tag), etc.

• Process media (Computer vision, augmented reality, media indexing, etc.) • Media and metadata recording and recovery • Transform and adapt media (H.264, H.263, VP8, Ogg, and others) • Media rou8ng and mixing • Etc.

Mul8media infrastructure

• REST API •  JavaScript API •  Java API

APIs

•  LGPL 2.1

Is distributed through a flexible FOSS license

What’s Kurento

5

Kurento Media Server (KMS): the nucleus of Kurento

•  KMS is a middleware for media streams – Receives the stream – Process the stream –  Issues the stream

Send Receive Analyze Augment Enrich

Transform Transcode Record Process Replicate

Media Source

Media Sink

KMS

6

The Media API: The API for accessing KMS capabili8es

Send Receive Analyze Augment Enrich

Transform Transcode Record Process Replicate

Media Source

Media Sink

KMS

Java Media API

JavaScript Media API

REST API

Applica8ons define the processing of streams geeng through KMS

7

Media API: Media Elements and Media Pipelines

Sink%

SRC% Sink%

SRC%

SRC%

Sink%

Sink%

§ Media Element • Provides a specific media

functionality › Send/receive media › Process media › Transform media

• Exchange media through ›  Sources ›  Sinks

§ Media pipeline • Chain of media elements

implementing the desired media logic.

• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox

Media Element

Sink

SRC

8

Media API: trivial example (Java) MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(“file:///myFile.webm”).build(); H2pGetEndpoint h"pEndpoint = mp.newH"pGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(h"pEndpoint); h"pEndpoint.getUrl(); //URL where the media is made available

Media Pipeline (inside KMS)

H"pGetEndpoint

Media from file or URI

HTTP media streaming Si

nk

SRC

PlayerEndpoint

9

Is that enough? Think about the WWW development model

Process WWW request -‐  DDBB access -‐  Authen8ca8on -‐  XML processing -‐  Etc.

HTTP request: I want this resource

HTTP response: The resource

•  Intui8on behind the WWW model – Client asks what it wants

•  GET /this_resource HTTP/1.1 – Server side APIs execute the associated processing

You develop this using … •  Servlet •  JSP •  ASP •  PHP •  Ruby on Rails •  CGI •  Etc.

10

We need an equivalent model: the role of the Signaling Plane

•  We need “something” with the capability of nego8a8ng what, when and how media is exchanged

•  A protocol for “saying” –  I want “this media” …

•  Iden8fica8on of the media to exchange –  File in hard-‐drive, IP camera, user, etc.

–  in “this way” … •  Iden8fica8on of the processing of media

–  Augmented, analyzed, etc.

–  with “this format” … •  Quality

–  Codec, screen-‐size, frame-‐rate, etc.

–  at “this moment” •  Stream control

–  Play, stop, start, pause, etc.

11

Don’t get it? think about WWW development again …

Process WWW request -‐  DDBB access -‐  Authen8ca8on -‐  XML processing -‐  Etc.

HTTP request: I want this resource

HTTP response: The resource

Process media request -‐  Media API -‐  DDBB access -‐  Authen8ca8on -‐  XML processing -‐  Etc.

Signaling request: I want this media

Signaling response: The media is here

This is how you create WWW applica8ons

(Servlets, ASP, PHP, Rails, etc.)

This is how you create Kurento enabled applica8ons:

Mul8media RTC is just another feature of your

applica8on

You, as a programmer, create this

12

Abstrac8ng the signaling plane: The Content Handler

KMS

Sink

SRC

Sink

SRC

Sink

SRC

Sink

Media API

REST API (Open API protocol) The Content Handler Equivalent to a Servlet/ASP/PHP script Exposes APIs for specifying: -‐  when receiving “this request”… -‐  … execute “this logic” Developer can use the media API

Code building the media pipeline and execu8ng the applica8on logic the

developer wants

13

Kurento Applica8on Server: The container of Handlers

•  Is an extension of Java EE technologies. •  Compa8ble with all Java EE Servlet

containers •  Hold Handlers

–  Your specific applica8on code •  Receives signaling requests:

–  I want “this media” in this way … •  Dispatches request to the appropriate

handler –  @annota8ons based mapping

•  Generate an answer showing “where and how to find the media” –  URL where media is located –  Nego8ated SDP

Java EE compaJble container

HTTP Servlet

SIP Servlet

Web services

Kurento REST API

Specific handler implementa8ons

Media API DD.BB.

Kurento ApplicaJon Server (KAS)

Other java APIs.

14

Kurento Architecture: pueng it all together

Kurento Media Server (KMS)

Receive Video

Augmented Reality

Send Video

Computer Vision

Video Playing and Recording

Java EE compaJble container

HTTP Servlet

SIP Servlet

Web services

Kurento REST API

Specific handler implementa8ons

Signaling and WWW traffic

Med

ia

Med

ia

Signaling and WWW traffic

Media API DD.BB.

Kurento ApplicaJon Server (KAS)

Other java APIs.

15

Applica8on execu8on flow Client Code

Applica8on Server (KAS)

Media Server (KMS)

I want this media in this way … (JSON-‐RPC request)

Commands reques8ng the crea8on of a pipeline

What you want is here … (JSON-‐RPC answer)

Media negoJaJon

phase

Media exchange phase

1

2

Specific applica8on logic at the server-‐side

(Content Handler)

Media pipeline creaJon

Media exchange between client and server

16

Content Handler: trivial example //Specify the type of service provided by this handler: Player, Recorder, WebRTC, RTP, etc. @H2pPlayerService(path = "/player”) //Mapping of handler specified in path public class MyPlayerHandler extends H"pPlayerHandler { @Override public void onContentRequest(H"pPlayerSession contentSession) {

//Thie client wants the media this handler provides //Create the pipeline for providing the media

} @Override public void onContentStarted(H"pPlayerSession contentSession) {

//Media started flowing, you can execute addi8onal ac8ons } @Override Public void onSessionTerminated(H"pPlayerSession contentSenssion){

//Media exchange termianted, you can collect your resources }

17

Let’s develop with Kurento •  What you need

–  A Kurento instance •  You can install your own Kurento instance •  You can launch a Kurento instance at the FI-‐LAB (FI-‐WARE project)

–  h"p://lab.fi-‐ware.org •  Geeng help

–  FI-‐WARE catalogue •  h"p://catalogue.fi-‐ware.org/enablers/stream-‐oriented-‐kurento

–  Kurento web site •  h"p://www.kurento.org

–  Kurento mailing list •  h"ps://groups.google.com/forum/#!forum/kurento

–  Twi"er •  @kurentoms

•  FI-‐LAB –  Working instance of FI-‐WARE enabling free experimenta8on with

technology –  h"p://lab.fi-‐ware.org

•  Crea8ng a Kurento instance from an image –  Use latest version of Kurento images.

•  Crea8ng a Kurento instance using recipes –  Use Ubuntu 13.10 clear image –  Use latest version of Kurento recipes

•  Accessing the Kurento demo at FI-‐LAB –  h"p://kurento.lab.fi-‐ware.org:8080/fi-‐lab-‐demo

19

Kurento at the FI-‐LAB

Kurento Hello World: Playing a file with an HTML5 client

Media Pipeline

H"pGetEndpoint

Media from file or URI


nk

SRC

PlayerEndpoint

Media API

REST API (Open API protocol)

Handler code I want “this media”

Media is “at this URL”

20

Playing a file with an HTML5 client: Handler code

@H"pPlayerService(path = "/player”) public class MyPlayerHandler extends H"pPlayerHandler {

@Override public void onContentRequest(H"pPlayerSession contentSession) throws Excep8on {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint( "h2p://media.w3.org/2010/05/sintel/trailer.webm").build(); contentSession.setA"ribute("player", playerEndpoint); H2pGetEndpoint h2pEndpoint = mp.newH2pGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(h2pEndpoint); contentSession.start(h"pEndpoint);


PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getA"ribute("player"); playerEndpoint.play();

} }

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyPlayerHandler.java

21

Playing a file with an HTML5 client: client code <!DOCTYPE html>

<html> <head>

<script src="./js/kws-‐content-‐api.js"></script> <script>

var conn; func8on start() {

var op8ons = { remoteVideoTag : "remoteVideo"

}; conn = new kwsContentApi.KwsContentPlayer("./player", op8ons);

} func8on terminate() {

conn.terminate(); }

</script> </head> <body>

<bu"on onclick="start();">Start</bu"on> <bu"on onclick="terminate();">Terminate</bu"on> <br /> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/player.html

22

Playing a file with an HTML5 client: See the example working

h"ps://www.youtube.com/watch?v=n5BQlhYgGSo

23

Media Pipeline

Adding Computer Vision

H"pGetEndpoint Media from file or URI


nk

SRC

PlayerEndpoint

SRC

Sink

JackVaderFilter

Media API


Handler code I want “this media”

Media is “at this URL”

24

Adding Computer Vision: Handler code @H"pPlayerService(path = "/playerWithFilter”) public class MyPlayerHandler extends H"pPlayerHandler {

@Override public void onContentRequest(H"pPlayerSession contentSession) throws Excep8on {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint( "h"p://media.w3.org/2010/05/sintel/trailer.webm").build(); contentSession.setA"ribute("player", playerEndpoint); JackVaderFilter filter = mp.newJackVaderFilter().build(); H"pGetEndpoint h"pEndpoint = mp.newH"pGetEndpoint().terminateOnEOS().build(); filter.connect(h"pEndpoint); playerEndpoint.connect(filter); contentSession.start(h"pEndpoint);


PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getA"ribute("player"); playerEndpoint.play();

} }

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyPlayerWithFilter.java

25

Adding Computer Vision: Client code <!DOCTYPE html> <html>

<head> <script src="./js/kws-‐content-‐api.js"></script> <script>



}; conn = new kwsContentApi.KwsContentPlayer("./playerWithFilter", op8ons);

} func8on terminate() {

conn.terminate(); }

</script> </head> <body>

<bu"on onclick="start();">Start</bu"on> <bu"on onclick="terminate();">Terminate</bu"on> <br /> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/playerFilter.html

26

Adding Computer Vision: See the example working

h"ps://www.youtube.com/watch?v=yJAQs23eoXw

27

WebRTC loopback

Kurento Media Server

WebRTC Streaming

Media API

JSON-‐RPC API

Handler code I want “this media” (offer SDP)

Media is “here” (answer SDP)

Sink

SRC

28

Kurento ApplicaJon Server

WebRtcEndpoint

Client Browser

WebRTC loopback: Handler code

29

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyWebRtcLoopback.java

@WebRtcContentService(path = "/webRtcLoopback") public class MyWebRtcLoopback extends WebRtcContentHandler {

@Override public void onContentRequest(WebRtcContentSession contentSession) throws Excep8on {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build(); webRtcEndpoint.connect(webRtcEndpoint); contentSession.start(webRtcEndpoint);

} }

29

WebRTC loopback: client code

30

<!DOCTYPE html> <html>

<head> <script src="./js/kws-‐content-‐api.js"></script> <script src="h"ps://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script> <script>


var op8ons = { localVideoTag : "localVideo", remoteVideoTag : "remoteVideo"

}; conn = new kwsContentApi.KwsWebRtcContent("./webRtcLoopback", op8ons); }

func8on terminate() { conn.terminate();

} </script>

</head> <body>

<bu"on id="start" onclick="start();">Start</bu"on> <bu"on id="terminate" onclick="terminate();">Terminate</bu"on> <br /> <video id="localVideo" autoplay></video> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/webrtcLoopback.html

30

WebRTC loopback: see the example working

h"ps://www.youtube.com/watch?v=HaVqO06uuNA

31

WebRTC recording

Media Pipeline

WebRTC Streaming

Media API


Handler code I want “this media (SDP)”

Media is “at here (SDP)”

Sink

SRC

Sink

Media to

file or URI

32

WebRTC recorder: Handler code

33

Source h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java Source of handler playing the recorded video: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyPlayerRecording.java

@WebRtcContentService(path = "/webRtcRecorder") public class MyWebRtcRecorder extends WebRtcContentHandler {

@Override public void onContentRequest(WebRtcContentSession contentSession) throws Excep8on {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build(); RecorderEndpoint recorderEndpoint = mp.newRecorderEndpoint("file:///tmp/recording").build(); contentSession.setA"ribute("recorder", recorderEndpoint); webRtcEndpoint.connect(webRtcEndpoint); webRtcEndpoint.connect(recorderEndpoint); contentSession.start(webRtcEndpoint);

} @Override public void onContentStarted(WebRtcContentSession contentSession) {

RecorderEndpoint recorderEndpoint = (RecorderEndpoint) contentSession.getA"ribute("recorder"); recorderEndpoint.record();

} }

33

WebRTC recording: client code

34





}; conn = new kwsContentApi.KwsWebRtcContent("./webRtcRecorder", op8ons); }


} </script>

</head> <body>

<bu"on id="start" onclick="start();">Start</bu"on> <bu"on id="terminate" onclick="terminate();">Terminate</bu"on> <a href="./playerRecording">Play recorded content</a> <br /> <video id="localVideo" autoplay></video> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/webrtcRecorder.html

34

WebRTC recording: see the example working

h"ps://www.youtube.com/watch?v=ogN81PGkMuE

35

WebRTC one-‐to-‐many (Handler omi"ed for simplicity)

Media Pipeline

Sink

SRC

SRC

Sink

SRC

Sink

SRC

Sink

36

WebRTC one-‐to-‐many: Handler code

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyWebRtcOneToMany.java

@WebRtcContentService(path = "/webRtcOneToMany") public class MyWebRtcOneToMany extends WebRtcContentHandler {

private WebRtcEndpoint firstWebRtcEndpoint; private String sessionId;

@Override public synchronized void onContentRequest(WebRtcContentSession contentSession) throws Excep8on {

if (firstWebRtcEndpoint == null) { MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); firstWebRtcEndpoint = mp.newWebRtcEndpoint().build(); sessionId = contentSession.getSessionId(); contentSession.releaseOnTerminate(firstWebRtcEndpoint); firstWebRtcEndpoint.connect(firstWebRtcEndpoint); //Loopback contentSession.start(firstWebRtcEndpoint);

} else { MediaPipeline mp = firstWebRtcEndpoint.getMediaPipeline(); WebRtcEndpoint newWebRtcEndpoint = mp.newWebRtcEndpoint().build(); contentSession.releaseOnTerminate(newWebRtcEndpoint); newWebRtcEndpoint.connect(firstWebRtcEndpoint); firstWebRtcEndpoint.connect(newWebRtcEndpoint); //Latest client gives media to the master endpoint contentSession.start(newWebRtcEndpoint);

} }

}

37

WebRTC one-‐to-‐many: client code

38





}; conn = new kwsContentApi.KwsWebRtcContent("./webRtcOneToMany", op8ons); }


} </script>

</head> <body>

<bu"on id="start" onclick="start();">Start</bu"on> <bu"on id="terminate" onclick="terminate();">Terminate</bu"on> <br /> <video id="localVideo" autoplay></video> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/webrtcOneToMany.html

38

WebRTC one-‐to-‐many: see the example working

h"ps://www.youtube.com/watch?v=TBkrl3fmHWI

39

WebRTC game (Handler omi"ed for simplicity)

Media Pipeline

Sink

SRC

Sink

SRC

Sink

SRC

40

WebRTC game: Handler code

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyWebRtcWithFilters.java

@WebRtcContentService(path = "/webRtcWithFilters") public class MyWebRtcWithFilters extends WebRtcContentHandler {

@Override public void onContentRequest(WebRtcContentSession contentSession)throws Excep8on {

final MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); final WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build(); final PointerDetectorFilter pointerDetectorFilter = mp.newPointerDetectorFilter().build(); final FaceOverlayFilter faceOverlayFilter = mp.newFaceOverlayFilter().build(); PointerDetectorWindowMediaParam start = new PointerDetectorWindowMediaParam("start", 100, 100, 280, 380); start.setImage("h"p://ci.kurento.com/imgs/start.png"); pointerDetectorFilter.addWindow(start); pointerDetectorFilter.addWindowInListener(new MediaEventListener<WindowInEvent>() { public void onEvent(WindowInEvent event) {

faceOverlayFilter.setOverlayedImage( "h"p://ci.kurento.com/imgs/mario-‐wings.png", -‐0.35F, -‐1.2F, 1.6F, 1.6F); }

}); webRtcEndpoint.connect(pointerDetectorFilter); pointerDetectorFilter.connect(faceOverlayFilter); faceOverlayFilter.connect(webRtcEndpoint); contentSession.start(webRtcEndpoint);

}

}

41

WebRTC game: client code

42





}; conn = new kwsContentApi.KwsWebRtcContent("./webRtcWithFilters", op8ons); }


} </script>

</head> <body>

<bu"on id="start" onclick="start();">Start</bu"on> <bu"on id="terminate" onclick="terminate();">Terminate</bu"on> <br /> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/webrtcFilters.html

42

WebRTC game: see the example working

h"ps://www.youtube.com/watch?v=5eJRnwKxgbY

43

Complex applica8on

44

Result: h"ps://www.youtube.com/watch?v=PYCw9-‐4oWWA

Sink

SRC

Sink

Sink

SRC

Sink

SRC

Sink

SRC

Sink

SRC

WebRtcEndpoint

MirrorFilter PointerDetectorFilter

ChromaFilter FaceOverlayFilter

Media Pipeline

•  Beyond video-‐conferencing: receiving events from the media stream –  Immersive and personalized adver8sing –  Games and entertainment

Events (bu"ons)

Game or adver8sement logic

Change face Change background

Collabora8ons welcome h"p://www.github.com/kurento

Thank you very much for your a"en8on

Sugges8ons, comments and complains: [email protected]

45

Kurento cpmx

Technology

Transcript of Kurento cpmx