www.kurento.org

Real-time Multimedia Stream ProcessingDeveloping rich multimedia applications with KurentoDeveloping rich multimedia applications

with Kurento: a tutorial for Java developers

lulop@kurento.orghttp://www.kurento.org

www.kurento.org

Multimedia infrastructures for the Future Internet

Media ishere Media got

there

Media gotthere

AnalyzeTransform

Store

Transport

EnrichAugment

Adapt

SensorsContext

EventsMedia is

here

Traditional infrastructures

Future Internet infrastructures

2

www.kurento.org

Enrich, augment, adapt, analyze, transform, store: what’s the problem?

Complexity

3

www.kurento.org

Future Internet

Multimedia Infrastructure

Simple Development

APIs

Kurento: the equation

4

www.kurento.org

• Interoperable media exchange (multiplatform/multiprotocol)• WebRTC, RTP, HTTP (video tag), etc.

• Process media (Computer vision, augmented reality, media indexing, etc.)• Media and metadata recording and recovery• Transform and adapt media (H.264, H.263, VP8, Ogg, and others)• Media routing and mixing• Etc.

Multimedia infrastructure

• REST API• JavaScript API• Java API

APIs

• LGPL 2.1

Is distributed through a flexible FOSS license

What’s Kurento

5

www.kurento.org

Kurento Media Server (KMS): the nucleus of Kurento

• KMS is a middleware for media streams– Receives the stream– Process the stream– Issues the stream

SendReceiveAnalyze

AugmentEnrich

TransformTranscode

RecordProcess

Replicate

MediaSource

MediaSink

KMS

6

www.kurento.org

The Media API: The API for accessing KMS capabilities

SendReceiveAnalyze

AugmentEnrich

TransformTranscode

RecordProcess

Replicate

MediaSource

MediaSink

KMS

JavaMedia API

JavaScriptMedia API

RESTAPI

Applications define the processingof streams getting through KMS

7

www.kurento.org

Media API: Media Elements and Media Pipelines

Media Element

• Provides a specific media functionality

› Send/receive media

› Process media

› Transform media

• Exchange media through

› Sources

› Sinks

Media pipeline

• Chain of media elements implementing the desired media logic.

• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox

Media Element

Sink

SRC

8

www.kurento.org

Media API: trivial example (Java)MediaPipeline mp = contentSession.getMediaPipelineFactory().create();

PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(file:///myFile.webm).build();

HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();

playerEndpoint.connect(httpEndpoint);

httpEndpoint.getUrl(); //URL where the media is made available

Media Pipeline (inside KMS)

HttpGetEndpoint

Media fromfile or URI

HTTP mediastreaming Si

nk

SRC

PlayerEndpoint

9

www.kurento.org

Is that enough? Think about the WWW development model

Process WWW request- DDBB access- Authentication- XML processing- Etc.

HTTP request: I want this resource

HTTP response:The resource

• Intuition behind the WWW model – Client asks what it wants

• GET /this_resource HTTP/1.1

– Server side APIs execute the associated processing

You develop this using …• Servlet• JSP• ASP• PHP• Ruby on Rails• CGI• Etc.

10

www.kurento.org

We need an equivalent model: the role of the Signaling Plane

• We need “something” with the capability of negotiating what, when and how media is exchanged

• A protocol for “saying”– I want “this media” …

• Identification of the media to exchange– File in hard-drive, IP camera, user, etc.

– in “this way” …• Identification of the processing of media

– Augmented, analyzed, etc.

– with “this format” …• Quality

– Codec, screen-size, frame-rate, etc.

– at “this moment”• Stream control

– Play, stop, start, pause, etc.

11

www.kurento.org

Don’t get it? think about WWW development again …

Process WWW request- DDBB access- Authentication- XML processing- Etc.

HTTP request: I want this resource

HTTP response:The resource

Process media request- Media API- DDBB access- Authentication- XML processing- Etc.

Signaling request: I want this media

Signaling response:The media is here

This is how you create WWW applications

(Servlets, ASP, PHP, Rails, etc.)

This is how you createKurento enabled

applications:Multimedia RTC is just

another feature of your application

You, as a programmer,

create this

12

www.kurento.org

Abstracting the signaling plane: The Content Handler

KMS

Sink

SRC

Sink

SRC

Sink

SRC

Sink

Media API

REST API (Open API protocol) The Content HandlerEquivalent to a Servlet/ASP/PHP scriptExposes APIs for specifying:- when receiving “this request”…- … execute “this logic”Developer can use the media API

Code building the media pipeline and executing the application logic the

developer wants

13

www.kurento.org

Kurento Application Server: The container of Handlers

• Is an extension of Java EE technologies.• Compatible with all Java EE Servlet

containers• Hold Handlers

– Your specific application code• Receives signaling requests:

– I want “this media” in this way …• Dispatches request to the appropriate

handler– @annotations based mapping

• Generate an answer showing “where and how to find the media”– URL where media is located– Negotiated SDP

Java EE compatible container

HTTP Servlet

SIP Servlet

Web service

s

Kurento REST API

Specific handler implementations

Media API DD.BB.

Kurento Application Server (KAS)

Other java APIs.

14

www.kurento.org

Kurento Architecture: putting it all together

Kurento Media Server (KMS)

ReceiveVideo

AugmentedReality

SendVideo

Computer Vision

Video Playingand Recording

Java EE compatible container

HTTP Servlet

SIP Servlet

Web service

s

Kurento REST API

Specific handler implementations

Signaling and WWW traffic

Med

ia

Med

ia

Signaling and WWW traffic

Media API DD.BB.

Kurento Application Server (KAS)

Other java APIs.

15

www.kurento.org

Application execution flowClientCode

ApplicationServer (KAS)

MediaServer (KMS)

I want this media in this way …(JSON-RPC request)

Commands requestingthe creation of a pipeline

What you want is here …(JSON-RPC answer)

Media negotiation

phase

Media exchange

phase

1

2

Specific applicationlogic at the server-side

(Content Handler)

Mediapipelinecreation

Media exchange between client and server

16

www.kurento.org

Content Handler: trivial example//Specify the type of service provided by this handler: Player, Recorder, WebRTC, RTP, etc.@HttpPlayerService(path = "/player”) //Mapping of handler specified in pathpublic class MyPlayerHandler extends HttpPlayerHandler {

@Overridepublic void onContentRequest(HttpPlayerSession contentSession) {

//Thie client wants the media this handler provides//Create the pipeline for providing the media

}

@Overridepublic void onContentStarted(HttpPlayerSession contentSession) {

//Media started flowing, you can execute additional actions}

@OverridePublic void onSessionTerminated(HttpPlayerSession contentSenssion){

//Media exchange termianted, you can collect your resources}

17

www.kurento.org

Let’s develop with Kurento

• What you need– A Kurento instance

• You can install your own Kurento instance• You can launch a Kurento instance at the FI-LAB (FI-WARE project)

– http://lab.fi-ware.org

• Getting help– Kurento web site

• http://www.kurento.org

– Kurento mailing list• https://groups.google.com/forum/#!forum/kurento

– Twitter• @kurentoms

18

www.kurento.org

Kurento Hello World: Playing a file with an HTML5 client

Media Pipeline

HttpGetEndpoint

Media fromfile or URI

HTTP mediastreaming Si

nk

SRC

PlayerEndpoint

Media API

REST API (Open API protocol)

Handler codeI want “this media”

Media is “at this URL”

19

www.kurento.org

Playing a file with an HTML5 client: Handler code

@HttpPlayerService(path = "/player”)public class MyPlayerHandler extends HttpPlayerHandler {

@Overridepublic void onContentRequest(HttpPlayerSession contentSession) throws Exception {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create();contentSession.releaseOnTerminate(mp);PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint("http://media.w3.org/2010/05/sintel/trailer.webm").build();contentSession.setAttribute("player", playerEndpoint);HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();playerEndpoint.connect(httpEndpoint);contentSession.start(httpEndpoint);

}

@Overridepublic void onContentStarted(HttpPlayerSession contentSession) {

PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAttribute("player");playerEndpoint.play();

}}

Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyPlayerHandler.java

20

www.kurento.org

Playing a file with an HTML5 client: client code<!DOCTYPE html>

<html><head>

<script src="./js/kws-content-api.js"></script><script>

var conn;function start() {

var options = {remoteVideoTag : "remoteVideo"

};conn = new kwsContentApi.KwsContentPlayer("./player", options);

}function terminate() {

conn.terminate();}

</script></head><body>

<button onclick="start();">Start</button><button onclick="terminate();">Terminate</button><br /><video id="remoteVideo" autoplay></video>

</body></html>

Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/webapp/player.html

21

www.kurento.org

Playing a file with an HTML5 client: See the example working

https://www.youtube.com/watch?v=n5BQlhYgGSo

22

www.kurento.org

Media Pipeline

Adding Computer Vision

HttpGetEndpointMediafrom

file or URI

HTTP mediastreaming Si

nk

SRC

PlayerEndpoint

SRC

Sink

JackVaderFilter

Media API

REST API (Open API protocol)

Handler codeI want “this media”

Media is“at this URL”

23

www.kurento.org

Adding Computer Vision: Handler code@HttpPlayerService(path = "/playerWithFilter”)public class MyPlayerHandler extends HttpPlayerHandler {

@Overridepublic void onContentRequest(HttpPlayerSession contentSession) throws Exception {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create();contentSession.releaseOnTerminate(mp);PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint("http://media.w3.org/2010/05/sintel/trailer.webm").build();contentSession.setAttribute("player", playerEndpoint);JackVaderFilter filter = mp.newJackVaderFilter().build();HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();filter.connect(httpEndpoint);playerEndpoint.connect(filter);contentSession.start(httpEndpoint);

}

@Overridepublic void onContentStarted(HttpPlayerSession contentSession) {

PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAttribute("player");playerEndpoint.play();

}}

Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyPlayerWithFilter.java

24

www.kurento.org

Adding Computer Vision: Client code<!DOCTYPE html><html>

<head><script src="./js/kws-content-api.js"></script><script>

var conn;function start() {

var options = {remoteVideoTag : "remoteVideo"

};conn = new kwsContentApi.KwsContentPlayer("./playerWithFilter", options);

}function terminate() {

conn.terminate();}

</script></head><body>

<button onclick="start();">Start</button><button onclick="terminate();">Terminate</button><br /><video id="remoteVideo" autoplay></video>

</body></html>

Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/webapp/playerFilter.html

25

www.kurento.org

Adding Computer Vision: See the example working

https://www.youtube.com/watch?v=yJAQs23eoXw

26

www.kurento.org

WebRTC loopback

Media Pipeline

WebRTCStreaming

Media API

REST API (Open API protocol)

Handler codeI want “this media (SDP)”

Media is“at here (SDP)”

Sink

SRC

27

www.kurento.org

WebRTC loopback: Handler code

28

Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcLoopback.java

@WebRtcContentService(path = "/webRtcLoopback")public class MyWebRtcLoopback extends WebRtcContentHandler {

@Overridepublic void onContentRequest(WebRtcContentSession contentSession) throws Exception {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create();contentSession.releaseOnTerminate(mp);WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build();webRtcEndpoint.connect(webRtcEndpoint);contentSession.start(webRtcEndpoint);

}}

28

www.kurento.org

WebRTC loopback: client code

29

<!DOCTYPE html><html>

<head><script src="./js/kws-content-api.js"></script><scriptsrc="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script><script>

var conn;function start() {

var options = {localVideoTag : "localVideo",remoteVideoTag : "remoteVideo"

};conn = new kwsContentApi.KwsWebRtcContent("./webRtcLoopback", options);}

function terminate() {conn.terminate();

}</script>

</head><body>

<button id="start" onclick="start();">Start</button><button id="terminate" onclick="terminate();">Terminate</button><br /><video id="localVideo" autoplay></video><video id="remoteVideo" autoplay></video>

</body></html>

Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/webapp/webrtcLoopback.html

29

www.kurento.org

WebRTC loopback: see the example working

30

https://www.youtube.com/watch?v=HaVqO06uuNA

30

www.kurento.org

WebRTC recording

Media Pipeline

WebRTCStreaming

Media API

REST API (Open API protocol)

Handler codeI want “this media (SDP)”

Media is“at here (SDP)”

Sink

SRC

Sink

Mediato

file or URI

31

www.kurento.org

WebRTC recorder: Handler code

32

Source https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java

Source of handler playing the recorded video:https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyPlayerRecording.java

@WebRtcContentService(path = "/webRtcRecorder")public class MyWebRtcRecorder extends WebRtcContentHandler {

@Overridepublic void onContentRequest(WebRtcContentSession contentSession) throws Exception {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create();contentSession.releaseOnTerminate(mp);WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build();RecorderEndpoint recorderEndpoint = mp.newRecorderEndpoint("file:///tmp/recording").build();contentSession.setAttribute("recorder", recorderEndpoint);webRtcEndpoint.connect(webRtcEndpoint);webRtcEndpoint.connect(recorderEndpoint);contentSession.start(webRtcEndpoint);

}

@Overridepublic void onContentStarted(WebRtcContentSession contentSession) {

RecorderEndpoint recorderEndpoint = (RecorderEndpoint) contentSession.getAttribute("recorder");recorderEndpoint.record();

}}

32

www.kurento.org

WebRTC recording: client code

33

<!DOCTYPE html><html>

<head><script src="./js/kws-content-api.js"></script><scriptsrc="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script><script>

var conn;function start() {

var options = {localVideoTag : "localVideo",remoteVideoTag : "remoteVideo"

};conn = new kwsContentApi.KwsWebRtcContent("./webRtcRecorder", options);}

function terminate() {conn.terminate();

}</script>

</head><body>

<button id="start" onclick="start();">Start</button><button id="terminate" onclick="terminate();">Terminate</button><a href="./playerRecording">Play recorded content</a><br /><video id="localVideo" autoplay></video><video id="remoteVideo" autoplay></video>

</body></html>

Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java

33

www.kurento.org

WebRTC recording: see the example working

34

https://www.youtube.com/watch?v=ogN81PGkMuE

34

www.kurento.org

WebRTC one-to-many (Handler omitted for simplicity)

Media Pipeline

Sink

SRC

SRC

Sink

SRC

Sink

SRC

Sink

35

www.kurento.org

WebRTC one-to-many: Handler code

36

Source https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcOneToMany.java

@WebRtcContentService(path = "/webRtcOneToMany")public class MyWebRtcOneToMany extends WebRtcContentHandler {

private WebRtcEndpoint firstWebRtcEndpoint;private String sessionId;

@Overridepublic synchronized void onContentRequest(WebRtcContentSession contentSession) throws Exception {

if (firstWebRtcEndpoint == null) {MediaPipeline mp = contentSession.getMediaPipelineFactory().create();contentSession.releaseOnTerminate(mp);firstWebRtcEndpoint = mp.newWebRtcEndpoint().build();sessionId = contentSession.getSessionId();contentSession.releaseOnTerminate(firstWebRtcEndpoint);firstWebRtcEndpoint.connect(firstWebRtcEndpoint); //LoopbackcontentSession.start(firstWebRtcEndpoint);

} else {MediaPipeline mp = firstWebRtcEndpoint.getMediaPipeline();WebRtcEndpoint newWebRtcEndpoint = mp.newWebRtcEndpoint().build();contentSession.releaseOnTerminate(newWebRtcEndpoint);newWebRtcEndpoint.connect(firstWebRtcEndpoint);firstWebRtcEndpoint.connect(newWebRtcEndpoint); //Latest client gives media to the master endpointcontentSession.start(newWebRtcEndpoint);

}}

}

36

www.kurento.org

WebRTC one-to-many: client code

37

<!DOCTYPE html><html>

<head><script src="./js/kws-content-api.js"></script><scriptsrc="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script><script>

var conn;function start() {

var options = {localVideoTag : "localVideo",remoteVideoTag : "remoteVideo"

};conn = new kwsContentApi.KwsWebRtcContent("./webRtcOneToMany", options);}

function terminate() {conn.terminate();

}</script>

</head><body>

<button id="start" onclick="start();">Start</button><button id="terminate" onclick="terminate();">Terminate</button><br /><video id="localVideo" autoplay></video><video id="remoteVideo" autoplay></video>

</body></html>

Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java

37

www.kurento.org

WebRTC one-to-many: see the example working

38

https://www.youtube.com/watch?v=TBkrl3fmHWI

38

www.kurento.org

WebRTC game (Handler omitted for simplicity)

Media Pipeline

Sink

SRC

Sink

SRC

Sink

SRC

39

www.kurento.org

WebRTC game: Handler code

40

Source https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcWithFilters.java

@WebRtcContentService(path = "/webRtcWithFilters")public class MyWebRtcWithFilters extends WebRtcContentHandler {

@Overridepublic void onContentRequest(WebRtcContentSession contentSession)throws Exception {

final MediaPipeline mp = contentSession.getMediaPipelineFactory().create();contentSession.releaseOnTerminate(mp);final WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build();final PointerDetectorFilter pointerDetectorFilter = mp.newPointerDetectorFilter().build();final FaceOverlayFilter faceOverlayFilter = mp.newFaceOverlayFilter().build();PointerDetectorWindowMediaParam start = new PointerDetectorWindowMediaParam("start", 100, 100, 280, 380);start.setImage("http://ci.kurento.com/imgs/start.png");pointerDetectorFilter.addWindow(start);pointerDetectorFilter.addWindowInListener(new MediaEventListener<WindowInEvent>() {public void onEvent(WindowInEvent event) {

faceOverlayFilter.setOverlayedImage("http://ci.kurento.com/imgs/mario-wings.png",-0.35F, -1.2F, 1.6F, 1.6F);}

});

webRtcEndpoint.connect(pointerDetectorFilter);pointerDetectorFilter.connect(faceOverlayFilter);faceOverlayFilter.connect(webRtcEndpoint);contentSession.start(webRtcEndpoint);

}

}

40

www.kurento.org

WebRTC game: client code

41

<!DOCTYPE html><html>

<head><script src="./js/kws-content-api.js"></script><scriptsrc="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script><script>

var conn;function start() {

var options = {remoteVideoTag : "remoteVideo"

};conn = new kwsContentApi.KwsWebRtcContent("./webRtcWithFilters", options);}

function terminate() {conn.terminate();

}</script>

</head><body>

<button id="start" onclick="start();">Start</button><button id="terminate" onclick="terminate();">Terminate</button><br /><video id="remoteVideo" autoplay></video>

</body></html>

Source: https://github.com/Kurento/kmf-tutorial/blob/master/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java

41

www.kurento.org

WebRTC game: see the example working

42

https://www.youtube.com/watch?v=5eJRnwKxgbY

42

www.kurento.org

Collaborations welcomehttp://www.github.com/kurento

Thank you very much for your attention

Suggestions, comments and complains:lulop@kurento.org

43