Farsight 2 Video Conferencing Made Easy 691

8/3/2019 Farsight 2 Video Conferencing Made Easy 691

1/36

Farsight 2Video conferencing made easy

Olivier Crte


2/36

Origins of Farsight

aMSN

Free IM had no VoIP

Each proprietary IM protocol had its own thing

Philippe's end of studies project

Hacked into aMSN, gaim


3/36

Original goals of Farsight

Enable Free Software IM to do audio/video likeother platforms

Abstract the streaming from different IM

protocols Hide the complexities of media streaming (of

GStreamer)


4/36

State of Farsight 1

Current used in Telepathy Stream Engine

On Nokia Internet Tablets

Very stable Thin core, complex plugins

Complex RTP plugin

In maintenance-only mode Unmaintained plugins for MSN, Yahoo


5/36

RTP plugin

1 to 1 audio & video calls

Codec detection, simple negotiation

Transmitters plugins

Unicast UDP

Google's pseudo-ICE (libjingle)

Full DTMF (RTP events and sound)

Confort Noise (on Nokia Tablets)


6/36

Limitations

Only one to one calls

No lip-sync

Video support broke abstraction

Hard to use with non-trivial GStreamerpipelines

No sRTP

Hacks for Nokia Tablets (DSP codecs, CN)

No RTCP (No statistics)


7/36

Farsight 2: Goals

High level objects

Core: interface, helper libraries

RTP is reference, most standard, most capable

Also, MSN, Yahoo, etc

One GStreamer element per protocol

Elegance Complete automated test coverage

Good documentation


8/36

Different conferencing models

Media

Decentralised

Unicast

Multicast Centralized (mixing server)

Signalling

Centralized Decentralized (crazy!)


9/36

Focused signallingCentralised media


10/36

Focused signallingDistributed media (multicast)


11/36

Focused signallingDistributed media (multi-unicast)


12/36

Distributed signallingCrazy ideas!


13/36

High level objects

Codec

Candidate

Participant

One person with synchronized streams

Session

Stream

Conference


14/36

Session

One type of media (audio, video, etc)

One local media source

One microphone

One camera

File

etc

Multiple stream from other participants RTP session


15/36

Stream

One participant in one session

Use for communication with participant

Codecs

Candidates

Remote media comes out of here


16/36

Conference

The GStreamer element

Multiple synchronized sessions

Contains everything else


17/36

Conference

Session(audio)

Session(video 1)

ParticipantBob

ParticipantAlice

Session(video 2)

Stream StreamStreamStream Stream Stream


18/36

New RTP plugin

Keep good things from older versions

Codec detection

GStreamer elements: DTMF, RTP payloaders, etc

Use new GStreamer rtpmanager

Multi-party

Lip-sync

Complete RTP feature set Including full RTCP, SSRC collision detection, etc


19/36

Transmitters

Multi Unicast UDP (with STUN and UPnP)

Multicast UDP

Interactive Connection Establishment (ICE)

Standard (draft 19)

Google Talk compatible

Windows Live Messenger compatible

Including TURN relay support

Pidgeons, etc


20/36

Current status

Complete RTP implementation

All Farsight 1 features

Multi-party

Lip-sync

Python bindings

Automated tests

Unicast, Multicast, libnice (ICE) transmitters Complex encoding pipelines (for Comfort Noise)


21/36

Free codecs available

Audio Speex

AMR Narrowband & Wideband (patented)

PCMA / PCMU Vorbis, MP3 & AAC (not great choices for VoIP)

Video

Theora H.263 (probably patented)

H.264 (very patented)


22/36

10 way conference


23/36

Exampleimport pygst; pygst.require('0.10'); import farsight, gst, gobject, sys

loop = gobject.MainLoop()

pipeline = gst.Pipeline()

conference = gst.element_factory_make ("fsrtpconference")

conference.set_property ("sdes-cname", sys.argv[1] + "@1.2.3.4")

pipeline.add (conference)

session = conference.new_session (farsight.MEDIA_TYPE_VIDEO)

participant = conference.new_participant (sys.argv[2]+"@1.2.3.4")

stream = session.new_stream (participant, farsight.DIRECTION_BOTH, "multicast")

stream set remote codecs([farsight Codec(96 "H263 1998"


24/36

stream.set_remote_codecs([farsight.Codec(96, "H263-1998",

farsight.MEDIA_TYPE_VIDEO,

90000)])

candidate = farsight.Candidate()

candidate.ip = "224.0.0.110"

candidate.port = 3442

candidate.component_id = farsight.COMPONENT_RTP

candidate.proto = farsight.NETWORK_PROTOCOL_UDP

candidate.type = farsight.CANDIDATE_TYPE_MULTICAST

candidate.ttl = 1

Candidate2 = candidate.copy()

candidate.port = 3443

candidate.component_id = farsight.COMPONENT_RTCP

stream.set_remote_candidates ([candidate, candidate2])


25/36

gst.parse_bin_from_description(sys.argv[3] + ' ! videoscale', True)

pipeline.add (videosource)

videosource.get_pad ("src").link(session.get_property ("sink-pad"))

def _src_pad_added (stream, pad, codec, pipeline):

videosink = gst.element_factory_make ("xvimagesink")

pipeline.add (videosink)

videosink.set_state (gst.STATE_PLAYING)

pad.link (videosink.get_pad ("sink"))

stream.connect ("src-pad-added", _src_pad_added, pipeline)

pipeline.set_state(gst.STATE_PLAYING)

loop.run()


26/36

Demos

3-slide example

More complex multi-party example

Will it work ... ?


27/36

Current Users

Pidgin-vv branch

aMSN Voice chats

Telepathy-Farsight library

Next Generation Maemo Stream Engine


28/36

Extending Farsight

Codecs are regular GStreamerencoder/decoder elements

Custom negotiators for codecs that have

complex parameter negotiation functions (likeH.264)

Also, RTP payloaders and depayloaders

Low hanging fruits iLBC

G.729

H.261


29/36

Introducing GStreamer

Codecs are regular GStreamerencoder/decoder elements

Three major types of elements:

Sources (camera, microphone, file, etc) Sinks (screen, speakers, file, etc)

Transformational elements (encoder, decoders,etc, etc, etc)

Also, complex magical elements (playbin,farsight, etc)


30/36

Basic GStreamer

Elements

filesrc

mad

alsasink


31/36

Basic GStreamer

Elements

Bins

filesrcsink

mad

alsasink

bin


32/36

Basic GStreamer

Elements

Bins

Pads

filesrc

src

sink

mad

alsasink

srcsink

sink

bin


33/36

Extending Farsight

Non-Free codecs are possible (patents)

Implement plugins for other protocols

MSN Webcam just needs cleanup

MSN AV is very similar

Others?

Documentation:

http://www.gstreamer.net/

Th F t


34/36

The Future

Advanced RTP features

Secure RTP

RTCP AVPF (more feedback)

Easy library to write simple IM applications

Use it in all Free clients so they can gain AVcapabilities

Application integration using Telepathy

Empathy

aMSN 2

PSI


35/36

Mingle

Multiparty Jingle New XMPP extension to set up multi-party

conferences using MUCs

Better than SIP Will work with server-based and link-local

XMPP

Will make it possible to use all of Farsight'scapabilities

Will be presented at Linux.Conf.Au 2009 bySjoerd Simons


36/36

Thank you

Farsight is brought to you by Collabora

Questions?

http://farsight.freedesktop.org/

http://www.collabora.co.uk/IRC: #farsight @ freenode

Farsight 2 Video Conferencing Made Easy 691

Documents

Transcript of Farsight 2 Video Conferencing Made Easy 691