Internet Media Guides (IMGs) · IP + UDP/TCP + RTP/… vs. MPEG multiplex (or even analog)...

HELSINKI UNIVERSITY OF TECHNOLOGYNETWORKING LABORATORY

© 2007 Jörg Ott 1

Next Generation Session Announcements:

Internet Media Guides (IMGs)

© 2007 Jörg Ott


2

ObservationsSAP/SDP tied to IP-Multicast-based session modelOnly one distribution scheme: announcementOnly one type of service: convey multimedia session information

(Global) IP-Multicast has not prevailed as a distribution platformSAP rather experimentalWas often used for debugging Mbone connectivity

SummarySAP/SDP too limitedNot appropriate as a general solution for distributing session informationTraditionally linked to IP-only (and Multicast-only)

© 2007 Jörg Ott


3

Background: Ubiquitous Information Access

CellularNetworks

Internet+ IP Networks

BroadcastingNetworks

TV set / radio

Workstation

Laptop / tablet PC

PDA

Cellphones

…

…

Live Broadcast

Canned Program

Studio

File Server

Ticker Server

Web Server

…

© 2007 Jörg Ott


4

“Classic” Broadcasting & Internet Multimedia

Broadcasting has been a different world(including customer expectations, philosophy)

EncodingsAudio/Video largely compatible (but different quality expectations)Image/text formats/HTML vs. Videotex, MHP, specific markups, tables

Data transmissionIP + UDP/TCP + RTP/… vs. MPEG multiplex (or even analog)

AddressingIP addresses + ports vs. frequency/channel, PID, satellite position, pol., …

Interaction & control RTSP, HTTP, SIP, … vs. MHP

But there is a migration towards IP in various areasContent providers, transmission technologies, consumer equipment

© 2007 Jörg Ott


5

Platform/Network-Independent Content Provision

The same content shall be available via different networksPreferably without repeated authoring

“Content” used in a broad senseOriginal media: Audio / video broadcasts, web pages, files, news feeds, …Supplementary information: background, statistics, subtitles, ads, …

Content needs to be globally (or regionally) identifiableContent needs to be found

Descriptive metadata Availability (scheduling) metadata

Alternate access methods must be possibleNetwork + network-specific address

© 2007 Jörg Ott


6

Internet Media Guides (IMG)Definition of an IMG (from MMUSIC Charter)

Content:A collection of multimedia session descriptionsExpressed using SDP, SDPng or other metadata formatsIt is used to describe a collection of multimedia sessions (e.g. televisionprogramme schedules).

Distribution:The IMG must be delivered to a potentially large audience (push orpull), who use it to join a subset of the sessions described, and whomay need to be notified of changes to the IMG.

© 2007 Jörg Ott


7

IMG ≈ EPGGeneralized for arbitrary...

Types of mediaTypes of sessions and interactions: services!Classes of devices

Plurality of access methodsPhysical delivery(Reliable) Broadcast / multicast (push)Interactive retrieval (pull)Provision of full IMGs and of deltasNotification about changes

Network-independentFor the delivery of IMGsFor the (request and) transmission of actual media in sessions

The same IMGs should be usable everywhere.

© 2007 Jörg Ott


8

IMG ElementsIMG Metadata:SDP(ng),

MPEG-7, ...

IMG Source IMGreceiver

IMGtransceiver

Processing

IMG Metadata:SDP(ng),

MPEG-7, ...

IMG Metadata:SDP(ng),

MPEG-7, TVA

IMG SourceIMG senderIMG

receiverIMGreceiver

IMG sender IMGreceiver

IMG Transport

IMG MetadataEnvelope

© 2007 Jörg Ott


9

IMG Delivery Models / OperationsIMG

announcer

IMG ANNOUNCE

Broadcast / MulticastIMG SourceIMGlistener

IMGresolver

IMGquerier

IMG QUERY (Pull)

IMG RESOLVE

IMGnotifier IMG NOTIFY (w/ content)

IMG SourceIMGsubscriber

IMG SUBSCRIBE

IMGsender

IMGreceiver

IMG QUERYIMG NOTIFY (w/o content, w/ pointer)

IMG RESOLVE

Full IMG

Full IMG

Full IMG

Full IMGδ

δ

δ

δ

*p

© 2007 Jörg Ott


10

IMG Architecture

Point-to-Multipoint Point-to-PointIMG

Transport

IMG ANNOUNCEIMG SUBSCRIBE

IMG NOTIFYIMG QUERY

IMG RESOLVEIMG

Operations

IMG Envelope

IMGData Types

Complete Description, Delta Description, Pointer

#1 #2 … #nMetadataFormats

© 2007 Jörg Ott


11

IMG Envelope: Security RequirementsAuthentication + Integrity validation of contained metadata

Must work for complete and delta informationMust work across IMG transceivers

Aggregation, splitting, filtering of pieces of metadata

PrivacyMust be able to protect (parts of) contained metadata

User protection + access controlEnable (limited) IMG transceiver functionality

Interdependency with metadata formatsWhat to expect from metadata?Granularity of embedded metadata objectsDRM? metadata formats

© 2007 Jörg Ott


12

IMG EnvelopeContainer for metadata

Complete, delta, pointersIndependent of metadataLikely to become some kind of wrapper mechanismMetadata itself defined by other bodies

Generic management informationIdentification + version + validity informationContent-Type: to identify metadata formatSupport for security?

authentication + integrity informationPrivacy of content

MIME vs. XML

© 2007 Jörg Ott


13

Envelope Features (1)Container for metadata (independent of these)

Complete, delta, pointersMetadata itself defined by other bodies

Version numberDetermine the most recent (i.e., valid) copyReferenced as basis for delta encoding

Validity timePeriod: from, to

Metadata URIIdentifies the metadata element contained in the envelopeHelps to deal with fragments

Content-TypeDefines the type of metadata contents

© 2007 Jörg Ott


14

Envelope Features (2)Support for digital signatures (on parts of the envelope)Support for encryption

Only partly specified so farMay use S/MIME

Metadata contents:Inline External (via pointer)

Envelope { … }

Metadata

Envelope { … }

Metadata

© 2007 Jörg Ott


15

Envelope Encoding: XML vs. MIMEPresent focus: XML (also used by 3GPP MBMS)Example (with SDP as metadata)

<?xml version="1.0" encoding="UTF-8"?><metadataEnvelope

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:noNamespaceSchemaLocation="envelope.xsd"metadataURI="http/www.example.com/img001/session001.sdp"version="1"validFrom="2003-12-17T09:30:47-05:00"validUntil="2003-12-17T09:30:47-05:00"contentType="application/sdp"><metadataFragment>

v=0o=jo 2890844526 2890842807 IN IP4 10.33.57.27s=SDP Seminarc=IN IP4 224.2.17.12/127t=2873397496 2873404696a=recvonlym=audio 49170 RTP/AVP 0m=video 51372 RTP/AVP 31

</metadataFragment></metadataEnvelope>

© 2007 Jörg Ott


16

IMG MetadataPast focus on traditional contents

Conveying plain TV-schedulesStreaming in 3GPP Release 6

Broadening the scopeCover services in a more general fashionProvide region/location informationSupport personalized inquiriesAddress issues of cost

Make offers automatically comparable

Technical level: enable service discovery (and location)Business level: support adequate service selection

© 2007 Jörg Ott


17

IMG URNIMGs need to be identified globally

In particular, across different networks and providersMotivates the use of IMG URNsFormaturn:img: ProviderId : DateId : IMGResourceId [: FragmentId]

ProviderId: domain nameDateId: Point in time when the domain name was owned by the entityIMGResourceId: provider-selected stringFragmentId: some identifier for a piece of an IMG

Examplesurn:img:example.org:20051021:my-imgurn:img:example.org:20051021:my-img:subset

Mapping to URIs (e.g., HTTP, SIP) to be defined

© 2007 Jörg Ott


18

IMG TransportsNeed to provide mechanisms for IMG Operations

ANNOUNCEReliable multicast transport protocol: FLUTE + MUPPET

SUBSCRIBE / NOTIFYSession Initiation Protocol (SIP): Extensions for Subscription/Notification

QUERY / RESOLVEHTTP

Identify IMGs properly across protocols: IMG URN (yet tbd.)Mappings to individual protocols for actual processing

© 2007 Jörg Ott


19

IMG ANNOUNCE: Reliable Multicast

Layered Coding Transport (LCT)Single sender multicast transportDefines single or multi-object delivery across an LCT session

Provides identifiers for objects (TOI)Provides session identification (TSI)

LCT session comprises a group of channelsEach identified by the respective (multicast) transport address

Forward Error Correction (FEC)General container for various FEC schemesAlows to identify payload + provides in-band signaling of FEC parameters

Asynchronous Layered Coding (ALC)Simple combination of LCT and FEC

© 2007 Jörg Ott


20

IMG ANNOUNCE: FLUTE BasicsFile Delivery over Unidirectional TransportUses ALC (= LCT + FEC)

Fixed parameter sets for the protocol instantiation

Specifies semantics of objectsFilesFile Delivery Table (FDT)

FDTXML-based format to carry file attributes (name, location, size, etc.)

Carried as Transport Object ID = 0

Transmitted in a carrousel style together with files

© 2007 Jörg Ott


21

IMG ANNOUNCE: FLUTE FDTXML-based structured information

Example<FDT-Payload Expires="<date>" complete="true">

<FileContent-Location=TOI=Content-Length=Transfer-Length=Content-Type=Content-Encoding=Content-MD5=... plus some FEC stuff ... >

<File ...>...

</FDT-Payload>

© 2007 Jörg Ott


22

IMG ANNOUNCE: MUPPETSpecific usage of FLUTE for carrying IMG envelopesDefines various lower layer parametersDefines usage of multiple layers

© 2007 Jörg Ott


23

IMG QUERY / RESOLVE“Naturally” maps to HTTP GET + 200 OKHTTP URI: http://<hostname>/<resource>?param1&param2&...

Parameters identify IMG versiontype: full or delta IMG, pointerversion requesteddiffVersion: base for delta IMG

Querier response format selectionAccept: application/img-envelope+xml

Provide IMG in envelope formatAccept: text/plain, text/html

Provide a human-readable description of an IMG as optional fallbackAllow for directly returning the plain metadata without envelope?

200 OK carries response in bodyHTTP headers used accordingly

© 2007 Jörg Ott


24

IMG SUBSCRIBE / NOTIFYBased upon the Session Initiation Protocol (SIP)

Particularly its SUBSCRIBE / NOTIFY mechanismDetails to be discussed

SUBSCRIBE / NOTIFYRegister interest in (part of) an IMGReceive an immediate response and updates upon changesSoft-state based: subscription times out and needs refreshing

IMG usage of SIP SUBSCRIBE / NOTIFYDefine SIP event package: imgPresently suggests a MIME-based IMG envelope

Natural choice for SIP

Content-Type:, Content-Location:Content-ID: major.minor, Expires: valid-until

© 2007 Jörg Ott


25

Regionalization & Personalization with IMGs

IMGSender

IMGMetadata

IMGMetadata

IMGMetadata

IMGMetadata

IMGMetadata

IMGMetadata

IMGMetadata

IMGMetadata

IMGMetadata

IMGSender

IMGTransceiver

Localcontents

Globalcontents

Creation of IMGs

Creation of IMGs

IMGTransceiver

ContentBroker

Aggregation

IMGMetadata

Filtering +Augmenting

IMGTransceiver

Serviceprovider

Filtering

IMGreceiver

Set-top BoxConsumption

© 2007 Jörg Ott


26

TV-EPG DistributionTVNetworkWebsite

XMLTV

IMGSender

HTTP HTML

WebScraping

Tool

FreevoHTPC

IMG EnvelopeXMLTV

IMGReceiver

IMGTransport

© 2007 Jörg Ott


27

IMGs: “Final” RemarksContent formats: various

Simple tables in DVB/MPEG (backwards compatibility)XML-based data sets for IMGs

IMGs in use in 3GPP MBMSStalled in the IETF (further work abandoned)

TV industry going various other waysSpecific EPGs in DVBTV Anytime forumWeb/RSS-based program pages of TV magazines and broadcastersOpen source platforms use yet other formatsXMLTV


© 2007 Jörg Ott 28

Media Streaming in the Internet

Introduction to Media Streaming Real-time Streaming Protocol (RTSP)HTTP-based Streaming

© 2007 Jörg Ott


29

Real-time Media StreamingRetrieving content from a source where

the content is continuous in nature(e.g. audio, video),

the content is (potentially) presented to the user before it has been downloaded entirely, and

there is no human-to-human interaction involved (i.e. latencies are acceptable to a certain degree),

yet there may be a need for interactive streaming controls (possibly realized in a distributed fashion across sender and receiver)

Contrast: interactive, interpersonal communications

© 2007 Jörg Ott


30

Two Types of StreamingBroadcast streaming (non-interactive)

Sender transmits media stream according to its own scheduleReceivers “tune into a media stream” of interestedReceivers have no means to influence the transmissionSuitable for multicast / broadcast networks

Interactive streamingSender provides media stream to receivers “on demand”Receivers may start / stop transmissionReceivers may invoke further operations

Fast forward, search, play offset, …Suitable for P2P sessions or coordinated small groups

© 2007 Jörg Ott


31

Architectural ComponentsContent Description

Describe type of content, format, access methods, ...SDP, SDPng, IMGs, MPEG tables, proprietary formats, ...

Content Description Delivery / Access ProtocolDelivers Content DescriptionHTTP, SMTP, NNTP, SAP, proprietary protocols, ...

Content Access (= Media Streaming) ProtocolInitiates, controls, and terminates media streamsRTSP, proprietary protocols, ...

Content Delivery (= Media Transport) ProtocolCarries the actual contentRTP/RTCP, HTTP, proprietary protocols, ...

© 2007 Jörg Ott


32

Conceptual Overview

Clients

WebServer

MediaServer

Announce-ment

Server

2. ContentDescription

1. Reference toMedia Server

MediaServer

SAP HTTP RTSP

© 2007 Jörg Ott


33

Conceptual Overview

Clients

WebServer

MediaServer

Announce-ment

ServerMediaServer

Control sessions

Media sessions

© 2007 Jörg Ott


34

Variants of Media StreamingFrom a service provider

Via a broadcast networkBroadcastingAdvanced multicast-based video-on-demand

Specific support for the last mileTV-over-DSL (and other Internet access links)

Video-on-DemandIntegrated with the webUsing dedicated network links

In a private householdFrom a server to one or more home devices

Community-based: Peer-to-peerVia the Internet between consumersAssisting service providers

© 2007 Jörg Ott


35

Real-Time Streaming Protocol (RTSP)RFC 2326 (“buggy”, “underspecified”)draft-ietf-mmusic-rfc2326bis-15.txt

Interactive streaming control in the InternetMedia servers provide media streams to users on demandContent described by presentation descriptions

“Network Remote Control” of a media serverPLAY [and RECORD]Numerous options for media control

PAUSE, faster / slower playback, selection of ranges from a stream, ...

© 2007 Jörg Ott


36

RTSP ScenariosMediaServer

MediaServer

MediaServer

© 2007 Jörg Ott


37

Protocol CharacteristicsBorrows heavily from HTTP

Syntax, quite a bit of semantics, parts of the architecture

Important differencesServers may issue requests, too!

Symmetric communication

Servers are statefulDifferent methodsDifferent headers

But many HTTP headers re-used

Entities (=request/response bodies) only describe contentContent itself (=media) is carried out of band

e.g. in RTP; also support for interleaving of media with RTSP connection

Transport: TCP [or UDP]Reliability handled at the RTSP level

© 2007 Jörg Ott


38

RTSP Components

ContentDescription

MediaServer

Video

Audio - 1

Audio - 2

/movies/matrix

media-server.tkk.fi

Content Base

/movies/matrix/video/movies/matrix/audio/de/movies/matrix/audio/en

•rtsp://media-server.tkk.fi/movies/matrix/audio/en

© 2007 Jörg Ott


39

RTSP URIsSchemes:

rtsp: reliable, connection-oriented (TCP)rtspu: potentially unreliable, connectionless (UDP)rtsps: secure, reliable, connection-oriented (TLS)

General scheme:rtsp:// host / local identifier

HostShould be DNS nameSupport for IPv4; IPv6 now being added

Local IdentifierOpaque; may be used for aggregate / non-aggregate control

© 2007 Jörg Ott


40

Time in RTSPSMPTE Timestamps

SMPTE = Society of Motion Picture Television EngineersMeasured in hours, minutes, seconds, frames, fractions (subframes)

29.97 or 25 frames per second (default: 29.97)Human readable HHH:MM:SS:FF.ff 3:47:09:10.25

Normal Play Time (NPT ≠ NTP)Relative to beginning of streamIn seconds: SS.fff 10.74In human readable time: HHH:MM:SS.fff 3:47:09.314159

Absolute TimeUsing ISO 8601 format20021211T101435.89Z

(RTP Media Time)Media-specific clock for the RTP timestampSynchronized with absolute time via RTCP

© 2007 Jörg Ott


41

RTSP SessionsShared state between RTSP client and server

Establish by SETUP messageRemoved by TEARDOWN

Or due to some timeoutIndependent of underlying TCP connections

TCP connections may be closed and re-opened during a single RTSP sessionTypically bound to a single presentation

in case of SDP, valid for one SDP session (description) May contain several RTP sessions

e.g. one per media stream

© 2007 Jörg Ott


42

RTSP Request MessageSETUP rtsp://ms.tkk.fi/movies/matrix RTSP/1.0CSeq: 302Date: 10 Dec 2002 15:35:06 GMTSession: 47112344Transport: RTP/AVP;unicast;

client_port=4588-4589<CRLF>[Optional Message Body]

© 2007 Jörg Ott


43

RTSP Response MessageRTSP/1.0 200 OKCSeq: 302Date: 10 Dec 2002 15:35:07 GMTServer: Matrix-Server 0.4.2Session: 47112344Transport: RTP/AVP;unicast;

client_port=4588-4589;server_port=6256-6257<CRLF>[Optional Message Body]

© 2007 Jörg Ott


44

RTSP Protocol Operation: DESCRIBEObtain presentation description from server

e.g. SDPMedia initialization

Contains information about all embbeded media streamsSupport for aggregate / non-aggregate controlAllows a client to determinesuitability of contentChoose encoding if possible

Optional: description may be obtainedout-of-band

Client ServerDESCRIBE

200 OK + SDP

© 2007 Jörg Ott


45

RTSP Protocol Operation: ANNOUNCEUpdates the presentation description actively from the server

e.g. add or remove media streamsMay be issued at any time


200 OK + SDP

ANNOUNCE + SDP

200 OK

© 2007 Jörg Ott


46

RTSP Protocol Operation: SETUPInitiate an RTSP sessionReserve resources at the server

Server may redirect to other servers(e.g. if busy)

Convey transport parameters formedia sessions

Negotiate transport protocole.g. RTP/UDP vs. tunnelingEnable firewalls to open holes


200 OK + SDP

SETUP + transport

200 OK + transport

© 2007 Jörg Ott


47

RTSP Protocol Operation: PLAYStart streamingAllows to specify a variety of streaming operations

Range(s) to play= seek operationE.g. 10-20s; 30-45s; 60s-

Forward / backwardSpeed

+3.0- 2.5


200 OK + SDP

SETUP + transport

200 OK + transport

PLAY [range]

200 OK

© 2007 Jörg Ott


48

RTSP Protocol Operation: PAUSEInterrupt streaming

But keep resources allocatedMay take effect

Immediately orAt a specified point in time

PLAY may be used to resumestreaming


200 OK + SDP

SETUP + transport

200 OK + transport

PLAY [range]

200 OK

PAUSE [time]

200 OK [range]

© 2007 Jörg Ott


49

RTSP Protocol Operation: TEARDOWNStop streamingTerminate RTSP session

Free resourcesTakes effect immediately


200 OK + SDP

SETUP + transport

200 OK + transport

PLAY [range]

200 OK

PAUSE [time]

200 OK [range]

TEARDOWN

200 OK

© 2007 Jörg Ott


50

RTSP MethodsOPTIONSDESCRIBE, ANNOUNCESETUP, TEARDOWNPLAY, PAUSEREDIRECT

May be used by a server to refer a client to a different location

GET_PARAMETERRetrieve parameter value specified in the header (in the Session: context)

Returned in 200 OK response body as “Name: value” pairsMay be used for keep-alive purposes

SET_PARAMETERSet value of parameter(s) per response body (“Name: value” pairs)

[RECORD]Record a media stream at a serverUnderspecified, not really suppored, now removed from base spec

© 2007 Jörg Ott


51

RTSP General Header Fields(For reference only)

Cache-Control:Connection:CSeq:Date:Timestamp:Via:

© 2007 Jörg Ott


52

RTSP Request Header Fields(For reference only)

Accept:, Accept-Encoding:, Accept-Language:Authorization:Bandwidth:Blocksize:From:If-Modified-Since:Require:, Proxy-Require:, Supported:Referer:Scale:, Speed:, Range:Session:Transport:User-Agent:

© 2007 Jörg Ott


53

Some Response Status Codes100 Continue200 OK / 201 Created300 Multiple Choices301 Moved Permanently / 302 Moved Temporarily304 Not Modified305 Use Proxy400 Bad Request401 Unauthorized / 407 Proxy Authentication Required403 Forbidden404 Not Found405 Method Not Allowed / 406 Not Acceptable / 408 Request Timeout451 Parameter Not Understood454 Session Not Found455 Method vot valid in this State / 457 Invalid Range461 Unsupported Transport500 Internal Server Error / 501 Not Implemented / 551 Option not Supported

© 2007 Jörg Ott


54

Response Header Fields(For reference only)

Accept-Ranges:Proxy-Authenticate: / WWW-Authenticate:Public:Location:Range: / Scale: / Speed:Retry-After:RTP-Info:Transport:Unsupported:Vary:Session:

© 2007 Jörg Ott


55

EntitiesEntities contained in RTSP messages are typically presentation descriptions

e.g. an SDP message(Content-Type: application/sdp)Should always fully specify the media stream(s)

Header fields:Content-Length:, Content-Type:, Content-Encoding:,Content-Base:, Content-Location:, Content-Language:Allow:Last-Modified:, Expires:

© 2007 Jörg Ott


56

InterleavingRTSP should use RTP/UDP for media streaming

Not always feasible (e.g. firewall, see next slide)

Interleaving of RTSP and media dataEscape binary data (“$”)Define multiple “channels”Specify packet length in binaryYields a four byte header:

Interleaved with RTSP messagesStarts right after previous messageLength used to determine how many bytes to skip / pass

$ ch length

© 2007 Jörg Ott


57

RTSP 2.0Presently under development (well advanced)draft-ietf-mmusic-rfc2326bis-15.txt

Tons of editorial changes (readability, coherence, …!)Better state machine descriptionsUpdated (more coherent) semantics for various header fields

Significant alignment with SIP based upon experience gained there

RECORD disappeared from base specWas underspecified anyway

Support for NAT traversal upcomingdraft-ietf-mmusic-rtsp-nat-05.txt

© 2007 Jörg Ott


58

Firewall FriendlinessSeveral means to support RTSP across firewalls

Interleaving supportTransport: header indicates port numbers, IP addresses, …

Firewall logic does not need to parse SDP formatSOCKS support

Still may be insufficientFirewalls may block RTSP in the first place“Last resort”: HTTP tunneling

Really bad (dubious!)Boils down to a competition between firewall vendors and application developersDefeats the purpose of a firewall in the first place

Nevertheless: widely deployed (“HTTP streaming”)Apple, Microsoft, …

© 2007 Jörg Ott


59

RTSP: Implications for Session DescriptionsSession Announcements (SAP)

Session Descriptions (SDP) specifies fixed parameter setMay be updated by the server later on

HTTP-based retrieval of session informationSDP specifies fixed parameter set or alternativesClient gets to choose one of these

RTSP-based session initiationSDP from server describes set of alternativesClients may choose which one to useBoth sides may update their offering / choice later

Need for negotiating session parametersBoth side may provide suggestions, make choices, and update theseParticularly relevant for interactive communications

Generalized Offer/Answer model for SDP

© 2007 Jörg Ott


60

“HTTP Streaming”Tunneling media and control in an HTTP connection

Simplest caseStart replay before download is completeNo extensions neededMainly client-side operationBut: server needs to use appropriate media file format

Alternative: add additional headers (MS)Preserve packetization of media within a TCP connection

© 2007 Jörg Ott


61

Old(?) MS HTTP Streaming Format0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

+00 | Type | Chunk Length |+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

+04 | Sequence Number |+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

+08 | Flags (Unknown) | Chunk Length (again?) |+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

+12 | |: Chunk Data Block :| |+ +| |+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

© 2007 Jörg Ott


62

Sample Request Header (1/2)GET test.asf HTTP/1.0Accept: */*User-Agent: NSPlayer/4.1.0.3856“Host: media_hostPragma: no-cache,rate=1.000000,stream-time=0,stream-offset=0:0,

request-context=1,max-duration=0Pragma: xClientGUID={c77e7400-738a-11d2-9add-0020af0a3278}Connection: Close

© 2007 Jörg Ott


63

Sample Request Header (2/2)GET test.asf HTTP/1.0Accept: */*User-Agent: NSPlayer/4.1.0.3856Host: media_hostPragma: no-cache,rate=1.000000,stream-time=0,

stream-offset=0:0,request-context=2,max-duration=40“Pragma: xPlayStrm=1Pragma: xClientGUID={c77e7400-738a-11d2-9add-0020af0a3278}Pragma: stream-switch-count=1Pragma: stream-switch-entry=ffff:1:0Connection: Close

© 2007 Jörg Ott


64

Sample Response HeaderHTTP/1.1 200 OKContent-Type: application/octet-streamServer: Cougar 4.1.0.3920Cache-Control: no-cachePragma: no-cachePragma: features=“broadcast"

© 2007 Jörg Ott


65

Another Example: HTTP GETGET /media/Videos/200710/200710A0/29102007002.mp4 HTTP/1.1Content-length: 0User-Agent: Java/1.5.0_10Host: 192.168.1.100:50 004Accept: video/mp4, text/html, image/gif, image/jpeg, *; q=.2, */*; q=. 2Connection: keep-alive

© 2007 Jörg Ott


66

Another Example: 200 OKHTTP/1.1 200 OKCONTENT-TYPE: video/mp4CONTENT-LENGTH: 7667062

0040 0d 0a 0d 0a 00 00 00 1c 66 74 79 70 6d 70 34 ........ftypmp4 0050 32 00 00 00 00 6d 70 34 32 33 67 70 34 69 73 6f 2....mp423gp4iso0060 6d 00 74 b2 13 6d 64 61 74 00 00 18 83 f2 1b fb m.t..mdat.......0070 04 29 69 69 69 69 69 69 69 69 69 69 69 69 69 69 .)iiiiiiiiiiiiii0080 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 iiiiiiiiiiiiiiii0090 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 iiiiiiiiiiiiiiii00a0 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 iiiiiiiiiiiiiiii00b0 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 iiiiiiiiiiiiiiii00c0 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 iiiiiiiiiiiiiiii00d0 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 iiiiiiiiiiiiiiii00e0 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 iiiiiiiiiiiiiiii00f0 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 iiiiiiiiiiiiiiii0100 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 69 iiiiiiiiiiiiiiii

© 2007 Jörg Ott


67

Home Media Streaming ArchitecturesNo single coherent solution at this point

Different camps follow different approachesApple vs. other industry consortia vs. (operator) standardization bodies vs. …

But architectural similaritiesDevices need to support zero/autoconfiguration

No need for any kind of setup interaction (“plug and play”)Devices need to be able to find one anotherDevices need to determine each others’ capabilities

Self-descriptionsDevices need to discover resources available on/from another devicesDevices need to engage in communications and deliver media streams

Example: DLNA: Digital Living Network AllianceDesign for small scale and closed deployments (home networks)Uses Universal Plug and Play (UPnP) and UPnP AV

© 2007 Jörg Ott


68

Example: DLNAAutoconfiguration

DHCP (if a DHCP server is present)Zero configuration: IPv4 link local address configuration (RFC 3927)

Device discovery: UPnPBased upon something called “UHTTP”: HTTP syntax over UDP packetsSent in regular intervals (no scaling as with RTCP or SAP!)

NOTIFY * HTTP/1.1LOCATION: http://192.168.1.100:50004/MediaServer1/MediaServer1.xmlHOST: 239.255.255.250:1900SERVER: Symbian/9.2 UPnP/1.0 Nokia/N95NTS: ssdp:aliveUSN: uuid:d8c66d26-1b20-10e1-9c90-001CD45CCA96CACHE-CONTROL: max-age=1800NT: uuid:d8c66d26-1b20-10e1-9c90-001CD45CCA96

© 2007 Jörg Ott


69

Example: DLNADevice capability assessment

HTTP-based (over TCP) query-response protocolSimple Service Discovery Protocol (SSDP)Retrieval of an XML-based service description

Resource discoveryBased upon the device capabilities(e.g., media server profile)Simple Object Access Protocol (SOAP) RPC

XML-encoded synchronous RPCs carried over HTTP

Example: get a “directory listing”Using naming conventions for folders to locate contentsYields URIs to access each individual media resource

Media streamingHTTP streaming: GET on the URI of the media resource

Client ServerUHTTP NOTIFY

UHTTP NOTIFY

GET server.xml

200 OK + XML file

POST XML request

200 OK + XML response

Contains media URI

GET media URI

200 OK + media data


© 2007 Jörg Ott 70

Third party media resource control

© 2007 Jörg Ott


71

Media Resource Control Protocol (MRCPv2)Another protocol to control media resources

Based upon a proprietary version by Cisco et al. (MRCPv1, RFC 4443)

Enable a client to task a third entity to perform on its behalfMedia stream generation (basic and advanced speech synthesis)Media processing (recording, DTMF/speech recognition, speaker verification)

ControllingClient

RemotePeer

MediaResource

Server

Call signaling (e.g., SIP)

Media stream (RTP)MRCPv2

© 2007 Jörg Ott


72

MRCPv2 Overview (1)MRCPv2 defines a common framework for rather different application classes

CommonalitiesMedia stream consumption or generation by a media resource serverControl of the media stream generation or processing by the clientReport on media stream contents, characteristics, and resource server status

Text-based protocolStart line + headers + message bodyBorrows heavily from HTTP and RTSPYet, subtle differences (later)Message bodies identified by entity headers (using MIME types, etc.)

Symmetric operationBoth peers can initiate actions: Methods (client->server), Events (server->client)Headers + contents to parameterize operations or deliver results

© 2007 Jörg Ott


73

MRCPv2 Overview (2)Uses TCP as underlying transport (+ optional TLS)

Reliability required; limited real-time interaction requirements only (true?)Or do we assume sufficiently well interconnected clients and media resources

One of more TCP connections multiplexedConcept of logical channels

Uses RTP for media streamsExplicit correlation to TCP control channels in SDP using new grouping

Relies on SDP offer/answer (using SIP) for session setupConnection-oriented media (TCP, TLS) as well as RTP sessions

© 2007 Jörg Ott


74

MRCP Overview (2)

ControllingClient

MediaResource

Server

TCP connection 1

TCP connection 2

SIP RTP

SIP+SDPv=0o=sarvi …44526 …22808 IN IP4 126.16.64.4s=-c=IN IP4 126.16.64.4m=application 9 TCP/MRCPv2a=setup:activea=connection:newa=resource:speechsyntha=cmid:1m=audio 49170 RTP/AVP 0 96a=rtpmap:0 pcmu/8000a=recvonlya=mid:1

Answer: a=channel:32AECB234338@speechsynth

Correlation

© 2007 Jörg Ott


75

MRCP PackagesDifferent command sets defined for different packets

Building upon a small common subset of protocol elementsOtherwise largely independent of one anotherMethods and events, response codesHeader fieldsContent types (references to externally defined content formats)

One package type per applicationSpeech RecognitionDTMF RecognitionBasic synthesisSpeech synthesisSpeaker verificationRecording

Highly specialized for the specific application domainYou wonder why all this stuff goes into a single spec

© 2007 Jörg Ott


76

Simple Example: Recording (1)Methods

RECORD — start recordingSTOP — stop recordingSTART-INPUT-TIMERS — configuration

EventsSTART-OF-INPUT — media stream recording has begunRECORD-COMPLETE — recording done

Some useful headersSensitivity-Level — for silence suppressionMedia-Type — what to recordRecord-URI — where to store recordingTrim-Length — limit length of recordingCapture-on-Speech — wait for speechVarious timeouts for input sensing, end of recording, …

Message bodiesCaptured recording (unless stored at a URI)

© 2007 Jörg Ott


77

Simple Example: Recording (2)C->S: MRCP/2.0 386 RECORD 543257

Channel-Identifier:32AECB23433802@recorderRecord-URI:<file://mediaserver/recordings/myfile.wav>Capture-On-Speech:trueFinal-Silence:300Max-Time:6000

S->C: MRCP/2.0 48 456234 200 IN-PROGRESSChannel-Identifier:32AECB23433802@recorder

S->C: MRCP/2/0 49 START-OF-INPUT 456234 IN-PROGRESSChannel-Identifier:32AECB23433802@recorder

S->C: MRCP/2.0 54 RECORD-COMPLETE 456234 COMPLETEChannel-Identifier:32AECB23433802@recorderCompletion-Cause:000 success-silenceRecord-URI:<file://mediaserver/recordings/myfile.wav>;

size=242552;duration=25645

© 2007 Jörg Ott


78

More Media ControlMedia Gateway Control Protocol (MEGACOP)

Configuring (PSTN) media gateways for IP telephonyControlling media resource functions in 3GPP

Media Server Control Markup Language and ProtocolControlling conference serversControlling Interactive Voice Response (IVR) systems

MEDIACTL WG in the IETF (newly created last week)

Lots of non-IETF work (e.g., W3C)

Gains importance in the context of service creation for interpersonal communications (using SIP)


© 2007 Jörg Ott 79

Broadcast Streaming

From television broadcast to video-on-demand

© 2007 Jörg Ott


80

Topologies

Server Server

R R

© 2007 Jörg Ott


81

Media Stream EncapsulationTraditional DVB: MPEG Transport Streams (TS)

Uses 188 byte packets of which 184 bytes are payload (3 x 48 = 184)Multiplexes different “channels”

Provides multiplex for different media streams on a single “channel”Audio, video, dataTime synchronization, supplementary information, etc.

MPEG-TS-over-RTP (RFC 2250)Maintaining the per channel multiplexDifferentiating different “channels” via IP multicast addresses

MPEG audio/video/… in separate RTP streamsMultiplexed via IP/UDP onlyIndependent RTP encapsulation

Video: RFC 3016, RFC 3984Audio: RFC 3016, RFC 4184, RFC 4598

Binding channels together via, e.g., SDP descriptions

© 2007 Jörg Ott


82

Broadcasting SchemesAssuming no or little per receiver interaction

No control as in RTSP-based on-demand point-to-point streamingPossibly indicating a desire to receive a particular stream

But no/little direct impact on scheduling

Inherent tradeoff betweenUser satisfaction

measured in the waiting time until the stream starts from the beginningFlexibility to decide what to see and not to miss anything

Server and link capacity (transmission bit rate R_t) required to provision the media streams to a large audience

Further parametersReceiver-side link (and hardware reception) capacity per media streamReceiver-side memory required (= cost, feasibility), also complexityStream bit rate (for regular playback) R_s

© 2007 Jörg Ott


83

Traditional SchemesWhich are actually in use…

File-based distributionMovies are distributed using a reliable multicast protocol

Possibly involving some interactive repair schemes or just erasure coding/FECSelected genres (or specific movies) are stored on a user’s receiver device

Possibly encrypted if they are pay-per-viewUsually requires a hard drive as mass storage

Plus some replacement policy to manage storage spaceViewed by the user whenever convenientDrawbacks

Limited storage capacityLimited flexibility and choiceNot really streaming

Some of the drawbacks mitigated by pull-based peer-to-peer file distributionE.g., more choiceBut users still need to decide in advance what they want to view

© 2007 Jörg Ott


84

Simple Schemes: Two Extreme PointsSimple television-style broadcast (proactive scheme)

Broadcast as per announced scheduleReceiver has to be aware or may miss the streamServer load: O(1)

One stream for allStreamed at the stream bit rate R_t = R_s

Client load: O(1)Received at the stream bit rate R_r = R_sReceived and played back instantly (after dejittering)No memory needed

Simple interactive video-on-demand (reactive scheme)Unicast whenever requested by the userServer load: O(n)

One stream per receiver, streamed at the stream bit rateN independent streams for N receivers: R_t = N x R_s

Client load: O(1)Received at the stream bit rate R_r = R_sReceived and played back instantly (after dejittering)No memory needed

C1

R1R2R3

R1R2R3

C1C2C3

© 2007 Jörg Ott


85

User Demand vs. Server LoadMedia stream demand follows a Zipf distribution (s=0.729)

[Source: Boris Nikolaus, 2007]

© 2007 Jörg Ott


86

User Demand vs. Server Load (2)Requests for a specific media stream follow a Poisson distribution


© 2007 Jörg Ott


87

Variations of Reactive SchemesBatching

Aggregate requests and start streaming delayedAfter the number of requests exceeds a thresholdAfter a maximum waiting time passedVarious scheduling policies conceivable

FCFS, priorities, sort by request frequency, …Fix the maximum number of transmission channelsServer load: depends on the batching granularityClient load: O(1)

Patching Clients receive pieces of media streams for other clients

Implies that a broadcast/multicast transmission medium is usedReceivers buffer a fraction of the other media stream

Delays playoutSender transmits the missing pieces in parallel

These pieces played out immediatelyMay be expanded across multiple streams (“Tapping”, “Merging”)Virtual batching: exploits media stream exchange between receivers

R1R2R3

C1C2

R1R2R3

C1C2

© 2007 Jörg Ott


88

Pro-active Broadcasting SchemesNon-segmenting (keeps stream as a whole)Round-robin broadcasting

Server repeats broadcasts according to a scheduleOnly repeats a single movie (e.g., every two hours)Repeats a sequence of movies per channel

Receiver waits until the next scheduled timeServer load: O(1)Client load: O(1)

Staggered broadcastingServer continuously broadcasts each streamUses multiple channels with interleaved start timesServer load: O(C) (C is the number of channels)Client load: O(1)Mean waiting time: Movie length / # channels

R1R2R3

C1

R1R2R3

C1C2

© 2007 Jörg Ott


89

Pro-active Broadcasting Schemes (2)Segmenting schemes

Subdivide the media stream into segmentsEqual sizeDifferent sizes

SenderTransmits the segments on one or more channels

With different frequenciesAt different data rates

Segment transmission is ordered and Segment assignment to channels according to apre-defined schedule (determined via some algorithm)Load: O(C_t) (C_t = number of transmission channels)

ReceiverJoins the right channels at the right time (knowing the transmission scheme)Receives from one or more channels at the same timeBuffers segments for later playout (unless they will be sent again prior to playout)

Maximum required buffer size is a limiting factor for client equipmentLoad: O(C_r) (C_r = number of channels to be joined simultaneously)


© 2007 Jörg Ott


90

Size-Based: Pyramid Broadcasting

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

2 2 2 2 2 2 2 2 2

3 3 3 3 3 3

R1

Worst case delay: 2 x transmission duration of segment #1Channel bit rate C_t must be much higher than the stream bit rate C_s

Further variants:• Maya Temple Broadcasting• Skyscraper Broadcasting• and others…

© 2007 Jörg Ott


91

Bit rate-based: Harmonic Broadcasting

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

2 2 2 2 2 2 2 2 2

3 3 3 3 3 3

4 4 4 4

5 5 5

6 6 6

7 7

8 8

9 9

R1 1 2 3 4 5 6 7 8 9

Can be realized viaSegment frequencyor segment rate

© 2007 Jörg Ott


92

Harmonic Broadcasting (variant)

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

2 2 2 2 2 2 2 2 23 33 3 3 3

4 4 4 455 56 66 77

88 99

R1 1 2 3 4 5 6 7 8 9

3 33

5 6 7 7 5 6

Further schemes:Cautious harmonic broadcastingQuasi-harmonic broadcastingPolyharmonic broadcastingStaircase broadcasting (and related ones)Greedy broadcasting

Internet Media Guides (IMGs) · IP + UDP/TCP + RTP/… vs. MPEG multiplex (or even analog)...

Documents

Transcript of Internet Media Guides (IMGs) · IP + UDP/TCP + RTP/… vs. MPEG multiplex (or even analog)...