Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 ›...
Transcript of Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 ›...
![Page 1: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/1.jpg)
Lecture 7: Internet Streaming Media
Reji MathewNICTA & CSE UNSW
COMP9519 Multimedia SystemsS2 2007
![Page 2: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/2.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 2 – Reji Mathew
Notes on Previous Lecture
� RTCP Packets� SR and RR can be used for independent network management� Payload independent monitors that receive only the RTCP packets
to evaluate the performance of their networks for multicast distribution
![Page 3: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/3.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 3 – Reji Mathew
IntroductionThe story so far ….
� Video Coding – MPEG-4� Audio Coding – u-Law, AMR
� Packetization and Transport of media – RTP� Transport Control / QoS Feedback - RTCP
Lecture Outline� Literature Review
� Client Architecture : An Example
� Next � Further Protocols Enabling Streaming � SDP - Description� RTSP – Signaling� SIP
![Page 4: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/4.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 4 – Reji Mathew
Client Architecture : An Example� Reference
� “Client architecture for MPEG-4 streaming”, Haifeng Xu Diamand, J. Luthra, A. IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages 16-23
� Review of � Streaming Video, RTP, MPEG-4
� Client Architecture for streaming MPEG-4 video� Multilevel buffer system
![Page 5: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/5.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 5 – Reji Mathew
Client Architecture : An Example
“Client architecture for MPEG-4 streaming”, Haifeng Xu Diamand, J. Luthra, A. IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages 16-23
Multiple buffer architectureto allow efficient control of media
processing and presentation.
Three Modules
• Network Module : Packet buffer• Codec Module : Frame Buffer• Presentation Module : Image
Buffer
![Page 6: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/6.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 6 – Reji Mathew
Client Architecture : Multiple Buffers
“Client architecture for MPEG-4 streaming”, Haifeng Xu Diamand, J. Luthra, A. IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages 16-23
Packet Buffer
Frame BufferImage Buffer
![Page 7: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/7.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 7 – Reji Mathew
Client Architecture : Multiple Buffers� Packet Buffer
� Reorder packets� Identify missing packets (allow for easy insertion)� Ignores delayed packets (outside moving window)
� Frame Buffer� Stores video packets (smoothing out network jitter)� Allows dropping of frame (when lacking CPU resources)� Identify frames to be dropped (B, P)
� Image Buffer� Stores decoded frames for rendering � Smoothing decoders speed
![Page 8: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/8.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 8 – Reji Mathew
Client Architecture : Rendering
Rendered frames rates with and without the image buffer
“Client architecture for MPEG-4 streaming”, Haifeng Xu Diamand, J. Luthra, A. IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages 16-23
![Page 9: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/9.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 9 – Reji Mathew
Client Architecture : Conclusion� The MPEG-4 MLBA subsystem facilitates three
player-related activities:
� precise A/V synchronization,� client-based QoS management, and� improved rendering performance through an image buffer.
“Client architecture for MPEG-4 streaming”, Haifeng Xu Diamand, J. Luthra, A. IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages 16-23
![Page 10: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/10.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 10 – Reji Mathew
Lecture Outline� Session Description Protocol
� How to describe a multimedia session ?� SDP example
� RTSP – Internet VCR controls� Stop, Pause, Play, Fast Forward over the internet� RTSP Methods� Signal Timing Diagram� Protocol stack for a total streaming system
� SIP – Quick Overview
� MPEG-4 File Format� What’s so good about MP4?� File Format and Structure� Hinting
![Page 11: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/11.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 11 – Reji Mathew
Introduction� Example : An existing live multicast session
� Video and Audio streams� Transport - RTP/UDP/IP, Control – RTCP/UDP/IP
� A new client wanting to join the multicast session� Needs to know
� multicast IP address and port� Media streams in a session (e.g. video only or audio + video)� Payload format (e.g. MPEG-4 video, AMR audio)� Initialization data for video and audio decoders� Transport protocol used� Other information ….
� Need a way to describe a multimedia session� To enable new clients to easily join the session
![Page 12: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/12.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 12 – Reji Mathew
SDP� Session Description Protocol (SDP)
� IETF – RFC2327� www.ietf.org/rfc/rfc2327.txt
� For describing multimedia sessions� To communicate the existence of a session� To convey sufficient information to join a session
� Simple text format
� Defined to be general purpose� Can be used for a wide range of network environments� And applications
![Page 13: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/13.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 13 – Reji Mathew
SDP� SDP includes
� Session name and purpose� Time the session is active� The media comprising the session� Information to receive media (addresses, ports, formats)� Information about bandwidth to be used� Contact information of a person responsible for the session
� SDP is used by other signaling / initiation protocols� SIP : Session Initiation Protocol� RTSP : Real-time Streaming Protocol
![Page 14: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/14.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 14 – Reji Mathew
SDP
v= (protocol version)o= (owner and session identifier)s= (session name)i=* (session information)u=* (URI of description)e=* (email address)p=* (phone number)c=* (connection information)b=* (bandwidth information)z=* (time zone adjustments)k=* (encryption key)a=* (zero or more session attribute)t= (time the session is active)r=* (zero or more repeat times)
m= (media & transport address)i=* (media title)c=* (connection information)b=* (bandwidth information)k=* (encryption key)a=* (zero or more media attribute)
session level description
media level description
SDP session description consists of a number of lines of text of the form
<type>=<value>
<type> is always exactly one character and is case-significant.
![Page 15: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/15.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 15 – Reji Mathew
SDP
v=0o=mmvc 2890844526 2890842807 IN IP4 129.94.135.201s=Camera ONEi=Video stream for realtime surveillanceu=http://www.nicta.com/mmvc/demos/[email protected] (Reji Mathew)c=IN IP4 225.0.0.37/2t=0 0a=recvonlym=video 20000 RTP/AVP 98a=rtpmap:98 MP4V-ES/90000a=fmtp:98 profile-level-id=1; config=000001b001000001…a=orient:portrait
Version number
Originator information<username> <session id> <version> <network type> <address type> <address> (address of machine from which the session was created)
Session nameSession Information
(text description / title for session)URL for more information about the session
Contact person e-mail (person responsible not necessarily the creator)Connection Details
<network type> <address type> <connection address>IN (internet) IP4 (IP v4) address (multicast address/TTL)Time Information
<start time> <end time>(ntp time, “0,0” implies permanent session)
Session level attribute(operate in receive only mode)
Media Announcement<media> <port> <transport> <fmt list>(fmt : refers to media format specific information)(e.g. Dynamic payload number)
Media attribute (rtp map)<payload type> <encoding name>/<clock rate>(map RTP dynamic payload number to media format and clock rate)
Media attribute (format specific parameters)(info regarding mpeg-4 media, profile, level & initialization data)
Media attribute (orientation)(only used in some applications, example – landscape or portrait)
increased when a modification is made to the session data. Recommended that an NTP timestamp is used
![Page 16: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/16.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 16 – Reji Mathew
System Overview� Example – Using SDP to join a multicast session
� Request SDP file via HTTP� Retrieve information from downloaded SDP file� Receive RTP streams on SDP specifed address & port� Decode and display specified media
HTTP GET
SDP file
http://www.nicta.com/lecture.sdp
RTP Media Stream
RTCPMedia Server
Client
![Page 17: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/17.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 17 – Reji Mathew
System Overview� Example – Traffic Monitoring
� Continuous multicast streaming of video � A client can receive the stream
� by downloading SDP via http (web browser)� Provide SDP file to QuickTime player� Player initializes and waits for stream data
� Easily support multiple client
� Demo Link
![Page 18: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/18.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 18 – Reji Mathew
Lecture Outline� Session Description Protocol
� How to describe a multimedia session ?� SDP example
� RTSP – Internet VCR controls� Stop, Pause, Play, Fast Forward over the internet� RTSP Methods� Signal Timing Diagram� Protocol stack for a total streaming system
� SIP – Quick Overview
� MPEG-4 File Format� What’s so good about MP4?� File Format and Structure� Hinting
![Page 19: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/19.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 19 – Reji Mathew
RTSP� Now we can describe and transport media streams
� But how to control these streams� Start, Stop, Pause, Fast Forward, Rewind� “internet VCR”
� Solution – RTSP� Real Time Streaming Protocol� Establishes and controls one or more continuous media
streams - such as audio and video
� Similar in syntax and operation to HTTP/1.1� Client –Server protocol� Text based
� IETF RFC 2326
![Page 20: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/20.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 20 – Reji Mathew
RTSP� More Info
� www.ietf.org/rfc/rfc2326.txt� www.rtsp.org
� Example
Play Request (TCP/IP)
Play Response (TCP/IP)
Media Server Client
PLAY rtsp://audio.example.com/twister.en RTSP/1.0CSeq: 833Session: 12345678Range: smpte=0:10:20-;time=19970123T153600Z
RTSP/1.0 200 OKCSeq: 833Date: 23 Jan 1997 15:35:06 GMTRange: smpte=0:10:22-;time=19970123T153600Z
![Page 21: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/21.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 21 – Reji Mathew
RTSP� Protocol Properties
� Media Transport Independent� RTSP does not depend on the transport mechanism used to
carry the continuous media� Media transport can be via RTP or some other mechanism
� Transport-independent:� RTSP may use either an unreliable datagram protocol (UDP)
or a reliable stream protocol such as TCP
� Control of recording devices:� Can control both recording only and playback only devices, � As well as devices that can alternate between the two modes
![Page 22: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/22.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 22 – Reji Mathew
RTSP� Protocol Properties
� Extendable:� New methods and parameters can be easily added to RTSP.
� Easy to parse
� Multi-server capable:� Each media stream within a presentation can reside on a
different server.
� HTTP-friendly:� RTSP reuses HTTP concepts, so that the existing
infrastructure can be reused.
![Page 23: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/23.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 23 – Reji Mathew
RTSP� Protocol Operation
� Text based messages between client and server� Messages can be :
� Requests� Responses
� Request Messages
Request-line
Message - header
Message - header
…
CRLF
Message - body
Contains URI of the resource
The method to be applied to the resource
The protocol version in use
May or may not be present
General and request headers
![Page 24: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/24.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 24 – Reji Mathew
RTSP� Protocol Operation
� Response Messages
� Next : examples of Request and Response messages� Corresponding to five important request methods� Methods : Describe, Setup, Play, Pause, Teardown � Unicast example
Request-line
Message - header
Message - header
…
CRLF
Message - body
The protocol version in use
Status code (3 digit code)
Textual description of status code
May or may not be present
General and response headers
![Page 25: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/25.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Reji Mathew
RTSP : Describe Method� Retrieves the description of a presentation, identified by the request
URL, from a server. The DESCRIBE reply-response pair constitutes the media initialization phase of RTSP.
C->S: DESCRIBE rtsp://nicta.com/twister RTSP/1.0CSeq: 1
S->C: RTSP/1.0 200 OKCSeq: 1Content-Type: application/sdpContent-Length: 164
v=0o=- 2890844256 2890842807 IN IP4 172.16.2.93s=RTSP Sessioni=An Example of RTSP Session Usagea=control:rtsp://nicta.com/twistert=0 0m=audio 0 RTP/AVP 0a=control:rtsp://nicta.com/twister/audiom=video 0 RTP/AVP 26a=control:rtsp://nicta.com/twister/video
Method
Presentation URL
Version
CSeq header fieldSequence number for an RTSP request-response pairMust be present in all requests and responses
Same CSeq number as requestResponse line: Version, Status code “200” with description
SDP Information (message body)
Request line
Aggregate control possible (session level control attribute)
![Page 26: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/26.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 26 – Reji Mathew
RTSP : Setup Method� Specifies the transport mechanism to be used for the streamed media
C->S: SETUP rtsp://nicta.com/twister/audio RTSP/1.0CSeq: 2Transport: RTP/AVP;unicast;client_port=8000-8001
C->S: SETUP rtsp://nicta.com/twister/video RTSP/1.0CSeq: 3Transport: RTP/AVP;unicast;client_port=8002-8003Session: 12345678
S->C: RTSP/1.0 200 OKCSeq: 2Transport: RTP/AVP;unicast;client_port=8000-8001; server_port=9000-9001Session: 12345678
S->C: RTSP/1.0 200 OKCSeq: 3Transport: RTP/AVP;unicast;client_port=8002-8003; server_port=9004-9005Session: 12345678
unicast RTP/RTCP port pair on which the client has chosen to receive media data and control information
unicast RTP/RTCP port pair on which the server has chosen to send media data and control information
RTSP session ID
Same session ID usedSetup operates on each stream separately
![Page 27: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/27.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 27 – Reji Mathew
RTSP : Play Method� Tells the server to start sending data via the mechanism specified in
SETUP request/response messages
C->S: PLAY rtsp://nicta.com/twister RTSP/1.0CSeq: 4Range: npt=0-Session: 12345678
S->C: RTSP/1.0 200 OKCSeq: 4Session: 12345678RTP-Info: url=rtsp://foo/twister/video;
seq=9810092;rtptime=3450012
Aggregate control of all media streams in the presentation.The PLAY request starts streamingboth audio and video streams.
Range header used to specify time ranges (start / stop) for playback
Normal play time (NPT) indicates the stream absolute position relative to the beginning of the presentation.
![Page 28: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/28.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 28 – Reji Mathew
RTSP : Pause Method� Causes the stream delivery to be interrupted (halted) temporarily.
C->S: PAUSE rtsp://nicta.com/twister RTSP/1.0CSeq: 5Session: 12345678
S->C: RTSP/1.0 200 OKCSeq: 5Session: 12345678
Aggregate control of all media streams in the presentation.The PAUSE request halts the streaming of both audio and video streams.
![Page 29: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/29.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 29 – Reji Mathew
RTSP : Teardown Method� Stops the stream delivery for the given URI, freeing the resources
associated with it
C->S: TEARDOWN rtsp://nicta.com/twister RTSP/1.0CSeq: 892Session: 12345678
S->C: RTSP/1.0 200 OKCSeq: 892
![Page 30: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/30.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 30 – Reji Mathew
RTSP : Signal Timing DiagramMedia
Client
Media
Server
DESCRIBE
Response (SDP)
SETUP (Video stream)
Response
SETUP (Audio stream)
Response
PLAY
Response
TEARDOWN
Response
PAUSE
Response
![Page 31: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/31.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 31 – Reji Mathew
Summary of RTSP Methods� OPTIONS : get available methods
� SETUP : establish transport
� ANNOUNCE : change description of media object
� DESCRIBE : get description of media
� PLAY : start playback, reposition
� RECORD : start recording
� REDIRECT : redirect client to new server
� PAUSE : pause delivery, while keeping state
� SET PARAMETER : device or encoding control
� TEARDOWN : remove state / close
![Page 32: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/32.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 32 – Reji Mathew
RTSP : Comparison with HTTP� Similar to HTTP, but some important differences for RTSP are :
� Data delivery takes place out-of-band in a different protocol� e.g. RTP
� Both the media client and media server can issue requests� Client > Server : SETUP, PLAY, ….� Server > Client : ANNOUNCE, REDIRECT, ….
� Requests are not stateless : client & server state machines� client changes state on receipt of replies to requests� server changes state on receiving requests
Init Ready Playing Recording
![Page 33: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/33.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 33 – Reji Mathew
Protocol Stack
IP
UDP
RTP
Decoder
packetize
RTP
UDP
RTCPRTCP
MPEG-4 File / Encoder
MPEG-4Video
TCP
RTSP
SDP
TCP
RTSP
SDP
![Page 34: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/34.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 34 – Reji Mathew
Lecture Outline� Session Description Protocol
� How to describe a multimedia session ?� SDP example
� RTSP – Internet VCR controls� Stop, Pause, Play, Fast Forward over the internet� RTSP Methods� Signal Timing Diagram� Protocol stack for a total streaming system
� SIP – Quick Overview
� MPEG-4 File Format� What’s so good about MP4?� File Format and Structure� Hinting
![Page 35: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/35.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 35 – Reji Mathew
SIP� Session Initiation Protocol (SIP)
� Another session control protocol� IETF RFC 3261 www.ietf.org/rfc/rfc3261.txt
� Protocol that can establish, modify and terminate multimedia sessions.
� Applications – IP telephony, multimedia conferences
� SIP, like RTSP, � Uses text-based request/response transaction model
� Requests contain Methods and Header fields� Responses include 3 digit status codes (eg “200 OK”)
� Is Transport layer independent� Uses other protocols for media delivery (eg RTP).
![Page 36: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/36.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 36 – Reji Mathew
SIP� SIP Methods
� INVITE, ACK, CANCEL : for setting up sessions� BYE : for terminating sessions� REGISTER : for registering contact information� OPTIONS : for querying servers about capabilities
� INVITE� User agent client initiates a session with INVITE� Invitation can be accepted by a user agent server� SIP invitations convey session descriptions that allow
participants to agree on a set of compatible media types� Offer/answer model
� ACK� Response confirmation to INVITE
![Page 37: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/37.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 37 – Reji Mathew
SIP� INVITE Example
INVITE sip:[email protected] SIP/2.0
Via:SIP/2.0/UDPpc33.atlanta.com;branch=z9hG4bK776asdhds
Max-Forwards: 70
To: Bob <sip:[email protected]>
From: Alice <sip:[email protected]>;tag=1928301774
Call-ID: [email protected]
CSeq: 314159 INVITE
Contact: <sip:[email protected]>
Content-Type: application/sdp
Content-Length: 142
(Alice's SDP not shown)
![Page 38: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/38.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 38 – Reji Mathew
SIP � SIP supports user mobility
� With proxy servers to help route requests to user’s current location
� Registration function allowing users to upload current locations for use by proxy servers
![Page 39: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/39.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 39 – Reji Mathew
SIP
Bob
atlanta.com proxy biloxi.com proxy
Alice INVITEINVITE
INVITE
200 OK200 OK
200 OK
ACK
Media Session
200 OK
BYE
![Page 40: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/40.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 40 – Reji Mathew
SIPResponseExample
SIP/2.0 200 OK
Via: SIP/2.0/UDP server10.biloxi.com;branch=z9hG4bKnashds8;received=192.0.2.3
Via: SIP/2.0/UDP bigbox3.site3.atlanta.com ;branch=z9hG4bK77ef4c2312983.1;received=192.0.2.2
Via: SIP/2.0/UDP pc33.atlanta.com ;branch=z9hG4bK776asdhds ;received=192.0.2.1
To: Bob <sip:[email protected]>;tag=a6c85cf
From: Alice <sip:[email protected]>;tag=1928301774
Call-ID: [email protected]
CSeq: 314159 INVITE
Contact: <sip:[email protected]>
Content-Type: application/sdp
Content-Length: 131
(Bob's SDP not shown)
![Page 41: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/41.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 41 – Reji Mathew
Streaming Applications
RTP RTCP RTSP SIP
MPEGSDP
IP
Video on Demand VoIP
Video Conf (3G)IPTV
![Page 42: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/42.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 42 – Reji Mathew
Streaming Media Players
� Qucik Time Player� Real Player� Microsoft Media player� Packet Video
� Players for various platforms� PC� Pocket PC� 3G Phones
![Page 43: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/43.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 43 – Reji Mathew
Lecture Outline� Session Description Protocol
� How to describe a multimedia session ?� SDP example
� RTSP – Internet VCR controls� Stop, Pause, Play, Fast Forward over the internet� RTSP Methods� Signal Timing Diagram� Protocol stack for a total streaming system
� SIP – Quick Overview
� MPEG-4 File Format� What’s so good about MP4?� File Format and Structure� Hinting
![Page 44: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/44.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 44 – Reji Mathew
File Format� Now we can code and stream video and audio
� Next we look at file formats for exchanging multimedia
� MPEG-1 & MPEG-2 content typically exchanged as files that represent a stream ready to be delivered� Embedded absolute time stamps� Fragmentation of media for some preferred transport� Random Access could be difficult
� Need a new file format that can be used when� Storing captured media� Editing stored media� Combining multiple files� Delivering as File Download or Streaming
� A “life-cycle” file format
![Page 45: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/45.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 45 – Reji Mathew
File Format� MPEG-4 file format : MP4
� “life-cycle” file format defined as part of MPEG-4� Based on Apple’s QuickTime format (“mov files”)
� Due to MP4’s flexibility and extensibility� Industry acceptance for exchange of MPEG-4 video and audio
� Basis for Motion JPEG 2000 file format� Basis for 3GPP file format used in MMS
� Multimedia Messaging Service
![Page 46: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/46.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 46 – Reji Mathew
MP4 : Main File Format Concepts� The Media data is stored separately from Meta data
� Media data : Audio, Video samples� Metadata : Data describing the media
Examples : timing info, number of bytes required for a frame
� Timing information specified by relative numbers (durations) rather than absolute numbers� Allows editing to be easier – eg insertion of a new frame
� Able to store media data distributed over several files� Use URLs to point to media data stored at various
locations
![Page 47: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/47.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 47 – Reji Mathew
MP4 : Main File Format Concepts� The Media data is stored separately from Meta data� Timing information specified by relative numbers (durations)
rather than absolute numbers� Able to store media data distributed over several files
� Locating media data by means of data offsets and length information� Metadata tables mapping media sample number to location in
a file
� Support streaming protocols through optional hint tracks� Metadata information for packetization and header data� Example – hints for RTP streaming stored as a separate track
![Page 48: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/48.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 48 – Reji Mathew
MP4: File Structure� The MP4 file format is composed of object-oriented
structures called ‘atoms’. � A unique tag and a length identify each atom.
� Metadata atoms : � A hierarchy of metadata giving information such as
� index points, durations, and pointers to the media data. � This collection of atoms is contained in an atom called
the ‘movie atom’.
� Media Data Atoms :� The media data is contained in one or more ‘mdat’ or
media data atoms, � or could be located outside the MP4 file; referenced via
URL’s.
![Page 49: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/49.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 49 – Reji Mathew
MP4: File Structure� Metadata Hierarchy [5]
Movie Atom “moov”
Audio track “trak” Video track “trak” Hint track “trak” Other MPEG-4 track “trak”
Sample table “stbl”Contains tables of sample timing and location information
![Page 50: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/50.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 50 – Reji Mathew
MP4: File Structure� Media data “mdat” Atom
� Contains interleaved, time ordered video and audio samples referred to as Access Units (AU)
� Several frames from the same track are often stored contiguously in “chunks”.
mdat
Media data
Frame 1 Frame 2 Frame 3 Frame 1 Frame 2 Frame 4
Video chunk Audio chunk
![Page 51: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/51.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 51 – Reji Mathew
MP4: File Structure� Basic structure of MP4 file
� Metedata “moov” atom� Each stream has a track “trak” atom – which contain tables� Tables with info on AU’s
� Timing, size, location (offsets), synchronization points� Tables used to point to AU in “mdat”
� Media Data “mdat” atom� Contains AU of audio and video
� Putting it together� See next slide
![Page 52: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/52.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 52 – Reji Mathew
MP4: File Structure [4]
![Page 53: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/53.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 53 – Reji Mathew
MP4: Hinting� MP4 file format is a streamable format, as opposed to
a streaming format.
� The file format facilitates streaming without ever being streamed directly
� File format does NOT define an on-the-wire protocol, and is never actually streamed over a transmission medium.
� Metadata in the file known as ‘hint tracks’ provide instructions, telling a server application how to deliver the media data over a particular delivery protocol.
� There can be multiple hint tracks for one presentation, describing how to deliver over various delivery protocols.
![Page 54: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/54.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 54 – Reji Mathew
MP4: Hinting� Remember for streaming, need to format data into
packets, with each packet containing a� Header, and� Samples or Media data� Example – RTP for MPEG-4
� Hint Track� Tables pointing to header information
� The header information is stored in “mdat”� Example – RTP header information
� Then points to appropriate fragment of sample data� Example – one video frame for an RTP packet
� See next slide
![Page 55: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/55.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 55 – Reji Mathew
MP4: Hinting [4]
![Page 56: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/56.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 56 – Reji Mathew
MP4: Hinting� What to do to accommodate different transport
protocols ?� Multiple hint tracks for one media
� Advantage to the media server ?� The server can be “dum”
� No knowledge of transport protocol required� Header information already stored in the MP4 file
� No need to do media fragmentation� Pointers to appropriate media fragments already created� No knowledge of media codec required
![Page 57: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/57.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 57 – Reji Mathew
MP4: Hinting
clip.mp4
retrieve
Media Server
No need to parse the coded media bitstream
No need to perform fragmentation
No need to create headers
Low server complexity
Generic server designClient
![Page 58: Lecture 7: Internet Streaming Media - Computer Science and ... › ~cs9519 › lecture_notes_07 › L7... · COMP9519 Multimedia Systems – Lecture 7 – Slide 25 – Re ji Mathew](https://reader033.fdocuments.net/reader033/viewer/2022042311/5eda042a28db2d5ca249424f/html5/thumbnails/58.jpg)
COMP9519 Multimedia Systems – Lecture 7 – Slide 58 – Reji Mathew
References and Further Reading1. IETF RFC 2327 “SDP: Session Description Protocol”
2. IETF RFC 2326 “Real Time Streaming Protocol (RTSP)”
3. IETF RFC 3261 “SIP: Session Initiation Protocol”
4. David Singer, QuickTime Engineering “Quick Time etc. Introduction for UC Davis”, Feb 2004
5. MPEG-4 Overview, ISO/IEC JTC1/SC29/WG11 N4668 www.chiariglione.org/mpeg/standards/mpeg-4/mpeg-4.htm