Jason Suess Senior Technical Evangelist Microsoft Eric Schmidt Director Microsoft PC39.

Inside the OlympicsAn Architecture and Development Review

Jason SuessSenior Technical EvangelistMicrosoft

Eric SchmidtDirectorMicrosoft

PC39

Player Demo

Agenda

• Business imperative• What we built• Where do videos come from?• How we managed the ship…• Lessons learned

Live Event(Beijing)

International Broadcast

Center (Beijing)

30 Rock(New York)

DMZs (New Jersey,

Burbank)

Video Production WorkflowCDN

DistributionVideoPlayer

Limelight & Level3

The Business Perspective

Deliver every minute of every sport live from Beijing to the web

Where possible deliver “HD” qualityBreak the mold on user experienceMake it feel “solid-state”

Reach the broadest audience as possibleLeverage existing assets as much as possibleHigh dwell timeQuality impressions for their advertisersInnovate

The Result

1.3 billion page views50 million unique visitors70 million videos watched27 minutes of viewing per session600 million minutes of video delivery5,000 unique clips viewed per day during the final week35 million mobile views130,000 peak streams3.4 petabytes of video delivered

Video Production Workflow

Live Event(Beijing)


Center (Beijing)

30 Rock(New York)

DMZs (New Jersey,

Burbank)



Limelight & Level3

The Workflow & Delivery Perspective

2,200 events covering 25 sports with up to 35 simultaneous live feeds19 days of live production5,000+ videos to be produced

2000 new clips 3000 highlights/encores

Redundant, scalable content delivery

Content PackagesLive

2,200 hours and up to 35 concurrent streamsSynchronized with live commentary and play-by-play

RewindVOD archive of live streamsAvailable within an hour of the completion of the event

Highlights (light touch and craft edit)3-5min highlights of individual eventsVarying degrees of graphics and voice commentaryX hours

EncoresBroadcast replaysCut down to individual sports and broadcast commercials removedFull broadcast graphics and voice commentaryX hours

The Planning Complexity

Anticipate ContentWhenWhereSize

Determined CPU, power, cooling, storage and ingress/egress needsProduction schedule drove advertising schedule

Encoding Profiles

Built for 1024x768Needed to meet IOC advertising requirementsRestricted to a 40~meg (DS3) line of out BeijingWe didn’t know how hard to really push…

http://alexzambelli.com/blog/2008/08/21/an-inside-look-at-nbc-olympics-video-player/

High Medium High Medium Low Low PIPTotal Bitrate 1500

kbps1100 kbps 650 kbps 400 kbps 50 kbps

Video Bitrate 1450 kbps

1050 kbps 600 kbps 350 kbps 50 kbps

Video Resolution 848x480 592x336 424x240 320x176 128x96

Video Frame Rate full full full full half

High Low PIPTotal Bitrate 650 kbps 350 kbps 50 kbpsVideo Bitrate 600 kbps 300 kbps 50 kbpsVideo Resolution 592x336 320x176 128x96Video Frame Rate full full half

High LowTotal Bitrate 1100 kbps 400 kbpsVideo Bitrate 1050 kbps 350 kbpsVideo Resolution 688x384 320x176Video Frame Rate Full full

Encore/Highlights

Live/Rewind Ads

Where does video come from?(live)

Multiple cameras at each eventRouted to International Broadcast Center

HD PAL Source

Linked to live encoder farm 40 encoders – Digital Rapids

Linked 2 Windows Media Services boxesStreams reflected back to New York

Pushed through DMZ

Streams then reflected to CDNs

Live Event(Beijing)


Center (Beijing) 30 Rock

(New York)

DMZs (New Jersey,

Burbank) Limelight & Level3

Live & Rewind

Beyond live…Rewind encoding on liveHighlights factoryEncore publishingSchedulingAd booking

SharePoint Server

Metadata was the key

Common Message SchemaVideo IdentificationSchedulingCommand/ControlDelivery

Highlights Factory

Publish

CMS

Content PackagesLive

2,200 hours and up to 35 concurrent streamsSynchronized with live commentary and play-by-play

RewindVOD archive of live streamsAvailable within an hour of the completion of the event

Highlights (light touch and craft edit)3-5min highlights of individual eventsVarying degrees of graphics and voice commentaryX hours

EncoresBroadcast replaysCut down to individual sports and broadcast commercials removedFull broadcast graphics and voice commentaryX hours

Video Players

Motivations for WMP playerSilverlight install base concernsConcerns about corporatedeployment restrictionsProven track record/WMS pedigree

Motivations for SilverlightNext generation user experience

PIPOverlaysBetter navigationetc

Adaptive streamingCross-platform/cross-browser

Player Operation

Each of these meta-data files are written to origin by the CMS when the asset is published

index.xml

JSON File

Commentary File

http://ad.doubleclick.net/adi/nbcu.olympics/sport_sw_general_sl;area=sport;section=general;video=highlights;videoid=249000;type=sl;player=popup;sport=sw;athlete=2;tile=2;sz=300x250;ord=964055555.7812571

http://manifest.olympics.video.msn.com/d1/0822_HD_MUL_AU_CE523_MBR.elive

http://chunks-54.olympics.video.msn.com/nbcu/chunks/0822_HD_MUL_AU_CE523_MBR/629f379d85d767625601deb2052776ec9f54d3db3220b1c3592b6b3cc1d5.vid

The Life of a Video

Load the player

JSON Schema

Content Protection

Geo-fencingGeo based IP restriction

TokenizationURL in JSON is tokenized with a hash based on time (ttl) and secret key

"http://manifest.olympics.video.msn.com/d1/0824_HD_MUL_AU_CE529_MBR.elive?

e=1225296772&p=42&h=21f9ee892f5093c624a0f95f846c7a7b&token=c3RhcnRfdGltZT0yMDA4MTAyOTE2MDI1MiZlbmRfdGltZT0yMDA4MTAyOTE2MTI1MiZkaWdlc3Q9MGQ1Y2M3NTFkYTE4YmY1OGM1

MTEzOTQ0MTA2ZGNmZDM=

The Life of a Video

Load the playerLoad the JSON

Edge side include calculates and inserts tokens

Player is handed video URLRequest “video URL”Geo checkToken checkCDN responds with actual URL

Live -> WMS Non-live -> Manifest file

Unzip, decrypt

Silverlight plays URL

CDN Distribution

Live Event(Beijing)


Center (Beijing)

30 Rock(New York)

DMZs (New Jersey,

Burbank)



Limelight & Level3

Two CDNs

Provisioned identicallyURL structureCName attributes (token authentication, geo-restriction, etc.)Content

Dynamic traffic routing with Internet Traffic Manager (ITM)Granular control

Content typeRegionStraight percentages

Traffic split (active-active)Limelight 95% (primary)Level3 5% (warm backup)

Level3’s ITM

Level3

Limelight

Video Requestshttp://manifest.olympics.video.msn.com/...

CDN Provisioning Detailslive.olympics.video.msn.com

Live WMS streamsToken authentication and geo-restriction

wmp.olympics.video.msn.comOn-demand WMS deliveryToken authentication and geo-restriction

intl-wmp.olympics.video.msn.comOn-demand WMS deliveryToken authentication but no geo-restriction

manifest.olympics.video.msn.comHTTP delivery of adaptive streaming manifestsToken authentication and geo-restriction

intl-man.olympics.video.msn.comHTTP delivery of adaptive streaming manifestsToken authentication but no geo-restriction

chunks-xx.olympics.video.msn.comHTTP delivery of adaptive streaming “chunks”No token authentication or geo-restriction

Video Player Development

Live Event(Beijing)


Center (Beijing)

30 Rock(New York)

DMZs (New Jersey,

Burbank)



Limelight & Level3

Working back from 08-08-08

06/07 – site vision and scoping meetings08/07 – IBC and data center design11/07 – player vision and scoping meetings01/08 - user experience design02/08 - traffic forecasting03/08 - development hit full stride04/08 - functional testing and spin up of CDN workflow05/08 – Beta 106/08 – Beta 207/08 - final build out in Beijing & stabilization08/08 - go live

The Ultimate Team EffortNBC (New York and Beijing)

Video workflowVendor coordination Overall monitoring

Microsoft (Redmond)CDN coordinationAdaptive streaming deliverySilverlight deliveryDemand generation (MSN)Monitoring

DeltaTre (Italy)Content management systemWeb site design and developmentClient HTML and Javascript

Schematic (New York, Atlanta, Costa Rica)

Video player and user experience designImplementation of Silverlight and WMP player

Ad logicLimelight Networks (Lead) & Level3 (Back-Up)

WMS and HTTP capacity build outGeo fencing & Tokenization

Digital Rapids & AnystreamEncoding/ Meta dataWorkflow synchronization

SigniantContent distribution to CDN’s and publishing pointsWorkflow synchronization

DoubleClickIn-Stream and companion ad deliverySilverlight 2 integration

OmnitureSite and player meteringAnalytics

KeynoteWMS and adaptive streaming monitoring

SliqLive stream and publishing point monitoring

Pick the right partners and make the investment up frontConstant scrum like communication to keep everyone moving the same direction

Player Design

Interaction Model > Wireframes > Comps > Motion Tests

Player Architecture

Bridge Player instantiation, translations layer

Silverlight XAP

Dart.dll

Elivelibrary.dll

Host ComponentServer communication, user interaction HTML

Business Abstraction LayerServer communication, user interaction HTML

Presentation LayerXAML, animation, UI logic

Ad Model Logic

Silverlight Player

Bridge Player instantiation, translations layer

Ad Model Logic

Presentation LayerHTML, CSS, WMP

Ad Call Preparation

WMP Player

Javascript/HTML

C#/XAML

Logging

Monitoring & Game Time Operations

Monitoring

NBC implemented an all-up view for all critical systems (storage, CPU health, power management, etc.)

But that was only part of the story…

Delivery and CDN healthConcurrent streams404 errorsOutbound trafficCache efficiency… It was really hard to monitor the health of chunked delivery and understand what was happening at the client level, so….

Inline Monitoring

Real-time Telemetry

Less than 5 min. delayAggregate view, with drill down to source log

Drill-down

Targetting Core Issue

Physical Architecture – ORCSWeb.com

Silverlight 2.0 Clients•60% IE, 40% other

IP Load Balancer (Redundant Pair)•5.4 million hits during peak day•40 million unique users

Web Tier (15 Web Servers)•WS2K8, IIS 7.0, ASP.NET, WCF•SQL2K8 SSB Initiator Queues

Data Tier (Write Only)•WS2K8, SQL2K8•SQL2K8 SSB Target Queue, RawLog Table

Data Warehouse (Read Only)•WS2K8, SQL2K8, SSIS, SSRS, SSAS•RawLog Dimensional Model

24 Hour "War Rooms"

New YorkBeijingPhoenixRedmondTorino

All connected by NBC’s virtual meeting space/call bridge system

Real-time system to tell us when we had media playback problems (many different potential causes)

Use data to diagnoseMultiple people with different domain expertise to fix

Lessons Learned

Scrumming builds better teams, meet face to faceEveryone should know all roles and all architectural touch pointsReduce complexity via common schemaEven when traffic volume decreased, the number of unique video’s viewed grew, providing the right UI is essential to make the long-tail workLong-tail delivery hides issuesThe industry needs better telemetry and monitoring solutions“Chunked” workflow presented new challengesWe left some on the table in terms of bandwidth and video profile estimates

IIS Smooth Streaming will help content publishers more efficiently tweak their workflows

Why were we successful?

Windows Media pedigreeStrong partnersThe quality of MSN’s audienceSilverlight was ready for primetimeThe competency, passion and resources to get it done

Please use the microphones provided

Q&A

Evals & Recordings

Please fill

out your

evaluation for

this session at:

This session will be available as a recording at:

www.microsoftpdc.com

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market

conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Jason Suess Senior Technical Evangelist Microsoft Eric Schmidt Director Microsoft PC39.

Documents

Transcript of Jason Suess Senior Technical Evangelist Microsoft Eric Schmidt Director Microsoft PC39.