Streaming Architectures for Streaming Architectures for Very Large CollectionsVery Large Collections

Jim DeRoestJim DeRoestDirector Streaming Media Technologies, ResearchChannelDirector Streaming Media Technologies, ResearchChannel

Assistant Director, University Computing ServicesAssistant Director, University Computing ServicesUniversity of WashingtonUniversity of Washington

deroest@cac.washington.eduderoest@cac.washington.edu206 543-6343206 543-6343

Who We Are And SupportWho We Are And Support

ResearchChannel/UWTV - ResearchChannel/UWTV - Exploring digital convergence Exploring digital convergence and educational outreach.and educational outreach.

National Echostar and CATV to over 5 million National Echostar and CATV to over 5 million homeshomesInternational VoD and live streams – I2 AbileneInternational VoD and live streams – I2 AbileneExperiments - First cinema quality HD over IPExperiments - First cinema quality HD over IP

KEXP – KEXP – UW and EMP collaboration exploring digital radioUW and EMP collaboration exploring digital radio

Uncompressed 1.4Mb live streamUncompressed 1.4Mb live stream Real time web playlist generationReal time web playlist generation

World music format – DJ choiceWorld music format – DJ choice

Faculty, Staff, Student, Clinical, K20Faculty, Staff, Student, Clinical, K20Other Affiliate/Collaborative ProjectsOther Affiliate/Collaborative Projects

Managing ContentManaging Content

Large Collection ProblemLarge Collection Problem Disparate Asset TypesDisparate Asset Types

HD, Broadcast TV, audio libraries, pictures, rich textHD, Broadcast TV, audio libraries, pictures, rich text Instantiation: capture, encoding, metadata, storageInstantiation: capture, encoding, metadata, storage

Multiple sources, formats, schemaMultiple sources, formats, schema InterfacesInterfaces

TV post production, radio DJ, viewer/listenerTV post production, radio DJ, viewer/listener Discovery and AdministrationDiscovery and Administration

Naming, categorization, relations, internationalizationNaming, categorization, relations, internationalizationDistributed management, sharing, searchingDistributed management, sharing, searching

Access ControlsAccess ControlsAuthentication, authorization, rights, privacy, securityAuthentication, authorization, rights, privacy, security

MediaWeb ProjectMediaWeb Project

18 month technology/requirements review18 month technology/requirements reviewRequirementsRequirements

Interviewed engineers, DJ’s, librarians and consumersInterviewed engineers, DJ’s, librarians and consumers

Reviewed vendor productsReviewed vendor products Proprietary, 70% solution, consumer specificProprietary, 70% solution, consumer specific

Created prototype during review processCreated prototype during review process Pluggable, scalable, standards based architecturePluggable, scalable, standards based architecture

Buy, build, both, call it quits?Buy, build, both, call it quits? Decision to go with prototypeDecision to go with prototype

RoboticTape

RoboticTape

DiskCache

DiskCache

Media Store

Asset Metadata

Asset Metadata

Remote StreamManager

Remote StreamManager

RequestDirector

Transcoding

MetadataCreation, Admin

Asset Management Services

TV Broadcast Services

Radio DJ Services

Delivery ServicesImage, Quicktime,

Windows, Real, HD, Webcasting …

UW MediaWeb – Digital Media Asset Management System

Core ServicesCore Services

Media StoreMedia Store HSM online/nearline multi-vendorHSM online/nearline multi-vendor

High Performance Storage System (HPSS)High Performance Storage System (HPSS) Gigabit interconnects.Gigabit interconnects. Distributed, redundant, recoverableDistributed, redundant, recoverable Block I/O stream to servers (Block I/O stream to servers (smoke and mirrorssmoke and mirrors

Metadata ServiceMetadata Service SQL, directory, imbedded objects, access control, rightsSQL, directory, imbedded objects, access control, rights

Stream ManagerStream Manager Maintains asset locality state and agingMaintains asset locality state and aging

Request Re-DirectorRequest Re-Director Re-direct requests to target stream or application serviceRe-direct requests to target stream or application service

FeedsFeeds

VideoVideoAirtime/Dubtime to 50Mb MPEG2 i-frame baseAirtime/Dubtime to 50Mb MPEG2 i-frame base

Louth/Mav triggers/APIsLouth/Mav triggers/APIs

Metadata generation Metadata generation Looking at Virage pluginsLooking at Virage plugins

AudioAudioCD ripping raw audio baseCD ripping raw audio base

Metadata map CDDB sites (Muse, AMG, …)Metadata map CDDB sites (Muse, AMG, …)

DJ metadata supportDJ metadata support

Transcode from base formats as neededTranscode from base formats as needed

MetadataMetadata

Schema – Schema – How rich can we affordHow rich can we affordConsumer specificConsumer specificStandards – Dublin Core, MPEG7Standards – Dublin Core, MPEG7Vendor (Muse, AMG, Louth playlists, …)Vendor (Muse, AMG, Louth playlists, …)Draft submitted for comment I2-VidMid VoD and ViDeDraft submitted for comment I2-VidMid VoD and ViDe

Access/Rights controlsAccess/Rights controlsVendor specific, MPEG21 Vendor specific, MPEG21 Auth*2 – Auth*2 – NetID Pubcookie, Shibboleth, …NetID Pubcookie, Shibboleth, …Imbedded – Honored by clientsImbedded – Honored by clientsTrustTrust

Metadata ObjectsMetadata ObjectsSQL (base), LDAP, Imbedded objects/methodsSQL (base), LDAP, Imbedded objects/methods

DeliveryDelivery

Streaming ServersStreaming ServersStandard file system or IO API requiredStandard file system or IO API required

Stream initiated as first blocks arrive to local cacheStream initiated as first blocks arrive to local cacheTape, disk, OS and network speed vs. hybrid file systemsTape, disk, OS and network speed vs. hybrid file systems

Selectable stream stub supportSelectable stream stub supportFirst +30 seconds of clip hide mount time content on tapeFirst +30 seconds of clip hide mount time content on tape

Low bit rate (no MPEG1)Low bit rate (no MPEG1)Quicktime, Real, Windows MediaQuicktime, Real, Windows Media

High bit rateHigh bit rateWindows Media – KEXP 1.4Mb live streamWindows Media – KEXP 1.4Mb live streamOptivision – RC 5.6Mb live stream to remote sitesOptivision – RC 5.6Mb live stream to remote sitesMPEG2 – Concurrent Computer Xstreme 5.6Mb VoDMPEG2 – Concurrent Computer Xstreme 5.6Mb VoDMPEG4 – Looking at IBM Videocharger – MPEG4 – Looking at IBM Videocharger – interface issuesinterface issuesHD – In-house (>270Mb), other vendor lower bit ratesHD – In-house (>270Mb), other vendor lower bit rates

MediaWeb Project StatusMediaWeb Project Status

Prototype operational with simple searchPrototype operational with simple searchGeographically separated/mirrored server environmentGeographically separated/mirrored server environmentVideo: RC/UWTV content (~700 hrs MPEG2, lower bit rates)Video: RC/UWTV content (~700 hrs MPEG2, lower bit rates)Audio: Sample audioAudio: Sample audio

Refining schemaRefining schemaObject experimentsObject experimentsImbedded controls (MPEG7)Imbedded controls (MPEG7)Reviewing large vendor schemasReviewing large vendor schemasDeploying admin toolsDeploying admin tools

Building feed systems (RC, +35,000 disc KEXP library)Building feed systems (RC, +35,000 disc KEXP library)Incorporating faculty/student mgt (expiration, quotas, …)Incorporating faculty/student mgt (expiration, quotas, …)Evaluating other large content requestsEvaluating other large content requestsOngoing evaluation, scaling, benchmarking architectureOngoing evaluation, scaling, benchmarking architectureLooking for opportunities to collaborate!Looking for opportunities to collaborate!

ExperimentsExperiments

UW CSci animation teaching toolUW CSci animation teaching tool““How Networks Change Teaching: A Case Study”How Networks Change Teaching: A Case Study”Louis Fox, Kris Martin, Barbara Mones-HattalLouis Fox, Kris Martin, Barbara Mones-HattalI2 VIMM Friday, October 5I2 VIMM Friday, October 5thth 1:30EDT 1:30EDT

KEXP live stream archivesKEXP live stream archivesKEXP automated playlistKEXP automated playlistEMP live webcasting and archivalEMP live webcasting and archivalUW Catalyst collection interfaceUW Catalyst collection interfaceHD IP VideoconferencingHD IP Videoconferencing