Research & Development

White Paper

WHP 346

June 2019

Framework for web delivery of immersive audio experiences using device orchestration

Kristian Hentschel, Jon Francombe

BRITISH BROADCASTING CORPORATION

White Paper WHP 346

Framework for web delivery of immersive audio experiences using device orchestration

Kristian Hentschel, Jon Francombe

Abstract

This demonstration introduces the use of orchestrated media devices and object-

based broadcasting to create immersive spatial audio experiences. Mobile

phones, tablets, and laptops are synchronised to a common media timeline and

contribute one or more individually delivered audio objects to the overall mix. A

rule set for assigning objects to devices was developed through a trial

production—a 13-minute audio drama called The Vostok-K Incident. The same

timing model as in HbbTV2.0 media synchronisation is used, and future work

could augment linear television broadcasts or create novel interactive audio-

visual experiences for multiple users. The demonstration will allow delegates to

connect their mobile phones to the system. A unique mix is created based on the

number and selected locations of connected devices.

This document was originally published in the Adjunct Proceedings of TVX 2019,

Salford, UK. https://doi.org/10.6084/m9.figshare.8175281.v1.

As of publication, the demonstration described is available to view online at

https://vostok.virt.ch.bbc.co.uk/.

Additional key words: device orchestration; immersive audio; object-based

broadcasting.

© BBC 2019. All rights reserved. Except as provided below, no part of this document may be

reproduced in any material form (including photocopying or storing it in any medium by electronic

means) without the prior written permission of BBC except in accordance with the provisions of the

(UK) Copyright, Designs and Patents Act 1988.

The BBC grants permission to individuals and organisations to make copies of the entire document

(including this copyright notice) for their own internal use. No copies of this document may be

published, distributed or made available to third parties whether by paper, electronic or other

means without the BBC's prior written permission. Where necessary, third parties should be

directed to the relevant page on BBC's website at http://www.bbc.co.uk/rd/pubs/whp for a copy of

this document.

White Papers are distributed freely on request.

Authorisation of the Head of Standards or Chief Scientist is required for publication.

Framework for Web Delivery ofImmersive Audio Experiences UsingDevice Orchestration

Kristian HentschelBBC R&DMediaCityUK, Salford, UKkristian.hentschel@bbc.co.uk

Jon FrancombeBBC R&DMediaCityUK, Salford, UKjon.francombe@bbc.co.uk

The Adjunct Proceedings of ACM TVX 2019, Manchester UK, June 2019. Copyrightis held by the author/owner(s).

AbstractThis demonstration introduces the use of orchestrated me-dia devices and object-based broadcasting to create immer-sive spatial audio experiences. Mobile phones, tablets, andlaptops are synchronised to a common media timeline andcontribute one or more individually delivered audio objectsto the overall mix. A rule set for assigning objects to deviceswas developed through a trial production—a 13-minute au-dio drama called The Vostok-K Incident. The same timingmodel as in HbbTV2.0 media synchronisation is used, andfuture work could augment linear television broadcasts orcreate novel interactive audio-visual experiences for multi-ple users. The demonstration will allow delegates to con-nect their mobile phones to the system. A unique mix iscreated based on the number and selected locations ofconnected devices.

Author KeywordsDevice orchestration; immersive audio; object-based broad-casting.

ACM Classification KeywordsH.5.5 [Sound and Music Computing]: Systems; H.5.3 [Groupand Organization Interfaces]: Collaborative Computing;C.2.4 [Distributed Systems]: Distributed Applications

IntroductionMedia experiences are increasingly taking place on mobiledevices—either as the main device or as a secondary de-vice being used at the same time. The Internet of Thingsoffers ever more connectable devices, many with mediaplayback capabilities. Device orchestration is the concept ofusing these diverse devices to create or augment an immer-sive media experience.

Current immersive audio experiences in the home rely onchannel-based delivery matched to common reproductionsystems, such as stereo or 5.1 surround loudspeaker con-figurations. However, only 11.5% of users in the UK reporthaving a surround system, according to a recent survey byCieciura et al. [1]. Alternatively, headphones can be used toreproduce binaural audio, but this is limited to a single lis-tener, and wearing headphones is not always appropriate.

This demonstration introduces a new approach that uses aweb-based framework alongside object-based audio (seesidebar) to deliver immersive audio with commodity mobiledevices. Listeners can connect their mobile phones andselect a location; their device becomes part of the soundsystem and plays appropriate content. A trial production (ashort audio drama) was used to motivate the developmentof the system, providing requirements for an orchestrationruleset, accuracy of synchronisation, and the user interface.

Prior work

Audio in object-basedbroadcasting

Object-based broadcasting isthe idea of storing the mediaassets that constitute a pro-gramme alongside metadatathat describe how those as-sets should be assembled.This enables flexible ren-dering; for example, contentmight be modified to suit thereproduction device, the userpreferences, or for interactiveor variable length applica-tions. In object-based audio,the objects are the differentsound sources from the mix,and the metadata gener-ally include details such asposition and level (but mayalso include more detailedsemantic descriptions [10]).

Synchronised second-screen applications for televisionbroadcasts have been enabled by the development of Hy-brid Broadcast Broadband TV (HbbTV 2.0). Previous demon-strations focused on textual or visual content, although al-ternative audio tracks (such as audio description or direc-tor’s commentary) have also been shown [9].

Mobile phones have been used for orchestrated audio re-production before, with demonstrations largely falling intotwo categories: (i) in interactive performance scenarioswhere individual sounds are triggered by each user [8];and (ii) for reproduction of channel-based spatial audiomixes requiring very accurate synchronisation betweenloudspeakers [5]. However, object-based broadcasting en-ables a new way to create immersive experiences by us-ing synchronised devices to reproduce different aspectsof a scene. This approach has performed well in percep-tual tests; the listening experience achieved with smallloudspeakers in non-standard locations was shown to becomparable to a surround sound system by Woodcock etal. [11].

Web framework for audio device orchestrationAn orchestrated immersive audio scene consists of individ-ual audio assets synchronously played back from multipledevices. The prototype system architecture for achievingthis is outlined in Figure 1. A single main device creates asession: additional devices can join by entering a pairingcode and selecting a location zone. All devices maintain awall clock synchronised to a central server. Metadata de-scribe timings and placement constraints for every object.The main device runs the placement algorithm and assignsobjects to devices. Each device requests only the audioobjects assigned to it, and dynamically mixes them on thedevice.

The web platform was selected for prototyping the expe-rience because it is easily accessible to a wide audience.WebSockets are used for asynchronous communication in-cluding wall clock synchronisation. The Web Audio API isused for rendering multiple synchronised audio streams. Auser interface template for pairing devices and controlling

Audio segments JSON metadata JavaScript code ImagesStatic Assets Web Server

Cloud-SyncSynchronisation

Messaging

Aux Device

Aux Device

Main Device

Media

Control

Figure 1: System diagram; one main and two auxiliary devicesrequest audio assets and metadata, synchronise to a server, andexchange messages. The main device decides which audioobjects should be played by each device.

playback was written in JavaScript using the React frame-work and can be customised for specific productions.

Sequence 1

Object 1

Placement rules

Item 1 Item 2 Item N

Object N

Placement rules

Item Item Item Item

...

Sequence N

Object 1

Placement rules

Item Item Item

...

...

Figure 2: A sequence metadatafile describes how objects shouldbe distributed to available devices,and when each rendering itemshould be played.

Devices exchange messages with a server to measure theround trip time and determine their hardware clock’s off-set and speed difference. Multiple clock synchronisationlibraries are available [6, 4]. The Cloud-Sync solution devel-oped in the 2-IMMERSE project [7] was chosen because itdirectly corresponds to the HbbTV2.0 timeline model andoffers a scalable server implementation. A synchronisedwall clock is used to derive media timelines for every indi-vidually scheduled audio object. The service also providesmessaging between devices in a session and notificationswhen a device joins or leaves.

The content metadata hierarchy is illustrated in Figure 2.An object-based audio mix is exported from a digital au-dio workstation (DAW) with a full-length mono audio file foreach object. Most objects are silent most of the time, sothe silence is removed and the objects are split into render-ing items. An experience may transition between multiplecontinuous sequences i.e., different parts or sections of thecontent. Each is defined by a JavaScript Object Notation(JSON) metadata file describing the objects and placementrules, and the start times and download locations for indi-vidual rendering items. Long running rendering items aredelivered using Dynamic Adaptive Streaming over HTTP(DASH). The browser on each device downloads, plays,and combines the rendering items for objects assigned toit using the Web Audio API. Additional effects, such as dy-namics compression, may be applied here. The metadatahierarchy is illustrated in Figure 2.

The main device maintains a list of auxiliary devices in thecurrent session. It runs a placement algorithm to assigna combination of object identifiers to each device whena device joins, leaves, or changes location. The assign-ments are distributed to all devices in the session througha publish-subscribe message broker included in the Cloud-Sync service. Each device only downloads the renderingitems for those objects currently assigned to it, shortly be-fore they are played out.

A user interface template for starting sessions, connect-ing devices, selecting their location, controlling the audioplayback, and transitioning between different content se-quences was also created. This shows how the frameworkcan be integrated with React to build a responsive web ap-plication. The template was customised and extended tocreate the interface for a trial production, discussed in thefollowing section.

DemonstrationA trial production, the 13-minute audio drama The Vostok-K Incident, motivated the development of the framework.Its main story sequence consists of a stereo bed and 61additional objects: character dialogue, musical elements,ambient sound, and sound effects. The experience beginswith a 30-second loading loop, so users can connect theirdevices without missing the beginning of the story.

Attendees may connect their own devices to augment thestereo reproduction, and listen to a section of the drama.The main screen shows connection instructions and trans-port controls. Devices are paired using a six-digit code, anda location is selected from a stylised map. The audio mixadapts to the number and location of devices. Artwork re-lated to the objects is shown on each device.

Figure 3: The Vostok-K Incidentwas released on BBC Taster inSeptember 2018. The photographsabove show the user interface:after pairing via a QR code or linkand numerical code, anapproximate location is selected(top); then each device displaysartwork related to the audio objectsit has been assigned (bottom).

The object-based trial production [3] used regular radiodrama recording and sound design techniques, and wasinitially mixed in stereo. A prototype production system con-nected to the digital audio workstation (DAW) allowed theproducers to audition the effect of different loudspeaker lay-outs and metadata rules during the immersive mix. For thisdrama, six different rules were used to allocate objects todevices. The rules covered:

• the location from which objects should be played (forexample, some objects might only be allowed intoloudspeakers in front of the listener);

• the type of loudspeakers from which objects shouldbe played (for example, some objects were only re-produced if devices were connected whilst otherswould fall back into the stereo bed); and

• the relationships between objects (for example, someobjects would only play if others were not playing,and some objects precluded any others from beingplayed from the same device).

Evaluation and future workThe flexible framework for delivering immersive audio toorchestrated devices created here could be extended inmultiple ways. It could integrate visual content, played backon the Web or a smart TV; it could make use of Internet-connected lights and similar devices for additional effects;and it could allow for interactive exploration of non-linearcontent.

The Vostok-K Incident was released on BBC Taster, BBCR&D’s public-facing trial platform, and was accessed 3121times. A questionnaire completed by 210 users indicatedthat the experience was positive (four out of five stars) andthat the framework was successful (only 15% rated ‘the setup’ as the worst thing about the experience). Interactionlogs for 2174 sessions were analysed further by Francombeet al. [2]. Preliminary results show that about 65% of ses-sions that reached the first minute of the main sequencehad at least one device connected, and that almost 20% ofthose that listened to the whole piece connected three ormore devices.

AcknowledgementsThe authors would like to acknowledge the work of mem-bers of The Vostok-K Incident production team: JamesWoodcock and Richard Hughes (University of Salford), andEloise Whitmore, Tony Churnside, and Ed Sellek (NakedProductions). The production was supported by the EPSRCProgramme Grant S3A: Future Spatial Audio for an Immer-sive Listener Experience at Home (EP/L000539/1).

They would also like to thank Rajiv Ramdhany (BBC R&D)for his assistance in integrating the Cloud-Sync service de-veloped as part of the 2-IMMERSE project.

REFERENCES1. Craig Cieciura, Russell Mason, Philip Coleman, and

Matthew Paradis. 2018. Survey of Media DeviceOwnership, Media Service Usage, and Group MediaConsumption in UK Households. In Audio EngineeringSociety Convention 145 (eBrief 456). New York, NY,USA.

2. Jon Francombe and Kristian Hentschel. 2019.Evaluation of an immersive audio experience usingquestionnaire and interaction data. In 23rd InternationalCongress on Acoustics. Aachen, Germany. Acceptedfor publication in September 2019.

3. Jon Francombe, James Woodcock, Richard Hughes,Kristian Hentschel, Eloise Whitmore, and TonyChurnside. 2018. Producing audio drama content foran array of orchestrated personal devices. In AudioEngineering Society Convention 145 (eBrief 461). NewYork, NY, USA.

4. Matt Hammond. 2018. Open-source DVB CSS librariesavailable. (2018). https://2immerse.eu/open-source-dvb-css-libraries-available/

5. Hyosu Kim, SangJeong Lee, Jung-Woo Choi, HwidongBae, Jiyeon Lee, Junehwa Song, and Insik Shin. 2014.Mobile maestro: enabling immersive multi-speakeraudio applications on commodity mobile devices. InProceedings of the 2014 ACM International JointConference on Pervasive and Ubiquitous Computing -UbiComp ’14 Adjunct. DOI:http://dx.doi.org/10.1145/2632048.2636077

6. Jean-Philippe Lambert, Sébastien Robaszkiewicz, andNorbert Schnell. 2016. Synchronisation for DistributedAudio Rendering over Heterogeneous Devices, inHTML5. In Proceedings of the 2nd Web AudioConference (WAC-2016). Atlanta, GA, United States.

7. Rajiv Ramdhany and Christoph Ziegler. 2019.Cloud-Sync Media Synchronisation Service. (2019).https://github.com/2-IMMERSE/cloud-sync

8. Sébastien Robaszkiewicz and Norbert Schnell. 2015.Soundworks–A playground for artists and developers tocreate collaborative mobile web performances. InProceedings of the 1st Web Audio Conference(WAC-2015). Paris, France.

9. Vinoba Vinayagamoorthy, Rajiv Ramdhany, and MattHammond. 2016. Enabling Frame-AccurateSynchronised Companion Screen Experiences. InProceedings of the ACM International Conference onInteractive Experiences for TV and Online Video. NewYork, NY, USA. DOI:http://dx.doi.org/10.1145/2932206.2932214

10. James Woodcock, Jon Francombe, Andreas Franck,Philip Coleman, Richard Hughes, Hansung Kim, QingjuLiu, Dylan Menzies, Marcos F Simón Gálvez, Yan Tang,Tim Brookes, William J. Davies, Bruno M. Fazenda,Russell Mason, Trevor J. Cox, Filippo Maria Fazi, PhiipJ. B. Jackson, Chris Pike, and Adrian Hilton. 2018a. AFramework for Intelligent Metadata Adaptation inObject-Based Audio. In AES International Conferenceon Spatial Reproduction - Aesthetics and Science.Tokyo, Japan.

11. James Woodcock, Jon Francombe, Richard Hughes,Russell Mason, William J Davies, and Trevor J Cox.2018b. A quantitative evaluation of media deviceorchestration for immersive spatial audio reproduction.In AES International Conference on SpatialReproduction - Aesthetics and Science. Tokyo, Japan.