Post on 12-May-2015
description
Multimedia on the web.HTML5 Audio and Video
Christian Heilmann, MIT Boston, January 2012
I’m Chris.
What we will cover:★ Quick history of Multimedia on the web★ Annoyances with Flash★ HTML5 audio and video★ Painful stuff – codecs and conversion★ Embedding★ Controlling★ Transforming★ Realtime changes★ Awesome audio stuff
Quick history of Multimedia on the web
First, there were images.
JPG, GIF, PNG (later), WBMP, ICO
Connections were bad.
Progressive JPG, interlaced GIF, <img src=”foo.jpg” lowsrc=”foogrey.gif” alt=”a foo, what else?”>
Animation was done with animated GIFs or with JavaScript animation.
Audio was mostly Midi background music.
(and if you think about using that today I will hit you!)
Other things we did were Java Applets.
And then we gotRealPlayer.
And many others...
★ Quicktime / Quicktime VR★ Microsoft Media Player★ Shockwave★ Acrobat had some image
editing features.★ iPix / VRML and many other
forgotten ones...
They all had the same issues.★ End users must install a
plugin.★ You need to upgrade the
plugin constantly.★ There is limited interaction
with the rest of the page.
Another big issue is and was security – plugins are one of main attack vectors for malware.
In the end, one plugin prevailed over all the others: Flash.
With aggressive marketing, clever partnerships and a rich dev environment Flash became the multimedia choice for the web.
And there is the DRM stuff...
Annoyances with Flash
+
=
Apple and Flash movies means a lot of times your fan will start up.
Why would I want yet another editor to build web content?
<object width="640" height="385"><param name="movie" value="http://www.youtube.com/v/SJixW2u4IvQ?fs=1&hl=en_US" /><param name="allowFullScreen" value="true" /><param name="allowscriptaccess" value="always" /><embed src="http://www.youtube.com/v/SJixW2u4IvQ?fs=1&hl=en_US" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="640" height="385" /></object>
WTF is all this?
Flash is a black box inside a document.
Alien
Therefore it is also an accessibility problem.★ Other browsers than IE don’t allow you
to access Flash with a keyboard.★ You are at the mercy of the developers
to make their movies keyboard accessible.
★ Which is bad, as audio and video can help a lot of people to understand things.
Flash can talk to JavaScript via APIs.
You are at the mercy of the API developers though – what they don’t make public, you can’t reach.
HTML5 audio and video
HTML5 audio and video make things much simpler:★ Better accessibility★ Doing one thing well★ Much simpler API★ Allows for styling and overlays★ View-source “hackable”
Screencast of Oprah’s web site
http://www.oprah.com/own
Painful stuff – Codecs and conversion
So, we understood that there is a need for an open high-fidelity technology if we really want the web to be a media.
What we didn’t quite consider yet is something that both accelerated and hindered innovation for a long time.
Copyright
Videos on the web are encoded to make them smaller – there is no such things as an “AVI File”...
They are containers for video in a certain compression and audio in another.
The issue was that all these compression formats had their own copyright and didn’t work with open technology like HTML5.
Therefore we needed new formats:
★ H.264 “MP4”★ OGG/Vorbis★ VP8 / WebM
Our job is now to convert the videos we record to these open formats – which can be annoying.
The main annoyance is that different browsers support different formats, which means we need to provide several versions :(
Audacity: http://audacity.sourceforge.net/
WebM tools: http://www.webmproject.org/tools/
Evom: http://thelittleappfactory.com/evom/
VLC: http://www.videolan.org/vlc/
Ogg convert: http://oggconvert.tristanb.net/
Firefogg: http://firefogg.org/
TinyOgg: http://tinyogg.com/
ffmpeg:http://ffmpeg.org/
http://www.mirovideoconverter.com/
Embedding
Embedding audio and video in an HTML5 document is easy:<audio src=”foo.ogg”> If your browser didn’t suck, you’d have audio here.
</audio><video src=”foo.ogv”> If your browser didn’t suck, you’d have video here.
</video>
That doesn’t do anything yet<video src=”foo.ogv” controls> If your browser didn’t suck, you’d have video here.
</video>
★ Controls appear on mouse hover or tabbing
★ Keyboard enabled controls
★ Video can be styled in any way.
Controls differ from browser to browser...
Firefox
Opera
Safari
Chrome
Full Screen
So to give it to all browsers... <video controls> <source src="http://www.archive.org/{...}_512kb.mp4" type="video/mp4"></source> <source src="http://www.archive.org/{...}nsters.ogv" type="video/ogg"></source> <p>Your browser doesn't like HTML5 video, watch the movie on <a href="http://www.archive.org/{...}_monsters"> archive.org</a>. </p> </video>
Things to consider:
★ Use MP4 always as the first option to support iOS devices (iPads, iPhone and so on...)
★ If you omit the type attribute, the browser loads the meta data of each source!
Media queries can save bandwidth: <video controls> <source src="http://www.archive.org/{...}_512kb.mp4" type="video/mp4" media="(min-device-width:800px)"></source> <source src="http://www.archive.org/{...}_low.mp4" type="video/mp4"></source> <source src="http://www.archive.org/{...}nsters.ogv" type="video/ogg"></source> <p>watch the movie on <a href="{...}_monsters"> archive.org</a>.</p> </video>
Other attributes:
★ poster – define a picture to show before loading.
★ height/width – oh, well...★ loop – automatically restart★ preload (auto/none/
metadata)
But what if you don’t like the controls?
Controlling
HTML5’s Media API gives you control:
★ load() – load a new media.★ canPlayType(type) – returns probably,
maybe and “” (really!)★ play() – play the movie★ pause() – pause the movie.★ addTrack(label,kind,language) -for
subtitles
Video details:width / height / videoWidth / videoHeight / poster
Controls:controls / volume / muted
Tracks:tracks
Network state:src / currentSrc / networkState / preload / buffered
Ready statereadyState / seeking
Playback statecurrentTime / startTime / duration / paused / defaultPlayBackRate / playbackRate / played / seekable / ended / autoplay / loop
Writing a play button is simple:var audio = document.getElementsByTagName('audio')[0];var play = document.getElementsByClassName('play')[0];play.addEventListener('click',function(e){ var t = e.target; if(audio.paused){ audio.play(); t.innerHTML = 'pause'; } else { audio.pause(); t.innerHTML = 'play'; } e.preventDefault();});
However, simply checking properties is not safe!
Any JS use gets much safer by listening to events.
HTML5 video events:loadstart / progress / suspend / abort / error / emptied / stalled / play / pause / loadedmetadata / loadeddate / waiting / playing / canplay / canplaythrough / seeking / seeked / timeupdate / ended / ratechange
video.addEventListener('play', playEvent, false);video.addEventListener('pause', pausedEvent, false);video.addEventListener('ended', function () { this.pause();}, false);function playEvent() { play.innerHTML = 'pause';}function pausedEvent() { play.innerHTML = 'play';}play.onclick = function () { if (video.ended) { video.currentTime = 0; } if (video.paused){ video.play(); }else{ video.pause(); }};
http://www.w3.org/2010/05/video/mediaevents.html
You can use these events to sync animation or trigger changes with audio and video.
http://isithackday.com/spirit-of-indiana/
Transforming
http://www.flickr.com/photos/jiazi/1061447777
I am a film buff and I like to find easter eggs in movies. Every pixar movie for example has A113 in it – the room in the uni a lot of pixar employees had their animation lectures in. To find things like that I sometimes zoom the screen and scan around.
I thought this should be possible in HTML5.
http://isithackday.com/mit/transforming-video.html
Using SVG and other technologies you can change the look and feel of videos even more.
http://people.mozilla.com/~prouget/demos/round/index.xhtml
http://people.mozilla.com/~prouget/demos/mashup/video.xhtml
Realtime changes
http://introducinghtml5.com/examples/ch05/animate-with-video.html
http://people.mozilla.com/~prouget/demos/green/green.xhtml
http://people.mozilla.com/~prouget/demos/DynamicContentInjection/play.xhtml
http://people.mozilla.com/~prouget/demos/tracker/tracker.xhtml
Awesome audio stuff
https://developer.mozilla.org/en/Introducing_the_Audio_API_Extension
https://wiki.mozilla.org/Audio_Data_API
http://hacks.mozilla.org/2011/01/html5guitar/
http://audioscene.org/scene-files/humph/sfxr/