HTML5 Media

Audio and Video in Mobile Devices and Media Profiles

Support for HTML5 audio and video, especially video, in mobile devices is varied and can be challenging for web page authors and designers.

Challenges of a Mobile Environment

There are known quirks about the use of the HTML5 media elements in mobile devices. For instance, Apple has been a big fan of HTML5 from the beginning, deciding against support for Flash on iOS devices in favor of HTML5 video. However, some things that work on the desktop don’t in an Apple mobile environment. As an example, using the poster attribute caused the video element to fail in iOS 3, though this problem has been fixed in iOS 4. Another interesting little quirk was iPad’s only checking the first source element, so you needed to place the MP4 video first in the list (again, since corrected).

In addition, the iOS environment has its own native application for playback control, so it ignores the controls attribute.

Then there are the issues of how to test your HTML5 media applications. Most of us can’t afford to buy half a dozen devices (some of us can’t afford to buy any) and emulators don’t really work when it comes to testing out hardware and resource limitations.

Note

A good article on the issues of mobile testing is “Testing Apps For SmartPhones and Mobile Devices (Without Buying Out the Store)” at http://www.softwarequalityconnection.com/2011/03/testing-apps-for-smartphones-and-mobile-devices-without-buying-out-the-store/.

Most importantly, the video capability itself is limited in mobile environments. There is the resolution/size issue, of course, but there are also issues with containers and codecs. Mobile devices don’t have the processing power our computers have, which means that the file sizes are larger (because of simpler compression techniques). At the same time, mobile devices have data access limitations as well as issues with storage, so larger files aren’t mobile-friendly.

There’s also the challenge associated with the sheer number of mobile operating systems, mobile browsers, and devices—especially devices.

At this time, the iOS supports H.264, and the Android OS supports H.264 and WebM (though without hardware acceleration). Since Google is making a move away from H.264, we can assume the Android OS will, eventually, drop support for H.264. Maybe. In addition, the upcoming release of Windows Phone 7 from Microsoft, codenamed “Mango”, supposedly includes support for HTML5 video. Since Windows Phone 7 is Microsoft, we have to assume it will have H.264 support. Nokia is transitioning to Windows Phone 7, but is not offering HTML5 video and audio in its next release of its built-in Symbian operating system. However, you can run Opera Mobile on Symbian/S60, and get HTML5 video and audio support. Opera supports only Ogg and WebM. Blackberry supports H.264 video, but not the HTML5 video element—you’ll have to use a link.

What we can take away from all of this is that to support mobile devices, you’ll need to provide appropriately sized video files, as well as include support for both WebM/Ogg Theora and H.264. But not just any H.264. You need to provide videos encoded with the right profile.

Media Profiles and Codec Parameters

Since H.264 was designed to meet the needs of large television sets to small mobile phones, H.264 incorporates a concept known as a profile. Each profile defines a set of optional features, balanced against the file size. The more the video relies on the hardware, the smaller the file size. H.264 supports 17 profiles, but the ones we’re interested in are baseline, main, extended, and high. As you would expect, the hardware requirements for each increases from baseline to high.

Different devices support different protocols. Microsoft supports all H.264 profiles, but Safari only supports the main profile, because that’s all QuickTime supports by default. Mobile devices, such as those running iOS and the Android OS, run the baseline profile. If your site needs to provide both mobile and larger videos, you may want to encode several versions with different H.264 videos. It’s actually simple to ensure the right encoding, because most conversion tools provide device profile presets (more on this later in the chapter).

Note

The WHATWG Wiki provides a page giving several different type codec parameters, at http://wiki.whatwg.org/wiki/Video_type_parameters.

In order to ensure that each device knows which video works best for it (without having to load the video’s metadata and extract the information), you can provide the information directly in the source element’s type attribute. An example of the syntax to use is the following, for an Ogg Theora video file:

<source src='videofile.ogg' type='video/ogg; codecs="theora, vorbis"' />

The syntax is container first, then the codecs in video, audio order.

The codec specification for WebM is as simple as Ogg, but the same cannot be said for H.264 because of all of the profile possibilities. The audio codec is low-level AAC (mp4a.40.2), but the video codec is profile and level based. From the WHATWG Wiki that collects the type parameters, the video codec for H.264 can be any one of the five following codecs:

• H.264 Baseline: avc1.42E0xx, where xx is the AVC level

• H.264 Main: avc1.4D40xx, where xx is the AVC level

• H.264 High: avc1.6400xx, where xx is the AVC level

• MPEG-4 Visual Simple Profile Level 0: mp4v.20.9

• MPEG-4 Visual Advanced Simple Profile Level 0: mp4v.20.240

The profile part is easy, because when you use conversion tools, most have presets predefined for each of the profiles. However, the AVC level isn’t as simple to discover. According to a paper on H.264 (a PDF is available at http://www.fastvdo.com/spie04/spie04-h264OverviewPaper.pdf), the AVC level is based on picture size and framework, and also added constraints for picture number reference and compression rate.

In Example 1-8, several different video files are listed in individual source elements, with both the codec and container information in the type attribute. The two H.264 videos represent a desktop capable video encoded with the main profile, while the mobile version is encoded with the baseline profile. The user agent in each environment traverses the list of source elements, stopping when it reaches a container/codec and profile it supports.

Example 1-8. Several video sources, each with different container/codec strings in the source type attribute

<!DOCTYPE html>
<head>
   <title>Video</title>
   <meta charset="utf-8" />
</head>
<body>
   <video controls>
      <source src="videofile.mp4"
                 type='video/mp4; codecs="codecs="avc1.4D401E, mp4a.40.2"' />
      <source src="videofilemobile.mp4"
                 type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"' />
      <source src="videofile.webm"
                 type='video/webm; codecs="vp8, vorbis"' />
      <source src="videofile.ogv"
                 type='video/ogg; codecs="theora, vorbis"' />
   </video>
</body>

Note

The Android Developer SDK documentation contains a listing of supported media types and recommended encodings at http://developer.android.com/guide/appendix/media-formats.html. The iOS Developer Library has a “Getting Started” section for audio and video at http://developer.apple.com/library/ios/#referencelibrary/GettingStarted/GS_AudioVideo_iPhone/_index.html. The announcement of an integrated IE9 into Windows Phone 7, including HTML5 media support, can be found at http://blogs.msdn.com/b/ie/archive/2011/02/14/ie9-on-windows-phone.aspx.