HTML5 AS A NEW PLATFORM FOR ADVANCED USER EXPERIENCE

Hiromasa Yanagihara Shigeyuki Sakazawa Yasuhiro Takishima

KDDI R&D Laboratories, Inc.

ABSTRACT In this paper, we focus on the HTML5 platform that can provide an advanced user experience on multiple devices through one source. Furthermore, we propose the sophisticated applications that are able to make good use of HTML5 features and also introduce the evolution of HTML5 as a service infrastructure.

1. INTRODUCTION HTML early (HyperText Markup Language) is a markup language that describes the rules of certain documents published on the Web by setting the hyperlinks between documents with a feature that allows you to browse through multiple documents. The version of HTML evolved later, CSS (Cascading Style Sheets) improves the representation of the design by modifying the HTML document, and JavaScript allows you to control the content of the HTML document dynamically. As a result, the expressive performance and usability of HTML documents have been greatly improved. This has also led to the evolution of markedly improved performance and functions of the Web browser, making it possible for the Web browser to be used as an application execution environment.

In 2004, engineers belonging to Apple, Mozilla, Opera interested in HTML technology and related issues established WHATWG (Web Hypertext Application Technology Working Group) [1] as a voluntary body. As the next generation of HTML, HTML5 was formulated based on a new technical specification and WHATWG proposed its inclusion to the W3C, the body that develops standards for HTML. The formal W3C recommendation for HTML5 is scheduled to be issued in 2014. The implementation of HTML5 is proceeding in stages in many Web browsers; as a result, Web applications using HTML5 are already coming into wide use. Bird's-eye view of a family of specifications related to HTML5 is illustrated in Fig.1[2].

Prior to the issuance of a formal recommendation, browser vendors have implemented HTML5 on their own based on the tentative recommendation. At the present time, as many Web applications using HTML5 work in common between browsers, expectations are growing that it will serve as a common platform for advanced applications running on a variety of devices.

In particular, the number of mobile terminals equipped with a Web browser using HTML5 had reached more than 100 million units in 2010, and is expected to be 2.1 billion units in 2016. In addition, 34% of the top 100 Web sites are using HTML5, and 75% of Web site developers have the intention to use HTML5 in future [3].

Under such circumstances, the HTML5 platform can provide an advanced user experience through one source on multiple devices. In this paper, the features of HTML5 are introduced in section 2, and concrete examples of HTML5 evolution as a service infrastructure are presented in section 3.

Fig.1 Bird's-eye view of a family of specifications related to HTML5

2. OVERVIEW OF HTML5 In HTML5, various functionalities have been specified, not only to realize rich documents representing hyperlinks, lists, tables, images, and so forth, but also to realize a wide variety of applications. In addition to HTML5, a number of related technologies listed below have been developed by W3C. It is thus common to consider HTML5 more broadly by including the following technologies: (1) Application Cache: specification that allows the

operation of the Web application in an offline environment

(2) Local Storage: a small database with key-value type (Web Storage), Indexed Database (IndexedDB), etc.

Proceedings of the IIEEJ Image Electronics 　　　　and Visual Computing Workshop 2012

Kuching, Malaysia, November 21-24, 2012

(3) Device Access: Location information (Geolocation API), orientation of terminal (Screen Orientation API), etc.

(4) Application Communication: asynchronous communication (WebSocket API), event push (Server-Sent Events)

(5) Multimedia: playback functionality for audio (HTMLAudioElement) and video (HTMLVideoElement)

(6) Graphics: Vector Graphics (SVG: Scalable Vector Graphics), canvas element, etc. In section 3.3, a geographical information platform is proposed using SVG.

(7) External collaboration and performance improvement: XMLHttpRequest Level 2, multi-threaded (Web Workers)

(8) Cascading Style Sheet, Level 3 (CSS3): Animations, Transitions, Transforms (2D and 3D),

(9) Automatic adaptation to a display device (Media Queries), Web fonts (WOFF: Web Open Font Format), etc. In section 3.1, we propose a cloud system using CSS3 and automatic adaption.

With these new functionalities, offline-operable applications, response improvement by multi-threaded operation, and a rich user interface employing vector graphics and 3D representation are available as Web applications, none of which can be realized with the existing HTML (HTML 4.01, XHTML 1.0).

On the other hand, since advanced functionalities and behaviors are realized, which are not supported by traditional Web content, some new concerns are beginning to emerge. First, since any Web application is able to run on any Web browser, there have been concerns about security. This is because any application can access the database in a browser and a variety of devices installed on a terminal, and thus there may be concerns about misuse of the device or the browser, or leakage of personal information. In HTML, in order to prevent the use of resources on another site or the use of local data by unauthorized third parties, restrictions on access from script not belonging to a site with the same domain (origin), called the same-origin policy, are specified. A Web browser is implemented to operate in accordance with the policy. Moreover, as shown in Fig. 2, it is known that traffic, especially the number of connections, is likely to increase because of dynamic cooperation with external resources by a Web application. In addition, by using WebSocket, multiple transactions can be processed on a single TCP connection, and SPDY [4] is being developed by Google as a technology to succeed HTTP; it optimizes only the channel without any change to HTML itself. However, the expansion and the improvement of a Web server are necessary in order to use such technologies.

At the W3C, standardization work is proceeding towards the final recommendation of HTML5 in 2014, and some additional features such as audio and video streaming, content protection, application coordination (Web Intents) are being considered. In addition, the

specification of HTML.next, the next generation of HTML, has already started. Standardization work at the W3C has been promoted mainly by the HTML Working Group (WG) [5] and the Web Applications WG [6]. Also, many other WGs such as CSS WG, SVG WG and the Device APIs WG, which develops the specifications for device cooperation with a terminal, are involved in specifying HTML5 and HTML.next. In recent years, in addition to the standardization work by each WG, community groups (CG) and business groups (BG) have been established in order to exchange views on standardization and perform a preliminary study. The number of established CG/BG is 95 as of August 21, 2012, and the active exchange of ideas and proposals to the WG is ongoing.

Fig. 2 Bottleneck factors that affect the system

3. HTML5 AS A SERVICE PLATFORM

Conventional HTML is basically the framework for information presentation. And the presentation shows structured text information with a variety of layouts including multimedia data such as images and graphics upon request from the Web browser to the Web server. Although it is possible to provide interactive services such as a scrolling map by sequentially performed requests in accordance with a user operation on a browser, the essential feature of all of this is the presentation.

HTML5, as introduced in the previous section, significantly expands presentation skills by CSS3 and Canvas. In addition, a high-efficiency communication mode and offline Javascript execution are supported, and these make it possible for HTML5 Web content to operate in a manner resembling an application. There is also evolution on the server side. A kind of application market, and flexible service construction using Web Intent to combine service functions in a server create a kind of “eco system”. For this reason, HTML5 is becoming an infrastructure and service layer that goes beyond the framework of the presentation of information.

In this section, the following three service layer topics are introduced 1) intelligent cloud, 2) expansion of HTML5 coverage from a PC or smartphone to an embedded device, and 3) geographical information platform to be used in everyday life.

3.1. Intelligent Cloud The advantage a cloud has is the collection, processing and presentation of information. In this subsection, we focus on photo services for end users from the perspective of leveraging advanced HTML5 presentation tools. For collection, it is convenient to store large amounts of information in the cloud with backup management. Reductions in the cost of building high volume servers and the spread of high-speed communication infrastructure make it possible to provide such a service at a low price. For server side processing, complex computation and data access such as face recognition and indexing to images are executed at the server, and it reduces power consumption at smartphones and also shortens the response time. For presentation, the cloud approach makes it possible to show pictures on any device such as PCs, smartphones and tablets if they are connected to networks.

For photo processing in the cloud, here, we introduce the comic-style layout display technology. In this approach, the time series of pictures is preserved in order to allow a story expressed by pictures to be presented in a comic-style layout, making the story easy to understand. Further, by using a slideshow display in conjunction with music, it is also possible to provide an audiovisual effect.

Fig.3 shows a schematic representation of the system. When photos and music material have been uploaded, the system converts the photos into plurality resolutions in order to realize “responsive images”. In responsive images, the most appropriate resolution and file size are chosen and sent to the receiver according to the resolution of the terminal display and NW bandwidth. Thus, transmission of an unnecessarily large file can be avoided. For music files, the beat of the music is analyzed from the audio signal, and it is added as a metadata of the music. Then, the system outputs the layout information to draw the slide in XML format. The layout information includes the point position of the frame pieces, cut-out position of the photos, visual effects assigned to the photo, the style information for the display timing, and number of pages.

Next, the flow is explained in detail according to the method proposed by Myodo et al. [7][8]. The system applies the following processing to a group photo in the order of: (i) Automatic picture selection so that the number of frames and the number of pages meet the requirement. (ii) Detection of critical areas in the photo. (iii) Template generation that defines the size and order of the frame. (iv) Optimal template selection in which the amount of cut-out area from the critical areas are minimized. It also specifies the location of the cut-out portion from the photo. Furthermore, in order to use the concept of “responsive Web design”[9] that began to spread with HTML5, the proposed system will switch the display layout on the browser automatically, depending on the display resolution. For responsive Web design, the following information is stored in a

style sheet: the resolution of the photo, display resolution to be switched, background material, number of frames, and the number of pages, which consists of multiple layout patterns. In addition, in the proposed system, visual effects are also applied in order to improve the subjective image quality. Employed effects are fade-in and zoom-in to the face area. Subsequently, time adjustment is applied; i.e., the beat of the music to match the timing of each visual effect.

Derived layout control information is stored in XML format for each style set to meet the requirements of responsive Web design. URI of photos and music as well as display position and comments are also stored in XML. These XML data are converted into CSS3 by Javascript when browsing slide show.

Resultant comic-style layout is shown in Fig. 4 for both cases (a) Double page layout and (b) Single page layout.(b) Single page layout

.

(a) Komawari slideshow generator

(b) Komawari slideshow viewer

Fig. 3 System configuration of comic-style layout generation

UI system

・select photos・select a music・insert comment

upload and save resources resources

photos music

comments

prepare variousresolution  images

extract music beat

photo

photos

resourcesmusic

music beat

decide slideshow parameters    

photos

beat

commentsstyleURI

calculate layout

apply transitions

decide  timing 

output XMLs

resources・photos・music beat・comments・style・background imagesURIs

generate view page

resources

resourcesHTML

PC

resources

slideshow parameters(XML)

javascript

send  URI of the view pageview page

HTMLXML

script resourcesHTMLJavascript

XML

resources

resources

get XML

get resources

screen size and shape

PC tablet smartphone

play back slideshow 

XML

Request XML

photoes

Photos

(a) Double page layout

(b) Single page layout

Fig. 4 Example of comic-style layout

3.2. Extension to embedded devices HTML5 content browsing is not limited to PCs, smartphones, and tablet, but also expanded to include embedded system equipment [10]. One typical example is the STB. Another one is a “head unit” for car navigation systems.

For an STB in cable television services, hybrid video and Web content services are progressing rapidly in addition to conventional multi-channel broadcasting services. Considering the fact that a Web browser is the most suitable way of displaying Web content, STBs and TV sets should support a Web browser. For a head unit, it is vital to be able to provide map-related information that is continuously updated and to offer a variety of multimedia services, hence deployment of the HTML5 browser is attracting attention. At this time, accessing information such as driving conditions from inside the vehicle is necessary in order to provide a service similar

to a conventional car navigation system. Further, in the configuration of the system, one option is to have a communication module inside the head unit; the other option is to exclude it in order to reduce costs. In the latter case, communication functionality needs to be provided by other devices such as the user’s smartphone, and the device relays data to the head unit.

The system configuration is shown in Fig. . While the head unit has a communication function for WiFi, it does not have communication network access capability. In addition, it is equipped with an HTML5 browser to display Web content. The head unit is linked with the smartphone. The smartphone has communication capability for both a mobile phone communication network and WiFi. In this system, a simple tethering operation can be used to access the mobile phone network; here the smartphone works like a simple pipe. However, this configuration has the following disadvantages. When you enter the mountains while riding in the car and can no longer have access to a mobile phone network, the head unit is not able to provide services. The other disadvantage is that there is a time-consuming problem at the mobile phone communication network if the head unit accesses large-capacity information such as a map image or video content. This can be resolved by storing large-volume content in the smartphone in a WiFi broadband environment such as at the home before getting into the car. As a further benefit, a user can enjoy planning the next trip using the smartphone, find out information about what to see and do, and the information is automatically transferred into the head unit through the smartphone. At the same time, the system can learn a user’s preferences according to the user’s Web search history and travel log, and then the system can pre-fetch relevant information and store this information in the smartphone. Thus, the system can provide useful information while on the road with no latency [11] . This form of use is shown in Fig. .

Fig. 5 Empowering HTML5 head unit with

smartphone

Head unitSmartphoneCloud

Wi-Fi3G/WiMAX/LTE

Inside car

Inside house

Wi-Fi

FTTH/CATV

HTML5 content is downloaded to Smartphone in Wi-Fi area

HTML5 application works on head unit.

Fig. 6 Infotainment system

3.3. Geographical information platform One of the most useful applications for smartphones is information services based on a user’s location. A typical example is personal navigation, in which, for example, a user can see a map, his/her current position, and the directions to a restaurant. At this time, the map will be displayed with some icons to show the location of nearby restaurants, however, restaurant information is not included in the original map. The system overlays the restaurant icons based on the latitude and longitude information at the time of the user’s request.

As a data format for presenting the above maps efficiently, SVG is one option. SVG is a scalable vector graphic, which is a representation format of the image that can be displayed beautifully even when scaled. It should be noted that the data size becomes excessively large if the entire country of Japan is expressed by a single map file. Hence, a tiling approach is used, wherein the entire area is subdivided into smaller areas, allowing the user to download the required areas and join them together to make a larger map. The map data also has the latitude and longitude information to enable overlay [12][13].

Although various data formats that existed in the past, these forms of geographic information have recently been promoted in the W3C standardization [14] with a view to providing services based on a Web browser. W3C is developing the next generation standard for SVG, referred to as "SVG 2", and there is a contribution regarding SVG map technique to be included in SVG 2. That contribution consists of two parts. 1) Incorporation of an SVG map display function into a prototype web browser based on Chromium in order to display SVG map content natively. 2) Development of an SVG map platform that runs with JavaScript in order to view it on some major browsers, in which traditional SVG is enhanced by using a tiling and layering approach. An example of an SVG map application is shown in Fig. 7. It is an information service system that shows the areas where mobile phone services have been suspended when a severe natural disaster occurs.

Fig. 7 Geographical information representation

4. CONCLUSION

Initially developed as a hypertext framework for a computer, to which expressive power and dynamic control functions such as CSS and JavaScript were added, HTML5 has been able to meet the demands of local application functions such as device access, local storage and background processes. For the future, there are high expectations that HTML5 will play a major role and evolve further as an application platform for single-source, multi-terminal devices that can be used widely in a variety of cloud services.

5. REFERENCES [1] “http://www.whatwg.org/” [2] “http://www.slideshare.net/dynamis/toward-firefox-os” [3] Jeff Jaffe, “The Open Web Platform for Entertainment”, “http://www.w3.org/2012/06/signage/slides/W3C_Jeff/jj-entertainment-1.pdf”, (Jun. 2012). [4] M. Belshe, R. Peon, “SPDY Protocol, Internet-Draft”, “http://tools.ietf.org/html/draft-mbelshe-httpbis-spdy-00”, (Feb. 2012). [5] “http://www.w3.org/TR/tr-groups-all#tr_HTML_Working_Group” [6] “http://www.w3.org/TR/tr-groups-all#tr_Web_Applications_Working_Group” [7] Emi Myodo, Satoshi Ueno, Koichi Takagi, Shigeyuki Sakazawa, “Automatic comic-like image layout system preserving image order and important regions”, Proc. ACM Multimedia 2011, pp.795-796, November 2011. [8] Emi Myodo, Koichi Takagi, Shigeyuki Sakazawa, “Comic-like Photo Slideshow System by using HTML5”, Proc. ITE General Conference 2012, August 2012. (in Japanese) [9] Marcotte, Ethan, “Responsive Web Design”, A BOOK APART, 2011 [10] Nikkei Electronics, pp.25-39, July 9, 2012. (in Japanese) [11] “http://en.responsejp.com/article/2012/05/30/175242.html” [12] “http://www.slideshare.net/totipalmate/webgis-20110311-takagi” [13] “http://www.slideshare.net/totipalmate/webgis-20120525-takagi” [14] “http://www.w3.org/Graphics/SVG/”