BFlavor: an Optimized XML-based Framework for Multimedia Content Customization

7/27/2019 BFlavor: an Optimized XML-based Framework for Multimedia Content Customization

1/6

BFlavor: an Optimized XML-based Framework

for Multimedia Content Customization

Davy Van Deursen, Wesley De Neve, Davy De Schrijver, and Rik Van de Walle+

Department of Electronics and Information Systems - Multimedia LabGhent University - IBBT

+Ghent University - IBBT - IMECSint-Pietersnieuwstraat 41, B-9000 Ghent, Belgium

e-mail: [email protected]

Abstract. During recent years, several languages havebeen developed that allow to automatically create de-scriptions containing information about the high-level

structure of binary multimedia resources. Such an ap-proach makes it possible to tackle the diversity of thecurrent networks, terminals, and multimedia formatsin a transparent way. This paper discusses the funda-mentals of BFlavor, a novel language for describing thestructure of binary multimedia resources. BFlavor hasbeen developed to combine the strengths of MPEG-21BSDL and XFlavor and to avoid their weaknesses. Thethree different scalability axes of the H.263+ coding for-mat (i.e., temporal, spatial, and Signal-to-Noise Ratioscalability) are exploited by making use of MPEG-21BSDL, XFlavor, and BFlavor. We show through exper-iments that BFlavor maintains the constant memoryconsumption of XFlavor, generates compact descrip-

tions, and outperforms XFlavor and MPEG-21 BSDLin terms of execution times.Index Terms -BFlavor, content adaptation, H.263+,MPEG-21 BSDL, XFlavor

1 Introduction

Today, in the world of multimedia, there is a hugeheterogeneity in terminals, networks, and multime-dia formats used. These differences have to be takeninto account when multimedia resources have tobe delivered to different devices. Scalable codingmakes it possible to tackle this tremendous diver-

sity. It enables the extraction of multiple (lowerquality) versions of the same multimedia resourcewithout the need of a complete encode-decode pro-cess. This is in line with the vision of the UniversalMultimedia Access (UMA) philosophy: a multime-dia resource only needs to be created once, afterwhich it can be published to all possible terminalsusing all possible different networks. It is importantto realise that an efficient solution for the hetero-geneity does not only imply the usage of scalablecoding, but also the usage of a complementary con-tent adaptation system. One could build a content

adaptation system operating directly onto the bit-stream. However, such an implementation will beerror prone and the adaptation system will only be

capable to support one bitstream format. There-fore, a generic approach is needed for the adapta-tion of (scalable) bitstreams. Such an architecturehas to support multiple multimedia formats andit should be possible to extend it with new multi-media formats. These adaptive multimedia systemscan be realised by relying on textual descriptions ofthe high-level structure of scalable bitstreams (usu-ally in XML format). In this paper, a new languageis introduced for the creation of these textual de-scriptions.

The outline of this paper is as follows. In Sec-tion 2, the concept of bitstream structure descrip-tion languages is elaborated. Two existing lan-guages, in particular MPEG-21 BSDL and XFlavorare discussed. Section 3 introduces BFlavor (BSDL+ XFlavor), our new bitstream structure descrip-tion language. In Section 4, this new approach isvalidated by presenting some experimental results,hereby targeting the H.263+ coding format. Fi-nally, conclusions are made in Section 5.

2 Bitstream Structure Description

Languages

2.1 Overall Approach

A possible architecture for the customization of(scalable) bitstreams is shown in Fig. 1. Suchan adaptation system can be realized by relyingon Bitstream Syntax Descriptions (BSDs). A bit-stream structure description language makes it pos-sible to describe the structure of a specific bit-stream format. A text-based BSD is generated bya bitstream-to-BSD parser which interprets thestructure of the bitstream. The BSD typically con-tains information about the high-level structure


2/6

(information about packets, headers, or layers ofdata) of a (scalable) bitstream. Such a BSD can beseen as an intermediate format that acts as an ab-straction layer for the bitstream. The most elegantmanner to store the text-based BSD is by usingthe ubiquitous XML specification. The XML for-malism does not only allow repurposing many ex-isting tools for manipulating XML-based BSDs, italso allows a straightforward integration with othermetadata standards, such as the MPEG-7 speci-fication. According to a given set of constraints,the BSD is transformed, resulting in a customizedBSD. Because of the high-level nature of the BSD,only a limited knowledge about the structure ofa bitstream is required to perform this transfor-

mation. Finally, the customized BSD is used by aBSD-to-bitstream parser to generate an adaptedbitstream. The major advantage of a BSD-drivenapproach for the customization of (scalable) bit-streams is its generic architecture. To support anew multimedia format, only a description of thestructure of a bitstream format in the specific lan-guage has to be created while the same genericparsers can still be used.

2.2 MPEG-21 BSDL

The MPEG-21 Bitstream Syntax Description Lan-

guage (MPEG-21 BSDL) is a tool of part 7 (DigitalItem Adaptation, DIA) of the MPEG-21 specifi-cation. It is built on top of the World Wide WebConsortiums (W3C) XML Schema Language andis able to describe the structure of a (scalable) bit-stream in XML format [1]. The primary motiva-tion behind the development of MPEG-21 BSDLis to assist in customizing scalable bitstreams [2].The Bitstream Syntax Schema (BS Schema), whichcontains the structure of a certain media format, isused by the BintoBSD Parser to generate a BSDfor a given (scalable) bitstream. After the BSD istransformed, an adapted bitstream is created by us-

ing the BSDtoBin Parser, which takes as input theBS Schema, the customized BSD, and optionallythe original bitstream.

2.3 XFlavor

The Formal Language for Audio-Visual Ob-ject Representation, extended with XML features(XFlavor) is a declarative C++-like language todescribe the syntax of a bitstream on a bit-per-bit basis. XFlavor was initially designed to sim-plify and speed up the development of software that

description of the

structure of a

bitstream

customized

bitstream

BSD-to-bitstream

parser

customized

BSD

Transformation

BSD

scalable

bitstreambitstream-to-BSD

parser

set of constraints

(e.g., usage

environment

descriptions)

bitstream structuredescription language

optional

Fig. 1. A BSD-driven content adaptation framework

processes audio-visual bitstreams by automaticallygenerating a parser for these bitstreams. By ex-tending this automatically generated parser withXML features, it was possible to translate the syn-tax of a bitstream in XML format [3]. The XFlavorcode, which contains a description of the syntax ofa certain media format, is translated by the Fla-vorc translator to Java or C++ classes. This setof classes is compiled to a coding format-specificparser. XFlavor comes with the Bitgen tool for cre-ating an adapted bitstream, hereby guided by thecustomized BSD. Note that the Flavorc translator

is also able to generate an XML Schema which canbe used to validate the BSD that is created by theautomatically generated parser.

3 BFlavor: Harmonizing MPEG-21

BSDL and XFlavor

Although MPEG-21 BSDL and XFlavor can beused as stand-alone tools [4], a harmonized ap-proach can combine the strengths of the two tech-nologies. In XFlavor, the XML schema is onlyused for validation purposes. The bitstream gen-

erator (i.e., Bitgen) only uses information fromthe BSD and thus is independent of the XFlavorcode. Hence, the complete bitstream data are ac-tually embedded in the BSD, resulting in poten-tially huge descriptions. On the contrary, MPEG-21 BSDL makes use of a specific datatype to pointto a data range in the original bitstream whenit is too verbose to be included in the descrip-tion (i.e., by making use of the language constructbs1:byteRange). This results in BSDs containingonly the high-level structure of the bitstream. Thestrengths of XFlavor are the fast execution speed


3/6

c l a s s P i c t u r e {

B a s e _ l ay e r b _ l a ye r ;

E n h a n c e me n t _ l a ye r e _ l a y er ;

}

c l a s s B i t s t r ea m {

while(1)

P i c t u re p i c t ur e ;

}

Fig. 2. BFlavor code example

and the low and constant memory consumption ofthe coding format-specific parser, while the Binto-BSD Parser of MPEG-21 BSDL struggles with anunacceptable execution speed and increasing mem-ory consumption (see Section 4) caused by an in-efficient XPath evaluation process. This is due to

the fact that the entire description of the bitstreamstructure is kept in system memory in order to allowthe evaluation of arbitrary XPath 1.0 expressions.

BFlavor, our novel bitstream structure descrip-tion language, bridges the gap between MPEG-21BSDL and XFlavor. It is developed to combine thestrengths of MPEG-21 BSDL and XFlavor, i.e., togenerate a compact high-level BSD at a fast execu-tion speed and with a constant memory consump-tion. It is built on top of XFlavor by defining a num-ber of restrictions and extensions (such as MPEG-21 BSDL is build on top of W3C XML Schema). Byusing the automatically generated parser of BFla-vor, it is possible to generate BSDs that can be fur-ther processed by the BSDtoBin Parser of MPEG-21 BSDL. Note that one could also modify theMPEG-21 BSDL language specification in orderto achieve a feasible execution speed and a con-stant memory consumption [5]. The performanceof this approach, together with the performance ofMPEG-21 BSDL and XFlavor, will be compared toBFlavors performance in Section 4.

3.1 Working of BFlavor

The BFlavor adaptation chain is illustrated inFig. 4. The BFlavor code describes the structureof a specific bitstream format. In Fig. 2, an ex-ample is given of a BFlavor code fragment. It de-scribes Pictures that contain a Base layer and anEnhancement layer. The BFlavorc translator usesthis code to generate Java source classes that canbe compiled to a coding format-specific parser. Sofar, the XFlavor approach has been followed. Fromthis point, a switch is made to the MPEG-21 BSDLapproach. The coding format-specific parser gener-ates a BSD which can be further processed by the

Fig. 3. An example of a BSD before and after a trans-formation

BFlavor code

customized

bitstream

BSDtoBin

customized

BSDtransformationBSD

bitstream

coding format

specific parserJava classes

BFlavorc

BS Schema

Fig. 4. The BFlavor adaptation chain

efficient and format-agnostic BSDtoBin Parser of

MPEG-21 BSDL. An example of a such a BSD isillustrated in Fig. 3. The BSDtoBin Parser needs aBS Schema. This schema contains the structure ofa specific bitstream format, analogous to the BFla-vor code. Therefore, the BFlavorc translator is alsoable to generate a BS Schema from the BFlavorcode. Thus, after the transformation of the BSD,the BSDtoBin Parser uses the customized BSD, thegenerated BS Schema, and the original bitstreamto generate a customized bitstream. As a resultof our approach, it is possible to generate BSDswith the fast execution speed and the low memoryconsumption of the coding format-specific parser

of XFlavor, while the generated BSDs are compactbecause they only contain a description of the high-level structure of the (scalable) bitstream.

3.2 Implementation details

In Section 3.1, the general working of BFlavor wasillustrated. In order to realize the construction ofBFlavor (i.e., the generation of a parser producingBSDs usable by the BSDtoBin Parser of MPEG-21 BSDL), a few restrictions and extensions haveto be defined on top of XFlavor. This section will


4/6

discuss these restrictions and extensions, as well asthe methodology needed to implement them.

Restrictions and extensions on top of XFla-

vor Section 3.1 has shown that the Flavorc trans-lator has to be able to generate Java source classesand a BS Schema using the BFlavor code. Thisis only possible when a number of restrictions aredefined for this BFlavor code. Because the BFla-vor code is a C++-like code, it contains variables.These variables can be parsable (i.e., they retrievetheir value from the bitstream) or non-parsable(i.e., regular C++ variables). The parsable vari-ables are similar to the xsd:element language con-

struct in MPEG-21 BSDL. Non-parsable variablescannot be translated to an MPEG-21 BSDL lan-guage construct which implies a prohibition onthe use of non-parsable variables in the BFla-vor code. Other language constructions in XFla-vor which cannot be translated to MPEG-21 lan-guage constructs are the map and bac construc-tions as well as multi-dimensional arrays. Hence,only one-dimensional arrays are allowed in BFla-vor as they can be translated to elements with abs2:nOccurs attribute. A last restriction is theuse of the built-in operators of XFlavor. Onlythe nextbits() operator can be translated to an

MPEG-21 BSDL element (i.e., an element with thebs2:ifNext attribute). This implies that the otherbuilt-in operators (i.e., nextcode(), numbits(),lengthof(), isidof(), and skipbits()) are pro-hibited in BFlavor.

XFlavor also needs some extensions to be able togenerate Java source classes and a BS Schema. Afirst extension is the possibility to include informa-tion about the root element, namespace, and targetnamespace in the BFlavor code. A second extensionis related to datatypes. In MPEG-21 BSDL, onecan have elements and types, while XFlavor onlyhas classes. As such, it must be possible to signal

in the XFlavor code that a class represents a typeor an element. There are three options to signal anXFlavor class as a type: the class definition has tobe translated to a complexType element; the classdefinition has to be translated to a simpleType el-ement, hereby providing information about whatprocedural object to use (i.e., by making use ofthe bs0:implementation attribute in MPEG-21BSDL); or the class definition has to be translatedto a simpleType element, hereby providing infor-mation about the length of a start or end code. Adisadvantage of XFlavor was the lack of possibility

%targetns{H_263%targetns}

%ns{h263%ns}

%root{Byte_stream%root}

%align{%align}

c l a s s S t u f f in g { }

%payload{0000FC-0000FF%payload}

%payload{000080-000083%payload}

c l a s s G O B _ P ay l o a d { }

c l a s s P i c t ur e {

b i t ( 2 2) p i c t u r e_ s t a r t _ co d e = 0 b 0 0 0 0 0 0 0 00 0 0 0 0 0 0 01 0 0 0 0 0 ;

b i t ( 8 ) t e m p o r al _ r e f e re n c e ;

P t y p e p t y p e ;

i f ( p t y pe . s o u r c e _ fo r m a t = = 7 )

P l u s _ He a d e r p l u s _ he a d e r ;

b i t ( 5 ) p q u a n t ;

i f ( p t y pe . s o u r c e _ fo r m a t ! = 7 )

b i t ( 1 ) c p m ;

i f ( p t y pe . s o u r c e _ fo r m a t ! = 7 & & c p m = = 1 )

b i t ( 2 ) p s b i ;

i f ( p t y pe . o p t i o n a l _ pb f r a m e s_ m o d e = = 1 ) {

b i t ( 3 ) t e m p o r al _ r e f e r en c e _ b ;

b i t ( 2 ) d b q u an t ;

}

do {

b i t ( 1 ) p e i ;

i f ( p e i = = 1 ) b i t ( 8 ) p s u p p ;

} w h i l e ( p ei = = 1 ) ;

S t u f fi n g e s t u ff ;G O B _ P ay l o a d p a y l o ad ;

}

c l a s s B i t s t re a m {

while(1)

P i c t ur e p i c t u re ;

}

Fig. 5. Partial BFlavor code for the H.263+ coding for-mat

32

0

13

015

< p a y l o ad > 1 0 3 8 9 9 < / p a yl o a d >

Fig. 6. Example of BSD for a H.263+ bitstream

to refer to the original bitstream. BFlavor solvesthis problem by defining a payload datatype thatcan be translated to a simpleType element with re-striction to the bs1:byteRange datatype of MPEG-21 BSDL. The last extension is a datatype which

is able to make the bitstream byte-aligned. This ismapped to the non-normative bs0:fillByte con-struct in MPEG-21 BSDL.

Methodology The extensions are implemented byusing the verbatim codes of XFlavor. These codesare enclosed within the verbatim delimiters %x{and %x} in the XFlavor code. The symbol x herebydenotes the type of the verbatim code. In Fig. 5,a partial BFlavor fragment for the H.263+ cod-ing format is illustrated. Specific verbatim codesare used to signal the root element, namespace,


5/6

0,0

5,0

10,0

15,0

20,0

25,0

0 500 1000 1500 2000 2500 3000 3500 4000

#Pictures

Executiontime(s)

BintoBSD_m

XFlavor

BFlavor

Fig. 7. Execution speed (SNR)

and target namespace. Classes that represent atype will be preceded by the specific verbatim codeassociated with that type. In the example, theStuffing class represents an alignment type, whilethe GOB Payload class is a payload type. The valueof the verbatim codes is used to pass some addi-tional information about the datatype. For exam-ple, the payload datatypes contain some start codesas additional information as illustrated with theGOB Payload class.

The modifications to create BFlavor on top ofXFlavor have to be made inside the Flavorc trans-

lator. The working of this translator is as follows.The XFlavor code is parsed by making use of aparser generated by Lex/Yacc. Based on the in-ternal representation, generated by the Lex/Yaccsoftware, an XML Schema or a set of Java or C++classes is generated. To implement the restrictionsand extensions in the Flavorc translator, modifica-tions have to be made in the Lex and Yacc descrip-tion, as well as in the core of the Flavorc translator.In the Lex and Yacc description, the new verba-tim codes have to be inserted in order to be recog-nized by the generated Lex/Yacc-parser. The coreof the Flavorc translator has to be modified in two

manners. Firstly, the generated XML Schema rep-resents a BS Schema. Secondly, the generated Javaor C++ classes are able to generate an XML docu-ment which can be further processed by MPEG-21sBSDtoBin Parser.

4 Experimental Results

To validate our harmonized approach, a compari-son of MPEG-21 BSDL, XFlavor, and BFlavor isprovided in terms of execution times, memory con-sumption, and file sizes. The languages are used

0,0

10,0

20,0

30,0

40,0

50,0

60,0

300 900 1800 3600

#Pictures

Filesize(MB)

MPEG-21 BSDL

BFlavor

XFlavor

Fig. 8. BSD file sizes (Temporal)

Table 1. Bitstream characteristics

Base Layer Enhancement Layer#Pic . Res. QP # Pic. Res . Q P

Spat. 300 QCIF 13 300 CIF 13Temp. 300 QCIF 13 600 QCIF 13

SNR 300 QCIF 25 300 QCIF 5

to exploit the scalability properties of H.263 Ver-sion 2 bitstreams. H.263 Version 2, also knownas H.263+, supports both temporal, spatial, andSignal-to-Noise Ratio (SNR) scalability. The Fore-man sequence with QCIF (176x144) resolution was

used as test sequence. Table 1 shows the bitstreamcharacteristics for the encoded test sequence with300 pictures. The characteristics for the sequenceswith 900, 1800, and 3600 pictures are similar. Thetechnologies generate a BSD of the bitstream whichcontains the base layer and the enhancement layer.During the transformation, the enhancement layeris dropped by using eXtensible Stylesheet LanguageTransformations (XSLTs). Finally, a bitstream isgenerated from the customized BSD which containsonly the base layer. All tests were done five times,after which an average was taken.

The experiments were done on a PC having an

Intel Pentium D 2.8 GHz CPU and 1 GB of sys-tem memory at its disposal. The operating systemused was Windows XP Pro SP2, running Sun Mi-crosystemss Java 2 Runtime Environment (Stan-dard Edition version 1.5). SAXON 8 was used forapplying XSLTs to BSDs. The memory consump-tion was registered by relying on the JProfiler 4.0software package. The TMN-3.0 H.263 codec wasused to encode/decode the test sequences.

The results of the BSD generation speed areshown in Table 2. Note that BintoBSDr standsfor the original BintoBSD Parser (v1.2.1), while


6/6

Table 2. BSD generation speeds

BintoBSDr BintoBSDm BFl. XFl.#Pic. (s) (s) (s) (s)

300 278.6 2.7 0.3 1.0Spat. 900 2332.2 6.6 0.7 3.1

1800 9127.7 12.4 1.3 5.83600 N/A 18.4 1.9 9.2300 232.6 2.4 0.2 0.6

Temp. 900 2013.1 6.4 0.6 1.61800 8712.6 11.2 1.0 3.23600 N/A 19.8 1.5 5.8300 270.8 2.7 0.3 1.0

SNR 900 2288.5 6.6 0.7 2.81800 8902.2 12.1 1.2 5.53600 N/A 23.5 2.3 11.1

Table 3. Adaptation and bitstream generation times

XSLT (s) Bitstream Generation (s)#Pic. BSDL BFl . XFl. BSDL BFl. XFl.

300 0.6 0.6 2.6 0.6 0.6 1.6Spat. 900 1.1 1.1 5.1 0.7 0.7 4.7

1800 1.6 1.6 9.7 1.0 0.9 9.33600 2.3 2.3 14.7 1.2 1.2 14.2300 0.5 0.5 2.2 0.6 0.6 1.5

Temp. 900 0.9 0.8 3.1 0.7 0.7 6.11800 1.4 1.3 8.1 0.9 0.9 12.13600 2.5 2.7 15.5 1.3 1.3 24.3300 0.6 0.6 2.1 0.6 0.6 1.6

SNR 900 1.1 1.0 4.2 0.7 0.7 5.61800 1.6 1.7 7.8 1.0 0.9 11.23600 2.6 2.7 16.2 1.3 1.4 16.3

BintoBSDm stands for a modified version of theBintoBSD Parser which supports the BSDL ex-

tensions for achieving a usable adaptation frame-work [5]. It is clear that the BintoBSDr Parser can-not be used in practice because of the quadraticincreasing execution time and the increasing mem-ory consumption (35.0 MB and 62.0 MB for abitstream having 900 and 1800 Pictures respec-tively). This is due to the fact that the entire BSDis kept in memory in order to allow the evalua-tion of XPath expressions. The BintoBSDm Parser,the BFlavor-based parser, and the XFlavor-basedparser all maintain a low and constant memory con-sumption (less than 2 MB). Furthermore, Fig. 7illustrates that these parsers show a linear behav-

ior in terms of execution speed. One can see thatthe BFlavor-based parser outperforms the XFlavor-based parser and the BintoBSDm Parser. Table 3shows the transformation and bitstream generationtimes for the adapted BSDs. The most notable isthe poor transformation speed for the BSDs gen-erated with XFlavor. This is because all the infor-mation to create the adapted bitstream has to beincluded in the customized BSD, resulting in largeBSDs. Fig. 8 illustrates the difference in BSD filesizes between XFlavor, MPEG-21 BSDL, and BFla-vor.

5 Conclusions and Future Work

We have presented BFlavor, a new solution for

XML-driven content adaptation. BFlavor is de-signed to combine the strengths of XFlavor andMPEG-21 BSDL and to avoid their weaknesses. Ex-perimental results have shown that BFlavor main-tains the constant memory consumption of XFlavorand outperforms XFlavor and MPEG-21 BSDL interms of execution times. Thanks to its possibil-ity to refer to the original bitstream, BFlavor isalso able to describe the high-level structure of abitstream, resulting in compact BSDs that can befurther processed by MPEG-21s BSDtoBin Parser.Future work will focus on the exploitation of scal-ability of the H.264/AVC SVC coding format byrelying on BFlavor. This will require further devel-opment of BFlavor.

Acknowledgements

The research activities as described in this pa-per were funded by Ghent University, the Inter-disciplinary Institute for Broadband Technology(IBBT), the Institute for the Promotion of Innova-tion by Science and Technology in Flanders (IWT),the Fund for Scientific Research-Flanders (FWO-Flanders), the Belgian Federal Science Policy Office

(BFSPO), and the European Union.

References

1. Devillers, S., Timmerer, C., Heuer, J., Hellwagner,H.: Bitstream Syntax Description-Based Adaptationin Streaming and Constrained Environments. IEEETrans. Multimedia 7 (3) (2005) 463470

2. De Schrijver, D., Poppe, C., Lerouge, S., De Neve,W., Van de Walle, R.: MPEG-21 Bitstream SyntaxDescriptions for scalable video codecs. (accepted forpublication in Multimedia Systems Journal)

3. Hong, D., Eleftheriadis, A.: XFlavor: Bridging Bitsand Ob jects in Media Representation. In: Proceed-

ings, IEEE Intl Conference on Multimedia and Expo(ICME), Lausanne, Switzerland (2002)4. De Neve, W., Van Deursen, D., De Schrijver, D.,

De Wolf, K., Van de Walle, R.: Using BitstreamStructure Descriptions for the Exploitation of Multi-layered Temporal Scalability in H.264/MPEG-4AVCs Base Specification. In: Proc. of PCM 2005,Chejudo, Korea, Springer-Verlag (2005) 641652

5. De Schrijver, D., De Neve, W., De Wolf, K., Van deWalle, R.: Generating MPEG-21 BSDL DescriptionsUsing Context-Related Attributes. In: Proceedingsof the 7th IEEE International Symposium on Mul-timedia, Irvine, USA (2005) 7986

BFlavor: an Optimized XML-based Framework for Multimedia Content Customization

Documents

Transcript of BFlavor: an Optimized XML-based Framework for Multimedia Content Customization