Direct Mapping of X3D Scenes to MPEG-7 Descriptions

TEI of Crete Multimedia Lab

Direct Mapping of X3D Scenes to MPEG-7 Descriptions

Markos Zampoglou, Patti Spala, Konstantinos Kontakis, Athanasios G. Malamos, J Andrew Ware


What is our motivationi-Promotion: we work on a platform for VR web advertising. The goal of this project is to join cloud stored X3D objects to more complicated scenes and to publish them in the way that we used to publish GIF Banners!

The first questions we had to answer was how can we search and retrieve the resources according to technical as well as functional and contextual characteristics. The resources might be the 3D geometry as well as Routes, Textures (images or videos), audios and Sounds embedded on the 3D.

What is missing is an efficient annotation of X3D in MPEG7.

MPEG-7 defines several descriptors to facilitate the description of 3D entities involving geometries, textures, animations and content metadata. However, the majority of 3D descriptors are centered on vertices representation of 3D objects, lacking in the ability to provide optimized semantic descriptions based on XML encoded 3D objects. However the existence of the 3D descriptors helped us to extend them to fit in our case allowing us to still use the older and the newer descriptors at the same time.

I-Promotion is a project granted by the European Union and the Hellenic General Secretary of Research and Technology under the ”COOPERATION 2009 /09SYN-72-956” Framework.


The i-Promotion platform

HTTP

SOAPCROSS3D Service Protocols

HTTPSOAP

I Promotion Service Protocols (MPEG7, MPEG21, etc)

Virtual machinesCloud

I-promotion coreCROSS 3D Hosts

3D Model Repositories Rendering enginesRendering engines

Device Interfaces

Mobile WEB Digital TV

Client Devices

Mobile WEB

3D Model Repositories Rendering enginesRendering engines

Device Interfaces

Mobile WEB Digital TV

Device Interfaces

WEB/HTML5

Digital Media

Client Devices

HTTPHTTP

Mobile WEB Digital Media

RTSPWeb-socketsHTTPHTTP/SOAP

WEB/HTML5

3D worlds Authoring

tool

3D banners Promotion

Manager

Media framework


The i-Promotion platform architecture

• A cloud-based repository for 3D scenes• A remote rendering server

– supplementing a potential local rendering approach• Service-oriented content distribution• HTML5-based clients

– PCs, mobile devices, wall-to-wall projections…• Supplementary modules for content adaptation

– Device Interfaces, Media Framework• Additional tools

– 3D Worlds authoring tool, 3D banners promotion manager


The i-Promotion platform:Standards conformance

• MPEG-7: Content description for 3D scenes– Content-based search

• MPEG-21: Multimedia Item declaration & adaptation– Scene organization (thumbnails, descriptors, rights)– Content adaptation for extreme device flexibility

• ???? : Scene structure composition• X3D: 3D scene format

– Offers a choice between server-based rendering or local, HTML5 compatible rendering (via X3DOM)


MPEG-7 for 3D scenes

• MPEG-7 – Part 3: Visual includes descriptors for 3D model shape.

• Efforts towards full 3D scene description (shape, color, structure, light etc.)– 3DSEAM: X3D scene structure and object location (Bilasco et al.

2006)– Multimedia Interaction Model (Chmielewski 2012)– MPEG-7 Shape and semantics for MPEG-4 3D scenes

(Bonhomme et al. 2007)– Various RDF and OWL Semantic annotations


What MPEG 7 does support in 3DGeometry DescriptorsMPEG-7 Part 3, Visual, defines two shape descriptors for 3D objects: the Shape3D Spectrum descriptor and the MultipleView Descriptor Container.

Motion DescriptorsMPEG-7 Part 3, Visual, defines three descriptors, The Camera Motion descriptor, The Motion Trajectory descriptor and the Motion Activity descriptor.

Texture and Color DescriptorsIn order to describe textures, MPEG-7 Part 3, Visual, provides three texture descriptors: the TextureBrowsing Descriptor, the HomogeneousTexture Descriptor and EgdeHistogram Descriptor. The color description tools presented in, MPEG-7 Part 3, Visual, include four distinct color descriptors to annotate different color features, the DominantColor Descriptor, the ScalableColor Descriptor, and the ColorLayout and ColorStructure Descriptors. Meanwhile, two additional supporting descriptors are included, the ColorSpace and ColorQuantization Descriptors, which are used in conjunction with the DominantColor Descriptor, providing supplementary details to representative colors.


MPEG-7 Extension DescriptorsMPEG-7 Part Name Usage

MPEG-7Part 3 – Visual

BoundingBox3D Position and size of a 3D object in the scene

Geometry3D Types of primitive or complex geometries and their curvature.

Metadata3D Additional metadata information (such as MatadataFloat, MetadataInteger, MetadataSet).

Interactivity3D Interactions with other objects in the scene or with the end user, (based on X3D ROUTE elements).

MotionTrajectory Animation characteristics of a moving object: type of animation, through the Interpolator or Follower nodes bound to that object, along with the motion path keypoint values.

Viewpoint3D Viewpoint nodes’ position, orientation, animation, and coordinates.

Lighting3D The type of X3D Light nodes used within an X3D Scene, their illumination specifications and their scope (global or not).

MPEG-7Part 5 – MDS

Profile3D The X3D profile.

Script3DIf the file contains Script nodes, specifies the script class location (internal or external) and the scripting language used.


Automatic extraction of MPEG-7 descriptions

• X3D and MPEG-7 are both XML-based standards• We use XSLT for the direct extraction of (extended-)

MPEG-7 compliant descriptions from X3D scenes– Both scenes are kept: the original X3D scene is retained for

rendering and display, while the new MPEG-7 description is used for search and retrieval


XSLT Mapping

X3D Scene Document

(in XML format)

XSLT Processor

XSLT Style Sheet

MPEG-7 XML Document

X3D Scene and Model Annotation

Repository

3D Media Graphics Repository


Extensions in MPEG-7 Part 3

Visual


BoundingBox3D Elements

Name DefinitionBoundingBox3DType Descriptor

Volumetric coordinates as obtained from the X3D BoundingBox element

BoundingBox3DSizeWidth, Height and Length coordinates of the BoundingBox, passed into the BoxWidth, BoxHeight and BoxDepth attributes. Default values are (-1, -1, -1) as in the X3D bounding box specification.

BoundingBox3DCenter

Center coordinates of the BoundingBox. Each of the three values contained in bboxCenter is passed into the BoxCenterW, BoxCenterH and BoxCenterD attributes. Default values are (0, 0, 0) as in the X3D bounding box specification.


BoundingBox3D


BoundingBox3DType FieldsMPEG-7 Description Fields Value Type Input from X3D Fields

BoundingBox3DType Descriptor - Created if an X3D Grouping Node contains valid

bboxCenter and bboxSize attributes

BoundingBox3DSize - Created if an X3D Grouping Node contains a valid bboxSize attribute

BoxWidth Float Position[1] of the value of the X3D bboxSize attribute String

BoxHeight Float Position[2] of the value of the X3D bboxSize attribute String

BoxDepth Float Position[3] of the value of the X3D bboxSize attribute String

BoundingBox3DCenter - Created if an X3D Grouping Node contains a valid bboxCenter attribute

BoxCenterW Float Position[1] of the value of the X3D bboxCenter attribute String

BoxCenterH Float Position[2] of the value of the X3D bboxCenter attribute String

BoxCenterD Float Position[3] of the value of the X3D bboxCenter attribute String


BoundingBox Mapping


Geometry3D Elements

Name Definition

Geometry3DType Describes the types of primitive or complex shapes contained in an X3D file.

Geometry3D Describes one shape from the X3D file.

Metadata3D Specifies the additional metadata values, such as MetadataDouble, MetadataFloat, and MetadataSet.


Geometry3D


Geometry3DType Fields

MPEG-7 Description Fields Value Type Input from X3D Fields

Geometry3D Descriptor - Created if an X3D Shape element

contains an X3D Geometry nodecreaseAngle Float Value of the X3D creaseAngle

attributeconvex Boolean Value of the X3D convex attributeObjectType String Value of the X3D Geometry node type

(node name)DEF String Value of the X3D DEF attributeMetadata3D - Created if an X3DMetadataObject

exists within the X3D Geometry node.


Geometry3D Mapping


Metadata3DType ElementsName Definition

Metadata3DType Additional metadata values contained within an X3D node, such as MatadataFloat, MetadataInteger, MetadataSet.

name The name of the metadata type being described, from the Name attribute of the X3DMetadataObject.

type Specifies the type of metadata being described, such as MatadataFloat, MetadataInteger, MetadataSet.

referenceThe value of the reference field from the X3DMetadataObject. If the field is not provided or is empty, the meaning of the name field is considered implicit to the characters in the string.

valueAll metadata values as derived from the corresponding X3DMetadataObject. In the case of a MetadataSet, the value element contains all internal elements and values.

Ref The XPath location from the X3D XML graph of the metadata element being described.


Metadata


MetadataSet


Metadata3DType Fields


Metadata3D Descriptor -

If an X3D Metadata element such as MatadataFloat, MetadataInteger, MetadataSet, exists.

name String Value of the X3DMetadataObject name attributetype String Value of the X3DMetadataObject type attributereference String Value of the X3DMetadataObject reference

attribute

valueInteger/ Float/ String/ Double/

nested Metadata3D element

Value of the X3DMetadataObject attributes’ value. Depends on the X3DMetadataObject type (Float, Integer, String). In the case of X3D MetadataSet, the value attribute holds the X3D MetadataSet children

Ref XPath XPath route to the specific X3DMetadataObject.


MetadataSet Mapping


MotionTrajectory ElementsName DefinitionMotionTrajectoryType

Specifies the motion trajectory of a moving object, as defined in the MPEG-7 ISO Standard. A MotionTrajectory Descriptor is formed for each X3D Interpolator node.

CoordRef Allows the definition of a CoordRef or CoorfDef element, depending on the coordinate system. The CoordRef element is used to reference a two-dimensional coordinate system. It is therefore not used in the case of X3D descriptions.

CoordDef If the attribute value is set to meter, then the trajectory is considered three-dimensional and the coordinate system is local to the specified trajectory

Params the spatio-temporal key-points and interpolations used to express the trajectory data, as per official MPEG-7 ISO. The X3D Interpolator node key and keyValue attributes are provided in the Params element to describe the trajectory of the objects’ motion. In the case of dynamic components such as the X3DFollowerNode, annotation of the trajectory can only be described by their initial estimation through their initialValue and initialDestination attributes, as a smooth motion trajectory between dynamically produced vertices created within the scene during the rendering process.


MotionTrajectory


MotionTrajectory FieldsMPEG-7 Description Fields Value Type Input from X3D Fields

MotionTrajectory Descriptor - Created if when an X3D Interpolator exists.motionType String The X3D Interpolator motion activity type.motionComponent String The X3D Interpolator node name.

CoordDef - Child element of MPEG-7 MotionTrajectory Descriptor

Units String Value of “meter”, to specify 3D content held in Params element

Params - Child element of MPEG-7 MotionTrajectory Descriptor

KeyTimePoint - Child element of the Params element in the MPEG-7 MotionTrajectory Descriptor

MediaRelIncrTimePoint IntegerValue of key attribute of the X3D Interpolator node. One element for each X3D Intepolator node key attribute.

InterpolationFunctions - Child element of the Params element in the MPEG-7 MotionTrajectory Descriptor

KeyValue FloatValue of the keyValue attribute of the X3D Interpolator node. One element for each X3D Intepolator keyValue attribute.


The MotionType attributeX3D Interpolator Node MPEG-7 Motion Type

ColorInterpolator LinearCoordinateInterpolator LinearCoordinateInterpolator2D LinearNormalInterpolator LinearOrientationInterpolator LinearPositionInterpolator LinearPositionInterpolator2D LinearScalarInterpolator LinearSplinePositionInterpolator Non-Linear SplineSplinePositionInterpolator2D Non-Linear SplineSplineScalarInterpolator Non-Linear SplineSquadOrientationInterpolator Non-Linear OrientationEase-In-Ease-Out Ease-In-Ease-OutX3DFollowerNode Smooth


MotionTrajectory Mapping


Interactivity3D Elements

Name DefinitionInteractivity3DType

Describes the interactions that occur within an X3D file, specifically from the X3D ROUTE elements.

TriggerSourceThe source that triggers a specific animation. If the node is triggered internally depending on other nodes or script functions, the TriggerSource is defined as an InternalTrigger.

RouteThe corresponding X3D ROUTE node. Specifies the nodes it connects together and their equivalent node types.


Interactivity3D


Interactivity3D FieldsMPEG-7 Description Fields Value Type Input from X3D Nodes

Interactivity3DType -Specifies interactions contained within an X3D file, as defined in the X3D ROUTE node.

TriggerSource String“UserTrigger” or “InternalTrigger”, depending on the X3D fromNode attribute

Route XPath XPath route to the specific X3D ROUTE node.

fromNode String Value of the X3D fromNode attributefromNodeType String The X3D Node Type of the X3D

fromNode attributetoNode String Value of the X3D toNode attributetoNodeType String The X3D Node Type of the X3D toNode

attribute


Interactivity3D


MetadataHeader Fields


MultimediaContent Descriptor -

Created when the MPEG-7 DescriptionMetadata Descriptor is formed.

id String String value of the X3D document file name.

Multimedia-

Created when the MPEG-7 MultimediaContent element is formed.MediaLocator

MediaUri String String value of the X3D document file path.


MetadataHeader MultimediaContent Descriptor Mapping


Viewpoint3D Elements

Name DefinitionViewpoint3DType Position, orientation, animation properties and

coordinate system of each X3D Viewpoint node.Viewpoint3D Contains the description, fieldOfView, isAnimated and

animatedBy attributes.Viewpoint3DPosition Consists of three sub-elements, Orientation, Position

and CenterOfRotation, providing coordinates of the X3D Viewpoint.

Orientation Orientation coordinates of the Viewpoint node.

Position Positional coordinates of the Viewpoint node.

CenterOfRotation If the X3D Viewpoint node contains a centerOfRotation attribute, the value is passed into the CenterOfRotation.


Viewpoint3D


Viewpoint3D fieldsMPEG-7 Description

Nodes Value Type Input from X3D NodesViewpoint3DType -Viewpoint3D -description String String value of the description provided in the

corresponding X3D Viewpoint nodefieldOfView Float Float value of the X3D Viewpoint fieldOfView

attributeisAnimated Boolean Determines whether the X3D Viewpoint is

bound to an X3D animation nodeanimatedBy String The animation node type bound to the

Viewpoint nodeViewpoint3DPosition - The specific positional characteristics of the

X3D Viewpoint node within the X3D file.Orientation Float Vector The X3D Viewpoint orientation attributePosition Float Vector The X3D Viewpoint position attributeCenterOfRotation Float Vector The X3D Viewpoint CenterOfRotation attribute


Viewpoint3D Mapping


Lighting3D Elements

Name DefinitionLighting3DType Describes each X3D Lighting node contained in a scene.Lighting3D One Lighting node from the corresponding X3D file. Lighting3DCharacteristics Defines specific characteristics on the X3D Lighting node.Lighting3DSpecs The exact specifications on the X3D Lighting node

definition within the scene. It consists of four sub-elements, namely Attenuation, Colour, Direction, and Location.

Attenuation The attenuation properties of the X3D Lighting node. DominantColor Tcolour properties of the X3D Lighting node. Direction The directional lighting properties of the lighting node. Location The location properties of the illuminated geometry.


Lighting3D


Lighting3D FieldsMPEG-7 Description Nodes Value Type Input from X3D NodesLighting3DType -Lighting3D - One Light node from the X3D documentname String The DEF name provided in a Light nodelightingType String The Light type: Directional, Point or SpotLight.isOn Boolean The X3D Light node “on” attributeusedGlobally Boolean The X3D Light node usedGlobally attributeLighting3DCharacteristics - Specific characteristics of the Light nodeambientIntensity Float The X3D Light node ambientIntensity attributeintensity Float The X3D Light node intensity attributebeamWidth Float The X3D Light node beamWidth attributecutOffAngle Float The X3D Light node cutOffAngle attributeradius Float The X3D Light node radius attributeLighting3DSpecs - Specifications of the X3D Light nodeAttenuation Float Vector The X3D Light node attenuation attributeDirection Integer Vector The X3D Light node direction attributeLocation Integer Vector The X3D Light node location attribute

DominantColorMPEG-7:

DominantColorType

MPEG-7 DominantColor descriptor formed by the X3D Light colour attribute


Lighting3D Mapping


Extensions in MPEG-7 Part 5

Multimedia Description Schemes


Profile3D & Script3D


Profile3D Elements

Name DefinitionProfile3DType Indicates the profile type as defined in the

corresponding X3D file. (Core, Interactive, MPEG_interactive, Immersive)


Profile3D Fields

MPEG-7 Description

FieldsValue Type Input from X3D Fields

Profile3D Descriptor String The X3D root node profile attributeVersion Descriptor Float The X3D root node version attribute


Profile3D Version Mapping


Script3D Elements

Name DefinitionScript3DType The scripting type of the X3D Script node. externalScript The external scripting class that performs the

corresponding script functionalities on the X3D file described.

internalScript Specifies that the Script node contains internal script functions and does not contain an outer scripting class reference. The internal script type is Javascript by default, as specified by the X3D standard.


Script3D Fields


Script3D Descriptor - One for each Script nodeinternalScript String “JavaScript”, if no url attribute is

present in the X3D Script node

externalScript String“Java” or “Jscript”, if a url attribute is present and has a url with a .class or .js extension respectively


Script3D Mapping


Use of the ContentCollection Element

• An MPEG-7 DescriptionCollection element is created for each top-level X3D/Scene/X3D Grouping Node element that contains at least one Shape element.

• Subsequently, an MPEG-7 Content element is formed as a child element of the ContentCollection element previously defined, for each internal (child) X3D Grouping or Transform Node containing a Shape Node.

• The process for generating the Content elements is repeated recursively for all internal Grouping nodes at any level and a Content element is also created once for the top-level ContentCollection element.


Use of the ContentCollection Element

• The XSLT texture retrieval template utilizes the same algorithm as the Grouping retrieval.

• The main difference is that each MPEG-7 ContentCollection Descriptor is created for every X3D Grouping node that contains an X3D Shape node with a child X3D Appearance object.

• If the Appearance node contains any type of X3D Texture Objects (e.g. ImageTexture or MovieTexture), then an MPEG-7 Content element is created.


ContentCollection FieldsMPEG-7 Description Fields Value Type Input from X3D FieldsCollection Descriptor - Created if a top-level X3D Grouping

node existsContentCollection Descriptor -

name StringThe X3D DEF attribute in the X3D Grouping node. If no DEF attribute exists, a unique name is generated by the XSLT process.

id StringA unique id, generated by the XSLT process depending on the position of the X3D Grouping node

Content -One for each X3D Grouping Node at any sub-level. A content element is also created for the top-level Grouping node.

Multimedia - Created when the MPEG-7 Content element is formed.MediaLocator -

MediaUri XPath XPath route to the specific X3D Grouping node.


ContentCollection mapping


Content Fields for X3D Appearance nodes

MPEG-7 Description

FieldsValue Type Input from X3D Fields

Content - Created for each X3D Appearance node that contains an X3D Texture object.

Multimedia - Created when the MPEG-7 Content element is created.MediaLocator -

MediaUri URL String String value from the URL attribute of the X3D Texture object.


Content Mapping forX3D Appearance nodes


http://www.medialab.teicrete.gr:8080/X3DtoMPEG7Tool/mainPage.html

Direct Mapping of X3D Scenes to MPEG-7 Descriptions

Documents

Transcript of Direct Mapping of X3D Scenes to MPEG-7 Descriptions