Predictive 3D streaming

16
Multi-Resolution 3D Human Brain Visualization through Streaming Vani V 1 , Pradeep Kumar R 2 and Mohan S 3 1 Department of Information Technology, Dr.N.G.P.IT, Affiliated to Anna University, Coimbatore, India. 2 Department of Computer Science Engineering, Adithya IT, Affiliated to Anna University, Coimbatore, India. 3 Department of Computer Science Engineering, Dr.N.G.P. IT, Affiliated to Anna University, Coimbatore, India. [email protected],[email protected], [email protected] Abstract. The well established techniques for video and audio streaming have become the primary inspiration to focus our research towards 3D Streaming. In 3D streaming, the model is buffered and rendered instead of complete download. The central idea of this paper is to visualize the typical 3D human brain model at different resolutions based on the client’s requirement at a rapid rate with necessary interactions. This is achieved by initially streaming the simplified version of 3D human brain and then the refinements are streamed based on the client’s requirement. The experiment results and analysis affirms the fact that this type of multi resolution streaming reduces the initial time to present the rendered 3D model to the requested clients. Experiments are further conducted to compare the quality of the multi resolution models that are being rendered with respect to the original model. Keywords: Multi-Resolution model, 3D Streaming, 3D Rendering, 3D Visualization 1 Introductio

Transcript of Predictive 3D streaming

Page 1: Predictive 3D streaming

Multi-Resolution 3D Human Brain Visualization through Streaming

Vani V1, Pradeep Kumar R2 and Mohan S3

1 Department of Information Technology, Dr.N.G.P.IT, Affiliated to Anna University, Coimbatore, India.2 Department of Computer Science Engineering, Adithya IT, Affiliated to Anna University, Coimbatore, India.3 Department of Computer Science Engineering, Dr.N.G.P. IT, Affiliated to Anna University, Coimbatore, India.

[email protected],[email protected],[email protected]

Abstract. The well established techniques for video and audio streaming have become the primary inspiration to focus our research towards 3D Streaming. In 3D streaming, the model is buffered and rendered instead of complete download. The central idea of this paper is to visualize the typical 3D human brain model at different resolutions based on the client’s requirement at a rapid rate with necessary interactions. This is achieved by initially streaming the simplified version of 3D human brain and then the refinements are streamed based on the client’s requirement. The experiment results and analysis affirms the fact that this type of multi resolution streaming reduces the initial time to present the rendered 3D model to the requested clients. Experiments are further conducted to compare the quality of the multi resolution models that are being rendered with respect to the original model.

Keywords: Multi-Resolution model, 3D Streaming, 3D Rendering, 3D Visualization

1 Introduction

In recent years, 3D real time computer graphics [1] over the web has given rise to the development of 3D Virtual Environments (VE) [2]. The 3D virtual environment which comprises of set of 3D scenes and the scene in turn consists of set of 3D complex meshes is available on the web and accessed by multiple users at the same time. Therefore, an efficient mode of transmitting the 3D complex meshes across the web has become the need of the hour. In order to cater to the need, the streaming [3] approach which is promising to deliver the content as per the client request has evolved and few attempts are already made by the researchers. However, in this paper, an attempt is made specifically to stream and deliver the 3D human brain with the required quality requested by the client who could be a medical student/practitioner who wants to visualize the 3D human brain and study in detail about its anatomy by interacting with it. The system he uses to stream and render the 3D human brain may have limited buffering capacity from few KBs to MBs.

Mohan S. et al. (Eds.): Proceedings of the International Conference ICSIP 2012, , pp. springerlink.com © Springer India 2012

Page 2: Predictive 3D streaming

1 Vani V et al.

Therefore, based on the buffering limits of the client, the corresponding LODs can be streamed from server to the client. Also, in order to ensure the quality of the rendered 3D mesh compared with the original mesh a detailed experiment is conducted and results are analyzed. In the process of developing a 3D visualization of human brain with required resolution through steaming firstly, Geometric Level Of Details (GLOD) [8] library which was developed on top of OpenGL is used at the server end. The GLOD, which focuses on the geometry details of the mesh, has included polygon reduction technique [4] to reduce the level of details from the original mesh and gives different levels of meshes by reduction process. This process is carried out and required levels of multi resolution meshes are identified and stored in the server for a specific 3D human brain scene which has got only one complex 3D mesh. Secondly at the server end, the quality of the derived LODs with respect to the original mesh (LOD14) is also estimated. Finally the real time streaming is implemented with serialization and de-serialization technique and results are analyzed. In streaming, if the requested scene is the 3D human brain then the initial LOD (Level 1) with very less geometrical information is streamed by computing the total space required and the time spent for streaming. Then, if the total space estimated is available, the streamed data is buffered and rendered at the client end with the rendering time estimated. On the client end, the client can interact (move, rotate, zoom in/out) with the rendered LOD with the help of keyboard and pointing device. Also, either based on the special key press (+ / -) or based on the Euclidean distance between updated camera position and a random point taken from the screen space next/previous LOD would be streamed and rendered if the space criteria is satisfied. This process is repeated by ensuring that the levels remain between 1 and 14.The rest of the paper is organized as follows. In Section 2, we present the overview of Multi resolution 3D model. Section 3 describes the distance measures used to check the quality of the original and reduced multi-resolution mesh. In Section 4, with the 3D human brain model being considered as a specific case, we discussed the proposed work which includes the streaming and 3D visualization process. In Section 5, we discussed the experiment results and analysis. Section 6 concludes by reflecting on the insights we gained from our model and its implications.

2 Multi-Resolution 3D Model – An Overview

Highly detailed geometric models are necessary to satisfy the demands of user in 3D virtual world for realism in computer graphics [4]. But, due to the increasing complexity of the models, it makes the virtual world developers feel expensive to store, transmit, and render requested 3D data by the client. In order to store, transmit and render the complex models at the rapid rate, the multi-resolution models [15] are developed whenever, a representation at low resolution is adequate to meet the needs of the application. Multi-resolution models offer the possibility to manipulate representations of objects at different levels of detail (LOD) [5, 6, 7] and accuracy, depending on the needs of each specific application. In addition to the benefits of speeding up visualization, multi-resolution techniques also support the interactive modeling of detailed objects.

Page 3: Predictive 3D streaming

Multi-resolution 3D visualization 2

2.1 Types of Hierarchies

The level of detail pipeline consists of three basic stages [8]: geometric simplification, adaptation, and rendering. The simplification process takes in “flat” geometry and produces a multi-resolution hierarchy [5, 7] from it. There are several types of multi-resolution hierarchies, the classification of hierarchies as 1) Discrete 2) Continuous 3) View-dependent or 4) Hierarchical is based on the multi resolution granularity as shown in Fig.1.

1) Discrete hierarchies (Discrete Level of Detail (DLOD)): refers to the creation of several static levels of detail [5, 7] which are swapped out directly for each other and it is analogous to mip-maps for texturing.

2) Discrete hierarchies are extremely computation efficient as far as rendering is concerned. The DLOD is the simplest and most common form of simplification hierarchy.

It encodes multiple LODs at very coarse granularity. In this, each successive LOD would be half the complexity of the predecessors and the choice of granularity doubles the storage requirements with respect to higher resolution model. Each LOD may also contain the error value indicating the fidelity/quality of it. This kind of simplification is unaware of viewing directions since, the LODs are estimated offline during preprocessing. Therefore, it uniformly reduces object detail. Thus, Discrete LODs are also referred to as isotropic or view-independent LODs

The benefits of DLOD are described as below: LODs are easily compiled into an optimized form for efficient rendering

with the help of processes such as o Triangular strip reconstruction o Re-ordering of vertices based on vertex cache sizeo Indexed vertex arrays

The management of one or more discrete hierarchies is not too computationally expensive within the interactive application (i.e.,) with the given circumstances for each hierarchy appropriate DLOD can be chosen

DLODs are most useful when the data comprises of one or more objects that are relatively small in spatial extent. In this case, single choice of resolution is appropriate for the entire model.

DLOD is used by a class of models ranging from 3D scanned objects to that of virtual environments.

3) Continuous hierarchies (Continuous Level of Detail (CLOD)): creates a progressive data structure [5, 7] from which it can extract a continuous spectrum of detail at run time and it is also referred as dynamic levels of detail. Though it is more computationally complex during rendering, it addresses many of the limitations of DLOD.The benefits of CLOD are described as below: It allows more exact choice of number of primitives to use for an object It enables more subtle transition between LOD It provides convenient representation for progressive transmission of

data.

Page 4: Predictive 3D streaming

3 Vani V et al.

Since it allows only linear progression and not selective refinement, it is used for same class models as that of DLODs

4) View dependent LOD extends CLOD by creating a hierarchical data structure [5, 7] from which it extracts a mesh tailored to the given viewpoint. . The benefit of the view-dependent hierarchy is overwhelming though there is a run time overhead in adjusting the detail across the model on the fly.

5) Hierarchical Level of Detail (HLOD) [5, 9 and 10] works at scene level with multiple objects and has combined discrete hierarchy with view dependent hierarchy. This LOD hierarchy is more coarse-grained as compared to a typical view-dependent hierarchy and roughly corresponds to the notion of hierarchical scene structure. This coarse-grained representation is beneficial to visualize complex scenes with large number of objects, reducing per object overhead for given rendering frame rate both in LOD selection process as well as the number of primitives to be rendered.

Fig. 1. Types of hierarchies [5], Continuous LOD is large no. of LODs encoded by representing small changes from one LOD to next. LOD in the view-dependent hierarchy is represented as a cut across the tree. HLOD is similar to view-dependent, but it is much more coarse-grained, allowing faster management and more efficient rendering. A cut across an HLOD hierarchy essentially selects a set of discrete LODs to represent the scene.

3 Distance Measures

One of the efficient geometric error metrics [11, 12, 13, 14] used to calculate the quality of the rendered 3D LOD is the distance. It gives the distance between the simplified vertices and the planes of the original surface. Root Mean Square (RMS) difference is estimated between two LODs and it is treated to be simplest possible estimation. Also, the Hausdorff distance is considered to be the most popular

Discrete Continuous View Dependent Hierarchy

Page 5: Predictive 3D streaming

SendCompute LODs & its Quality

Serialize

Server

De-Serialize

Render

Client

User Input

Receive

distance measure which compares a pair of LODs with different connectivity and calculates the similarity of two point sets by computing one sided

Multi-resolution 3D visualization 4

distanceDist ( LO D1 , LO D2 ), where Dist is the Euclidean distance between two points p(x , y , z)∧p ' (x’,y’,z’).

Dist ( p , LO D2 )= minp'∈ LOD2

(√( x−x' )2+( y− y ')2+( z−z ')2)

Dist ( LO D1 , LO D2 )= maxp∈ LOD1

Dist ( p , LO D2 )

The Dist in general is not symmetrical i.e Dist ( LO D1 , LO D2 )≠ Dist ( LO D2 , LO D1 )

Where Dist ( LO D1 , LO D2 ) is referred as forward distance and Dist ( LO D2 , LO D1 ) is referred as backward distance.

The two sided (symmetric) distance which provides a more accurate measurement can be estimated by taking the

Max(Dist ( LO D1 , LO D2 ) Dist ( LO D2 , LO D1 ))

Therefore, the Hausdorff distance has been used to find the geometric error between a pair of 3-D mesh models which is shortly referred as LODs.

5 Proposed Work

The proposed work attempts to visualize the 3D human brain streamed from the server and rendered at the client. Fig 2 gives the overview of the 3D Streaming & Rendering system which has got two major modules. 1) Server Module2) Client Module

Fig. 2. Overview of 3D Streaming & Rendering System

Page 6: Predictive 3D streaming
Page 7: Predictive 3D streaming

13. Display Appropriate Error & Terminate

6. Start Streaming Timer & Serialize LOD

7. DeSerialize LOD and Stop the Streaming Timer

8. Compute & Display Total Streaming Time and Required Buffer Space

10. Store and Render de-serialized 3D mesh with necessary interactions

4. Check for 3D Scene Requested with LODs Estimated

2. Is connection OK?

3. Set Camera Position & Request for 3D Scene to Visualize

Setup Connection between Clients and Server

No

Yes

5. Is the Scene LOD Available?

9. Is Space Available?

No

Yes

No

Yes

11. Interact with the Rendered Scene

12. Notify the interaction to Server with the updated move to stream next level of refined mesh

A

A

A

5 Vani V et al.5.1 Server Module

In this module, required levels of LODs are estimated and its quality with respect to the original well refined mesh is measured. This measure would give the client an insight on the quality of the rendered mesh based on his request. Based on the client’s request, the initial LOD is streamed by serializing the 3D data. Serializing is the process of packing the 3D data which comprises of set of vertices and faces for the level of LOD that is selected for streaming. Further, the server module streams the LODs based on the interactions made by the client.

Fig. 3. Work Flow of the Streaming System

Page 8: Predictive 3D streaming

Multi-resolution 3D visualization 65.2 Client Module

In this module, client establishes a connection with the server. Once the connection is successfully established, it requests for a scene from the server and if the scene is present then the initial LOD is streamed from the server. In client end, the streamed data is de-serialized in order to get the vertices and faces of the corresponding LOD and then it is rendered. Also the time taken for rendering process is also estimated. Fig. 3 illustrates overall workflow of the system with the client and server interactions.6 Experiment Results and its Implications

Experiments are conducted to achieve 3D visualization of human brain through streaming of multi resolution meshes based on the client request and further interactions. The experimental setup was run on an Intel Core2 Duo CPU P8600 @ 2.4 GHz with 4 GB RAM and ATI Radeon 1GB graphics card system for both server and client. For experimental purpose, we have considered 14 levels of LOD from coarse mesh (LOD1) to refined (LOD14) mesh. Table 1 gives multi-resolution 3D human brain models. Table 2 describes the attributes of the LODs considered which includes number of vertices and faces considered in each LOD with its corresponding % of vertices and % of faces with respect to LOD14 (original mesh). Fig 4 shows the % of vertices and % of faces represents the corresponding LODs. Also, in order to understand the quality of the mesh (LOD) being rendered, we have considered hausdorff distance measure which would estimate the quality of the LOD being rendered with respect to LOD14 (original mesh).Fig. 5, highlights the quality of the LODs streamed and rendered for 3D visualization using one sided and two sided hausdorff distance.

Table 1. Multi-Resolution 3D Human Brain Models (Coarse LOD to Refined LOD)

LOD1 LOD2 LOD3 LOD4

LOD5 LOD6 LOD7 LOD8

LOD9 LOD10 LOD11 LOD12

LOD13 LOD14

Page 9: Predictive 3D streaming

7 Vani V et al.

As specified in section 3, hausdorff distance is a generic distance measure which defines the distance between two empty sets and it is useful in measuring the error between surfaces. We have considered both one sided and two sided hausdorff distance measure as some times one sided distance would end up in less accurate measure than that of the two sided distance as depicted in Fig 6. However, according to the results highlighted for our LODs, it clearly shows that Dist ( LO D2 , LO D1 ) >>

Dist ( LO D1 , LO D2 ) in all comparisons. Where LO D 2 is considered to be the original mesh (LO D14) and LO D1 is substituted with LO D1 toLO D13. Table 3 and Fig. 7 highlight the turnaround time required to stream each LODs. The results shown in Table1 – Table 4 and Fig.4,5& 7 clearly indicates that by exploiting the human perception characteristics, it is enough if we stream and render LO D9 which could present all the parts of the brain with approximately 3% of the vertices and 3% of the faces and at the error rate of .09 distance from LO D 9 and LO D14 and .23 distance from LO D14 and LO D9. Also, the buffering capacity required is only 576 KB which is only 2.68% of the total capacity required to transmitLO D14 . From Table 1 and Fig 7, we could infer that if it is sufficient to stream and render LO D9 then the turnaround time required is only 3.78 secs where we can save 8.5 secs.

Table 2. Attributes of 3D Human Brain LODs

LOD No. of Vertices

No. of Faces

% Vertices

% Faces

LOD1 12 16 0.0041 0.0027LOD2 39 71 0.0133 0.0121LOD3 74 142 0.0252 0.0241LOD4 289 574 0.0983 0.0976LOD5 578 1148 0.1966 0.1952LOD6 1152 2296 0.3918 0.3905LOD7 2299 4594 0.7819 0.7813LOD8 4593 9188 1.5622 1.5625LOD9 9186 18376 3.1244 3.125LOD10 18375 36752 6.2497 6.25LOD11 36748 73504 12.4988 12.5LOD12 73500 147008 24.999 25LOD13 147004 294016 49.9993 50LOD14 294012 588032 100 100

Page 10: Predictive 3D streaming

A

LOD2

LOD1

B

Fig. 4. % of Vertices and Faces Rendered w.r.t. LODsMulti-resolution 3D visualization 8

Fig. 5. Quality of the LOD Estimated using (a) One Sided and (b)Two Sided Hausdorff Distances

Fig. 6. In this case Dist ( LO D1 , LO D2 ) will remain smaller than Dist ( LO D2 , LO D1 ) since

here Dist ( A , LO D2 )≪ Dist ( B , LO D1 )Table 3. Streaming and RenderingTime of Discrete LODs

LODStreamin

g Time(sec)

Rendering

Time(sec)LOD1 0.024 1.51LOD2 0.029 1.526LOD3 0.064 1.458LOD4 0.092 1.554LOD5 0.07 1.742LOD6 0.108 1.8LOD7 0.146 1.96LOD8 0.184 2.39LOD9 0.19 3.593LOD1

0 0.208 4.008

LOD11 0.293 6.196

LOD12 0.333 8.1

LOD13 0.666 9.24

LOD1 1.275 11.034

Fig. 7. Tur

Page 11: Predictive 3D streaming

4 naround Time of Discrete LODs

7 ConclusionIn this paper an attempt has been made to build a 3D human brain visualization system through streaming by exploiting human perception and multi-resolution capabilities. It is demonstrated with experimental results that the rendering latency and streaming

9 Vani V et al.time are reduced to a greater extent by streaming and rendering appropriate multi resolution model instead of transmitting the original mesh itself. 3D visualization is made possible with the predominant data being streamed and rendered with necessary interactions for further refinement if needed. The results achieved is overwhelming and would have certainly a greater impact when we have to stream and render much more complex scenes with time and space constraints enforced on it.

References

[1] Don Brutzman, Mike Macedonia, Theresa-Marie Rhyne and Steve McCanne,“Internetworked 3D Computer Graphics: Beyond the Bottlenecks and Roadblocks”.ACM SIGCOMM '97. (1997).

[2] Blanc, A. L., Bunt, J., Petch, J., and Kwok, Y. “The virtual learning space: an interactive 3D environment”. In Proceedings of the Tenth international Conference on 3D Web Technology (Bangor, United Kingdom). Web3D '05. ACM, New York, NY, 93-102. DOI= http://doi.acm.org/10.1145/1050491.1050505, March 29 - April 01. (2005).

Page 12: Predictive 3D streaming

[3] Soumyajit Deb and P. J. Narayanan. “Design of a geometry streaming system”. In Proc. ICVGIP, pages 296-301. (2004).

[4] Hugues Hoppe.:Progressive meshes. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques (SIGGRAPH '96). ACM, NY, USA, 99- 108. DOI=10.1145/237170.237216 (1996).

[5] Cohen, Jonathan D. and Dinesh Manocha.: Model Simplification for Interactive Visualization. Visualization Handbook. 13 pages. Eds. Chris Johnson and Chuck Hansen. Elsevier Butterworth-Heinemann. Chapter 20, pp. 393-410. (2005).

[6] Huimin Ma, Tiantian Huang, Yanzhi Wang, Multi-resolution recognition of 3D objects based on visual resolution limits, Pattern Recognition Letters, Volume 31, Issue 3, Pages 259-266, ISSN 0167-8655, 10.1016/j.patrec.2009.08.015.(2010).

[7] Li Xin, Research on LOD Technology in Virtual Reality, Energy Procedia, Volume 13, Pages 5144-5149, ISSN 1876-6102, 10.1016/j.egypro.2011.12.142.(2011).

[8] Cohen, Jonathan D., David Luebke, Nathaniel Duca, Brenden Schubert, and Chris Niski.: GLOD: A Minimal Interface for Geometric Level of Detail. 10 pages. Johns Hopkins University Graphics Lab Technical Report. (2004).

[9] Carl Erikson, Dinesh Manocha, and William V. Baxter, III. 2001. HLODs for faster display of large static and dynamic environments. In Proceedings of the symposium on Interactive 3D graphics (I3D '01). ACM, NY, USA, 111-120. DOI=10.1145/364338.364376. (2001).

[10] Jean Pierre Charalambos. Hoq-Driven HLOD Refinement: Using Hardware Occlusion Queries Information to Drive Hierarchical Level-Of-Detail Refinement. VDM Verlag, Saarbrücken, Germany, Germany. (2010).

[11] Nicolas Aspert, Diego Santa-Cruz, Touradj Ebrahimi, MESH : Measuring Errors Between Surfaces using the Hausdorff Distance In Proc. of the IEEE International Conference in Multimedia and Expo (ICME) vol. 1, pp. 705-708, Lausanne, Switzerland, August 26-29. (2002).

[12] Abdullah Bulbul, Tolga Capin, Guillaume Lavoué , Marius Preda, Assessing Visual Quality of 3D Polygonal Models, IEEE Signal Processing Magazine, vol. 28, No. 6, pp.80-90.(2011).

[13] P. Cignoni, C. Montani, and R. Scopigno, “A Comparison of Mesh Simplification Algorithms,”Computers & Graphics, vol. 22, no. 1, pp. 37-54.( 1998).

[14] P. Cignoni, C. Rocchini, R. Scopigno: Metro: Measuring error on simplified surfaces. Computer Graphics. Forum 17, 2, 167–174. (1998).

[15] Eric Shaffer, Michael Garland, A Multi-resolution Representation for Massive Meshes IEEE Transactions on Visualization and computer graphics, VOL. 11, NO. 2.(2005).