Web3D 2013 Tutorialmetagrid2.sv.vt.edu/~npolys/Web3D2013/Med_Vol... · Web3D 2013 Tutorial: Medical...
Transcript of Web3D 2013 Tutorialmetagrid2.sv.vt.edu/~npolys/Web3D2013/Med_Vol... · Web3D 2013 Tutorial: Medical...
Web3D 2013 Tutorial:
Medical and Volume Rendering with X3D
Nicholas F. Polys, Ph.D. Andy Wood Virginia Tech
Peter Leskovsky, John Congote,
Esther Novo, Luis Kabongo, Ph.D. VicomTech
Topics
• Scenarios and Motivation
• Image Formats
• Processing Pipelines
• Presenting and Publishing (X3D, X3DOM)
Volume Rendering more than medical imaging
• Geology
• Meteorology
• FlowVisualization
• Microscopy
• Paleontology
• Non-invasive sensing:
– Transportation security
– Manufacturing QA
….
Web3D Medical Working Group
• Reproducible rendering for stakeholders throughout the healthcare enterprise
• An n-D Presentation must include: – Structured and interactive virtual environment
display (2D & 3D objects and time series)
– Platform-independent, royalty-free technology to enable vendor innovation
– Can be rendered with or without stereoscopy
– Openly-published
X3D Volume Rendering
• Necessary and Sufficient node set for industry’s Greatest Common Denominator:
– Volume Component: render styles
– X3D version 3.3
• Two independent implementations:
– www.h3d.org
– www.instantreality.org
Volume Processing and Presentation Tools
• Data – Sample xxxxx.dcm
– X3D Content Examples http://www.web3d.org/x3d/content/examples/Basic/VolumeRendering/index.html
– Volvis.org
– http://www.osirix-viewer.com/datasets/
• Warning: some are compressed w/ jpg2000 !
• Tools – ImageJ : http://rsbweb.nih.gov/ij/
• Plugins: DICOM reader, DICOM header inspector
– Seg3D.org
– Slicer.org; ITK-Snap
– X3D-Edit 3.3
• Viewers – H3D.org
– InstantReality.org
– MedX3DOM
X3D Volume Rendering
• Composable Render Styles covering the state of the art – Formalizes parameters and transfer functions for 3D rendering & blending
• BoundaryEnhancementVolumeStyle • CartoonVolumeStyle • ComposedVolumeStyle • EdgeEnhancementVolumeStyle • OpacityMapVolumeStyle • ProjectionVolumeStyle • ShadedVolumeStyle • SilhouetteEnhancementVolumeStyle • ToneMappedVolumeStyle
– Greatest Common Denominator
• Assign different RenderStyles to different segments, blend two volumes – BlendedVolumeStyle – SegmentedVolumeData – IsoSurfaceVolumeData
• Clipping Planes are already specified in X3D 3.2 Rendering Component
Silhouette Opacity Map Cartoon
Medical Interchange Profile
• X3D 3.3 • Minimal X3D node set (footprint) to meet DICOM requirements:
– Core – Time – Networking – Grouping – Rendering – Shape – Geometry3D – Geometry2D – Text – Lighting – Texturing – Interpolation – Navigation – Environmental effects – Event utilities – Texturing3D – Volume rendering
Includes polygon, line and point rendering; metadata on any node
ISO Process
• [ X3D 3.3 ] ISO/IEC DIS 1 19775-1:2008
–Clause 33 : Texturing3D Component
–Clause 41 : Volume Rendering Component
–Annex L: Medical Interchange Profile
• Final Draft International Standard (FDIS)
• Change document and unified spec drafts available to Web3D & DICOM members (* now published)
Online Videos & Examples
• Web Video summary: – Extensible 3D (X3D) Volume Rendering
• https://snoid.sv.vt.edu/medical/X3DVolumes/videos/VolumeVis-X3D-collected.mp4 (65 mb)
• X3D Examples – http://www.web3d.org/x3d/content/examples/Basic/
VolumeRendering/index.html
• For other other Videos, Images and Scenes using the VolumeData and VolumeRenderStyles of X3D 3.3 Clause 41, please visit: – https://snoid.sv.vt.edu/medical/X3DVolumes/
Volume Presentation
Many techniques:
• Volume rendering
– 3DSplatting, ray tracing, pixelshaders
– Established CPU and GPU algorithms
• Surfaces – actual meshes
• Segments – identifying voxels as groups
• ISOSurfaces – rendered at a threshold
I: Scenarios and Motivation
X3D Presentation Demo
• Nicholas Polys, Andy Wood Virginia Tech
Screenshots
Example Volume Rendering Styles
(Head MRI, XML encoding)
<Transform DEF='backdrop' >
<VolumeData dimensions='.75 1 1' > <ImageTexture3D containerField="voxels" url='"./Segments/masked-vispart.nrrd"'/>
<OpacityMapVolumeStyle />
</VolumeData>
</Transform>
Example Volume Rendering Styles
(Head MRI, optic segment)
<ISOSurfaceVolumeData surfaceValues='.15' dimensions='.75 1 1' >
<ImageTexture3D containerField="voxels" url='"./Segments/masked-optic.nrrd"'/>
<CartoonVolumeStyle />
</ISOSurfaceVolumeData>
(Head MRI, cerebrum segment)
<VolumeData dimensions='.75 1 1' >
<ImageTexture3D containerField="voxels" url='"./Segments/masked-cerebrum.nrrd"'/>
<ComposedVolumeStyle>
<CartoonVolumeStyle />
<EdgeEnhancementVolumeStyle gradientThreshold='.8' edgeColor='0 0 .5' />
</ComposedVolumeStyle>
</VolumeData>
I: Scenarios and Motivation
Image Formats
• 2D image slices with order and spacing define the voxels
• Typical image stacks as:
– Tiff
– Png
– dcm
• Sizing, smoothing, resampling can happen in 2.5D space (image stack)
Image J
• Many useful (and programmable) functions to deal with image stacks:
• Invert, Blur, resample, resize reduce bit depth
• ImageJ : http://rsbweb.nih.gov/ij/ • Lots of Plugins: eg DICOM reader, DICOM header
inspector
III: Processing Pipelines
DICOM.dcm
• Medical.nema.org
• Ongoing work Item for n-Dimensional Presentation State
• Working Group 11 (Display)
• Extensive use cases documented
• MPR and CPR parameters of members surveyed
• Supplement draft in preparation
NRRD
• Nearly Raw Raster Data
• http://teem.sourceforge.net/nrrd/
• Nrrd and the teem library facilitates the pre-processing tasks (cropping, quantizing, resampling), as well as the basic analysis tasks which help to understand the structure of a dataset (histograms, maximum intensity projections, connected components)
II: Image Formats
Seg3D
Segment3D – a Gui tool With multiple methods for segmentation : - Neighborhood Growth - Thresholding
Export .nrrd - all-in-one - separate files Seg3d.org
III: Processing Pipelines
Slicer
• Slicer.org
ITKSnap
• http://www.itksnap.org
X3D Edit 3.3
• X3D-Edit is an open-source Extensible 3D (X3D) Graphics authoring tool for simple error-free editing, authoring and validation of X3D scenes.
• Lots of translation and QA features
• Stand-alone or Netbeans plug-in
• https://savage.nps.edu/X3D-Edit/
Volume acquisition
DICOM data
Surfacing
Viewer
workstation
Compose WCS Model (Scene) for Rendering;
assign appearances, views, etc.
Segmentation
nDPS
Processing Scheme for DICOM
Use Cases
X3D
Segmentation: Supp 111
Mesh: Supp 132
Other 3D data
(e.g. CAD, X3D)
Volume Processing Pipelines Source Data
X3D
<ImageTexture2D>
<VolumeData>
<ImageTexture3D> -voxels
<IsoSurfaceVolumeData>
<ImageTexture3D> -voxels
<SegmentedVolumeData>
<ImageTexture3D> -voxels
<ImageTexture3D> -segment identifiers
Image Processor (ImageJ)
Volume Data Processing
(TEEM)
Segmentation • Seg3D • Slicer • ITKSnap
DICOM Plugin
Image Stack
DICOM Stack
Raw Volume
NRRD Volume
Segment IDs
Individual Segments
Volume Image Data Source Data
X3D
<ImageTexture2D>
<VolumeData>
<ImageTexture3D> -voxels
<IsoSurfaceVolumeData>
<ImageTexture3D> -voxels
<SegmentedVolumeData>
<ImageTexture3D> -voxels
<ImageTexture3D> -segment identifiers
Image Processor (ImageJ)
Volume Data Processing
(TEEM)
Segmentation • Seg3D • Slicer • ITKSnap
DICOM Plugin
Image Stack
DICOM Stack
Raw Volume
NRRD Volume
Segment IDs
Individual Segments
Source Data
• Stacks of 2D images with order and spacing data generated by imaging machinery
• Some typical file formats for image stacks: – Tiff
– Png
– Dcm
• Image processing techniques can accomplish much of the work for rendering in 2.5D – Smoothing, (simple) segmentation, resampling,
contrast, etc.
Working with Images
• Resolution – number of pixels in x and y dimensions • Channels – color images are 3 or 4 channel (RGB or RGBA), typical
volume stack is single channel (greyscale) • Bit Depth – number of bits to encode value of a pixel, per channel • Volume data becomes large quickly:
– 512 512x512 8-bit greyscale images: 134 million voxels and 128MB (raw)
– 1024^3 volume, 16-bit RGB: 1.07 billion voxels and 48GB (raw)
• ImageJ – Image processor designed to work with stacks – Many useful (and scriptable) functions: invert, blur, resample, resize,
mask, etc. – Website: http://rsbweb.nih.gov/ij/
Volume Data Source Data
X3D
<ImageTexture2D>
<VolumeData>
<ImageTexture3D> -voxels
<IsoSurfaceVolumeData>
<ImageTexture3D> -voxels
<SegmentedVolumeData>
<ImageTexture3D> -voxels
<ImageTexture3D> -segment identifiers
Image Processor (ImageJ)
Volume Data Processing
(TEEM)
Segmentation • Seg3D • Slicer • ITKSnap
DICOM Plugin
Image Stack
DICOM Stack
Raw Volume
NRRD Volume
Segment IDs
Individual Segments
Volume Data Formats
• Raw voxel data (image stack) – Requires external metadata
• DICOM (.dcm) – Digital Imaging and Communication in Medicine:
http://medical.nema.org
– [MORE POINTS HERE]
• NRRD (.nrrd) – Nearly Raw Raster Data: http://teem.sourceforge.net/nrrd
– Wrapper for raw or compressed image stacks with plain text metadata
– TEEM library for pre-processing and analysis
Segmentation Source Data
X3D
<ImageTexture2D>
<VolumeData>
<ImageTexture3D> -voxels
<IsoSurfaceVolumeData>
<ImageTexture3D> -voxels
<SegmentedVolumeData>
<ImageTexture3D> -voxels
<ImageTexture3D> -segment identifiers
Image Processor (ImageJ)
Volume Data Processing
(TEEM)
Segmentation • Seg3D • Slicer • ITKSnap
DICOM Plugin
Image Stack
DICOM Stack
Raw Volume
NRRD Volume
Segment IDs
Individual Segments
Segmentation
• Transfer functions and rendering techniques are powerful, but not always sufficient
• Segmentation - mark features of interest for special rendering treatment
• Some techniques: – Hand segmentation (painting)
– Threshold (used in many approaches)
– Volume growing
– Connected components
Segmentation Tools
• Seg3D – Large selection of segmentation algorithms and tools – Layered segment masks with multi-layer operations and export
flexibility – http://www.sci.utah.edu/cibc-software/seg3d.html
• ITKSnap – Active contour segmentation (volume growing) and manual
tools – http://www.itksnap.org/pmwiki/pmwiki.php
• 3D Slicer – Volume manipulation and segmentation, with a focus on
registration (multiple volumes) and rendering – http://slicer.org/
Volume Rendering With X3D Source Data
X3D
<ImageTexture2D>
<VolumeData>
<ImageTexture3D> -voxels
<IsoSurfaceVolumeData>
<ImageTexture3D> -voxels
<SegmentedVolumeData>
<ImageTexture3D> -voxels
<ImageTexture3D> -segment identifiers
Image Processor (ImageJ)
Volume Data Processing
(TEEM)
Segmentation • Seg3D • Slicer • ITKSnap
DICOM Plugin
Image Stack
DICOM Stack
Raw Volume
NRRD Volume
Segment IDs
Individual Segments
Volume Rendering With X3D
• Volume rendering component added in X3D 3.3 – [ISO document info here]
• Examples found at: http://www.web3d.org/x3d/content/examples/Basic/VolumeRendering/
• Browsers: – H3D (full support): http://www.h3dapi.org/ – InstantReality (partial support):
http://instantreality.org/
• [not sure about officially supported file types, per browser]
Rendering a Volume: VolumeData Node
• Most basic volume rendering node in X3D
• Contains the voxels to be rendered and render styles to do so
<VolumeData dimensions='1.28 1.28 1.0' > <!-- VolumeRenderStyle node here (optional) --> <ImageTexture3D containerField='voxels' url=' "path_to_dataset" '/> </VolumeData>
Rendering a Volume: IsoSurfaceVolumeData
• Similar to the basic VolumeData node, but renders a surface across voxel gradients
<IsoSurfaceVolumeData dimensions='1.28 1.28 1.28‘ surfaceValues='.15' contourStepSize='0' surfaceTolerance='0' containerField='children'> <CartoonVolumeStyle colorSteps='32' /> <ImageTexture3D containerField='voxels' url=' "skull.nrrd" '/> </IsoSurfaceVolumeData>
Rendering a Volume: SegmentedVolumeData
• Allows different render styles for segments of interest
Applying Distinctive Rendering Techniques
• VolumeStyle nodes allow different rendering techniques/filters – Of the form *VolumeStyle
– Example: <ToneMapVolumeStyle />
• Child to a volume data node – Exception: composing and blending
• Default style – OpacityMapVolumeStyle with a linear, alpha-only transfer function – Interpolates voxel opacity linearly by intensity
Opacity Map VolumeStyle
• The default style, the basis for all enhancement styles • Has only one field, “transferFunction”
– Two or three dimensional texture – One to four components
<OpacityMapVolumeStyle> <ImageTexture3D containerField='transferFunction' url='"engineTransferSchnitt.png"' /> </OpacityMapVolumeStyle>
Default With Transfer Function
Projection VolumeStyle
• A raycasting technique • Depending on the value of the “type” field, returns either the MAX,
MIN, or AVERAGE of the voxel values along the ray • If “intensityThreshold” is specified, returns the first local min/max
above/below the threshol
<ProjectionVolumeStyle type='MAX' enabled='true' intensityThreshold='0' containerField='renderStyle'/>
Max Projection Threshold Max Default Style
Per-Voxel VolumeStyles
• View-Dependent
– Opacity Map (default)
– Enhancement Styles
• Boundary, Edge, Silhouette
– Cartoon
• Lighting-Dependent
– Tone Map
– Shaded
Opacity Map
Edge Enhanced Cartoon
Shaded Tone Map
Boundary Enhancement Style
• Modifies voxels based on how quickly their surface normals are changing:
<BoundaryEnhancementVolumeStyle boundaryOpacity='0.9' opacityFactor='0.9' retainedOpacity='0.2'>
Default Boundary Enhanced
Edge Enhancement
• Voxels are colored based on how close to perpendicular their normal are to the view, outside of a threshold.
• Useful for surface features, not internal features.
<EdgeEnhancementVolumeStyle enabled='true' edgeColor=‘1 0 0 1‘ gradientThreshold='0.4' containerField='renderStyle'/>
Default Edge Enhanced
Silhouette Enhancement
• Modifies the color and opacity of voxels based on their normal values • Unlike edge enhancement, it can be used to reveal internal features
<SilhouetteEnhancementVolumeStyle silhouetteBoundaryOpacity='1' silhouetteRetainedOpacity='.5' silhouetteSharpness='10' enabled='true' containerField='renderStyle'/>
Default Silhouette Enhanced
Cartoon VolumeStyle
• Renders voxels based on the normal value as one of a specified number of color steps between an orthogonal (plane surface) color and parallel color:
<CartoonVolumeStyle enabled='true' colorSteps='4' orthogonalColor='1 1 1 1' parallelColor='0 0 0 1' containerField='renderStyle'/>
2 color steps 4 color steps 8 color steps
Shaded VolumeStyle
• Voxel appearance is controlled by a material node, similar to normal geometry (relative to light source)
• Can be computationally expensive
<ShadedVolumeStyle lighting='true' shadows='true' enabled='true' phaseFunction='Henyey-Greenstein' containerField='renderStyle'> <Material ambientIntensity='0.8' diffuseColor='0 .5 1‘ shininess='0.08' specularColor='1 1 1'/> </ShadedVolumeStyle>
Default Shaded
Tone Mapped VolumeStyle
• Uses the Gooch shading model to color voxels based on their orientations relative to a light source, between a warm (facing light) and cool (facing away) color
<ToneMappedVolumeStyle warmColor='0 0 1 0' coolColor='1 1 0 0' />
Default Tone Mapped
Combining Styles: ComposedVolumeStyle
• Apply multiple rendering styles to a volume
• Styles are applied strictly in the order they appear
• All per-voxel style nodes are composable
– ProjectionVolumeStyle is not composable
Combining Styles: ComposedVolumeStyle
<ComposedVolumeStyle enabled='true' ordered='false' containerField='renderStyle'> <SilhouetteEnhancementVolumeStyle silhouetteBoundaryOpacity='1' silhouetteRetainedOpacity='.1' silhouetteSharpness='10' enabled='true' containerField='renderStyle'/> <EdgeEnhancementVolumeStyle edgeColor='.5 0 0' gradientThreshold='.8' enabled='true' containerField='renderStyle'/> </ComposedVolumeStyle>
+ Style1 (Edge Enhance) Style2 (Silhouette) Composed Styles
Multipart Volumes: SegmentedVolumeData
• Requires two textures: voxels (as normal) and segment identifiers for each voxel
• Specify multiple render styles, in identifier/segment order (default is opacity map) – If more segments than styles, last style applies to
remaining segments
• Individual segments may be turned on or off using “segmentEnabled” field – Styles still required to maintain indexing
Multipart Volumes: SegmentedVolumeData
<SegmentedVolumeData dimensions='2.304 2.304 1.116' containerField='children'> <ImageTexture3D containerField='voxels' repeatS='false' repeatT='false' repeatR='false' url=' "mri_ventricles.nrrd" '/> <ImageTexture3D containerField='segmentIdentifiers' repeatS='false' repeatT='false' repeatR='false' url=' "mri_ventricles_segment.nrrd" '/> <OpacityMapVolumeStyle enabled='true' containerField='renderStyle'/> <ToneMappedVolumeStyle enabled='true' coolColor='0 0 1 0' warmColor='1 1 0 0' containerField='renderStyle'/> </SegmentedVolumeData>
Voxels
Segments
Styles
Default Segmented Volume
Multipart Volumes: Multiple Volumes
• Use different VolumeData nodes to represent each segment
• More computationally expensive than SegmentedVolumeData, but more flexible
– Can manipulate individual segments
– More fine-grained rendering control
• Requires masked volumes (more pre-processing)
Multipart Volumes: Multiple Volumes
Segment1
Segment2
Style 1
Style 2
<VolumeData dimensions='.75 1 1' > <ImageTexture3D containerField="voxels" url='"./Segments/masked-halfhead.nrrd"'/> <OpacityMapVolumeStyle /> </VolumeData> <VolumeData dimensions='.75 1 1' > <ImageTexture3D containerField="voxels" url='"./Segments/masked-cerebrum.nrrd"'/> <ShadedVolumeStyle lighting="TRUE" shadows="TRUE" > <Material diffuseColor='0 .5 1' specularColor='1 1 1' ambientIntensity='0.8' shininess='0.08' /> </ShadedVolumeStyle> </VolumeData>
1
2 3
4 5 Composited Volume
Multipart Volumes: Blended Render Style
• Blends multiple volumes/segments in one VolumeData node
• BlendedVolumeStyle node contains its own voxels and render style nodes – Contained voxels rendered according to styles,
then blended with parent
– Weight of blending is configurable
• BlendedVolumeStyle is composable and nestable!
Multipart Volumes: BlendedVolumeStyle
+
Volume1
Volume2
Style2
Default Style1
Volume1 (Default) Volume2 (Tone Map) Blended Volume
<VolumeData dimensions='512 512 452' containerField='children'> <BlendedVolumeStyle weightConstant1='0.51' enabled='true' weightConstant2='0.5' weightFunction1='CONSTANT' weightFunction2='CONSTANT' containerField='renderStyle'> <ToneMappedVolumeStyle enabled='true' coolColor='0 0 1 0' warmColor='1 1 0 0' containerField='renderStyle'/> <ImageTexture3D containerField='voxels' repeatS='false' repeatT='false' repeatR='false' url=' "internals.nrrd" '/> </BlendedVolumeStyle> <ImageTexture3D containerField='voxels' repeatS='false' repeatT='false' repeatR='false' url=' "body.nrrd" '/> </VolumeData>
Multichannel Microscopy
Using Channels as segments, each with its own style
• https://www.cellimagelibrary.org/
• Online Survey:
– Which of the renderings is most helpful to understand the…?
Preliminary Results Proximity Connectivity
Containment N= 50
Presenting & Publishing
X3D WebGL based Volume Rendering
John Congote, Luis Kabongo, Esther Novo,
Peter Leskovsky ([email protected])
Welcome to San Sebastian
WHERE WE CAME FROM
Volume Visualization (back in 2011)
• Pure WebGL implementation
<HTML><HEAD><META http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<TITLE>Real-Time Interactive Visualization of Volumetric Data with
WebGL</TITLE>
<SCRIPT type="text/javascript" src="sylvester.js"></SCRIPT>
<SCRIPT type="text/javascript" src="glUtils.js"></SCRIPT>
<SCRIPT type="text/javascript" src="glAux.js"></SCRIPT>
<SCRIPT type="text/javascript" src="volumerc.js"></SCRIPT>
</HEAD><BODY onload="volumerc_main('./raycast-color.frag','./aorta-
low.jpg','./tf.png');" bgcolor="white">
<CENTER>
<H1>Real-Time Interactive Visualization of Volumetric Data with
WebGL</H1>
<canvas id="canvas_win" width="512" height="512" style="border:1px solid
black"></canvas>
</br>
<button type="button" onclick="volumerc_main('./raycast-
bw.frag','./aorta-low.jpg','./tf.png');">Aorta LowRes B/W</button>
<button type="button" onclick="volumerc_main('./raycast-
color.frag','./aorta-low.jpg','./tf.png');">Aorta LowRes Color</button>
<button type="button" onclick="volumerc_main('./raycast-
bw.frag','./aorta-high.jpg','./tf.png');">Aorta HighRes B/W</button>
<button type="button" onclick="volumerc_main('./raycast-
color.frag','./aorta-high.jpg','./tf.png');">Aorta HighRes
Color</button>
<P>Rotation with the mouse, and Alt+Mouse to Zoom</P>
</BR>
…
WebGL Solution
• Achievements:
– volume rendering of 3D texture adapted into a 2D atlas
– Use of color/opacity transfer function
– Existing VR shaders adapted & simplified for performance and compatibility issues
From WebGL to …
• Drawbacks: – Hard to maintain as the code was specific to each site-
page-solution-dataset …
– Existing shaders were not necessarily and easily portable
– Hard to integrate in applications, limited interaction, …
• We needed something to: – Facilitate loading volume information
– Integrate existing basic interaction solutions
– Let us focus on rendering styles instead of rendering tricks
– Use a standardised structure to allow portability
MedX3DOM (2012)
• Implementation to support advanced medical visualization on the Web without plugins
• MEDX3D standard implemented into the X3DOM framework
– MEDX3D: extension of the X3D ISO standard to
support advanced medical visualization functionality
– X3DOM: framework for integrating and manipulating X3D scenes as HTML5/DOM elements
Methodology for MedX3DOM Node generation for two components
• Texturing3D • X3DTexture3DNode
• ComposedTexture3D
• ImageTexture3D
• PixelTexture3D
• TextureCoordinate3D
• TextureCoordinate4D
• TextureTransformMatrix3D
• TextureTransform3D
• ImageTextureAtlas
• VolumeRendering • X3DComposableVolumeRenderStyleNode
• X3DVolumeDataNode
• X3DVolumeRenderStyleNode
• BlendedVolumeStyle
• BoundaryEnhancementVolumeStyle
• CartoonVolumeStyle
• ComposedVolumeStyle
• EdgeEnhancementVolumeStyle
• IsoSurfaceVolumeData
• OpacityMapVolumeStyle
• MPRVolumeStyle
• ProjectionVolumeStyle
• SegmentedVolumeData
• ShadedVolumeStyle
• SilhouetteEnhancementVolumeStyle
• ToneMappedVolumeStyle
• VolumeData
Implemented/In progress Defined
Not implemented Abstract
Atlas Image Type
• Atlas
– Composed image
– Parameters • Number of slices
• Number of rows
• Number of columns
Ex: Brain Atlas Image
The texture atlas is the mosaic of all the slices of a volume in one image, the order of the images is given by the rows and columns.
New Node ImageTextureAtlas
ImageTextureAtlas node defines a texture map by specifying a single image file that contains slides of complete 3D data. Parameters indicate the information necessary for mapping voxels to the geometry.
ImageTextureAtlas: ImageTexture { SFInt32 [in] numberOfSlices 0
SFInt32 [in] slicesOverX 0
SFInt32 [in] slicesOverY 0
}
Modified OpacityMapVolumeStyle
OpacityFactor: Global contrast value of the volume data LightFactor: Global radiance of light value of the volume data
OpacityMapVolumeStyle : X3DComposableVolumeRenderStyleNode { SFBool [in,out] enabled TRUE SFNode [in,out] metadata NULL [X3DMetadataObject] SFNode [in,out] transferFunction NULL [X3DTexture2DNode,X3DTexture3DNode] SFFloat [in,out] opacityFactor 0.01 [0,1] SFFloat [in,out] lightFactor 0.3 [0,1]
}
X3DOM (Simple Example) <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-
strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="X-UA-Compatible" content="chrome=1" />
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<title>Hello World</title>
<link rel="stylesheet" type="text/css" href="x3dom.css" />
<script type="text/javascript" src="x3dom.js"></script>
</head>
<body>
<h1>XHTML Hello World</h1>
<p>With X3D-namespace and case senstive element/node names. Works with self-closing tags (e.g.
Viewpoint and Material)</p>
<X3D xmlns="http://www.web3d.org/specifications/x3d-namespace" showStat="false" showLog="false"
x="0px" y="0px" width="400px" height="400px“ altImg="helloX3D-alt.png">
<Scene>
<Viewpoint position='0 0 10' />
<Shape>
<Appearance>
<Material diffuseColor='0.603 0.894 0.909' />
</Appearance>
<Box DEF='box'/>
</Shape>
</Scene>
</X3D>
</body>
</html>
X3DOM (Simple Example)
MEDX3D <?xml version="1.0" encoding="utf-8"?>
<X3D version="3.2" profile="MedicalInterchange">
<Scene>
<Group>
<NavigationInfo type='"EXAMINE" "WALK"'/>
<Viewpoint description="first" position="0 0 0.6"/>
<Background skyColor="1 1 1"/>
<Transform scale=".001 .001 .001">
<!-- The volume -->
<VolumeData DEF="volume” rayStep="0.01“ useSlicing="false“
dimensions="250 250 156">
<Image3DTexture containerField="voxels“ url='"data/S76280/IM00010“>
</VolumeData>
</Transform>
</Group>
</Scene>
</X3D>
MedX3DOM <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN“ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-
strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="X-UA-Compatible" content="chrome=1" />
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<title>Hello World</title>
<link rel="stylesheet" type="text/css" href="x3dom.css" />
<script type="text/javascript" src="x3dom.js"></script>
</head>
<body>
<X3D xmlns='http://www.web3d.org/specifications/x3d-namespace'
showStat='true' showLog='true' width='500px' height='500px'>
<Scene>
<Background skyColor='0.0 0.3 0.65'/>
<Viewpoint description='Default' zNear='0.0001' zFar='100'/>
<Transform>
<VolumeData id='volume' dimensions='4.0 4.0 4.0'>
<ImageTextureAtlas containerField='voxels' url='media/volume/aorta4096.png'
numberOfSlices='96' slicesOverX='10' slicesOverY='10'/>
<OpacityMapVolumeStyle lightFactor='0.01'>
<ImageTexture containerField='transferFunction' url='transfer.png'/>
</OpacityMapVolumeStyle>
</VolumeData>
</Transform>
</Scene>
</X3D>
</body>
</html>
Volume Visualization 2012
Aorta Example
<X3D xmlns='http://www.web3d.org/specifications/x3d-namespace’ showStat='true' showLog='true' width='500px' height='500px'>
<Scene>
<Background skyColor='0.0 0.0 0.0'/>
<Viewpoint description='Default' zNear='0.0001' zFar='100'/>
<Transform>
<VolumeData id='volume' dimensions='4.0 4.0 4.0'>
<ImageTextureAtlas containerField='voxels' url='media/volume/aorta4096.png' numberOfSlices='96' slicesOverX='10' slicesOverY='10'/>
<OpacityMapVolumeStyle lightFactor='0.02' opacityFactor='0.4'>
<ImageTexture containerField='transferFunction' url='media/volume/transfer/transfer.png'/>
</OpacityMapVolumeStyle>
</VolumeData>
</Transform>
</Scene>
</X3D>
http://x3dom.org/x3dom/test/functional/volrenOpacityTestTF_aorta.xhtml
Skull Example
<X3D xmlns='http://www.web3d.org/specifications/x3d-namespace'
showStat='true' showLog='true' width='500px' height='500px'>
<Scene> <Background skyColor='0.0 0.0 0.0'/> <Viewpoint description='Default' zNear='0.0001'
zFar='100'/> <Transform> <VolumeData id='volume' dimensions='4.0 4.0
4.0'> <ImageTextureAtlas containerField='voxels'
url='media/volume/skull-at.png' numberOfSlices='256' slicesOverX='16' slicesOverY='16'/>
<OpacityMapVolumeStyle> </OpacityMapVolumeStyle> </VolumeData> </Transform> </Scene> </X3D>
Brain Example
<X3D xmlns='http://www.web3d.org/specifications/x3d-namespace'
showStat='true' showLog='true' width='500px' height='500px'>
<Scene>
<Background skyColor='0.0 0.0 0.0'/>
<Viewpoint description='Default' zNear='0.0001' zFar='100'/>
<Transform>
<VolumeData id='volume' dimensions='4.0 4.0 4.0'>
<ImageTextureAtlas containerField='voxels' url='media/volume/brain-at_4096.jpg' numberOfSlices='226' slicesOverX='16' slicesOverY='16'/>
<OpacityMapVolumeStyle>
</OpacityMapVolumeStyle>
</VolumeData>
</Transform>
</Scene>
</X3D>
Aorta Example
<X3D xmlns='http://www.web3d.org/specifications/x3d-namespace'
showStat='true' showLog='true' width='500px' height='500px'>
<Scene> <Background skyColor='0.0 0.0 0.0'/> <Viewpoint description='Default' zNear='0.0001'
zFar='100'/> <Transform> <VolumeData id='volume' dimensions='4.0 4.0
4.0'> <ImageTextureAtlas containerField='voxels'
url='media/volume/aorta4096.png' numberOfSlices=‘96' slicesOverX='10' slicesOverY=‘10'/>
<OpacityMapVolumeStyle> </OpacityMapVolumeStyle> </VolumeData> </Transform> </Scene> </X3D>
MPR Style (2012)
<X3D xmlns='http://www.web3d.org/specifications/x3d-namespace’ showStat='true' showLog='true' width='500px' height='500px'>
<Scene>
<Background skyColor='0.0 0.0 0.0'/>
<Viewpoint description='Default' zNear='0.0001' zFar='100’ position="0.85713 1.49578 9.85028" orientation="-0.85521 0.50269 0.12614 0.17466" />
<Transform>
<VolumeData id='volume' dimensions='4.0 4.0 4.0'>
<ImageTextureAtlas containerField='voxels' url='media/volume/aorta4096.png’ numberOfSlices='96' slicesOverX='10' slicesOverY='10'/>
<MPRVolumeStyle id='style' positionLine='0.5'>
</MPRVolumeStyle>
</VolumeData>
</Transform>
</Scene>
</X3D>
http://x3dom.org/x3dom/test/functional/volrenMPR_aorta.xhtml
WORK IN PROGRESS
How to get an Atlas? …
• Do it by hand with an image editor – Time consuming
• Use an image viewer that can create a thumbnail atlas for you – A bit tricky to use, little flexibility
• Write your own C++/… application – A bit long, requires effort and little glory
• Use a python script like the one we made for you
NEW Python Script for Atlas
• Get it here: http://www.volumerc.org/documentation.html
• Requires Python, PIL • Use it as is:
– Parses all slices in a folder and merges then into a PNG atlas at different resolutions (full, 81922, 40962, 20482, 10242)
– Automatic data rescale – DICOM compatibility, uses
Window/Level values (requires pydicom and numpy)
• Personalize it for your own data format: – Implement loadMyData function
Not optimised!! (memory)
Usage: command <InputFolder> <OutputFilename>
- <InputFolder> must contain only one set of
PNG files to be processed
- <OutputFilename> must contain the path and
base name of the desired output, extension
will be added automatically
How to extend it …
• Follow X3DOM extension guide:
– http://x3dom.org/docs/dev/components.html
• Example:
– Implement the iso surface volume raycasting
• Process:
– Check the supporting nodes within X3D specification
• Found! -> ISOSurfaceVolumeData
– Take the X3DOM source code from github
– Extend functionality
VolumeRendering.js BEFORE
/* ### ISOSurfaceVolumeData ### */ x3dom.registerNodeType( "ISOSurfaceVolumeData", "VolumeRendering", defineClass(x3dom.nodeTypes.X3DVolumeDataNode, function (ctx) { x3dom.nodeTypes.ISOSurfaceVolumeData.superClass.call(this, ctx); this.addField_MFNode('renderStyle', x3dom.nodeTypes.X3DVolumeRenderStyleNode); this.addField_SFNode('gradients', x3dom.nodeTypes.X3DTexture3DNode); this.addField_MFFloat(ctx, 'surfaceValues', []); this.addField_SFFloat(ctx, 'contourStepSize', 0); this.addField_SFFloat(ctx, 'surfaceTolerance', 0); }, { nodeChanged: function() {}, fieldChanged: function(fieldName) {} } ) );
VolumeRendering.js AFTER /* ### ISOSurfaceVolumeData ### */
x3dom.registerNodeType(
"ISOSurfaceVolumeData",
"VolumeRendering",
defineClass(x3dom.nodeTypes.X3DComposableVolumeRenderStyleNode,
function (ctx) {
//Some initialization code… },
{
nodeChanged: function() {},
fieldChanged: function(fieldName) {},
uniforms: function() {
//Define the uniforms for the shaders },
textures: function() {
//Prepare the textures },
vertexShaderText : function() {
//Generate vertex shader },
fragmentShaderText : function (numberOfSlices, slicesOverX, slicesOverY) {
//Generate fragment shader }
}
)
);
Usage <X3D xmlns='http://www.web3d.org/specifications/x3d-namespace'
showStat='false' showLog='true' width='500px' height='500px'>
<Scene>
<Background skyColor='0.0 0.0 0.0'/>
<Viewpoint description='Default' zNear='0.0001' zFar='100'/>
<Transform>
<VolumeData id='volume' dimensions='4.0 4.0 4.0'>
<ImageTextureAtlas containerField='voxels'
url='media/volume/aorta4096.png' numberOfSlices='96'
slicesOverX='10' slicesOverY='10'/>
<ISOSurfaceVolumeData surfaceValues='0.2'
contourStepSize='0.2' lightFactor='0.02' opacityFactor='0.4'>
<ImageTexture containerField='transferFunction'
url='media/volume/transfer/transfer.png'/>
</ISOSurfaceVolumeData>
</VolumeData>
</Transform>
</Scene>
</X3D>
IN PROGRESS Result
ISO=20%
ISO=40%
ISO=60%
ISO=80%
ISO=80% Lighting OFF
FUTURE
DICOM, NRRD, RAW Format??
• Javascript is very slow to process binary data
• The loaders of this kind of volume data are based in C and the conversion or reimplementation is difficult
• Other types of volume formats exists, like MHD, NetCDF, …., imposible to implement all.
• MEDX3DOM is for the web, using web formats
MEDICAL IMAGING
• DICOM format is ´´the´´ standard for medical imaging, but internally is a container of several kind of data, sound, 2D images, 3D images, metadata,etc.
• WADO: It is a webservice which provides medical images through the web stored in a PACS server.
• MEDX3DOM: support WADO (Web Access to DICOM Objects)
MEDX3DOM Future …
• Next improvements – Integration Combination with mesh models
– Implementation of different styles (MIP, X-Ray, Composed …)
– Lighting (Phong, Global illumination …)
– Animation (4D timesteps, video, flow animation …)
– Data transfer optimization (streaming, compression,…) … collaborative visualization
Volume Rendering beyond medical imaging
• Geovisualization
• Volume Attention Maps
• Meteorology
• FlowVisualization
• Confocal microscopy
• ….
Keep updated
• Links:
– http://demos.vicomtech.org
– http://www.volumerc.org
– http://www.web3d.org/x3d/specifications/x3d
– http://www.x3dom.org
Thank you!
• Acknowledgements: – Web3D Consortium (X3D / MEDX3D)
– Fraunhofer IGD (X3DOM)
– John Edgar CONGOTE CALLE (main architect and developer of this initiative) ... now Developer Relations Engineer @Unity3D