Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats

Toby Cornish, M.D., Ph.D.

Department of Pathology

Johns Hopkins Medical Institutions

Baltimore, MD

Overview• JPEG-2000 is a highly-efficient compression

standard well-suited to storage of whole slide images

• JP2 and Aperio SVS are two WSI file formats that employ JPEG2k compression

• Although the file formats are different, the data is essentially stored in the same format

• Few tools exist for manipulating these formats • It should be possible to open, manipulate, and

interconvert these formats using readily available, free and open libraries

JPEG-2000

Compression

Codestreams

File Formats

JP2

Aperio SVS

Design

Access SVS Data

Convert SVS to JP2

Results

File Formats

JP2

Aperio SVS

Design

Access SVS Data

Convert SVS to JP2

Results

JPEG-2000

Compression

Codestreams

JPEG-2000

• A standard for wavelet-based image compression

• Joint Photographic Expert Group as a replacement for JPEG

• ISO/IEC 15444-1 and 15444-2 (parts 1 and 2)

• Part 1 defines a self-contained codestream (JPC) as well as a file format (JP2) that can wrap the codestream

JPEG-2000, cont.

• JPIP (ISO/IEC 15444-9) is a related standard for streaming of large JPEG2k codestreams

• Several encoders/decoders both free and proprietary for JPEG2k– Vary widely in the implementation of the standard, but

most support JPEG2k part 1

• Mostly used for high capacity image storage in videogames, satellite imagery, and, more recently, medical imagery

JPEG-2000, cont.

• Features:– Highly efficient compression

– Native tiling support for large images

– Native multiresolution decompression from a single codestream

– Support for an arbitrary number of image components (i.e. channels)

– JPEG2k codestreams are self-contained and can be embedded in other file formats

File Formats

JP2

Aperio SVS

Design

Access SVS Data

Convert SVS to JP2

Results

JPEG-2000

Compression

Codestreams

JPEG-2k codestreams

• The codestream are capable of indepently being decoded into an image

• All codestreams are tiled, but the simplest and most common form is the single tile codestream

• Single tile images are good for small images

• Multiple tile images better for handling large image data such as WSI

JPEG2k codestream, single tile

1

One codestream – One tile

JPEG2k codestream, single tile

SOC

QCD

COD

SIZ

Main header

Tile header 1

Packet 1

Packet n

SOT

COM

SOD

EOC

start of codestream

image and tile size

coding style defaults

quantization defaults

comments

start of tile

start of data

end of codestream

EOC

Tile data 1

JPEG2kcodestream

JPEG2k codestream, multiple tiles

One codestream – Multiple tiles

11 12

13 14 15 16

1 2 3 4

5 6 7 8

9 10

JPEG2k codestream, multiple tilesSOC

QCD

COD

SIZ

Packet 1

Packet n

SOT

COM

SOD

start of codestream

image and tile sizecoding style defaults

quantization defaultscomments

start of tilestart of data

EOC end of codestream

Packet 1

Packet n

SOTSOD

start of tilestart of data

Main header

Tile header 1

EOC

Tile data 1

JPEG2kcodestream

Tile header n

Tile data n

Design

Access SVS Data

Convert SVS to JP2

Results

JPEG-2000

Compression

Codestreams

File Formats

JP2

Aperio SVS

JP2

• Wrapped JPEG2k codestream

• Composed of mandatory and optional header boxes

• Extensible via UUIDs (Universal Unique Identifiers) and/or XML

JP2 file format, cont.

JPEG2k signature box

File type box

JP2 header box

Contiguous codestream box

JPEG2k codestream

XML boxes

UIUD boxes

Main header

Tile header 1

EOC

Tile data 1

Tile header n

Tile data n

Design

Access SVS Data

Convert SVS to JP2

Results

JPEG-2000

Compression

Codestreams

File Formats

JP2

Aperio SVS

Aperio SVS file format• Developed by Aperio, Inc. (San Diego, CA)

• Essentially, a TIFF file with a tiled layout

• Tile data are JPEG2k codestreams

• Can consist of multiple images (pages) derived from slide scanning with one Image File Directory (IFD) per image

• Image metadata is stored in Tags

• The highest resolution data is always tiled

SVS/TIFF structure

ImagePyramid

Thumbnail

Label

Macro

SVS/TIFF structure, cont.Byte order

Version number

IFD offset

Tag entry count

Tag 1

Tag n

IFD 1

Image data

IFD 2

IFD 3

IFD 4

IFD 5

IFD 6

Tag entry count

Tag 1

Tag n

Image data

SVS/TIFF Tiling

Multiple tiles – one codestream per tile

11 12

13 14 15 16

1 2 3 4

5 6 7 8

9 10

SVS/TIFF tiling, cont.

IFD 1

Tile 1

Tile n

Tile 2

Tile 3

Tiff Tiles

JPEG2k codestream

Tag entry count

Tag 1

Tag n

Image data

Main header

Tile header 1

EOC

Tile data 1

SVS/TIFF v JP2• Differences:

– JP2 contains one JPEG2k codestream, which contains multiple tiles

– SVS/TIFF contains multiple tiles, each of which contains one codestream with exactly one tile

• Similarities:– The tile data is essentially the exact same, but packaged

– Should permit direct copying of tile data between formats

Design

Access SVS Data

Convert SVS to JP2

JPEG-2000

Compression

Codestreams

File Formats

JP2

Aperio SVS

Results

Design

• Open source libraries (LibTiff, ImageMagick/Jasper)

• C programming language for portability

• First two goals:

– Read SVS/TIFF image data using custom software

– Convert SVS/TIFF files to JP2 without decompressing

Read SVS/TIFF

Tile 1

Tile n

Tile 2

Tiff Tiles

Main headerTile header

EOC

Tile data 1

Main headerTile header

EOC

Tile data 2

Main headerTile header

EOC

Tile data n

IFD

SVSImage data

TIFF Header

Decode

Decode

Decode

SVS/TIFF to JP2 conversion

Tile 1

Tile n

Tile 2

Tiff TilesIFD

SVSImage data

TIFF Header

Main headerTile header

EOC

Tile data 1

Main headerTile header

EOC

Tile data 2

Main headerTile header

EOC

Tile data n

Main headerTile header

EOC

Tile data 1

Tile header

Tile data 2

Tile header

Tile data 2

JPEG2kCodestream

JP2 Header

Mergedcodestream

Issues• SVS format “quirks”

– Somewhat unusual component subsampling

• Limits the number of decoders that will read the data without complaining

– SVS tiles are transformed into YCrCb color space

• Requires a transform to RGB color space before use

• A compliant JP2 reader should do this, but not all do

Results

JPEG-2000

Compression

Codestreams

File Formats

JP2

Aperio SVS

Design

Access SVS Data

Convert SVS to JP2

Results

• Several tools were created, allowing:

• Direct access to SVS tile data– Manipulate, extract, and montage tiles– See LOCI Bioformats project

• Direct conversion of SVS files to JP2– Requires the SVS tile dimensions to be multiples of 64

– Embedded ICC profiles not propogated

Future work• Implement software in Java

• Create a JP2 SVS/TIFF converter

• Investigate limitations: – Why must tile dimensions be multiples of 64?– Why do most decoders (notably the JAI decoders) fail to

decode JPEG2k codestreams from SVS?

• Include ICC profiles

• Develop a generic LibTIFF plugin for encoding/decoding JPEG2k image data

AcknowledgementsJohns HopkinsKristen Lecksell

University of Wisconsin-Madison, LOCIMelissa LinkertCurtis Rueden

ReferencesTaubman and Marcellin, JPEG2000 Image Compression

Fundamentals, Standards and PracticeAperio Technical Docs (http://www.aperio.com/documents/)

LibrariesLibTIFF (http://www.remotesensing.org/libtiff/)ImageMagick (http://www.imagemagick.org/script/index.php)Jasper (http://www.ece.uvic.ca/~mdadams/jasper/)