Digital Microscopy and Imaging UpdateDigital Microscopy and Imaging Update

Michael Feldman, MD, PhDAssociate Professor of Pathology

Assistant Dean for Information TechnologyUniversity of Pennsylvania School of Medicine

(Feldmanm@mail.med.upenn.edu)

Digital Imaging OutlineDigital Imaging Outline

Digital Imaging• What is it? Why use it?

How do you use it (What are you trying to achieve)? • Integrate into clinical practice• Education• Archive• Research

Hardware• Camera’s (Gross and microscopic)• Server’s for storage• Computer to manage, manipulate, display and utilize data• Whole slide (virtual) digital slide vs static

Software• Storage• LIS Integration• Manipulation

– How to make it look better…

• Web• Education

Why are we here?Why are we here?

What do we hope to learn today?

Who is using digital imaging?

How and where is it being used in your practice?

Why are you using digital imaging technology?

Any burning questions out there about this technology that we can discuss today?

Digital Imaging Digital Imaging (What are we talking about?)(What are we talking about?)

What is digital imaging?Use of digital imaging technologies in place of analog systems to capture our daily work for archive, publication, education and research

Implicit in this definition is a need for pathologists to understand (or hire folks who do) hardware and software as well as provide users with technology that enables this technology

Define “Use case” for imaging: • Why use imaging, what are the goals, what are the needs, what is the cost…

Why switch from the Why switch from the Good Old DaysGood Old Days……

35mm slide workhorseEveryone knows how but not everyone does it wellFolks already have large collectionsCost is ~$4-5K for 35mm camera set up for a microscopeCost for gross images <$1K for good 35 mm camera

Pathology is a Visual DisciplinePathology is a Visual Discipline

Areas with Primary Image Data:• Gross Pathology – Autopsy and Surgical Pathology• Microscopic Imaging – Autopsy, Surgical Pathology,

Cytopathology, Hemepath, Microbiology• Electron Microscopy• Immunofluorescence Microscopy• Molecular Diagnosis –gels, microarray, SNP

Current ParadigmCurrent Paradigm

Sinard et al, Human Pathology 2002

Pathology WorkstationPathology Workstation

Sinard et al, Human Pathology 2002

So, why switchSo, why switch……Possible Use CasesPossible Use Cases

Cost• ROI for digital imaging is very positive• Gross room ROI 3-4 months• Microscope imaging 1-2 years

Convenience• Immediately available and easily modifiable• Reusable content• Shareable content among faculty and housestaff

Usability• Education

– Image retrieval, Online education, Content based Image Retrieval

• Publication – require electronic formats• Research

– Quantitive– Higher throughput

• Collaboration and consultation– Online

Digital Imaging OutlineDigital Imaging Outline

Digital Imaging• What is it? Why use it?

How do you use it (What are you trying to achieve)? • Integrate into clinical practice• Education• Archive• Research

Hardware• Camera’s (Gross and microscopic)• Server’s for storage• Computer to manage, manipulate, display and utilize data• Whole slide (virtual) digital slide vs static

Software• Storage• LIS Integration• Manipulation

– How to make it look better…

• Web• Education

What are you trying to accomplish?What are you trying to accomplish?

Do not underestimate this statement!• Your hardware and software needs will vary depending on what you want

to accomplish as will the cost of the project

How are you planning on using the digital images?• Integrate into report (LIS integration vs custom solution)• Education – Med school (UME), residents (GME), attending (CME)?• Publications – which formats• Consultation – real time vs static image transfer• Research – archive vs image process vs CAD vs Histocytometry• Clinical support – case conferences, tumor boards, etc

What areas are amenable to Digital Images?What areas are amenable to Digital Images?

Gross Room – Autopsy and Surgical PathologyExcellent place to start

• Large volume and good case mix• PA’s can assist with technology• Easier to enforce standards within a small group• Education - Images are great for clinical conferences, publications,

education of medical students and house staff• Cost can be low – consumer level digital camera at least 3 megapixel or

better will do very nicely in these areas• 3 Megapixel will do a great job on images printed at up to 5x7. If you think

you will need 8x10 then go for a 5 megapixel camera

Size Really Does MatterSize Really Does Matter……

Pixels – basic unit of digital camera sensor• Image is derived from red, green and blue information at each pixel• Most cameras produce 8 bits (256 shades) of information at each color

(red, green and blue) to create a total of 24 million color variations• 8 bits = 1 byte so…Since each pixel contains 8 bits of data for red, green

and blue, a single pixel with 24 bits of color information takes up 3 bytes of data

Size – Megabyte (million), Gigabyte (billion) and Terabyte (Trillion)

• A single image from a digital camera will occupy 3 bytes of data, – 2 megapixel camera = 6 Megabyte image– 3 mgeapixel = 9 Megabytes image– 5 megapxiel = 15 Megabyte image– 12 megapixel = 36 Megabyte image

• These camera’s can produce large files!

File formatsFile formats

The image produced by camera is stored as a file, of which there are many different file types.Loss less formats:

• RAW – pure binary data from the camera sensor. Each manufacturer has it’s own raw format (images are big)

• TIFF – standard established years ago. May be used without compression for lossless image (images are big)

Lossy Formats• JPEG – (Joint Photographic Experts Group ) Again, an older standard which uses a

mathematical algorithm to reduce the image size (throws out some data). – Variable amount of compression can be used from very little to a whole lot.– Personal experience suggests 10-20:1 compression produces so little change that it is imperceptible

(Photoshop setting of quality = 75 in a jpeg file produces ~15:1 compression

• JPEG2000 – newer version of jpeg uses wavelet compression. More compression with less degradation of image quality. Not widely adopted yet but could be interesting in future. Also does lossless compression

Autopsy PathologyAutopsy Pathology

Built In Macro for Very Close WorkBuilt In Macro for Very Close Work

Pseudo Melanosisof Esophagus

Small lymphoid aggregate with anthracotic pigment

Lymphoid aggregates are < 1 mm in size

Tiff Tiff vsvs JPEG (2X) with Leica D480JPEG (2X) with Leica D480

TIFF

JPEG – 20:1

Tiff Tiff vsvs JPEG (10X) with Leica D480JPEG (10X) with Leica D480

TIFF

JPEG – 20:1

Tiff Tiff vsvs JPEG (40X) with Leica D480JPEG (40X) with Leica D480

TIFF

JPEG – 20:1

Gross ImagingGross Imaging

3.3 Megapixel camera (This is minimum requirement)• Generates uncompressed 2048x1536 pixel images 9 Mb uncompressed• Images displayed as well as the printouts are jpeg compressed to 400-500

Kb (15-20:1 compression ratio). Compression is done in the camera

Key features:• PA’s are a constant in the laboratory and can help educate other users• Yearly conference to teach how to use the camera• Monthly gross image conference for residents to use the images for

teaching gross pathology (helps folks realize benefit from their efforts)• Integrated into daily work for resident’s, PA’s• Images transferred and stored in central image database server archive and

indexing• Resulting images are used in web based education materials for housestaff:

– Web based Interesting cases– Web based Board review– Web based support for Clinical conferences (Breast, Liver, ENT pathology)– Web based image query for search and retrieval

Microscopic ImagingMicroscopic Imaging

More challenging than gross photographyTechnical Issues:

• Static vs wide field full digital slides• Camera’s

– Which type (consumer camera, dedicated microscope camera, single chip, progreesive scan)– How many pixels is enough?– S/N ratio and dynamic range of cameras

End User Issues:• Ease of use – how many seconds does it take to make a good picture?• This is more important than the technical stuff• Good way to test this is to get system in house and play with it for a few weeks.

Alternately, go visit someone with the setup you are interested in buying• Know the end user – match the system to the person

Always extended test before purchasing!

Image and AcquisitionImage and Acquisition

Microscope with glass slide:• Start with good scope and slide - garbage in, garbage out...

Your eye can resolve very small differences. No camera, digital or otherwise is currently better.Color film captures information at approximately 6000 dpi

• Film 6000 dpi• High end digital camera 3000 dpi

Intermediate digital camera 1500 dpiLow resolution digital camera 600 dpi

How many pixels do you need?• Depends on what you are doing with the image?

Resolution and Imaging SlidesResolution and Imaging Slides

•

Dirk Soenksen CEO Aperio technology

CameraCamera’’s and Technologys and Technology

Standard Photograph - gold standardDigital Camera

• Single chip• Triple chip• Progressive scanning

Whole slide scanning (Digital or Virtual Slide)

Digital Camera Digital Camera -- Single chip CCDSingle chip CCD

CCD - Single chip

Histology Slide

Microscope Lens

Digital information

Each pixel measures a color intensity.

A filter is used to split the signal into Red, Green and Blue (RGB) signals at each pixel (Bayer = GRGB). The information is then combined in software to create final image which is not true colorSPOT, Leica, Olympus…

3 Images taken with filter switching RGB at each pixel. Final image is true color.SPOT 3 shot

Filter for RGB

Digital Camera Digital Camera -- Triple chip CCDTriple chip CCD

CCD - Three chip

Histology Slide

Microscope Lens

Digital information

Each pixel measures a color intensity.

A prism and filters are used to split the signal into Red, Green and Blue (RGB) signals for each of the three CCD arrays.

The information is the combined to give a true color at each pixel in the CCD array

Final image is true color at each pixel

In general used for high end analog video

Prism to split light

Digital Camera Digital Camera -- Progressive scanningProgressive scanning

Progressive scanning

Histology Slide

Microscope Lens

Digital information

Each pixel measures a color intensity.

A filter is used to split the signal into Red, Green and Blue (RGB) signals at each pixel.

The information is the combined to give a true color at each pixel in the CCD array.

10-12 megapixel cameras

Slow acquisition, vibration, small stepping motors

Leica, Zeiss, Nikon

CCD Array is moved across field to captureinformation between pixels that a stationary CCDwould miss

Virtual slides (Stitching)Virtual slides (Stitching)

BLISS Bacus Labs

Microbrightfield

Digital Slide (Virtual)Digital Slide (Virtual)

- A 1 cm2 piece of tissue imaged at 40X (0.25 mm FOV)

- Creates a 40x40 grid = 1600 Images- Each Image is 1 Mb- Total Image Tiled together is 1.6 Gb

uncompressed- With compression 150-200 Mb- We view 250,000 Pieces of glass/year- If we scanned 33% of our glass

(80,000 pieces of tissue) at 200 Mb/slide = 16 Terabytes/year! (That is 23, 73 GB HD’s)

Whole slide scanner (Whole slide scanner (ScanScopeScanScope from Aperio Inc.)from Aperio Inc.)

Digital Slides (Wide field whole slide)Digital Slides (Wide field whole slide)

DMetrixDMetrix’’s arrays array--microscope technologymicroscope technology

Problem with current microscopy paradigm: • High detail means small field of view; and• Small field of view means long time for imaging

Solution: Parallel imaging with the equivalent of 80 or more microscopes in one instrumentResult: High throughput

• True if: only one image per slide, and• True if: multi-plane, multi-color, multi-modality image

set per slide

Virtual slide scanningVirtual slide scanning

Many vendors in space today• Dmetrix – microsocope array• Aperio – linear scanner• Olympus/Baccus - tiles• Hammatsu - tiles• Zeiss/HistoRx - tiles• Bioimagene - tiles• Clarient/Trestle - tiles

Digital Imaging OutlineDigital Imaging Outline

Digital Imaging• What is it? Why use it?

How do you use it (What are you trying to achieve)? • Integrate into clinical practice• Education• Archive• Research

Hardware• Camera’s (Gross and microscopic)• Server’s for storage• Computer to manage, manipulate, display and utilize data• Whole slide (virtual) digital slide vs static

Software• Storage• LIS Integration• Manipulation

– How to make it look better…

• Web• Education

Now that you have the image, what nextNow that you have the image, what next……

Image Storage: Department server with 200 GB RAID 5 array and tape backup

Database – Client Server (ThumbsPlus from www.cerious.com) running against Microsoft SQL 7 or 2000

Autopsy images are 0.3 MB (300 KB) each. We capture 1000-2000 images/year. That will require 0.3 – 0.6 GB space/year

Surg Path gross images are small – 0.3 MB (300 KB). We capture 400-500 images/month 1.5 – 2 GB space/year

17 Faculty and resident’s with digital microscope cameras – 15-20,000 images/year at 0.3 MB = 5-6 GB/space/year

Combined, we will use < 10 GB of space each year.

Carefully think through and consider your needs!

Image DatabaseImage Database

ThumbsPlus from Cerious SoftwareScalable

• Single user ($89.95/seat) runs Access database• Multiuser – Scalable to enterprise level using Microsoft SQL database

– Added cost of central file server and database server and software (5-7K)

Integrates with cameras and scanner’s – TWAINCustomizable user fields, annotation, keywordsControlled vocabularyGallery view work’s like digital light boxCatalog offline disks (CD, DVD…)

Screen Shot ThumbsPlusScreen Shot ThumbsPlus

Expl

orer

Vie

wPr

evie

w/T

ask

Thum

bnai

l Are

a

Image PropertiesImage Properties

Web Search PageWeb Search Page

Simple client for web enabled image searchEasier to understand than thumbs client search functionGreat to be able to distribute light weight thin search client

To Do:• Shopping cart metaphor being adopted to help folks collect multiple

images as they shop for data• Keep pushing folks to carefully label and annotate images

Web based Search pageWeb based Search page

Board Review Board Review –– AnswerAnswer’’ss

Medical School Case AuthoringMedical School Case Authoring

Med School Case AuthoringMed School Case Authoring

Client side viewingClient side viewing

What software is used to pull images from server• Java Applet from web page (Aurora, Microbright, Bacus) – cross

platform• ActiveX (Zoomify, Bacus, Aperio) – integrate into web or custom

application (windows)• Stand alone application (Aperio) – windows based• HTML (Aperio) using zoomify format • Quicktime (Zoomify)• Flash (Zoomify)

Advantages of AppletAdvantages of Applet

Whether Flash, Java, Quicktime• Cross platform viewing Mac and PC• Easier to support across enterprise for software updates• Portable to devices including (PDA’s)• Lightweight (<100 Kb)

AnnotationAnnotation

Serving up standardized microscope quality high resolution images is start

• Uniformity• No bad slides, cracked, not representative• No slide handling and management• Slides always available

Even better if we can author content on slides and with the slides into educational lessons

Annotation ContAnnotation Cont’’dd

Ability to annotate image can be used in multiple manners

• Simply label slide – simple answer

• Prior to lesson - Author material with questions and no answer’s• After lesson – provide same images with answers and pointers

• Allow students to label images with questions independent of faculty annotations

– Questions– Interactive – like a threaded discussion that embeds digital slides

• Which fit’s your pedagogy?

How do you teach with glass vs. digital?How do you teach with glass vs. digital?

Glass: Search and identify with you assisting and facilitating in a small groupDigital: SameDigital: Compare and contrast multiple slides at onceDigital: Before and after scenarioDigital: support’s asynchronous learning, threaded discussion

• Student to student • Student to faculty

Digital: expert learning system with intelligent tutor (Rebecca Crowley, University Pittsburgh)Digital: distributed multiheaded microscopeDigital: share content

Usage Data from Penn PilotUsage Data from Penn Pilot

Usage stats from one course section (4 labs):• 1.6 million hits (each time move slide registers as hit• Only 12 manually tiled slides used• Avg 22 minutes per visit• More 50% viewed outside campus• 150 students viewed slides from > 200 computers• Pushed > 2 Gb data during these 4 small group sessions

Feedback from PilotFeedback from Pilot

Uniformity of imageAlways there to studySupported group studyImage quality excellentStudent’s liked it, more comfortable than search and find on microscopeWanted more of it

New Value in Going DigitalNew Value in Going Digital

Image Processing• CAD (computer assisted diagnosis)• Automated and quantitative IHC – single color• Spectral imaging – multicolor immunostaining “Histocytometry” or flow

cytometry for your slide• CBIR – content based image retrieval• Distance consultation/collaboration

Light has no color. “Color” is an observer interpretation.These two yellows appear identical to the human eye, yetthey have very different spectral components.

Color and Spectra

MSI vs. Traditional Color Imaging

Inte

nsity

Spectral

400 500 600 700

Spectroscopy captures the entire spectrum (light intensity as a function of wavelength).

Ired

= 253Igreen

= 203Iblue

= 112

RGB

But with RGB-based instrumentation, this complex spectrum will be described using only 3 values (bins) averaged over large spectral regions.

Inte

nsity

“extract” spectra of individual stains

3. Image processing to resolve individual stains based on spectra

teach computer thespectral profile ofchromogen/fluorophoreA, B, C

A B C

breast tumor stainedfor p-ERK (DAB) & hematoxylin

p-ERK- stromal cells (grey)

segmentation of nuclei(based on hematoxylin)

p-ERK+ tumor cells (yellow)

6. Computational assignment of immunostains to each nucleus

Data display & Analysis:Data display & Analysis:Frequency histogram of intensity of pFrequency histogram of intensity of p--ERK staining of stromal and tumor cell ERK staining of stromal and tumor cell

nuclei in a breast tumornuclei in a breast tumor

tumor cell nuclei

stromal cell nuclei

A B

C DFr

actio

n of

nuc

leii

0

50

100

150

0 20 40 60 80 100 120

0

50

100

150

0 20 40 60 80 100 120 140 160 180 200

A B C

E F G

D

L

H pERK

Ki6

7

pERK

Ki6

7

0

50

100

150

0 20 40 60 80 100 120

P

pERK

Ki6

7

0

50

100

150

0 20 40 60 80 100 120

pERK

Ki6

7

I J K

N OM

pERK Ki67

Don’t hesitate to ask if you are uncertain