Image Analysis: Basics and Practice...Dr. Xianke Shi, Zeiss Research Microscopy Solutions. 14....

Image Analysis: Basics and Practice

Dr. Xianke ShiZEISS Research Microscopy Solutions

2Dr. Xianke Shi, Zeiss Research Microscopy Solutions

Microscopy Images: A Picture is worth a thousand words!

How Many Cells are DAB positive?

A: 1-10%B: 10-20%C: 20-30%D: 30-40%


Microscopy Images: A Picture is worth a thousand words!

How Many Cells are DAB positive?

A: 1-10%B: 10-20%C: 20-30%D: 30-40%

𝐷𝐷𝐷𝐷𝐷𝐷 =234

234 + 418% = 35.9%


Visual perception is not quantitative, and sometimes not reliable

What is the grey level difference between black and white surface (8 bit image)?

A: 0B: 10C: 20D: 30

Use your finger to block the center region (red arrow indicated). There is no grey level difference!

Imagine you are judging protein expression level by visually compare neighboring tissue brightness!



Internet sensation from 2015

#blackandblueVS#whiteandgold



Humans recognize, machines measure• Human vision is qualitative and comparative• It is very powerful in recognizing patterns• The performance for intensity, distance and

size measurements is poor

Machine based image analysis improves:• Objectivity• Reproducebility• Speed


The earliest surviving Analog Fotograph of the World

First Photograph

• In the year 1826• Bitumen plate• Exposure time: 8 h • Photographer:

Joseph Nicéphore Nièpce

http://upload.wikimedia.org/wikipedia/commons/5/5c/View_from_the_Window_at_Le_Gras%2C_Joseph_Nic%C3%A9phore_Ni%C3%A9pce.jpg


The first digital photograph

First Digital Photograph

• In the year 1975• 20 year before the first digital

camera by Kodak• Resolution: 176x176 pixels• Photographer: Russel Kirsch• His 3 months old son• Digitally scanned from analog

photograph


Digital Imaging: the foundation of Image Analysis

A digital image is a two dimensional array of number matrix. Each number refers to as a pixel (2D) or voxel (3D).The word pixel is an abbreviation for picture element.Higher dimensions is common for microscopy:

Z-stack (z)Time (t)Spectral Channels (λ)Tiles and Position (p)


Digital Imaging: the foundation of Image Analysis

For microscopy imaging, additional Metadata is usually built into the raw data:• Objective information (model, magnification, numerical aperture, refractive index)• Dimensional information (pixel size, frame size X/Y/Z)• Coordinate information (absolute stage position X/Y/Z, well number, position number)• Acquisition parameters (camera exposure, binning, zoom, gain, offset, pixel dwell time)• Environment information (Temperature, humidity, CO2 concentration)• Image information (Acquisition date, time, duration, name)• Many more…

https://docs.openmicroscopy.org/bio-formats/5.9.2/metadata/ZeissCZIReader.htmlThe Zeiss .czi file format has 476 metadata entries:

Those metadata is useful for image analysis, additional image processing, and programming based microscopy automation!

https://docs.openmicroscopy.org/bio-formats/5.9.2/metadata/ZeissCZIReader.html


Understand Digital ImageBit depth and grey levels

• With 8-bit images 0 to 255 (“grey values”).• With 14-bit 0 to 16383 grey values• With 16-bit 0 to 65535) grey values

More Bits:• More accurate representation of the

intensity values• More time it takes to transfer data, slower

image acquisition


Understand Digital ImageColor image

A standard RGB color image is a combination of 3 layers of Red, Green, and Blue channels.It can be also defined by HLS color space for: Hue, Lightness, and Saturation.

Each color channel has its own grey value


Understand Digital ImageRGB image vs Multichannel images

x

y

x

y

Multichannel imageMultiple channels, each with 1 Histogram

RGB imageOne channel with 3 Histograms

DAPIeGFP

…


Understand Digital ImageHistogram

A histogram is a graphical representation of the pixel grey value distributions of an image.

Normalized number of pixels

Pixel grey values

Black pixels (low grey level)More such pixels

Brightest pixels (highest grey level)Less such pixels

Logarithmic scale

Linear scale


Understanding Digital ImagesHistogram: RGB image vs Multichannel images

A histogram is a graphical representation of the pixel grey value distribution of an image.

Fluorescence Multichannel image

RGB Image

• Multiple histograms• Each histogram has its own display setting• Default gamma 1.0

• RGB histograms• One display setting• Default gamma 0.45


Understand Digital ImageHistogram and adjustment of display curves for visual enhancement

Display curves of the histogram can be adjusted to better visualize the imageSuch adjustment won’t change the grey value of the image


Image Analysis and its Workflow

Pre-Processing Segmentation Feature

DetectionData

Presentation

• Smoothing• Sharpening• Noise reduction• Shading correction• Histogram Stretching

• Intensity• Size• Shape• Coordinates

• Thresholding• Variance based• Machine learning

• Line plot• Scatter plot• Heatmap

“Image analysis is the extraction of meaningful information from images; mainly from digital images by means of digital image processing techniques.” --- Wikipedia

For Microscopy, it is also a essential part of microscopy automation.


Pre-Processingoverview

• Reduction of acquisition artefacts which could influence the results, e.g. Shading and noise • Improving image quality for better target identification, e.g. edge enhancement• Can be applied for segmentation purpose only, then subsequent parameter measurement still performed on raw data

Typical pre-processing steps:

• Shading Correction • Backgroud subtraction• Smoothing• Noise reduction• Spectral unmixing• Deconvolution• Histogram stretching• …

shading correctionoriginal

Corrected image after thresholdingoriginal after thresholding


DetectionData

Presentation


Pre-ProcessingPixel based Image Processing (aka filtering)

input image output image

• One pixel in the input image is processed and is assigned to the same pixel in the output image

• The new greyvalue can also be calculated from all pixels in a small neighborhood around the input pixel

• The neighborhood size is usually called a kernel

3x3 kernel

• This procedure is repeated for all pixels of the input image

• The kernel operator can be linear (e.g. Gaussian and mean filter) and non-linear (e.g. median)


88 81 152 227 255 214 211 145

219 185 255 193 193 142 162 140

255 242 240 216 194 109 125 118

119 66 142 93 96 125 204 173

125 71 43 64 102 75 94 163

166 94 100 45 89

147 108 100 75 71

67 83 87 104 54

⊗ 𝟏𝟏𝟏𝟏𝟏𝟏

1 2 1

2 4 2

1 2 1

=

163 147 171 193 200 191 178 160

169 171 182 186 183 171 161 151

175 171 171 167 159 150

3x3 Gauss

Pre-ProcessingExample of 3x3 kernel Gaussian filter


SegmentationThe most critical step

• Segmentation: partitioning a digital image into multiple segments, i.e. find groups of pixels that belong together

• Binary process: the object of interest is 1, the rest is 0. The result is a “mask” of the segmented objects.

• The “mask” is usually one additional layer on the original image. The mask can be further refined by operations on the binary mask (e.g. dilation, separation,…)

• The segmentation can be based on different image information (e.g. intensity, color, pattern, pixel position,…)


DetectionData

Presentation


SegmentationThe most critical step

Threshold (dividing the histogram basedon intensity)

- Fluorescence images

Variance(local change in intensity, usually linked to edges)

- transmission images

Intellesis (machine learning)

- Everything (computationally heavy)- Nothing else works- Ease of use


Segmentation by global thresholdingThe most common method

# pixels

Gray valueIntensity based separationThreshold between two populations

Foreground

Background

Original After Threshold


Segmentation by global thresholdingManually

Manually define upper/lower threshold

3 sets of threshold for RGB image

Useful software tools for direct mouse-click-on-object and thresholding


Segmentation by global thresholdingAutomatically

There are 5 different automatic thresholding methods in ZEN:• Otsu

(objects and background separation with minimum sum of variance for each group)• Max Peak

(separation at histogram maximum value)• Iso Data

(separation at the middle of two histogram maximum values)• Triangle

(separation at the sum of the average histogram value)• Three Sigma

(separation at the sum of the average histogram value with 3 sigma)

The success rate of auto thresholding depends very much on image SNR.Improve image SNR during acquisition is vital to the final result!As a simple guide, draw a profile along the object, and its SNR should be >5 for auto thresholding to work efficiently.


There is no absolute good or bad automatic thresholding method. Each of them is applicable to different type of images.It is usually good to start with Otsu.

Original Otsu Max Peak Iso data Triangle Three sigma

2202-6866 324-6866 2202-6866 813-6866 4851-6866

Segmentation by global thresholdingAutomatically


Segmentation by Variance-based thresholdingsegmentation based on local variance

Transmission based images, like BF, DIC, Ph, PGC, has little grey value difference between object and background. Histogram thresholding based segmentation usually won’t work.

Profile tool

PGC

One population Two populations

VS

Higher localvariance


Measure stdev

Set threshold 100-1000

Necessary binary process

Segmentation by Variance-based thresholdingExample in ZEN


Classify Segmented Image

Input Data

Feature ExtractorsLabel

TrainedModel

Retrain

Segmentation by Machine LearningA nutshell


Segmentation mask refinementBinary operation

Roeder et. al. Development (2012)

The “segmented mask” can be refined / cleaned up by morphological operations.

This affects only the mask itself, but not the original image.

• Dilate: enlarge the boundary of the mask by counts/pixels

• Erode: shrink the boundary of the mask by counts/pixels

• Open: Erode then dilate: smoothens and removes isolated pixels.

• Close: Dilate then erode, smoothens and fills small holes.


Separate objects that are connected.Acts only on the binary image.

• MorphologyErode then dilate, objects that are separated won’t be merged again

• WatershedsSeparate objects assuming they are roughly the same size, might wrongly split elongated objects

Original

Count 6

Count 9

Morphology

Count 1

Count 3

Watersheds

Segmentation mask refinementobject separation


Segmentation mask separationSeparate via Morphology

original

erosion 3erosion 1 erosion 2

• First an erosion process is applied to all objects, which reducestheir size and separates them.

• The larger the object, the more counts needed for separation. Inthis example 3 erosion counts are necessary to separate allobjects completely.

dilation 3dilation 1 dilation 2

• After erosion, the objects are enlarged again by a dilation processto get back the original size.

• This dilation is done without allowing objects being merged again.

• Objects with different sizes cannot be handled properly• Erosion and dilation might change the original object shapes

original 60 counts18 counts 26 counts


Segmentation mask separationSeparate via Watershed

Step 1: “Distance transformation” of the binary image(The distance to the closest edge is calculated for each object pixel and is coded as a grey value)

Step 2: grey value as height within a “landscape”(object is a basin)

Step 3: “landscape” is filled with water, and “watershed” can be detected between two touching basins.

Step 4: Objects are separated at “watershed”

• Powerful to separate objects with different sizes• Do not change original object shape• Hole within binary object confuses the algorithm

(“fill hole” necessary)


Segmentation summaryA proper segmentation is usually a combination of a few processes

Lowpass smoothing Unsharp Mask sharpening Fill holes

Min area set 150 Watersheds separation Lowpass and watersheds

Original


Feature DetectionWhat to measure?


DetectionData

Presentation

Do I measure all objects?Or conditioned objects?

e.g. measure only metaphase cells

If >1 object of interest, are they independent or associated? And how associated?

e.g. measure total mitochondria expression per cell

Do I measure parameters for a group of objects? Or a single object?

e.g. counting total cells per image, and measure intensity per cell


Feature DetectionParameters of group vs single

Group parameters:• How many nucleus (count): 10• How much area do they occupy

(Area percentage): 5%

Single object parameters:• Area: 20 um2

• Mean Intensity: 500• Roundness: 0.9


Feature DetectionObject filtering

• Filtering further refine/limit the segmented objects.• It allows detecting only specific objects, rather than all segmented objects • e.g. detect only metaphase cells and not interphase cells.

Courtesy of internet

FL image

Metaphase cell

What conditions to use to isolate metaphase cells?(In metaphase, chromosomes condense)• Area• Mean intensity• roundness

Setup filtering condition


Feature DetectionObject association

You want to quantify the total green structures (area and intensity) per cells

A counter stain channel is needed for cell identification (aka master channel)

If object of interest is outside master channel, binary process is needed

A second segmentation is performed within the master channel and parameters measured

The final data structure can be very complicated, when there are:• Multiple objects of interest• Multi-channel fluorescence data• Multi-dimensional data: time, z-stack, positionsPlan the data structure carefully, and minimize detected parameters


Data Presentation Pre-Processing Segmentation Feature

DetectionData

Presentation

Display your data in a way that highlights the question you want to ask.

Line chart for time seriese.g. scratch assay

Histogram for distribution analysis

e.g. cell size distribution

Scatter plot for relationship analysis

e.g. multiple gene expression

Heatmap for HCSe.g. cell growth comparison


Image Analysis and Microscopy Automation

“...Ultimately, the goal is to have a general platform for a wide range of applications that automatically detects the object and acts accordingly. ....”

Life science researchers, these days, require higher number of microscopy images at ever faster speed.

Microscopy automation is essential to achieve such ambitious goal.

Image analysis is one indispensable step inside microscopy automation.

Image Analysis: Basics and Practice...Dr. Xianke Shi, Zeiss Research Microscopy Solutions. 14....

Documents

Transcript of Image Analysis: Basics and Practice...Dr. Xianke Shi, Zeiss Research Microscopy Solutions. 14....