SEEING IS BELIEVING?

Chris Glasbey

Biomathematics and Statistics Scotland

Image analysis is the extraction of information from pictures

Which line is the longer? Who is this?

2

The human eye/brain is a superb image analyser

So, why use a computer?

• better for quantitative tasks (repeatable/non-subjective)

• cheaper, faster and less tedious

• capable of different techniques

3

An example of non-automated image analysis:

Cross-section of turbinate bone

=⇒Using sno-pake and black ink.

4

But automating image analysis is hard, because all a computer ‘sees’ is:

132 128 126 123 137 129 130 145 158 170 172 161 153 158 162 172 159 152139 136 127 125 129 134 143 147 150 146 157 157 158 166 171 163 154 144144 135 125 119 124 135 121 62 29 16 20 47 89 151 162 158 152 137146 132 125 125 132 89 17 19 11 8 6 9 17 38 134 164 155 143142 130 124 130 119 15 46 82 54 25 6 6 11 17 33 155 173 156134 132 138 148 47 92 208 227 181 111 33 9 6 14 16 70 180 178151 139 158 117 22 162 242 248 225 153 62 19 8 8 11 13 159 152153 135 157 46 39 174 207 210 205 136 89 52 17 7 6 6 70 108167 168 128 17 63 169 196 211 168 137 121 88 21 9 7 5 34 57166 170 93 16 34 63 77 140 28 48 31 25 17 10 9 8 22 36136 111 83 15 48 69 57 124 55 86 52 112 34 11 9 6 15 3049 39 46 11 83 174 150 128 103 199 194 108 23 12 12 10 14 3426 24 18 14 53 175 153 134 98 172 146 59 13 14 13 12 12 4621 16 11 14 21 110 126 47 62 142 85 33 10 13 13 11 11 1517 14 10 11 11 69 102 42 39 74 71 28 9 13 12 12 11 1818 19 11 12 8 43 126 69 49 77 46 17 7 14 12 11 12 1924 30 17 11 12 6 73 165 79 37 15 12 10 12 13 10 10 1624 40 18 9 9 2 2 23 16 10 9 10 10 11 9 8 6 1043 40 25 6 10 2 0 6 20 8 10 16 18 10 4 3 5 739 34 23 5 7 3 2 6 77 39 25 31 36 11 2 2 5 217 16 9 4 6 5 6 36 85 82 68 75 72 27 5 7 8 04 8 5 6 8 15 65 127 135 108 120 131 101 47 6 11 7 42 9 6 6 7 74 144 170 175 149 162 153 110 48 11 12 3 5

11 9 3 7 21 127 176 190 169 166 182 158 118 44 10 11 2 58 0 5 23 63 162 185 191 186 181 188 156 117 38 11 12 25 333 5 6 64 147 182 173 190 221 212 205 181 110 33 19 42 57 505 3 7 45 160 190 149 200 253 255 239 210 115 46 30 25 9 59 4 10 16 24 63 93 187 223 237 209 124 36 17 4 3 2 17 8 13 8 9 12 17 19 26 41 42 24 11 5 0 1 7 4

Could you recognise Mona Lisa from this?

5

What are the data? microscopy medical scanner remote sensing

What is image size? 20µm 50cm 15km

What is measured? transmitted light X-ray absorption reflected light

How is objectsampled? focal plane cross-section perspective view

6

Three questions: 1. What is the 3-D shape of this diatom?

increasing focal plane

7

Question 2: How fat is this sheep?

⇒X-ray CT

SAC-BioSS CT Unit, led by Geoff Simm

8

Question 3: Can we distinguish between fish species?

haddock whiting

9

PLAN

0. Introduction

1. Microscopy

2. Medical scanners

3. Remote sensing (and fish)

10

1. Microscopy

Using computers, we can

• Visualise images in ways not possible optically

11

We can superimpose different microscope images of algae and bacteria

brightfield DIC

+

12

DIC fluorescent

+

(Nick Martin, SAC)

13

1. Microscopy

Using computers, we can

• Visualise images in ways not possible optically

• Count and measure more easily

14

1. Microscopy

Using computers, we can

• Visualise images in ways not possible optically

• Count and measure more easily

However, identifying objects to count and measure is more challenging if

• Objects vary in appearance

15

DIC image of yeast cells – example of template construction

↓ ↑

→ →integrate rotate differentiate

16

1. Microscopy

Using computers, we can

• Visualise images in ways not possible optically

• Count and measure more easily

However, identifying objects to count and measure is more challenging if

• Objects vary in appearance

• Optical effects distort what is seen

17

Brightfield image of cashmere fibres

These fibres are the same thickness, but look different

(Angus Russel, Macaulay Land Use Research Institute)

18

We use a mathematical model to understand how a fibre appears indifferent focal planes

→fibre oblique to focal plane

And use it to adjust for errors in measuring fibre thicknesses in images

19

1. Microscopy

Using computers, we can

• Visualise images in ways not possible optically

• Count and measure more easily

However, identifying objects to count and measure is more challenging if

• Objects vary in appearance

• Optical effects distort what is seen

Though optical distortions are sometimes an advantage

20

In answer to question 1: we can reconstruct the 3-D shape of this diatom

Brightfield images at series of focal planes

(Ed Breen, CSIRO – Mathematical and Information Sciences)

21

Step 1: compute local gradients in intensity

gradient: small large

22

Step 2: smooth the gradients

23

Step 3: at each location, choose image with maximum gradient

24

Step 4: reconstruct the 3-D surface

25

PLAN

0. Introduction

1. Microscopy

2. Medical scanners

3. Remote sensing (and fish)

26

⇒Ultrasound

(Geoff Simm, SAC)

27

To measure fat depth, we need to draw the red lines

28

We consider the pattern of pixel values on either side of a boundary, averagedover many images.

29

30

We compare the patterns with the different possible locations of a boundary,where darker values denote better fits:

top boundary bottom boundary

The best boundary is the darkest path between the two sides of each image

31

Illustration of Dynamic Programming:

Find connected path from left to right with minimum cost

32

One of 700 possible paths

33

Minimum cost paths to column 2

34

Minimum cost paths to column 3

35

All minimum cost paths

36

Minimum cost in final column

37

Trace path back to first column

38

Minimum cost path

39

Here we see the resulting automatic, and hand-drawn, boundaries:

40

In CT, pixel values range from −1000 to > +1000 Hounsfield units.

According to how we display these values we obtain different images.

41

For segmentation, we first ‘unroll’ the images

polar transformation boundary pattern

42

We then use level of X-ray attenuation to identify tissues

⇒automatic hand-drawn fat muscle bone

In answer to question 2: we predict sheep fatness from these measured areas

43

With a 3-D scanner there is more information but segmentation is harder

sheep skeleton leg muscles in pelvic region

44

PLAN

0. Introduction

1. Microscopy

2. Medical scanners

3. Remote sensing (and fish)

45

Landsat image of St Andrews, Scotland

It is rare to get such a cloud-free scene in Britain. Venus is even cloudier.

46

=⇒NASA Magellan mission SAR image of Mona Lisa crater, Venus

47

SAR (synthetic aperture radar) is also used on Earth

SAR image, East Anglia Map of field and road boundaries

(European Space Agency)

48

How the SAR image was obtained

x1

x2

h( , )x1 x2

φ1

f1

f2

We need to warp the SAR image to align with the map

49

Arad et al. (CVGIP: Graphical Models and Image Processing, 1994)

↘‖

→ →Venus Venus warped average

50

edges in SAR image

+

after warping

51

Estimated elevations

52

Resulting alignment

53

We can also use warping to identify fish

haddock 1 whiting 1

haddock 2 whiting 2

(Norval Strachan, Torry Research Centre)

54

Fish warping has a long history....

D’Arcy Thompson, On Growth and Form (1917)

55

haddock 1 haddock 2

↓ ‖

→warping haddock 1 ‘impersonating’

haddock 2

Dissimilarity between fish = difference after warp + distortion in warping

56

haddock 1 whiting 1

↓ ‖

→warping haddock 1 ‘impersonating’

whiting 1

57

average haddock average whiting

In answer to question 3: we can distinguish between fish species

Haddock are more similar to the average haddock

Whiting are more similar to the average whiting

58

Summary

Automated image analysis is a challenging problem,but offers the possibility of

• more accurate,

• faster,

• more sophisticated, measurements.

For further details, see

Glasbey, C.A. and Horgan, G.W. (1995)Image Analysis for the Biological Sciences. Wiley: Chichester.

and research papers on

http://www.bioss.ac.uk/~chris

59