Post on 25-Dec-2015
lakshman@ou.edu 1
Automated analysis of Spatial Grids
Valliappa.Lakshmanan@noaa.govPart of the short course on Artificial Intelligence at the American Meteorological Society Annual Meeting, Jan. 2006, Atlanta.
lakshman@ou.edu 2
Information from grids
Observed data are often on gridsSatellite imageryGround-based radarOther data can be gridded
You may need to extract information from these grids Is there a boundary?Where are the troughs?
lakshman@ou.edu 3
Spatial grids as images
Our spatial grids are often two dimensional
Automated extraction of information from images is a long-established field in machine intelligenceWe can treat spatial grids as imagesAnd apply image processing techniques to
our data.
lakshman@ou.edu 4
Image processing terminology
What you think of it as
What they call it Where is it used?
Grid Image
Grid point Pixel
Finding signatures
Pattern recognition Identifying machine parts on an assembly line
Finding storms Segmentation (ditto)
Tracking storms Optical flow MPEG compression
Smoothing, finding changes
Filtering Photo-editing software
Combining gridded data
Stereography Robotics
lakshman@ou.edu 5
Other terms for this field
There are subtle differences between image processing andData mining
Relationships between identified signaturesKnowledge discovery
A human draws the inferencesComputer Vision
Pattern recognition when it doesn’t work
lakshman@ou.edu 6
Limitations of image processing
Scenarios where pattern recognition succeeds: Highly controlled environments
Assembly lines, robotics Photography
Red, green, blue channels can be assumed independent IR channels can not!
Objects have sharp boundaries Storms, etc. do not
The most mature sub-fields Filters (smoothing, edge finding) Segmentation
lakshman@ou.edu 7
Workflow of AI applications
Artificial Intelligence applications that operate on spatial data usually follow these steps:
Gridding: take spatial data and make it a locally uniform resolution grid Filtering: smoothing, pattern matching, etc. Edge finding: find areas of drastic spatial change Segmentation: find contiguous areas of constant value Feature extraction: find features based on edges/regions Classification: classify features using a rule base, neural network, support vector
machine, etc. Normally done with all the features from all your data fields This will be covered by the other speakers
Gridding Filtering
Edge finding
Segmentation
Featureextraction
Classification(NN, etc.)
lakshman@ou.edu 8
A note on convention
Image processing uses matrix notation (0,0) is the top-left corner of the image (0,1) is the first row, second column.
As long as you don’t think of the first index as “x” and the second index as “y”, you will be okay In what direction does the first index
increase?Answer: South or downwards
lakshman@ou.edu 9
Image processing assumptions
Traditionally image processing assumes a Markov random process Two pixels are correlated if they are adjacent to
each other If you have pixels: a b c a and b are correlated as are b and c But the correlation between a and c is captured solely by
the correlation between a-b and b-c No “second-order” correlation term.
A simplifying assumption that usually causes no problems.
If your field is random (no correlation at all) or if the correlation is second-order image processing techniques may not work
lakshman@ou.edu 10
Image processing assumptions
An assumption that could cause problems: The field has uniform resolution Projected spatial grids are not uniform resolution.
Locally, most projected grids can be assumed uniform. Pixels 10 km apart don’t vary that much in size. Be careful with global operators on projected grids
Segmentation is a global operation
Filtering, edge finding, etc. are fine. Filters are local operations.
lakshman@ou.edu 11
Radar data
Radar data are NOT uniform resolution Size of a “pixel” varies with range Dramatic changes
Two options: Use varying-size windows for local operators
Hard to keep track! Convert to a Cartesian grid before applying image
processing techniques Will cause problems for local operators at far ranges
where data are “repeated” This is an issue with most projected data as well
lakshman@ou.edu 12
Missing Data
Some times, you will have spatial data where some pixels are missingCan you treat this data as some value?
Say zero?Will simplify your task considerablyCan dramatically speed up operations
If not, you can vary the formulae to use only those pixels for which you have data
Filtering results may have higher qualityOperations will be much slower
lakshman@ou.edu 13
Summary: Spatial grids
To treat spatial grids as images, Operate on the raw, un-projected data
Less chance of repeated values For “gridded” data, operate on grids at the resolution
closest to half the spacing between observations.
Change “missing” data to background value if available.
Use local operators (e.g: filters) freely Be careful with global operations (e.g: segmentation)
Convert computed features (such as area) to projected systems
lakshman@ou.edu 14
Gridding
From insitu observations to grids
lakshman@ou.edu 15
In-situ observations
Some environmental data are measured by in-situ instruments Not remotely sensed These measurements
are at geographically dispersed points
Need to be converted into grids
lakshman@ou.edu 16
Gridding observations
In-situ observations have to be interpolated on to grids to process as imagesWhat is the pixel resolution?What interpolation techniques are
available?
lakshman@ou.edu 17
Pixel resolution
If the chosen pixel resolution is too coarse, lose some of the observations
If the chosen pixel resolution is too fine, strong gradients may not be apparent
A good rule of thumb:Choose pixel resolution to be half the mean
distance between observation points
lakshman@ou.edu 18
Interpolation methods
Interpolation methods:Cressman analysisBarnes analysisKriging I_xy
I_i
lakshman@ou.edu 19
Cressman analysis
The value at each pixel of grid:
Every observation gets weighted based on its distance from grid point R is the “radius of influence” Higher the R, the more distant points are
considered
i iyyixxR
iyyixxR
iiyyixxR
iyyixxR
iI
xyI2)(2)(2
2)(2)(2
2)(2)(2
2)(2)(2
lakshman@ou.edu 20
Barnes analysis
The problem with a Cressman analysis:Even if a pixel is collocated with an
observation, the final value of pixel will be different
Due to (small) effect of pixels further awayBarnes analysis tries to account for this
lakshman@ou.edu 21
Barnes analysis equation
Barnes also changed the Cressman interpolation function to be exponential Smoother than
simple polynomials
i
iyyixx
i
iyyixx
i
e
eI
xyI22
2)(2)(
22
2)(2)(
lakshman@ou.edu 22
Barnes analysis: the technique
Perform Cressman analysis on obs Compute errors at observation points Perform Cressman analysis on errors
Give the error field a weight Add the two fields One pass of Barnes analysis
Can repeat Barnes analysis N-pass Barnes analysis Closer and closer to the value of the observations
at the observation points.
lakshman@ou.edu 23
Kriging
Kriging is a principled method of interpolation Compute correlation between observations Use variogram to determine scaling parameter Works only if the observations are very close to
each other M. A. Oliver and R. Webster "Kriging: a method of
interpolation for geographical information system", Int’l. J. Geographical Information Systems, 1990, VOL. 4, No. 3, 313-332
lakshman@ou.edu 24
Convolution Filters
A surprisingly powerful local operation
lakshman@ou.edu 25
Smoothing: an intuitive look
How would you smooth the image on the top right? So that the edges
aren’t quite so ragged?
To get an image like bottom right?
lakshman@ou.edu 26
Averaging is a smoothing operation
The most obvious way to smooth is to replace each pixel by a local average:
I used k=3 in the WDSS-II filter “average” http://www.wdssii.org/ The dataset I’m using for
demonstration is available online. See:
http://cimms.ou.edu/~lakshman/aitut/
kxi
kxi
kyj
kyj
ijkkxy II )12).(12(1
lakshman@ou.edu 27
How it works
What does this equation mean?
For every pixel x,y, look in a 2D neighborhood, add up all the values and divide by the number of pixels.
(2k+1) is the neighborhood size k is the half-size Because the neighborhood size
needs to be odd, most software will ask you for the half-size, k
kxi
kxi
kyj
kyj
ijkkxy II )12).(12(1
lakshman@ou.edu 28
Convolution
Another way to think about smoothing: At each pixel, you
assign weights to the neighboring pixels.
And compute a weighted average.
W is a (2k+1)x(2k+1) matrix shown on right. Called the convolution
matrix or kernel Technically, this is cross-correlation, but if you
are using symmetric kernels (as we almost always will be), cross-correlation and convolution are the same thing.
kxi
kxi
kyj
kyj
ijijxy IWI
111
111
111
9
1W
lakshman@ou.edu 29
Smoothing kernels
The “boxcar” kernel Poor characteristics in
transition areas. More spatial relocation of
maxima Smudges more
A better convolution kernel is where the kernel function drops off with distance.
A Gaussian convolution kernel offers the best space/frequency trade off
Frequency = raggedness Space = location
You can smooth a lot more effectively because the features don’t move or smudge as much.
Gauss:9
Average:3
lakshman@ou.edu 30
Gaussian Kernel
2
2)(2)(
22
1x
yx
eWxy
x and y range from –k to k. Since the Gaussian has infinite range, you need to choose k
appropriately. Choosing k to be 3 * largest sigma is a reasonable choice.
The sigmas are the scale Divide by total weight (The W above doesn’t add up to 1)
lakshman@ou.edu 31
Convolution kernels
By changing the weights in the convolution kernel, you can extract many features.
101
101
101
3
1W
lakshman@ou.edu 32
Matching filters
In general, make your convolution kernel be the feature you are trying to extract.
The above kernel returns large magnitudes for thin vertical lines But not just thin vertical lines Need to normalize by smoothed value also
Can use the matched filter idea to find objects
101
101
101
3
1W
lakshman@ou.edu 33
Scale and rotation
Convolution filters are: Scale dependent
What if the gradient is 5 pixels apart? Need a matching filter with columns 5 apart!
Orientation dependent What if the boundary is horizontal? Need a matching filter with that orientation.
Remember typical usage? Do you know the size of a machine part before
hand? How about a squall line?
lakshman@ou.edu 34
Scale and orientation
)(
22
22
2)sin(2)cos(
1 yx
yx
eWyx
xy
If you specify your convolution filter as Gaussian functions, it is easy to specify rotation and scale. x and y range from –k to k.
Since the Gaussian has infinite range, you need to choose k appropriately.
Choosing k to be 3 * largest sigma is a reasonable choice. Theta is the orientation from x-axis (anti-clockwise from south) The sigmas are the scale (use same sigma for circular kernels;
different sigmas for elliptical ones) Divide by total weight (The W above doesn’t add up to 1)
lakshman@ou.edu 35
Speed
The larger the scale of interest, the larger the half-size of the kernel. Will take a long time to compute. Since we need to compute weighted average at
every pixel.
Two speed ups possible Convolution can be performed in Fourier Transform
space. Convolution can be performed using separable
filters.
lakshman@ou.edu 36
Convolution with Fourier Transforms
To convolve with Fourier Transforms: Compute 2D FFT of original image
Can not have missing data Fill with background value somehow
Compute 2D FFT of convolution kernel The convolution kernel has to be same size as image (pad with zeros)
Multiply the FFT values pixel-by-pixel Take inverse 2D FFT of result.
The larger your kernel, the more dramatic the speed up. Can be 100-200 times faster. Seconds instead of hours!
Resources fftw is a free, nice Fourier Transform library. Lakshmanan, V: 2000: Speeding up a large scale filter. J. of Oc. and
Atm. Tech., 17, 468-473
lakshman@ou.edu 37
Convolution with separable filters
What if you have missing data? Can not use FFT method Problem with radar data in particular.
If the convolution filter is “separable”: Compute weighted average row-wise on original
image. Compute weighted average on the row-wise result
column-by-column. If your filter was 129x129, 60 times faster than
doing the standard way.
lakshman@ou.edu 38
What is a separable filter
A filter is separable if
For example, if the convolution filter is a simple Gaussian:
If you want to vary orientation, see: Lakshmanan, V 2004:
A separable filter for directional smoothing. IEEE Geosc. and Remote Sensing Letters, 1, 192-195
2
2
2
2
2
22
1112
yxyx
eeeW
WWW
xy
yxxy
lakshman@ou.edu 39
Edge finding
The “line-finding” filter is not a robust way to find edges
Similar problems to using boxcar for smoothing
Use Laplacian of a Gaussian (Canny)
Differentiate the smoothing kernel twice.
Convolve image with that kernel
Look for zero crossings in the convolution result.
J. Canny, ``A computational approach to edge detection", IEEE Trans. on Pattern Analysis and Machine Intelligence, 8(6),pp679-698 (1986)
101
101
101
3
1W
2
22
)2
1(
2
22
yx
eyx
Wxy
lakshman@ou.edu 40
Multi scale or orientation analysis Use filter banks:
To perform multi scale (multi resolution) or orientation analysis A bunch of filters at different scales and orientations Combine their results by a weighted average or maximum
Drawbacks: Repeated convolutions on original image. Results of the filters are not related. Can get very expensive
Simplifications possible: We can steer the results based on a few filters The resulting features are related
The squall line contains storm cells So far, we have only considered image-to-image
The resulting images are related. Use wavelets
Image
Filter 1
Filter 2 Pixel operation
Filter N
Result
lakshman@ou.edu 41
Wavelets
Wavelet analysis breaks up images smoother and smoother results on the right
(s0,s1) The lost “detail” information on the left
3 images of the size of the corresponding s Can not use any old filter
So that s1 is half the size of s0 Convolution filter has to be a wavelet function
Haar function (boxcar) Daubechies p-functions Many others
Fast Wavelet transformations exist Use wavelets only if you need this
decomposition Otherwise, use simple multi scale analysis
with lots of Gaussian filters. Gaussian filters have lots of nice properties. Plus everybody knows what they are!
S0 (NxN)
D1 (N/2 x N/2)3
S1 (N/2 x N/2)
D2 (N/4 x N/4)3
S2 (N/4 x N/4)
D3 (N/8 x N/8)3
S3 (N/8 x N/8)
lakshman@ou.edu 42
Texture
In convolution, we compute weighted average of the original image’s pixels
Can also do other operations inside neighborhoodTexture operatorsOperators based on relative ordering
lakshman@ou.edu 43
Texture operators
Texture is a good way to distinguish different objects Not just on data value, but on distribution of data values Lakshmanan, V., V. DeBrunner, and R. Rabin, 2000: Texture-
based segmentation of satellite weather imagery. Int'l Conference on Image Processing, Vancouver, 732-735.
2nd and 3rd order statistics Variance, standard deviation Homogeneity
Entropy Shannon’s measure of information content Bin the data values and form a histogram P is the fraction of pixels in each bin
i ii pp )log(
lakshman@ou.edu 44
Percent filters
Operators based on sorting Take the median value
Excellent smoothing filter Optimal when you have speck noise
Take the maximum value Effect of dilating the field spatially Minimum erodes the field spatially
Use percentiles 50% is the median (maximum noise reduction) 0% is the min (heavily noise dependent) 100% is the max Intermediate values trade off on noise reduction
lakshman@ou.edu 45
Dilation and Erosion filters
Using the second-highest or second-lowest value in the neighborhood is an excellent dilation/erosion filter.
Implemented this way in WDSSII.
origerode
dilate
lakshman@ou.edu 46
Filter Banks and Neural Networks
Classification using NN Use filter banks Convolution filters Texture operators Percentile operators Each pixel provides a
pattern May have to select pixels –
not use all of them Since a 1000x1000
image has 1 million patterns.
But is only data case
Image
Filter 1
Filter 2 Neural Network
Filter N
Result
lakshman@ou.edu 47
QCNN
Features computed in local neighborhoods of radar gates
Classified by NN Returns [0,1] Is it perfect?
If we use this directly as a mask, there will be holes in the result!
Lakshmanan, V., A. Fritz, T. Smith, K. Hondl, and G. J. Stumpf, 2005: An automated technique to quality control radar reflectivity data. J. Applied Meteorology, subm.
Effect ofPre-processing(don’t care)
NN randomerror
Effect of postprocessing
QC’ed
Original
lakshman@ou.edu 48
Actual QC technique
Effect ofPre-processing(don’t care)
NN randomerror
Effect of postprocessing
QC’ed
Original
lakshman@ou.edu 49
Images to Objects
So far, we have looked at ways of processing imagesResult is still image
Bunch of pixelsNeed to group pixels together into objects
Classify, track, etc. these objects if possible.
lakshman@ou.edu 50
Segmentation
From images to objects
lakshman@ou.edu 51
What is segmentation?
Finding groups of pixels that together comprise an object.
This is how we (humans) make sense of most images.
Very hard for a computer algorithm.Simple region-growing approachComplex vector segmentation approach
lakshman@ou.edu 52
Binary images
Region growing can be applied only to binary images.Apply to images that can be thresholdedPixels below (above) a threshold are not of
interest.Your image now has only 0’s (not
interested) and 1’s (interested)Threshold dependent on application
lakshman@ou.edu 53
Hysterisis
Alternative is to employ “hysterisis” You are interested in areas > 30 dBZ (T_i –
threshold of interest) But if you see a 30 dBZ pixel, you lose interest only
if pixel value < 20 dBZ (T_d – threshold of drop off) Less chance of too many disconnected regions. Employed in QCNN
Lakshmanan, V., A. Fritz, T. Smith, K. Hondl, and G. J. Stumpf, 2005: An automated technique to quality control radar reflectivity data. J. Applied Meteorology, subm.
Choose T_i and T_d by experiment.
lakshman@ou.edu 54
Region growing algorithm
Initialize: label, an image with all pixels = 0. curr_label = 0
Walk through image pixel-by-pixel. If label(x,y) = 0 and image(x,y) > T_i
Increment curr_label Perform region growing at x,y with curr_label
Region growing steps: Set label(x,y) = curr_label Check all 8 neighbors (i,j) of (x,y)
If image(i,j) > T_d, perform region growing at i,j Two pixels with the same label are part of the same region.
Can compute region properties based on all the pixels within a region.
lakshman@ou.edu 55
Vector segmentation
Region growing:Can lead to disconnected regions
Even with hysterisisNot hierarchical
Can not always use hierarchy of thresholds
Need a segmentation approach that:Can avoid disconnected regions Is hierarchical
lakshman@ou.edu 56
Watershed Segmentation
Watershed segmentation Sort pixels in ascending order
of magnitude; then, flood the plain:
Vincent, L., and Soille, P., Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations,PAMI(13), No. 6, June 1991, pp. 583-598.
Use saliency to form hierarchy
Watershed segmentation works very poorly in images with statistical noise (most meteorological images).
lakshman@ou.edu 57
Contiguity-enhanced segmentation
Explicitly trade-off contiguity of region with the idea that a region’s pixels ought to be similar Similarity of pixels based on a number of texture or
wavelet features Pixels are dissimilar if vector distance between
features is large Lakshmanan V. 2001: A Heirarchical, Multiscale
Texture Segmentation Algorithm for Real-World Scenes. Ph.D. thesis, U. Oklahoma, Norman, OK
The WDSS-II algorithm w2segmotionll implements this segmentation approach http://www.wdssii.org/
lakshman@ou.edu 58
K-Means Segmentation
Lakshmanan, V., R. Rabin, and V. DeBrunner 2003: Multiscale storm identification and forecast. J. Atm. Res., 367-380
lakshman@ou.edu 59
Hierarchical Segmentation
lakshman@ou.edu 60
Incorporating time
lakshman@ou.edu 61
Frames and sequences
A spatial grid at a point time is called a “frame”
A bunch of frames form a sequenceTerminology from video
Focus of most image processing researchUsed for security or monitoring applications
lakshman@ou.edu 62
Detecting change
To detect change between framesSimply difference the images pixel-by-pixel
Captures edges of moving objectsWorks well on video
Truck enters scene
Doesn’t work well when the objects themselves are changing
Such as storms
lakshman@ou.edu 63
Object association
One alternative is to associate objects Segment the frames Try to associate objects in one frame to next based
on: object similarity (use texture features) object overlap (fraction of pixels in common) Closeness of centroids
A linear programming problem Dixon, M. and Wiener, G., 1993: TITAN: Thunderstorm
Identification, Tracking, Analysis and Nowcasting: A Radar-Based Methodology. J. Atmos. Oceanic. Tech, 10, 6
lakshman@ou.edu 64
Object association (contd.)
Advantages of object association:Can extract underlying trends.Can advect each storm individually.
Disadvantages of object association:Segmentation results can change from
frame to frame.Splits/merges can not be handled easily.Accurate only for small stormsWorks only at one scale (not hierarchical)
lakshman@ou.edu 65
Optical flow
Another way is to treat the images are 2D fluid fields Find movement that minimizes some cost measure
to move from one frame to another. Cross-correlation
Implemented as w2imgmotion in WDSS-II Lagrangian motion
Turner B., Zawadzki, I. and Germann, U. 2004: Predictability of Precipitation from Continental Radar Images. Part III: Operational Nowcasting Implementation (MAPLE), J. Applied Meteorology, 43, 2, pp.231-248
Typically more accurate than object association. Can extrapolate motion vector to get forecasts. No way to extract trends for individual storms
lakshman@ou.edu 66
Hybrid technique
Object association gives us tracks/trends Optical flow gives us prediction accuracy A hybrid technique
Identify objects at current frame Match to image (not objects) in previous frame using cost
measure Extract trends from objects at current frame and images from
previous frames based on movement of object Lakshmanan, V., R. Rabin, and V. DeBrunner 2003:
Multiscale storm identification and forecast. J. Atm. Res., 367-380 Implemented as w2segmotion in WDSS-II
lakshman@ou.edu 67
Hybrid technique (contd.)
Advantage of hybrid techniqueCan use shape of identified storms to track
other fields Identify storms on radar imagesTrack satellite cloud-top temperature within
stormPossible because the exact shape of the
storm is knownThough it changes from frame to frame.So item of interest should not be at edges.
lakshman@ou.edu 68
GIS systems
Trends can include geographic propertiesJust grid the GIS data into the same
resolution at your spatial gridsThen, you can find the total population
affected by a storm, for example.Treat the GIS data as just another spatial
grid.
lakshman@ou.edu 69
Polygons
Gridding the GIS data is not always practical.County boundaries are polygons.Want to find things that happen within that
polygonMaximum hail within that polygon
How can we extract spatial properties within a earth-based polygon? Implemented as w2alarm in WDSS-II
lakshman@ou.edu 70
Polygon in/out tests
Put the polygon in the gridded reference
The polygon corners have to be converted to (x,y) locations.
We’ll assume the polygon is completely inside the grid.
Walk row-wise from the grid boundary to the
pixel of interest. count the number of times you
cross the sides of the polygon Odd: the pixel is inside the
polygon Even: the pixel is outside the
polygon Repeat column-wise
Has to be inside in both directions.
1
2
1
2
0
3
lakshman@ou.edu 71
Summary
We looked at how image processing plays an important role in most AI applications on spatial grids.
Filtering Finding edges Finding objects
Also at purely image-driven applications Tracking Extraction of spatial properties.
Gridding Filtering
Edge finding
Segmentation
Featureextraction
Classification(NN, etc.)
lakshman@ou.edu 72
Thank you!
http://www.cimms.ou.edu/~lakshman/
lakshman@ou.edu
http://www.wdssii.org/