Analyzing oriented patterns

COMPUTER VISION, GRAPHICS, AND IMAGE PROCESSING 37, 362-385 (1987)

Analyzing Oriented Patterns

MICHAEL KASS AND ANDREW WITKIN

Schhmberger Palo Alto Research, 3340 Hillview Avenue, Palo Alto, California 94304

Received September 3,1985; accepted November 4,1985

Oriented patterns, such as those produced by propagation, accretion, or deformation, are common in nature and therefore an important class for visual analysis. Our approach to understanding such patterns is to decompose them into two parts: the flow field, describing the direction of anisotropy; and the residual pattern obtained by describing the image in a coordinate system built from the flow field. We develop a method for the local estimation of anisotropy and a method for combining the estimates to construct a flow coordinate system. Several examples of the use of these methods are presented. These include the use of the flow coordinates to provide preferred directions for edge detection, detection of anomalies, fitting simple models to the straightened pattern, and detecting singularities in the flow field. 0 1987 Academic Press. Inc.

1. INTRODUCTION

A central focus in recent computational vision has been the decomposition of an intensity image into intrinsic images [l-3], representing such properties as depth, reflectance, and illuminance. These intrinsic properties are believed to be more meaningful than image intensity because they describe basic independent con- stituents of the image formation process. Thus, for example, in separating shape from illumination, we can recognize an invariance of shape regardless of changing illumination.

The advantages of decomposing what we see into its more-or-less independent parts extends beyond the image formation process to the shapes and patterns on which that process operates. For instance, decomposing a bent rod into a straight rod and a bending transformation reveals the similarity between a bent rod and one that has not been bent, or some other solid that has been bent the same way [4].

Just as we need to understand the image-forming process to decompose an image into intrinsic images, we need to understand the processes that generate patterns to decompose them into their intrinsic parts. But, while there is only one image- forming process, a staggering variety of processes shape and color the world around us. Our only hope of dealing with this complexity is to begin with some basic pattern classes that recur in nature and understand how to decompose and describe them.

One such class consists of oriented patterns: patterns which everywhere have a dominant local orientation. Orientation selective mechanisms have been widely studied by physiologists since Hubel and Wiesel’s [8] discovery of orientation selective cells in mammalian visual cortex (see [15] for a comprehensive example). There has also been considerable interest among psychologists in the perception of oriented patterns, particularly dot patterns [7]. Only recently have the computational issues involved received attention. Stevens [17] examined the grouping of tokens in Glass patterns based on orientation. While successful with Glass patterns, his methods were never extended to natural imagery. Zucker [20] investigated the

362 0734-189X/87 $3.00 Copyright 0 1987 by Academic Press, Inc. All rights of reproduction in any form reserved.

ANALYZING ORIENTED PATTERNS 363

estimation of orientation by combining the outputs of linear operators in what he believes to be a biologically realistic manner.

Rather than modeling the human visual system directly, we have focused on creating simple models of the types of physical processes which give rise to commonly occurring visual patterns. Guided by these models, we have attempted to derive perceptual computations which measure the important physically significant parameters of the models.

In this paper we consider oriented patterns such as those produced by propagation, accretion, or deformation. In order to recover the significant parameters, we must determine from the image (1) what is propagating, accreting or deforming, and (2) which way and how much. More precisely, we must estimate everywhere the direction and magnitude of anisotropy (which we will call the flow field,) and describe the residual pattern, independent of that field. Why this decomposition leads to simpler, more regular descriptions is best illustrated by example:

l A typical oriented pattern created by propagation is the streaked trail left by a paint brush dipped in variegated paint. The flow field describes the trajectory of the brush, while the residual pattern describes the distribution of paint on the brush.

l Accretion typically results in laminar structures, such as wood grain. Here, the flow field gives isochrones (the moving accretion boundary,) and the residual pattern describes the change in color or brightness of the accreting material over time.

l If an isotropic body is deformed, the flow field principally describes the bending and stretching it has undergone, while the residual pattern describes the undeformed body.

In all these cases, separate descriptions of the flow field and the residual pattern are appropriate because they describe different processes. The path of propagation for many physical processes is controlled by very different mechanisms than control the coloration of the trail left behind. Similarly, the mechanisms which control the shape of an accretion boundary are frequently unrelated to the processes controlling the color of the accreted material. Finally, the forces which deform a piece of material are often completely unrelated to the process which created the piece of material in the first place. By separately describing these processes, we can create descriptions of the whole which are often simpler than is possible without the separation, because each of the pieces may have different regularities.

In order to perform this separation, we propose a computation depicted in Fig. 1. The first step is to examine local image neighborhoods and compute their dominant local orientation. These orientations can then be linked together into integral curves which everywhere are aligned with the computed orientation. For the types of physical processes just described, these curves correspond to propagation paths, accretion boundaries, or formerly straight lines which have been deformed.

For a number of image interpretation tasks, having the raw orientations or the integral curves may be sufficient, but for others it is important to know how neighboring integral lines are related. A systematic way to do this is to combine integral curves into a coordinate system. One coordinate tells you which integral curve you are on, while the other tells you where you are along that curve.

364 KASS AND WITKIN

Local Integral Coordinate

Orientation - curves - system + FIOW Field

I Residual

w Straightener ) Panern

FIG. 1. Computation of flow field and residual pattern from an oriented image.

The local orientations, integral curves, and the coordinate system together completely specify the orientation structure or flow field of the image. If the original image intensities are viewed in the flow-field coordinate system, all the propagation paths, accretion boundaries, and deformed lines in the original are mapped to straight parallel lines. This simplified picture in which the changing orientation structure of the original has been removed, is what we mean by the residual pattern.

In Section 2 a simple computation is developed for determining local orientations. Based on a nonlinear combination of image gradients, the computation is fast and reliable, provided the local pattern does not have more than one dominant orientation. In addition to an estimate of the dominant local orientation, the computation provides a coherence measure which indicates the degree to which the local pattern is oriented. Several direct applications of these local measurements are provided including edge and anomaly detection.

In Section 3, we describe the combination of local orientation measurements into more global structures. Integral curves in the flow field are computed numerically, by following the estimated orientations from point to point. Direct applications of integral curves to image interpretation are provided. Then a coordinate system is constructed in which the integral curves are parameter lines. Transforming the image into these “flow coordinates” straightens the pattern, removing the effects of changing orientation. We present several examples of analysis and description of the flow field and the straightened pattern.

2. LOCAL ORIENTATION ANALYSIS

For intensity patterns created by anisotropic processes such as propagation, accretion, or deformation, variation in the flow direction is much slower than variation in the perpendicular direction. Anisotropy in such patterns will be evident in the local power spectrum. The high frequency energy will tend to cluster along the line in the Fourier domain perpendicular to the flow orientation.

A simple way to detect this clustering is to sum the energy in an appropriate region of the power spectrum and examine how the sum is affected by rotations. This can be done by examinin g the energy in the output of an appropriate orientation-selective linear filter. The orientation at which the energy is maximal can be expected to be perpendicular to the flow orientation.

Selection of the filter involves a number of tradeoffs. Very low spatial frequencies are affected more strongly by illumination effects than surface coloration, so they


FIG. 2. The power spectrum of the filter in equation 1 for o, = 20,.

are inappropriate for measuring textural anisotropy. Very high spatial frequencies are sensitive to noise and aliasing effects so they too are inappropriate. Hence some type of roughly bandpass filtering is required.

The orientation specificity of the filter is also important. If the filter is too orientation-specific then a large spatial neighborhood will be required in order to make a reliable measurement of the energy. Conversely, if the filter responds over a wide range of orientations then it will be difficult to localize the orientation very accurately. Thus there is a trade-off between angular and spatial resolution.

One reasonable choice for the frequency response of the filter is

F(r, f3) = [ee: - e-‘2a:]2Tircos(8). (1)

The filter is bandpass with passband determined by ui and ul. In our experience, ratios of the sigmas in the range of 2.0 to 10.0 work well. The orientation specificity or tuning curve is provided by the cosine dependence of the filter on 8. This appears to strike a reasonable balance between angular and spatial resolutions for the range of patterns we have examined. The filter’s power spectrum is shown in Fig. 2.

The cosine orientation tuning curve of the filter has some unusually good properties for computing the filter output at different orientations. The impulse response S(x, y) of the filter is

where

H(~, y) = o;2e-rz/a: _ a;2e-r2/20: I I

is an isotropic filter Let C = H * Z and let R,[S] denote a counterclockwise rotation of S by an angle 8. Then the convolution RJS]* Z is just the directional derivative of H * Z in the 8 direction. The directional derivative can easily be

366 KASS AND WITKIN

written in terms of the gradient so we have

R&]*I= (cose,sine)-vH*I. (2)

Thus a single convolution stices for all orientations. Since the filter S severely attenuates very low frequencies Re[S] * I can be safely

regarded as zero mean. Thus the variance in its output can be estimated by the expression

V(6) = W*(l?,[S]*I)2

where W(x, y) is a local weighting function with unit integral. We use Gaussian weighting functions W(x, y) because approximate Gaussian convolutions can be computed efficiently [6].

Using the gradient formulation of the filter output in Eq. (2) we can write the variance V(8) as

V(6) = w* [cos(8)C, -+- sin(6>cy]2. (3)

2.1. Interpretation of Filter Output

There remains the issue of interpreting V(8). Assume that there is only one axis of anisotropy. Then V(e) will have two extrema 7~ apart corresponding to that axis. Let V,( 0) = V( 8/2). Then V,(e) will have a single extremum in the interval 0 < 19 < 277. A computationally inexpensive way of estimating the position of this extremum is to consider V, as a distribution and compute its mean. Since V,(0) is periodic, it should be considered as a distribution on the unit circle. Hence its mean is the vector integral (cu, j3) = j~“V,(Q( cm 8, sin 8) dt3. The angle tan-l( p/a) is an estimate of the angle of the peak in V, and hence twice the angle of the peak in V. Thus the angle C$ of greatest variance can be written

= tan-’ (4)

= tan-1 ’ [V(8)sin(20) d8

,/ ?‘( B)cos(26) d6’ 0 ii

2.

These integrals are evaluated in Appendix A to show that the angle of anisotropy $I can be written

+ = tar-’ l w*2c,c,

w*(cx’- c,z) Ii 2 (5)


which directly yields a simple algorithm for computing $. Note that this computation is based on the assumption that there is a single axis of anisotropy. If more than one dominant orientation is present the result will not be meaningful.

2.2. Combining Gradient Orient&ions Notice that the right-hand side of Eq. (5) can be regarded as the orientation of a

locally weighted sum of vectors of the form J(x, y) = (C,” - Cy’, 2C,C,). These vectors are related in a simple way to the gradient vectors G(x, y) = (C,, C,). The magnitude of J(x, y) is just the square of the magnitude of G(x, y), and the angle between J(x, y) and the x axis is twice the angle between G(x, y) and the x axis. This follows easily from the observation that (C, + C,i)’ = C,’ - Cy’ + 2C,C,i.

One might be tempted to believe that smoothing the gradient vectors G(x, y) would be nearly as good a measure of anisotropy as smoothing the rotated squared gradient vectors J(x, JJ). This is emphatically not the case. Consider an intensity ridge such as I( x, y) = exp( -x2). The gradient vectors on the left half-plane all point to the right and the gradient vectors on the right half-plane all point to the left. Adding them together results in cancellation. By contrast, if they are first rotated to form the J vectors, they reinforce. The types of oriented patterns we are concerned with often have nearly symmetric distribution of gradient directions around the axis of anisotropy. In such patterns, if the gradients are added together directly, the cancellation is so severe that the result often has little relation to the direction of anisotropy. Thus the difference between rotating the gradient vectors or leaving them unchanged is often the difference between being able or unable to detect the anisotropy. Note also that smoothing the image first and then computing the gradients is exactly the same as computing the gradients and then smoothing. It will not avoid the difficulties of cancellation.

2.3. Coherence In addition to finding the direction of anisotropy, it is important to determine

how strong an anisotropy there is. If the orientation of the local J vectors are nearly uniformly distributed between 0 and 2a, then the orientation $I of slight anisotropy is not very meaningful. Conversely, if all the J vectors are pointing the same way then the indication of anisotropy is quite strong and $J is very meaningful. A simple way of measuring the strength of the peak in the distribution of J vectors is to look at the ratio x(x, y) = 1 IV* JI/( W * (Jj) which we will call the coherence of the flow pattern. If the J vectors are close to uniformly distributed, then the ratio will be nearly zero. If the J vectors all point the same way, the ratio will be one. In between, the ratio will increase as the peak gets narrower.

2.4. Summaty The computation of the flow direction and local coherence can be summarized as

follows. First the image 1(x, y) is convolved with the isotropic portion H(x, y) of the filter response. The result C(x, y) is then differentiated (by finite differences) to form C,(x, y) and C,(x, y). The resulting vectors (C,(x, y), C’(x, y)) are rotated by amputh Jdx, Y) = 2C,(x, Y)C,(X, VI ad JAx, VI = C,‘<x, Y) - C,‘<x, Y). The gradient magnitude J3(x, y) = [C:(x, y) + C,‘(x, y)]‘/* also has to be computed in order to measure the coherence. The next step is to convolve J,(x, y),

368 KASS AND WITKIN

FIG. 3. An image of wood grain with its flow field. Estimated flow directions are given by the red needles. The length of each needle encodes coherence. Notice that coherence is low within the knot at the center.

.&(x, y), and .&(x, y) with the weighting function W(x, y) to obtain Ji*(x, y). J**(x, y), and J3*(x, y). The angle +(x, y) of anisotropy and the coherence x(x, y) can then be computed from the formulas

and cpb, Y) = tan-‘( J,*h Y)/J,*(x, ~)),‘2

x(x, Y) = (J,*(x, Y)’ + J,*k ~)~)~‘~/Jj*(x, Y).

An example of this computation applied to a picture of a piece of wood is shown in Fig. 3. The flow direction +(x, y) + s/2 is displayed by the orientation of small needles superimposed on the image. The lengths of the needles are proportional to the coherence x(x, y). Note that the pattern is strongly oriented except near the knot in the middle.

2.5. Relation to Prior Work The flow computation just described bears an interesting relation to an early

proposal of Marr that information about local distributions of oriented edge elements be included in the primal sketch [lo]. If this proposal is combined with his later work with Hildreth on edge detection [ll] it results in a special case of the above computation. Marr and Hildreth define edges as zero-crossings in the Laplacian of the Gaussian smoothed image. The natural combination of Marr’s proposal with this definition of edge elements calls for examining the local density of zero crossings as a function of orientation. For stationary zero-mean Gaussian processes the square of the oriented zero-crossing density is approximately V( 8) (see Appendix B). Thus in the special case where the point spread function of the t&r is


S = (a/ax)v*exp(-(x2 + y2)/2a2) our computation can be viewed as computing the direction of minimal edge density in the Marr-Hildreth theory.

Zucker’s work on flow [20] is also related to a special case of the above computation. For biological reasons, he prefers to use oriented second derivatives of Gaussians as the initial filters. These have F(r, 0) = r*exp( - r2/2)cos2(r3). Instead of looking at the variance of the filter outputs as the orientation is changed, he combines the outputs in a biologically motivated relaxation process. Although quite different in detail, the computation described here has much in common with his technique.

2.6. A Coordinate Frame for Edge Detection

Oriented measurements have been widely used in edge detection. For example, Marr and Poggio [13] employed directional second derivative operators, whose zero-crossings were taken to denote rapid intensity changes. Due to the difficulty in selecting an orientation, Marr and Hildreth [ll] later abandoned this scheme, in favor of zero-crossings of the Laplacian, a nondirectional operator.

The flow field provides two meaningful directions-along and across the direction of flow-in which to look for edges within an oriented pattern. Zero-crossings of the second directional derivative in the $J direction (against the grain) should highlight edges that contribute to defining the flow field, while zero-crossings in the second derivative perpendicular to $I (with the grain) should highlight anomalous elements or terminations. The sum of these two derivatives is the Laplacian.

FIG. 4. Using flow coordinates for edge detection. Upper right: 2nd directional derivative along the flow direction. Upper left: 2nd directional derivative across the flow direction. The first of these suppresses the oriented structure, the second highlights it. Lower left: the sum of the directional derivatives is the Laplacian.

370 KASS AND WITKIN

FIG. 5. “Finding the needle in the haystack.” In this straw pattern, directional derivatives across the flow direction show elements aligned with the pattern (upper right.) Those along the Row direction show anomalous elements (lower right.) The Laplacian (lower left) shows both.

The two directional derivatives of the wood grain image are shown, with the Laplacian, in Fig. 4. Indeed, the derivative against the grain captures all the elements comprising the grain pattern, while the derivative with the grain does not appear meaningful. The Laplacian confuses these very different signals by adding them together.

The derivative along the grain can also be meaningful, where anomalous elements are present. In addition to being perceptually salient, such anomalies are often physically significant, with origins such as cracks, intrusions, or occlusions, that are distinct from those of the main pattern. In man-made structures, anomalies are often important because they indicate some variety of flaw.

Figure 5 shows a pattern of aligned elements (straw) with some anomalous elements. The directional second derivatives along and across the flow direction are shown, together with their sum which is just the Laplacian. Differentiating along the grain highlights anomalous elements, attenuating the rest thus finding the 3xedles” in the haystack. Differentiating across the grain suppresses the anomalies. The Laplacian shows both. A related demonstration is shown in Fig. 6, in which the anomalous elements have actually been removed by directional median filtering in the flow direction.

2.7. Orientation Edges

Where one oriented pattern abuts or occludes another, an abrupt change in orientation frequently occurs at the edge. The herringbone fabric shown in Fig. 7

371

FIG. 6. Right: the straw picture from Fig. 5. Left: the anomalous elements have been removed by directional median filtering in the flow direction. (This hack was suggested by Richard Szeliski.)

illustrates this effect. Detecting such edges by ordinary means is problematic, particularly where the mean intensities of the adjoining regions are similar. Marr [lo] introduced a method called “theta aggregation” which detected boundaries between oriented regions by grouping edge fragments symbolically according to their orientation.

Orientation boundaries may be detected more directly by examining the structure of the orientation field. Figure 8 shows the orientation field for the image of Fig. 7, encoded by grey level. This representation transforms the orientation boundaries into intensity edges which may be easily detected by conventional edge detectors.

FIG. 7. Herringbone fabric from Brodatz.

372 KASS AND WITKIN

FIG. 8. Orientation image of Fig. 7. The inset key shows the mapping from orientation to intensity.

Note that in this instance the transformed image is itself an oriented pattern. Applying orientation analysis again would reveal the pattern’s hierarchic organi- zation consisting of stripes made of stripes.

3. GLOBAL ORIENTATION ANALYSIS

3. I. Integral Curves The orientation field provides local orientation measurements, but the underlying

oriented structures are typically spatially extended in the flow direction-for instance, the grain lines and cracks in Fig. 3 or the pieces of straw in Fig. 5. To simplify the description or analysis of these extended structures we first need a more global description of the flow field. One such description is provided by linking the local orientation measurements into integral curves or “flow lines.”

An integral curve is a curve whose tangents everywhere coincide with the flow direction. Specifically, (x(u), y(u)) is an integral curve if everywhere

(dx/du, dy/du) . (sin+, -coscp) = 0. f6)

An integral curve through any starting point may be obtained by taking a small step in the flow direction, getting a new direction from the flow field, taking another step, and continuing. Figure 9 shows a family of integral curves superimposed on the wood image of Fig. 3. Note that the curves closely follow the grain lines.

In the previous section, we saw that local orientation measurements provide useful constraints on edge detection because the measured orientations tend to coincide with the tangents to edges and similar features. Integral curves provide a powerful global version of this constraint. Where edges are closely tracked by


FIG. 9. Integral curves superimposed on the wood image of Fig. 3. Although only a few curves are displayed here, an integral curve can be computed through each non-singular point in the image. Note that the curves closely follow the grain lines.

integral curves, as in Fig. 9, the search for edges may be restricted to a one-parameter family of integral curves, which can be precomputed. By combining information along integral curves-averaging a local edge measure, for instance-much higher signal-to-noise ratios are possible than using the local measurements alone. Since the integration of information closely follows the edge, this gain in signal-to-noise ratio entails little loss in spatial resolution. In effect, integral curves can be used to create large-scale curved operators that conform to the underlying pattern.

FIG. 10. Below: Speech spectrogram of the digit “nine” (from the Texas Instruments digit data base). Above: Spectrogram with automatically selected integral curves superimposed. Note how closely they track the formants.

374 KASS AND WITKIN

This approach is well suited to the detection and tracking of formants in speech spectrograms. Formants appear as bands of unusuahy high energy. To detect them using flow analysis we simply look for integral curves whose average energy is greater than that of their neighbors. We applied this technique to several utterances from the Texas Instruments digit data base [12]. Results for an utterance of the digit “nine” are shown in Fig. 10. The automatically selected integral curves appear as red lines-note that they track the formants closely. No attempt was made here to identify formant onsets and terminations, but an analysis of energy changes along integral curves may be applicable to the problem.

3.2. Flow Coordinates Having computed the family of integral curves from an orientation field, any

point in the image can be specified by stating which integral curve it lies on, and where it lies on that curve. Hence the integral curves can be used to build a coordinate system for the image, (u, u), with u = U(X, y) and u = u(x, y), that everywhere satisfies

Vu. (sin+, -cos(p) = 0. (7)

In such a system, the parameter u indicates which integral curve the point is on, while u indicates its location on the curve. The parameter lines of constant u are the integral curves. Viewed in this system, structures that follow integral curves become parallel straight lines.

Equation (7) does not determine a unique coordinate system, so additional constraints are required. In general, we have to decide whether to permit stretch along the integral curves, shear from one integral curve to the next, or a combination of the two. Ideally, the choice should be based on a physical model. For example, a bending phone book shears freely from page to page, but stretch in the plane of a page is negligible.

To forbid shear across integral curves, lines of constant u are chosen orthogonal to those of constant U:

vu - (cm +, sin+) = 0. (8)

This constraint allows features aligned in the flow direction to stretch and compress in the deformation from image to Ilow coordinate, while forbidding shear. Even with Eq. (S), an additional constraint is needed, because we are free to specify arbitrary scaling functions for the u and u axes. In the spirit of Eq. (g), we want to choose these functions to avoid the introduction of spurious compression or dilation. Although difIicuIt to do globally (one might minimize total strain), we will usually want to construct a fairly local coordinate frame around some point of interest. For this purpose, it sttfli~s to take that point as the origin, scaling the axes u = 0 and u = 0 to preserve arc length along them. Alternatively, u(x, y) may be chosen to preserve arc length on the integral curves. In that case, we must provide a function that determines the shear from one curve to the next, as well as a scaling function for the u axis.


FIG. 11. A flow coordinate grid obtained for the image of Fig. 3.

3.3. Residual Pattern

We have seen that the orientation field is an abstraction from the anisotropic pattern that defines it. A single spiral field, for example, could arise from a pattern composed of bands, irregular streaks, dot pairs, etc. The orientation field is useful because it describes the changing direction of anisotropy independent of the underlying pattern elements that define it.

Conversely, a description of the image in flow coordinates reveals the underlying pattern independent of the changing direction of anisotropy. Such a description would make it possible to recognize, for example, that two very different orientation fields are defined by the same kind of bands or streaks. Intuitively, the orientation field describes the way the pattern is bent, while viewing the image in flow coordinates straightens the pattern out.

Figure 11 shows a flow coordinate grid for the wood-grain image from Fig. 3. The grid lines were computed by taking steps of fixed length in the direction (cos $B, sin +) or (-sin+, cos +) for lines across or along the direction of flow using bilinear interpolation on the orientation field. Since Cp is always computed between 0 and 7~, we must assume that there are no spurious discontinuities in direction to track smoothly.

For many purposes it is unnecessary to compute the deformed image explicitly, but doing so vividly illustrates how flow coordinates can simplify the pattern. Figure 12 shows the deformation from image coordinates to flow coordinates in several stages. As the grain lines straighten, the knot shrinks and finally vanishes. The deformed images were anti-aliased using texture-map techniques [Ml. The fully deformed image shows, to a reasonable approximation, what the grain would have looked like had it not been subjected to the deforming influence of the knot.

376 KASS AND WITKIN

FIG. 12. Deformation, in stages, from image coordinates to flow coordinates: (a) the original image: (b) and (c) two intermediate stages, in which the grain’s curvature has diminished, and the knot compressed; and (d) the image as seen in Row coordinates. In (d), the grain lines are straight and the knot has vanished, showing approximately what the grain would have looked like had it not been deformed by the intrusion of the knot.

Thus far, we have separated the image into a flow field and a pattern derived by viewing the image in low coordinates. We argued earlier that the advantage of this decomposition, like the decomposition of an image into intrinsic images, is that the components are liable to be simpler and more closely tied to independent parts of the pattern-generating process than is the original image. To exploit the decornposi- tion, we need ways of analyzing, describing, and comparing both the flow field and the straightened pattern. These are difficult problems. In the remainder of this section, we present several illustrations of the utility of the decomposition.

3.4. Singularities

We have shown several ways in which viewing an oriented pattern in flow coordinates facilitates analysis and description of the pattern. It remains to describe

ANALYZING ORIENTED PA’MERNS 377

FIG. 12.-Continued

and analyze the flow field itself. The topology of a flow field, as of any vector field, is determined by the structure of its singularities, those points at which the field vanishes. Identifying and describing singularities is therefore basic to describing the flow field. The singularities provide the framework around which metric properties, such as curvature, may be described.

The perceptual salience of singularities is demonstrated in the simple patterns devised by Glass [7]. These patterns are made by subjecting random dot fields to small rotations, translations, and scalings, and superimposing the transformed field on the original. The systematic pairings of the original dots with their transformed images give rise to a powerful impression of flow, with singularities apparent at centers of rotation and scaling. Figure 13 shows the integral curves we obtained by orientation analysis on such a pattern.

A robust basis for identifying singularities is the index or winding number [16]. Suppose we follow a closed curve on a vector field. As we traverse the circuit, the vector rotates continuously, returning to its original orientation when the circuit is completed. The index or winding number of the curve is the number of revolutions

378 KASS AND WITKIN

FIG. 13. (a) A spiral Glass pattern and (b) its flow lines. The pattern is perceptually dominated by the singularity at the center. Since the flow field vanishes at a singularity, the flow lines obtained by integrating the flow field tend to become ill-behaved as they approach one.

made by the vector in traversing the curve. The index of a point is the index of a small circle as we shrink it around the point:

ind(x, r) = CpO&p”$ +(x + ECOS~, y + &sine) de

= liio & /2n o (-sin8,cosf?)

XV+(X + ECOSI~, y + ~sinB) de.

To compute the winding number numerically, we divide the flow field into suitably small rectagles, summing the rotation of C#I around each rectangle. As in computing the flow lines, we assume that C/J has no spurious discontinuities. Where the retit-is

319

FIG. 14. A fingerprint and its flow coordinate grid. The two red circles represent the major singularities, which were detected by measuring the flow field’s winding number.

non-zero, the rectangle surrounds a singularity. Figure 14 shows an example of the detection of singularities using winding number, for a fingerprint. We are currently working on classifying the singularities, and using them to describe the topology of the flow field.

3.5. The Constant-ProJie Model A simple class of oriented patterns are those with constant brightness or color

along the flow lines, as when a l-dimensional profile is swept along a curved trajectory with dilation or compression. Among patterns well approximated by this model are some arising from propagation, such as the pattern left by a paintbrush dipped in variegated paint, simple laminar structures such as wood grain, and patterns that have been subjected to such extreme stretch or anisotropic diffusion that variation in the flow direction essentially vanishes.

Such patterns may be represented by a l-dimensional intensity profile, using the flow coordinates to sweep the profile into a 2-dimensional pattern. To fit the model, we compute the intensity protile I(u) as the average intensity along lines of constant u (where u is the flow coordinate against the direction of flow). To reconstruct the pattern, we extend this constant value with interpolation along lines of constant u in the original image coordinates. The difference between the reconstructed and original images provides a distortion measure. Where the constant-intensity model is appropriate, we obtain a marvelously compact and useful representation of the pattern. Of course, this model describes only the simplest oriented patterns.

The construction of this representation is illustrated in Figs. 15 and 16. Figure 15a shows a seismic section. I(u) was computed along the flow lines in the area covered by a flow coordinate grid (Fig. 16), and the average value extended back along the flow lines to produce the reconstructed image shown in Fig. 15b. Although important variations along the flow direction are lost, the reconstructed image preserves major orientation features of the original.

3.6. Statistical Pattern Models A variety of more intricate visual textures may be described succinctly by

combining the flow field with a few statistical parameters. This approach is a hybrid of structural and statistical texture representations. The structure of the pattern is represented in the flow field, while its statistics are represented by a small number of parameters.

380 KASS AND WITKIN

FIG. 15. (a) A seismic section. (b) A constant-intensity reconstruction. A l-dimensional intensity profile was first produced by taking the average intensity along each flow line of a coordinate grid (see Fig. 16.) The image was then reconstructed by extending these constant values back along the flow lines. Although important variations along the flow direction are lost, major orientation features are preserved.

FIG. 16. The image of Fig. 15, with a flow coordinate grid superimposed


FIG. 17. Fingerprint of Fig. 14 with square cut out.

An example of a pattern amenable to this approach is the fingerprint in Fig. 14. In ordinary Cartesian coordinates the pattern is very complex. It autocorrelation is highly non-stationary and defies simple characterization. Many standard approaches to texture representations, such as co-occurrence matrices or second-order statistics, depend on stationarity and would therefore do a very poor job at characterizing this pattern except where the ridges are straight.

.In flow coordinates, however, the fingerprint becomes extremely regular. It consists of parallel lines of approximately constant width and separation which split and merge in a statistically regular way as a function of the divergence and convergence of the flow lines. We have developed a simple algorithm [19] based on these regularities which is able to synthesize fingerprints and related patterns from a flow field and a handful of global parameters.

As an illustration of the power of this type of representation, we removed a piece from the original fingerprint and then grew it back using flow-field analysis and the synthesis algorithm. The original fingerprint with a square cut out is shown in Fig. 17. Using the flow-field analysis, we interpolated orientations in the missing square. The result is a flow field which shows the same type of singularity as the original pattern because, given the boundary conditions, it is the only possibility without multiple singularities. Figure 18 shows the result of using these orientations to drive the synthesis algorithm and fill in the missing square. The interpolated pattern differs in detail from the original, yet it is an equally plausible way of completing the square. Figure 19 shows the real and synthetic patches blown up side-by-side for comparison.

4. CONCLUSION

We addressed the problem of analyzing oriented patterns by decomposing them into a flow field, describing the direction of anisotropy, and describing the pattern independent of changing flow direction.

382 KASS AND WITKIN

FIG. 18. (a) Natural fingerprint; (b) fingerprint filled in with syuthe ,tiC data.


FIG. 20. Above: picture of burlap fabric from Brodatz. Below: coordinate system obtained by multi-orientation local analysis.

A specific computation for estimating the flow direction was proposed. The computation can be viewed as (a) finding the direction of maximal variance in the output of a linear filter, (b) combining gradient directions locally, or (c) finding the direction of maximal edge density. The computation has been applied to a number of natural and man-made patterns with consistent success.

The flow field was then used to form a coordinate system in which to view the pattern. Two orthogonal families of curves-along and across the direction of flow -form the coordinate system’s parameter lines. Viewing the pattern in these flow coordinates amounts to deforming the pattern so that the flow lines become parallel straight lines. This deformation produces a pattern that is simpler, more regular, and therefore more amenable to analysis and description than the original one.

Several examples of the use of this decomposition were presented. These included the use of the flow coordinates to provide preferred directions for edge detection, detection of anomalies, fitting simple models to the straightened pattern, and detecting singularities in the flow field.

The simple orientation analysis of Section breaks down for patterns with several dominant local orientations such as the burlap fabric shown in the upper half of Fig. 20. Our ongoing work includes research into more sophisticated computations capable of dealing with the added complexity of such patterns. The lower half of Fig. 20 shows a coordinate system derived from the burlap pattern using an algorithm still under development which checks for the possibility of several peaks in V( 0). Note that having more than one local orientation provides additional useful information about 34limensional structure of a scene and deformations which it may have undergone.

In addition to extending the local orientation analysis to deal with multiple orientations, we continue to look for richer descriptions of the structure of flow fields as well as better methods of modeling and resynthesizing straightened patterns.

384 KASS AND WITKIN

APPENDIX A: DERIVATION OF E&UATION 5

The integrals in Eq. (4) can be evaluated fairly easily by expansion using Eq- (3) for V. The numerator of tan(2#) can be written

~*V(B)sin(20) d6 = W* [2C.j!jo”sin(s)cos3(t9) dt’

+ 4C,C, J

“,in’( B)cos2( 0) d6

+2C,$in3(t9cos(t9) de].

The first and third integrals are zero and the middle integral is 7r/8. Hence

J ‘“V( 8)sin(28) d6 = (a/2)W * C,C,. 0

The denominator of tan(2+) similarly is

j7( ~)cos(2l?) d6’ = W * [ 2C;iT2 cos4( 6) - cos2( 19) dB

+ 4CxC$in(4fI) de

+ 2Cy’ oT2 cos2( t9)sin2( r3) - sin2( t!I) dB] . J

Here the first integral is a/4, the second integral is zero and the third is --r/4. Hence the above expression can be simplified to

/?‘(6)cos(26) d6 = (n/4)W*(C,2 - C;). 0

Substituting for both the numerator and denominator of tan(2+) in Eq. (4) yields

APPENDIX B: EDGE DENSITY

There is a close connection between the flow computation described in Section 2 and the density of edges in the Marr-Hildreth theory of edge detection. Suppose the image 1(x, y) is a stationary zero-mean Gaussian process and let D(x, y) be the Laplacian of a Gaussian. The density A( 8) of zero-crossings in I?,& X, y) = D * Re[ I] along the x axis can be approximated by [9, 141

J ,‘K( w) dw

/ ( ) K w do


where K(w) is the power spectrum of a slice through B, along the x axis. Using Papoulis’ Fourier transform conventions, the numerator can be converted to a spatial integral through the identity

/ B,‘dx = &ozK(o) dw.

Similarly, the denominator can be converted with the identity

/B*dx = &C(w) dw.

If the integrals are computed locally with the windowing function W, then we have the following estimate for zero crossing density:

1 W*B,’ Aye) = -~

r* W*B*’

If W is radially symmetric, W * B * will not depend on B so the maximum zero-crossing density will occur at the maximum of W * B,‘. By assumption, the mean of the process is zero, so W* B,’ is the variance of D, * &[I]. Thus, for stationary zero-mean Gaussian processes, if S = D,, A*(S) = V(O).

REFERENCES 1. B. K. P. Horn, Understanding image intensities, Artq. Inrell. 8, 1977, 201-231. 2. H. Barrow and J. M. Tenenbaum, Recovering intrinsic scene characteristics from images, in

Computer Vision Sysremr. (A. Hanson and E. Riseman, Eds.), Academic Press, New York, 1978. 3. D. Marr, Vision, Freeman, San Francisco, 1982. 4. A. Barr, Global and local deformation of solid primitives. Comput. Graphics 18, 1984, 21-30. 5. P. Brodatz, Textures, Dover, New York, 1966. 6. P. Burt, Fast, Hierarchical Correlations with Gaussian-like Kernels, Dept. Computer Science Center

TR 860, Univ. of Maryland, Jan. 1980. 7. L. Glass, Moire effect from random dots, Nature 243, 1969, 578-680. 8. D. H. Hubel and T. N. Wiesel, Receptive fields, binocular interaction and functional architecture in

the cat’s visual cortex. J. Physiol., London 1(X,1962, 106154. 9. A. Papoulis, Probability, Random Variables and Stochastic Processes, McGraw-Hill, New York,

1965. 10. D. Marr, Early processing of visual information, Proc. R. Sot. London B 275,1976, 484-519. 11. D. Marr and E. Hildretb, Theory of edge detection, Proc. R. Sot. London B 207, 1980, 187-217. 12. G. Leonard, A database for speaker-independent digit recognition, in Proceedings, 1984 IEEE Int.

Con/. Acoustics, Speech Signal Process, San Diego, Mar. 1984. 13. D. Marr and T. Poggio, A computational theory of human stereo vision. Proc. R. Sot. London 2&l,

1979, 301-328. 14. S. 0. Rice, Mathematical analysis of random noise, Bell Syst. Tech. J. 23-24, 1944-1945. 15. P. H. Schiller, B. L. Finlay, and S. F. Vohnan, Quantitative studies of single-cell properties in

monkey striate cortex. II. Orientation specificity and ocular dominance. J. Neurophysiol. 39, 1976,1320-1333.

16. M. Spivak, DQ$erential Geometry, Publish or Perish, Berkeley, Cahf., 1979. 17. K. Stevens, Computation of locally parallel structure. Biol. Cybern. 29, 1978, 29-26. 18. L. Williams, Pyramidal parametrics, Compur. Graphics 17, No. 3, 1983. 19. A. Witkin and M. Kass, Synthesizing oriented patterns, in preparation. 20. S. Zucker, Computational and psychophysical experiments in grouping, Human and Machine Vision

(J. Beck, B. Hope, and A. Rosenfeld, Eds.), Academic Press, 1983.

Analyzing oriented patterns

Documents

Transcript of Analyzing oriented patterns