自動光學檢測(AOI) -...
Transcript of 自動光學檢測(AOI) -...
-
(AOI)
Textbook R. Jain, R. Kasturi and B. G. Schunck, Machine Vision, McGraw-Hill Inc., 1995
References 1. M. Sonka, V. Hlavac & R. Boyle, Image Processing,
Analysis and Machine Vision, Brooks/Cole Pub., 1999.
2. E. R. Davies, Machine Vision- Theory, Algorithms,
Practicalities, Morgan Kaufmann Pub., 2005.
3. R. C. Gonzalez and R. E. Woods, Digital Image
Processing, 2nd Ed., Prentice Hall, 2002
4. L. G. Shapiro and G. C. Stockman, Computer Vision,
Prentice Hall, 2001
5. Christopher M. Bishop, Neural Networks for Pattern
Recognition, Oxford University Press, 1995
1
-
Topics
1.
2. Illumination and shading
3. morphological
operations
4. SMT
5. mura BGA
6. LCD
7. SMTscanner
8.
9.
10. 2.5
11.
2
-
Chap. 1: Introduction (case study and basic notions)
Chap. 2: Operations on Binary images
PCB Circuit Defects Detection
Chap. 3: Segmentation and Region Representation
Chap. 4: Filtering and Morphology operations on gray
images
Chap. 5: Edge detection
Chap. 6: Contour representations and description
Solder Joints Classification
Chap. 7: Colour
Chap. 8: Depth and 2.5D profilomotry 2.5
Chap. 9: Pattern classification and Object Recognition
Chap. 10: Optics, camera, grabber and calibration
* The above Chapter numbers are the numbers of lecture
note, not the textbook.
3
-
AOI and Image processing
There are three typical applications of image
processing: Medical image, Video, and AOI
Image
quality
Computation
time
Precision Reliability/
robustness
Medical
image
Poor Hours-days Extremely
high
Extremely
high
Video Poor 30sec/frame Low Low-medium
AOI Best 22sec/4,000
solders
High Extremely
high
Medical Image:
4
-
Industrial Images
5
-
6
-
Chap. 1
1.1A Touch on AOI (by case studies)
This section begins with basic definitions and notions
of CCD image
Followed by brief introduction to industrial application
of AOI
1.1.1Basic definitions and notions
(1) Image acquisition system
Single layer color CCD is not popular in industrial
7
-
applications due to (a)inferior resolution, (b)distortion
due to interpolation.
Three layer color CCD is still too expensive.
B/W CCD still dominant industrial applications. Hence,
unless stated explicitly, images addressed in this course
are gray images (B/W) and binary images (special
form of B/W images).
Industrial CCD is different from consumer CCD
mainly in: (a)low S/N ratio, (b)low distortion,
(c)external trigger, (d)faster image capturing and
transmission rate.
Machine vision consists of three key technology:
(1)Lighting design to highlight desirable features and
to suppress unwanted details, (2)image processing to
extract characteristic features contained in an image,
and (3)pattern classification to categorize objects in the
image.
(2) Coordinate and matrix associated with an image
CCD camera consists of an array of pixels (such as 8
-
1024*768), each pixel records the light intensity using
0~255 (28) gray level, with 0 corresponds to black
and 255 corresponds to white.
Hence it is natural to treat a gray image as a matrix:
9
-
In the above, the ith row and jth column element of a
matrix corresponds to the pixel at the coordinate (i, j),
and the value of the element corresponds to the
intensity of the (i, j) pixel.
Even when a linear camera (line scan camera) is used,
the image can still be transmitted and stored as an array,
hence an image can be treated as a matrix.
(3) Global and local coordinate in AOI measurement
The length/pixel resolution of an image depends on the
amplification factor of the lenses before the CCD chip.
Given the amplification factor and the size a CCD chip,
one can calculate the length/pixel resolution.
Given the length/pixel resolution and the number of
pixels corresponds to a line in an image, one can decide
the true length of this line.
The global coordinate of an AOI system is usually
fixed by linear encoders. The origin of the local
coordinate nof the CCD can be calibrated against the
global coordinate when the system is set up. 10
-
Hence a machine vision system can be used for:
(1)measuring the length and size of an object, (2)detect
defects of an inspected component, (3)classify the
defects for in process quality control.
1.1.2Illustrative examples of AOI
Three illustrative examples are presented to show some
of most prevailing AOI problems currently encountered
in local industry.
[A] LCD inspection Under proper lighting condition, the image of circuit
and fiducial mark on LCD panel generally has fairly
good contrast, hence binary image can easily be
obtained. This example illustrates some basic steps in
the inspection of binary mages.
STEP 1: fiducial mark and true position of LCD When a board (PCB or LCD panel ) is loaded, there is
11
-
usually translation and orientation dislocation. To know
these position and orientation error, fiducial marks are
first examined and analyzed to give true position and
orientation (w. r. t. the global coordinate).
Normally the fiducial mark occurs within a
predetermined area with extremely high contrast
against its background.
To determine the true position of a LCD panel,
(1)Move the CCD camera to the nominal position
(expected to be the center of a fiducial mark) and take
out the image of the fiducial mark using a window.
Window size depends on the
precision of the loading
mechanism.
12
-
(2)Use a threshold to convert the image into a binary
image.
(3)Find out the centroid of the first fiducial mark.
(4) Find out the centroid of the second fiducial mark.
(5)Determine position error and orientation error of
the LCD panel using center positions of two fiducial
marks.
This example is an extremely easy task in AOI because:
(a) the value of the threshold can easily be determined 13
-
since the fiducial mark is designed and manufactured to
give very good contrast in any lighting condition.
Beside, (2)the image processing is extremely simple
since the fiducial mark is carefully designed to a
regular shape.
STEP 2: measurement of corner length After some examinations, one may one to find out the
length of the corner on the LCD panel.
In this case we need to change lighting condition
and/or proceed some image pre-processing on the
original LCD image to result in good contrast between
the corner, the LCD panel and the background.
The we use edge detector (standard image processing
algorithms) to detect three edges on the corner: 14
-
Then the three edges (lines) are curve fitted to obtain
their mathematical equations.
Intersection points of three lines are solved from the
equations to give the length of the corner.
In this case, AOI algorithms are basically operated on
gray images, and pre-processing on the gray image
might be required to improve the contrast before edge
detection.
STEP 3: short circuit and open circuit There are printed circuits on IC, PCB and LCD panel.
15
-
For the purposes of either Qc or repairing, it is often
required to pin point the location of short circuit and
open circuit.
In such applications it is often the case that images with
good contrast are attainable, hence we can easily apply
a suitable threshold to turn the gray image into a binary
image as follows.
If the width of circuits is known, the binary close
operation can be applied to the B/W image in order to
erase all circuit from the image and leaves only black
holes as follows.
16
-
To detect the white hole (open circuit), we need a
golden template shown below
Comparing the golden template with the test image
given above, the difference is
17
-
Remove the defects which have been identified as
black holes, it results in white holes.
This case uses binary morphological operations and
golden template. The use of golden template
assumes that the true position of the LCD panel has
been determined before we start the process.
[B] Defect detection of passive R/C components Defects may appear as different shade as follows
(1) Take the component out of its
background use a threshold.
(2) Detect defects by thresholding
18
-
To take the image of the component out of the original
image, we need to (a)Use a threshold for segmentation,
(b)use morphological operations to fill cavities in the
binary image of the component, (c)define a mask of
the component, (d)take out the gray image of the
component from the original image.
To detect the location of defects (area with a gray level
different from surrounding area), we then apply an
optimal or adaptive thresholding technique.
In this example, we mainly work with gray images with
the aid of operations on binary images.
[C] Defects classification of solder joints In the above examples, we do nothing but detect and
locate defects, and measure their size. However, we did
nothing to classify the defects. To complete such
tasks which require no classification, we need only to
borrow standard algorithms from existing image
processing libraries and use them smartly. 19
-
However, if the AOI is to serve the purpose of quality
control, we need to go further to classify the defects we
find. For instance, the most challenging task in AOI of
PCB is the classification of solder joints.
20
-
The types of these defects can easily be distinguished
in color images, but very difficult to tell the differences
from gray images.
Type of
joints
Vertical image Side image
good
open
Tombstone
skew
missing
To solve such problems, (a)carefully define features
that reliably tell differences between types of defects,
21
-
(b)design lighting condition to highlight the features
in gray images with good contrast, (c)clever image
processing to extract features of the defects, (d)apply
or develop suitable classification method (such as
fuzzy-neural network).
1.1.3: An overview of AOI and machine vision
AOI is a technique that embraces several disciplines:
(a)optics and lighting, (b)image processing, (c)pattern
classification in addition to (d)precision machine and
(e)automation and control.
Precision
machine &
automation
Optics and Lighting Pattern classification
Image processing
AOI
Statistics & QC
22
-
In regard of a project of AOI, its usually consists of the
following main steps
Define the problem and key features of the image
Design suitable lighting condition to highlight features
Proper pre-processing on gray images for segmentation
Proper pre-processing on gray images for thresholding
Operations on binary image for defect
detection, measurement & feature extraction
Defects classification
Statistical analysis and QC report
Lighting should be used to highlight the features.
Failing in doing so, remaining tasks may be difficult, if
23
-
possible, to be accomplished by mere image processing
algorithms.
When take time (cycle time for examining a piece) is
critical, we need three more expertise: (f)image
acquisition (CCD camera and lens), (g)image
transmission (grabber card and data bus), (h)software
coding.
1.1.4: Organization of this course
(1) Introduction (case study and basic notions)
(2) Operations on Binary images
(3) Segmentation and Region Representation
(4) Filtering and morphological operations on gray images
(5) Edge detection
(6) Contour representations and texture
(15) Pattern classification and Object Recognition
(9) Lighting, luminance and Shading
(10) Color images
(11) Depth and 2.5D profilomotry
(12) Optics, camera, grabber and calibration 24
-
1.3Fundamental Concepts and Terminology 1.3.1Image geometry (1.4)
x
Chip object y x P(x, y z)
r
z
P f
Camera lens On the left is the CCD chip (blue lines), in the center
the coordinate (x, y) marks the position of the CCD
camera lens with a focal length f, and P is a point on
the surface of an object in the real world, while P on
the left be its image on the CCD chip.
It can be shown that ' 'f x y
z x y= = , hence
f xz
='x and f yz
='y
where f and z are given 25
-
1.3.2Sampling and Quantization (1.5)
Sample means true length/pixel, which is initially
dependent on the amplification factor of the lens.
One can perform down sampling to reduce the pixels
of an image. For instance, in the following diagram, the
original image is down sampled by a factor of 2, i.e.,
the gray level of four pixels in the original image is
averaged to give a pixel in the resultant image.
Down
original down sampled
Down sampling results in an image which is smoother
and fuzzier.
The quantization level of an image is the quantization
level of its intensity. For instance, the quantization
level of B/W camera is generally 256 = 28.
26
-
1.3.3Levels of Computation1.7
There several levels of image processing, beginning
with the lowest level pixel-by-pixel operations, local
operation such as convolution masks, and global
operations (such as histogram analysis).
The higher the level of operation, the more pixels
involves which usually implies higher demand on
memory resources and computation time.
APixel level operation
Pixel level operation involves only one pixel at a time.
Two of the most popular pixel level operations are
thresholding and pixel- to-pixel transformation.
P(x, y) PB(x, y)
T (x, y) (x, y)
TP(x, y) PB(x, y)
27
-
BLocal level operation
Local level operation involves several pixels in the
original image to determine the gray level of a pixel in
the resultant image.
P(x, y) PB(x, y)
(x, y) T
For instance, in the operation of a convolution mask it
usually involves 9 pixels in the original image to give
the gray level of a pixel in the resultant image, such as
28
-
3 3
1 1
11 12 13
21 22 23
31 32 33
1, 1 1, 1, 1
, 1 , , 1
1, 1 1, 1, 1
( , ) ( , ) ( , )
i j i j i j
i j i j i j
i j i j i j
Bk l
P i j g i k j l p k l
g g gg
p p pg g
g g g
p p p
p p p
+
= =
+
+ + +
=
= + +
+ + +
+ + +
+
The larger the size of a convolution mask, the more it
demands memory and computation. For instance, a
mask of size demands 25/95 5 3 times resources than a mask. Hence, unless necessary, we usually
uses masks of only the size
3 3
3 3 .
CGlobal level operation
If the output of a certain operations involves all pixels
in an image, such an operation is global. For instance,
calculating the histogram of an image is a global level
operation.
Fortunately, histogram analysis involves only addition
(no multiplication), hence it does not demand too much
computation effort. 29
-
DObject level operation
An image may contains several objects. To find out the
size, length or other attributes of an object, it may
requires several operations in the object level to draw
all the features required for complete analysis.
30
-
1.4: Lighting and illumination CCD
CCD camera
CCD camera
PCB after reflow
31
-
CCD
camera infra-red coating
22 /W 1000
27 2000
50~104 5000
LED
CCD camera
CCD camera
and/or
LED
noise 32
-
A specular surface
Down
normal
surface 45O+/2
cameranormal surface
45O+/2 camera
normal
surface
33
-
B
normal surface
C
34
-
A
B
35
-
C
36
-
D
37
-
E
F
38
-
GLighting and FOV
The following comments apply equally to a field camera as well as a line camera.
When the front lighting is used, the incident angle of light usually significantly affects the quality of an AOI. However, if the length of a line camera (or a filed camera) is too long (in terms of the true length of the FOV), it will be difficult to create uniform luminance of the object except when the incident angle of light is almost 90o from the horizon. That is, too long a FOV restricts the incident angle of light.
Besides, when the FOV is too long, it also causes serious image distortion.
Practice I: Try to illuminate the same object using
different illumination condition and see how features in the captured image change.
39
-
1.5Case study -- classifying solder types
This subsection presents a method for classification of
solder joints in order to illustrate how lighting, image
processing and pattern recognition techniques are
integrated to work for an industrial problem in PCB.
1.5.1The need for AOI
Solder joints defects occupies 55% of PCB failure (the
others are 25% wrong polarity, 10% mis-aligned and
10% missing components).
Consider a production line of PC board consisting of
250 SMDs/board. Even under the idea production
condition that defected solder be at a low rate of 100
ppm (parts per million), the first throughput rate is only
90%.
If there exists a economic full inspection method, the
failed 10% PCB can be reworked to significantly
increase the profit. The need for full inspection
To inspect a PCB of 2000 solder joints, human
inspector takes 60~90 mins (25sec by AOI) at a best 40
-
recognition rate of 8090, which lasts for 4 hours,
then starting to decline until around 60%. While a good
AOI maintains at a recognition rate of above 90% for
24 hrs/day. The need for AOI inspection
1.5.2: History of AOI in solder inspection
M. R. Driels (1990) from Phillips Corp. used diffusive
lighting to study 50 features and selected among them
11 features to arrive at a G/B recognition arte of around
90%, but classification rate is less than 60%.
U. Michigan, Ann Arbor (1988) used diffusive light
and 27 features to arrived at a G/B recognition arte of
around 92% and a classification rate around 88%.
However, both approaches are too time consuming.
Carnegie Mellon University (1990) used 127 point
source light, 4 cameras and 9 features to arrive at a
claimed 97% recognition rate. But it takes 10 mins to
examine 2000 joints (not including scanning time).
Kim (2000) used 3 tiered light and 4 features + neuro-
network to arrived at 99% recognition rate in 4 mins. 41
-
It is easier to perform G/B inspection
than classification of defect types
Definitions of features and lighting are
keys to success.
1.5.3: Revisit physics for lighting design (2001 summer)
Solder joint obeys specular reflection
42
-
Simulation for optimal incident angle Insufficient solder Acceptable solder Excessive
20
27
35
1.5.4: Feature definition
1
2
2
1 _ _
1 _ _
1 _ _
III
I
IV
I
IV
I
AIf insufficient solder imageA
AElse If acceptable solder ima
f
f geAAIf excessive solder imageA
>
>
<
f
43
-
1.5.5: Experimental results (2002 Winter)
Total: 62 joints, mis-classified: 3 joints (4.8%), all on
the boarder line
(a)insufficient or acceptable (b)acceptable or excessive
44
-
1.5.6: Compared with the best (2003 Spring)
All 62 joints are used
Features X1 and X2 are supposed to divide solder joints
into 2 groups, but in fact they do not.
Recognition rate is 57.62% (compare to 95.16%)
45
-
46
Take away ambiguous solders
-
1.5.7: More to think about
Recognition rate depends on testing samples.
Recognition rate repeatability Illumination (lighting) is important, usually leads to
patents.
Features design is key to success (and failure)
Feature definitions have to be obtained by interviewing
customer, who can be very mis-leading.
Physic law beneath the phenomena is important,
without it testing samples can be NOT sufficiently
representative (leading to ill-defined features and
classification criteria), and there will be no way to
double check interviewed information.
47
-
AOI
AOI
AOI 12
image acquisition and data
transmission3algorithms
patterm recognition4Coding
5
IC TFT-LCD AOI
handle
AOI
AOI
image grabber CodingMill
AOI
48
-
survey
image grabber
coding Mill
Mill
coding
telecentric lens
10m/pixel coherence
distorsion
Take time image grabber
coding efficiency 49
-
AOI
algorithms
operators
operators
algorithms
algorithms coding (computation
efficiency)
programming
50