Development of Robotic System for Cucumber Harvesting

Seiichi ARIMA*1, Naoshi KONDO*2 and Hiroshi NAKAMURA*1

*1 Tobe Engineering Division, ISEKI & CO., LD. (Tobe, Iyo, Ehime,791-21 Japan)*2 Faculty of Agriculture, Okayama University (Tsushima-Naka, Okayama, 700 Japan)

Abstract

Cucumber plant is widely cultivated in Japan, and its fruit grows rapidly. Automation for cucumber fruit harvesting is more important than for other fruits. In this paper, a cucumber harvesting robot which consisted of a visual sensor, manipulator, hand and traveling device was described. First, the visual sensor should be able to discriminate green fruit from leaves and stems. However, the color of leaves and stems is similar to that of fruit. In addition, the 3-D position of fruit should be detected. Two sensor systems were, therefore, combined for use of the robot: one was a monochrome TV camera with both 550 nm and 850 nm wavelength interference filters to discriminate fruit using spectral reflectance; the other was a photo-electric sensor in which a PSD (position-sensitive device) array was used to measure the distance. Secondly, the robot needs to approach the fruit. The mechanism of the manipulator was investigated based on the morphological properties of cucumber plant so as to adapt a harvesting configuration with high manipulatability and the 6 DOF (degree of freedom) manipulator was manufactured on an experimental basis. Thirdly, the harvesting hand must grip the fruit as well as detect and cut the peduncle. The hand was also constructed experimentally based on the physical properties of cucumber plant. Finally, a harvesting experiment was conducted, after the robotic components were mounted on the traveling device with 4 wheels. The results obtained showed that the use of the robot was suitable for harvesting.

Discipline: Agricultural machineryAdditional key words: end- effector, image processing, plant training system, 3-D position

JARQ 30, 233-238 (1996)

Introduction

Recently, the rapid aging of the farming popula­tion, the shortage of farmers' successors and the resulting shortage of manpower have become a seri­ous problem in Japan. As a result, the need for automation and robotization of farm operations has increased. However, as most farm operations which had not been automated hitherto require highly sophisticated judgment based on human vision and sense of touch, the development of a robot requires that the machine be equipped with sensors that can simulate human vision and sense of touch as well as corresponding intelligence.

One statistical survey8> reported that a total of one thousand and several hundred working hours were required for the cultivation and harvesting of cucumber grown in a 10 a greenhouse. About half

of these hours are spent for harvesting and post-

harvesting operations in cucumber. In addition, com­pared with some other vegetables, the growth rate of cucumber plant is faster. Accordingly, delayed harvest by I or 2 days may lower the commercial value of harvested cucumber plants significantly. Therefore, the need for mechanization of harvesting operations is justified.

To date, several studies on cucumber harvesting robot have been carried out. In the case of the visual sensing part, for instance, a study was carried out in which 2 images were compared and the difference in shape between those 2 images was enhanced using the image processing technique to detect the position of fruit 7>. A study on the end-effector for harvest­ing which consisted of a gripper part, detector. cutter part and universal rotating part was carried out 1>.. In addition, in the case of a plant training system suitable for robot and the manipulator, a study was conducted on harvesting system in which cucumber plant was harvested after it was turned

234

toward the horizontal direction and the leaves cover­ing it were lifted upward by the action of air now21

•

Though the manipulator in this system lacks nexibil­ity, at the same time, the productivity per unit land is considered to be relatively low in this plant train­ing system. For these reasons, the proposed system may not be practical.

In this paper, a plant training system for robot use was studied first by considering the capability of the existing robots. Then, the basic mechanism of a manipulator suitable for such a plant training system and its physical properties, as well as those of the end-effector for harvesting and the visual sen­sor were studied. Based on the results obtained, devices were developed on a trial basis and a basic harvesting experiment was carried out.

Plant training system for ro bot use

In the conventional plant training system, cucum­ber plant grows in the upward direction. Accord­ingly, the fruit is usually shielded by the leaves which cover a large area, often making it impossible f,or humans to recognize the existence of the fruit. A robot with a visual sensor that cannot readily dis­criminate one object with a certain color from an other object with a similar color and determine the shape of an object may not be able to detect tihe fruit which is entirely or partly shielded by the leaves. In addition, the stems and leaves are more likely to become obstacles when the manipulator tries to approach the fruit because the degree of freedom of the manipulator is lower than that of human be­ings. Furthermore, as the peduncle and the main stem are close to each other, both the peduncle and the main stem could be cut together when harvesting is carried out by the end-effector.

A new plant training system in which fruit can be more easily separated from leaves and stems when

Fig. I. Inclined trellis training system

JARQ 30(4) 1996

the robot was used was developed31• As a result, the control method and the mechanism of the robot for this plant training system was simplified and har­vesting became easier. Fig. I depicts the plant train­ing system.

In this new system, only the fruit part is exposed to the section below the trellis by the inclination of the cucumber plant cultivated in the conventional manner and at the same time by holding the leaves and stems with a pole. In this study, the method was designated as the inclined trellis training system. In this system, since it became possible to separate the fruit from the leaves and stems, it was consi­dered that the method was more suitable for robot harvesting.

Basic mechanism o f cucumber harvesting ro bot

Plate I shows the cucumber harvesting robot which was experimentally developed during this study. This cucumber harvesting robot consists of a visual sensor capable of identifying the fruit and detecting its position, a manipulawr made up of a prismatic joint capable of sliding along the trellis and 5 rota­tional joints, an end-effector capable of gripping fruit and detecting and cutting the peduncle, and a travel­ing device 9l.

I) Discrimination and recognition of fruit Based on the measurements of spectral reflectance

for cucumber plant, it was found that in the range from the visible to near infrared region there were 2 wavelength regions where frujt exhibited unique characteristics: one near 550 nm where the reflec­tance of fruit was lower than that of other regions and the other at 750-900 nm where the renectance of fruit was higher than that of the other regions.

Plate I . Cucumber harvesting robot

S. Arima et lll.: Developme/11 of Robotic System for Cucumber Harvesting 23.:i

Object ~·"'"...__, Synchronous detector

Monochrome TV camera

Di.splay!

Display2

lroage input

output board

Floppy disk ..._ _____ _

... <i

:i C. E 8

Fig. 2. Block diagram of visual sensor system

Based on these characteristics of fruit, the 2-wavelength type visual sensor in which fruit was discriminated by 2 interference filters was developed 6>. Fig. 2 shows the block diagram of this visual sensor system. In this system, the light reflected by the object illuminated from both sides by a light with a thermodynamic temperature of 5,500 K penetrated into the monochrome TV camera through the inter­ference filter. As optical filters, 2 types of interfer­ence filters with a wavelength of 550 nm and 850 nm were used. They were alternatively used whenever the image was inputted.

Two images obtained by using the different filters were processed in terms of every gray-tone value of their pixels using the following equation in order to prepare the processed image.

r = Gray-lone value at 850 nm

Gray-tone value + Gray-tone value at 550 nm at 850 nm

The equation shows that the gray-tone value of fruit after calculation increased because the reflectance of the fruit at 850 nm was high. Meanwhile, as the reflectance of the other sections at 850 nm was lower than that of the fruit, their gray-tone values after calculation were also considered to remain low. Therefore, if this processed image is binarized by comparing its individual pixel values with the threshold values, an image which represents only the fruit can be obtained.

Next, the processing of noise removal , filling and labeling was applied to the binarized image which was obtained from the recognition results and the features of the individual lumpy figures were extracted from that binarized image. These features included the area, the Feret's diameter ratio and FLAG (origi­nal index). Here, FLAG was used as FLAG = I when more than 30 lines with an individual width in the horizontal direction ranging between 5 and 20 dots

3 5

11..1

Fig. 3. Result of labeling I : Blob number of fruits and leaf, 2,3.4,5: Blob number of leaves.

were present continuously while it was used as FLAG = 0 when they were absent. Regarding these 3 reatures, when the Feret's diameter ratio was larg­er than 2.20 or FLAG = I as well as when the area over 300 was met, the blob was considered to con­tain the fruit and the center line was extracted. Fig. 3 shows the results of processing of noise removal , fi ll ing and labeling applied to one binarized image. Herc, blob I satisfied these conditions. After the noise removal processing was applied to the image in which the center line was extracted and the link­age among neighboring lines was completed, when the length of the center line was found to exceed J 20 dots, the pixel array was considered to correspond to the fruit. In the case of a pixel array shorter than 120 dots, its linkage with the other pixel arrays in the 45° angle area under that pixel array should be sought. The operation to connect the pixel array to the most appropriate one among the other pixel arrays should be repeated until the length of the con­nected pixel array exceeds 120 dots. If there is no pixel array in the lower section, it is concluded that there is no fruit or that the fruit is too small for harvesting, thus ending the algorithm of fruit recog­nition. The results are shown in Fig. 4. In this image, since lines 2, 3, 4 and 6 were connected, their center line was determined and corresponded to the fruit.

Thus, with the use of the algorithm in which the center lines were extracted from lumpy figures and the line images were linked to one another, it

236

1 I

2

+-3

Fig. 4. Resul1 of rccogni1ion 1-6: Linc numbers.

Light C:....:::, projector ....._

Stc1111ir.Jg motor(!) I

\ Stcppit}g motor(g)

-..... Stcppil_!g motor@

Fig. 5. Mechanism of photo-elec1 ric sensor

became possible to recognize the existence of fruit even if it was shielded by obstacles such as stems and leaves.

2) Detection of /he position of fruit Fig. 5 shO\VS the mechanism of the photo-electric

sensor5>. This sensor has both a light projector and a light detector and its operation is based on the active range finder type mechanism. That is, uti liz­ing the angles of the light from the projector and the light penetrating into the detector, the sensor can determine the distance from it to Lhe object based on the principle of triangulation. The modulated near-infrared light from the LED (light emitt ing diode) of the light projector was deflected toward the vertical direct ion after it passed through the lens and was reflected by the rotating reflector driven

0 0

20

60 i!' _AA ~ /\

~

' ~ . .

J ~~

40 ,= .

,\

. ~ ~

" 20 40

JARQ 30(4) 1996

-

' '£ '\

,, ~

. ~~

f',.. "V •

60

Fig. 6. 3-dimensional shape of cucumber

by the stepping motor (D. Then, the light reflected by Lhe object was received by PSD after the light was reflected by the rotating reflector driven by the stepping motor @ which rotated while being syn­chronized with the rotation of Lhe projector side and passing through the lens. In addition, the sensor was designed so that the 3-dimensional information was obtained by rotating the entire unit accommodat­ing both the projector and the detector by the step­ping motor @ a nd deflecting it coward the vertical direction. Fig. 6 shows an example when a 3-dimensional shape was analyzed. T hus, the unique shape of 2 parts of fruits, the leaves and main stem could be clearly identified from this figure.

3) Manipulator for harvesting Fig. 7 shows the basic mechanism of the manipu­

lato r which was studied in considering its suitability for the plant training system described in the previ­ous chapter·1>. This manipulator consisted of a pris­matic joint capable of sliding a long the trell is and 5 rotational joints capable of taking various config­urations. The link length , offset length of the manipulator and the displacement of its individual joints were determined based on the inclined trellis training system shown in Fig. I. As a result of the setting of a relatively large joint displacement, it be­came possible for Lhe 5 DOF manipulator, except for the sliding joint, to put its upper arm on its forearm, which resulted in a considerable reduction of the entire size of the manipulator. In add.ition, the test results showed that to improve the manipulat­ability of this device, it was necessary to increase both the angle of the trellis and the sliding stroke.

S. Arima et al.: Develop111e111 of Roboric Sy~·rem for C11c11111ber Narvesring 237

X Link length and . . offset lcngth(mm) J oint displacement

I 0=130 0 O= 65"-' 90• I J= SO s J= 0--400 mm /2=300 0 2= -90-- 90'

L3=182 0.1= -90-- 90·

/4=182 0 4= -173- 125'

15= 0 Os= -85- 95•

Fig. 7. Basic mechanism of manipulator

4) End-effeclor for harvesting Fig. 8 shows the mechanism of the end-effector

for harvesting 4>. The end-effector which was ex­perimentally constructed for this study consisted of a gripper section and a detector,cutter part . In its operation, the gripper of the end-effecLOr first gripped the position about 3 cm below the lop end of fruit with a force of 6 N and then the detector-culler pan slid upward. Al the same time, the detector

(a) Gripper part

plate was raised while remaining in contact with Lhe frui t and the displacement was read wiih a poten­tiometer to detect the boundary between fruit and peduncle. If the detection was successful, the pedun­cle could be cut wiih a force of 12 N by the cuuer installed right under the detector plate. These basic specifications were determined by measuring the phys­ical. properties of cucumber fruit and peduncle.

Conclusion

The results described above, indicate that the sys­tem combining a 2-wavelength type visual sensor and a photo-electric sensor was effective as visual sens­ing pan for a cucumber harvesting robot although some minor modifications need to be made for the individual sensors.

The 6 DOF manipulator caused some positioning error due to the factor associated with its internal mechanism and accordingly there were cases in which the manipulator failed to reach the right position. In the harvesting test , however, it could reach almost any fruit located in the working range.

The end-effector which was experimentally con­structed according Lo the measurements of the values related to the physical properties of the different parts of the cucumber plant could easily harvest the fruit, if the cucumber plant was healthy and its fruit had a sufficiently long peduncle. When the length of the peduncle was short, however, the detector plate could not move up to the position of the peduncle,

Potentiometer for detection

pi\ -,,...u:::wi-L..w..,..,

Detector for Motor for cutting peduncle

iN=====l / u@l[~~ II

1, Cutter

UL _ : :_u (b) Detector · cutter part

Fig. 8. Mechanism of cucumber harvesting hand

238

.due to the obstruction caused by neighboring joints and leaves. As a result, in some cases, fruit harvest­ing was unsuccessful. It was considered, therefore, thal the change in the stroke direction of the detec­tor plate to a more obstruction-free direction a nd the introduction of a non-contact type device were necessary to improve the harvesting performance.

Through the basic harvesting experiment using lhis robot system, it was found that the component units of the robot functioned properly allhough some minor modifications might be required and the robot could perform a series of harvesting operations. It was also found that these operations were carried out more effectively in the inclined trellis training sys­tem. Thus, the suitability of the robot harvesting system was confirmed.

References

I) Amaha, K. et al. (1989): Development of a robotic hand for harvesting cucumber fruit. J. Agr. Mel., 45 (2), 93-97 [In Japanese with English summary].

2) Amaha, K. et a l. (1990): Harvesting or cucumber fruits by robot. !11 Proc. of 28th Meeting for Envi­ron. Control in Biol., 148- 149 (In Japanese].

JARQ 30(4) 1996

3) Ari ma, S. ct al. (1994): Studies on cucumber har­vesting robot (part I). J. Jpn. Soc. Agric. Mach., 56(1), 55-64 [In Japanese with English summary).

4) Arima, S. el al. (1994) : Studies on cucumber har­vesting robot (part 2). J. Jpn. Soc. Agric. Mach .. 56(6), 69-76 [In Japanese with English summary].

5) Arima, S. et al. (1995): Studies on cucumber har­vesting robot {pan 3). J. Jpn. Soc. Agric. Mach., 57 (1), 51-58 [ln Japanese with EngUsh summary].

6) Kondo, N. ct al. (1993): Studies on visual sensor for cucumber fruit detecting (part I). J. Environ . Co111rol in Biol., 31 (2), 93-100 [In Japanese with English summary].

7) Shono, K. et al. (1989): Detection of cucumber fruit position by image processing. J. Agr. Met. , 45 (2), 87-92 [In Japanese with English summary].

8) Statistics Information Department, Ministry of Agricul­ture, Forestry and Fisheries (1995): Vegetables produc­tion cost in survey report (1993). Statistics association of agricultural and forestry, Tokyo, 66-69 CTn Japanese).

9) Yamashita, J. el al. (1991) : Development or an automatic transport vehicle for use in greenhouses and its practical performance. J. Jpn. Soc. Agric. Mach ., 53(5), 75-84 [In Japanese with English summary].

( Received for publication, January 8, 1996)