Post on 28-Jul-2018
1
Autonomous and Human-Robot Collaborative Systems
Avital Bechar
for Field Operations in Orchards, Greenhouses and Field Crops
Institute of Agricultural Engineering, ARO, Volcani Center, Israel
2
Overview Background The Agricultural Research Organization Agricultural productivity and production (robotics
perspectives) Characteristics of the agricultural domain (robotics
perspectives) Basic principles (AgRobots) ARL activity Conclusions
AgRA TC Webinar, March 2015 ,24
3
Agricultural Research Organization
AgRA TC Webinar, March 2015 ,24
4 AgRA TC Webinar, March 2015 ,24
• Founded in 1921.• 1000 people: including 200 research
scientists and 220 graduate students.• 6 Institutes: Soil water and
environmental sciences; Plant protection; Animal Sciences; Plant sciences; Food sciences; and, Agricultural Engineering.
Agricultural Research Organization
5 AgRA TC Webinar, March 2015 ,24
Institute of Agricultural Engineering The only research organization in Israel whose activities
encompass a wide range of engineering and technological topics relating to all aspects of agriculture.
About 60 people, including 14 research scientists
6 AgRA TC Webinar, March 2015 ,24
Institute of Agricultural Engineering Two departments:
Sensing, information, and mechanization engineering Production, growing and environmental engineering
a
7
Agricultural production Cultivation and production processes in agriculture. Affecting factors: crop characteristics and requirements, the geographical/geological environments, climatic conditions, market demands the farmer’s capabilities and means.
Farm sizes increase and the number of farmers and agricultural workers decreases.
Human labor intensive and labor cost of 25-40%.
AgRA TC Webinar, March 2015 ,24
8 AgRA TC Webinar, March 2015 ,24
(http://www.thadw.us/agricultural-employment-since- 1870/ )
9 AgRA TC Webinar, March 2015 ,24
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
MetalBolts
Metalscrew nuts
Metalnails
MetalDiscs
Plasticparts
Rubberparts
Woodparts
Flowercuttings
CV
CV of different materials
CV=σ/µ
CV2 > CV1
10 AgRA TC Webinar, March 2015 ,24
11 AgRA TC Webinar, March 2015 ,24
12
Unstructured Environments
• Unknown a-priori• Unpredictable• Dynamic
AgRA TC Webinar, March 2015 ,24
13
The terrain, vegetation, landscape, visibility, illumination and other atmospheric conditions are not well defined; vary, have inherent uncertainty, and generate unpredictable and dynamic situations.
AgRA TC Webinar, March 2015 ,24
Unstructured Environments
14 AgRA TC Webinar, March 2015 ,24
Unstructured Objects
Variable and non-uniform:size
shape color
texturelocation
15 AgRA TC Webinar, March 2015 ,24
--++Objects
-+-+Env.
Agr.MedicalSpaceUnder-water
Military
Industry
16
Basic principles Main task: pruning, picking, harvesting, weeding... Supporting tasks: localization, detection, navigation…
Mobility and steering Sensing Path planning and navigation Manipulators and end effectors Control Autonomy and human-robot collaboration
AgRA TC Webinar, March 2015 ,24
17
Autonomy/Human-Robot collaboration
AgRA TC Webinar, March 2015 ,24
Autonomous robot are lack the capability to respond to ill-defined, unknown, changing, and unpredicted events, such as occur in unstructured environments.
Pareto principle: roughly 80% of a task is easy to adapt to robotics and automation and 20% is difficult (Stentz et al., 2002).
18
Hybrid Human-Robot Systems
AgRA TC Webinar, March 2015 ,24
Main Task
SupportingTask
1
SupportingTask
3
SupportingTask
4
SupportingTask
2
Subsystem1
Subsystem2
19 AgRA TC Webinar, March 2015 ,24
20 AgRA TC Webinar, March 2015 ,24
Lab members (current)
2 PhD students (IE, CE-AgEng) 3 MSc students (ME, IE) Mechanical Engineer Electrical Engineer Postdoc Agronomist
21 AgRA TC Webinar, March 2015 ,24
Projects (current) Autonomous greenhouse sprayer for specialty
crops (with BGU). A human-robot collaborative system for deciduous
tree selective pruning. a human-robot system for selective melon
collection (with Technion). an autonomous system for monitoring of diseases
in greenhouses (with BGU). Robotic sonar for yield estimation (with TAU). Characterization of Agricultural Tasks for the
Design of a Minimalistic Robot (with Technion).
22
Autonomous greenhouse sprayer Avital Bechar, Itamar Dar, Victor Bloch, Yael Edan,
Roee Finkelshtein, Guy Lidor, Ron Berenstein
AgRA TC Webinar, March 2015 ,24
23
The motivation
AgRA TC Webinar, March 2015 ,24
24
Plot geometry
100
m
115
170
AgRA TC Webinar, March 2015 ,24
25 AgRA TC Webinar, March 2015 ,24
R R∆
Sensing (Features)
26
FeaturesFeature Formula Feature Formula
R Red h H/(H+S+V)
G Green s S/(H+S+V)
B Blue v V/(H+S+V)
r R/(R+G+B) deltaH (H-S)+(S-V)
g G/(R+G+B) deltaS (S-H)+(S-V)
b B/(R+G+B) deltaV (V-S)+(V-H)
deltaR (R-G)+(R-B) C1 R-G
deltaG (G-R)+(G-B) C2 R-B
deltaB (B-G)+(B-R) C3 G-B
H Hue Real_ModHue
S Saturation imag_ModHue
V Value
)(cos)(2
21222
{GBRBRGBGR
BGR−−−++
−−−
27
For all featuresFind feature threshold value that maximizes the "splitting criterion“
Among all featuresChoose the one thatmaximizes the"splitting criterion“
Decision Tree - CART Breiman et al., 1984
AgRA TC Webinar, March 2015 ,24
28
Total success
Movie 1 LevelTS
2 LevelTS
3 LevelTS
Movie 1 0.834 0.834 0.886Movie 2 0.943 0.941 0.940Movie 3 0.617 0.834 0.848Movie 4 0.818 0.874 0.889Movie 5 0.922 0.927 0.920Movie 6 0.892 0.899 0.899Movie 7 0.932 0.925 0.930Average 0.851 0.891 0.902
Number of nodes 1 3 7
AgRA TC Webinar, March 2015 ,24
Decision tree
29
Judges Vote (~ Majority rule)
A customized CART variation, developed in this research
A “Judge” is single level CART (root node only)
Classification rule:
JudgesofNumberVoteJudges__
_
54321432132121Vote (M)55555444433322Judges (N)
AgRA TC Webinar, March 2015 ,24
30
Test set – "Judges Vote"Variation 2/2 2/3 3/4 3/5 4/5 2 Level (3 features)
Average 0.903 0.914 0.920 0.915 0.905 0.890
std 0.041 0.021 0.016 0.020 0.022 0.044
AgRA TC Webinar, March 2015 ,24
31
Algorithm Evaluation Platform
ArdunoCMP-03 Compass
AX3500 - Dual 60A
lifeCam NX-6000
180⁰
Lenovo R400
Servo SC-1256T
Motor DL-30Encoder Optical
E5
45
DATA
PWM
123
AgRA TC Webinar, March 2015 ,24
32 AgRA TC Webinar, March 2015 ,24
33
TXT1 Ein Yahav 261109 1st exp-fast.wmv
AgRA TC Webinar, March 2015 ,24
34
Modification of a commercial sprayer
An electric motor was installed on the steering wheel controlled by a Roboteq controller.
Installation of encoders on the steering pivot/axle and the front wheels.
PID control system. Control system inputs: platform
steering angle; desired direction from the adaptive algorithm and bearing.
Pure pursuit, carrot point 2m
AgRA TC Webinar, March 2015 ,24
35
The ‘autonomous unit’
Installed on the platform Connected to sensors and
actuators
AgRA TC Webinar, March 2015 ,24
36
Commercial Sprayer II
AgRA TC Webinar, March 2015 ,24
37 AgRA TC Webinar, March 2015 ,24
38
A H-R collaborative system for selective pruning
Avital Bechar, Victor Bloch, Roee Finkelshtain, Sivan Levi
AgRA TC Webinar, March 2015 ,24
39
Objective
Develop a human-robot integrated system for tree pruning and shaping Design of a cutting tool Develop a modelling technique Development of human robot interface and
methodology
AgRA TC Webinar, March 2015 ,24
40
Cutting tool alternatives
Chain saw Pruning shears Laser Water jet Disc saw Jigsaw
AgRA TC Webinar, March 2015 ,24
41
Cutting tool design
The cutting tool must be adapted to: Tree dimensions, branch diameter and strength Robot carrying ability, precision, energy source Pruning technique: cutting angle, velocity Tree structure: branch angles, depth inside the crown, obstacle
density, reaching ability
Agronomical requirements: Cutting angle 45° Reduce risk of wounds Cut disinfection (burned by high cutting speed)
AgRA TC Webinar, March 2015 ,24
42
Cutting tool selection & modification for a robotic arm Energy source, type and consumption Safety Weight Dimensions Precision and accuracy …
AgRA TC Webinar, March 2015 ,24
43
High accuracy requirements
Pruning shears: 3 directional dim. and 2 angular dim. Total required accuracy in 5D.
Disk saw: 1 directional dim. and 2 angular dim. Total required accuracy in 3D
AgRA TC Webinar, March 2015 ,24
44
Cutting tool design
0
10
20
30
40
50
60
70
80
3 6 9 12 15 18 21 24 27 30 33 36
Num
ber
of b
ranc
hes
Branch Diameter [mm]
80%82%84%86%88%90%92%94%96%98%
100%
3 6 9 12 15 18 21 24 27 30 33 36
Bra
nch
acc
umul
ativ
e pe
rcen
tage
Branch diameter [mm]
AgRA TC Webinar, March 2015, 24
45
Cutting tool design
AgRA TC Webinar, March 2015, 24
46
Tree Modeling
Simple and reliable method – mechanical
AgRA TC Webinar, March 2015 ,24
47
Tree Modeling
(Linker et al., 2014)
AgRA TC Webinar, March 2015 ,24
48
Path planningAssuming cutting points given: Find optimal reaching orientation Solve robot navigation problem in 6 dimensional C-space Find optimal order of cutting points
AgRA TC Webinar, March 2015 ,24
49
Primal collaboration 3D modelling of a tree a-priori HO marks cutting point on model Trajectory planning Branch pruning
Drawbacks The need for a-priori 3D modelling Long set up time Computation power (time, cost) Inaccurate Lack of information Not up to date
AgRA TC Webinar, March 2015 ,24
50
Concept and task description
Stage
SenseReason
PlanAct
Sub-Task
Images / modelCutting point detectionTrajectory planning and controlBranch pruning
Control
RH+ R
RR
AgRA TC Webinar, March 2015 ,2450
51 AgRA TC Webinar, March 2015 ,24
52 AgRA TC Webinar, March 2015 ,24
Brunch orientation (Two methods)
Movements (joints and linear)
53
21 subjects Age: 24 – 69 20 branches Two types of marking (1 click and 2 clicks) Two types of movements
Experiment
AgRA TC Webinar, March 2015 ,24
54
Results
0
1
2
3
4
5
6
2 clicks 1 click
Tim
e [s
ec]
click 1click 2
aa
bα<<0.01
AgRA TC Webinar, March 2015 ,24
55
Results
0
5
10
15
20
25
Cut sign move toscan
scan move tocut
return HR cycletime
Robotcycle time
Aver
age
time
[s] Joints
Linearic
AgRA TC Webinar, March 2015 ,24
56
Results Cutting point accuracy: 8-22 mm Branch orientation accuracy: mean: 9.4º, med: 5.75º
0%10%20%30%40%50%60%70%80%90%
100%
0
1
2
3
4
5
6
7
8
2 4 6 8 10 12 14 16 18 20 22 More
Num
ber
of b
ranc
hes
Error [degree]
Frequency
Cumulative %
AgRA TC Webinar, March 2015 ,24
57
Avital Bechar, Noa Schor, Sigal Berman, Aviv Dombrovsky, Yigal Elad and Timea Ignat
AgRA TC Webinar, March 2015 ,24
A Diseases Monitoring Robot
58
A cart roves between crop rows. Manipulator mounted on the cart is maneuvering into
a set of positions for sensing and detection. Sensors acquire data and fuse them to achieve high
precision. Early detection of two diseases: powdery mildew and
tomato spotted wilt virus (TSWV).
AgRA TC Webinar, March 2015 ,24
Scenario
59
No known algorithms for TSWV detection. Detection of more than one threat has not been
attempted thus far. Development based on a holistic approach integrating
the design of both motion and perception.
AgRA TC Webinar, March 2015 ,24
Challenges
60
A robotic manipulator (MH5L, Motoman). A custom-made end-effector. Sensory apparatus.
AgRA TC Webinar, March 2015 ,24
Apparatus
Manipulator
End-effector
Laser sensor
Multispectral camera
Color camera
61
TSWV Powdery mildew
AgRA TC Webinar, March 2015 ,24
Disease detection
62 AgRA TC Webinar, March 2015 ,24
Motion planning and execution
63 AgRA TC Webinar, March 2015 ,24
TSWV detection
64 AgRA TC Webinar, March 2015 ,24
Disease detection
65 AgRA TC Webinar, March 2015 ,24
Melons collecting robotAvital Bechar, Moshe Karagoden, Ariel Weinstock, Moshe Mann, Sasha Katzman, Victor Bloch, Guy Lidor, Roee Finkelshtein, Itzhak Shmulevich
66
A Cartesian robot. Two cylindrical rails, toothed belt axis and end limit
switches; Two Stepper Motors; Motor Controllers; Programmable Logic Controller (PLC); Frame (600 mm x1500mm).
Vacuum operated Gripper
AgRA TC Webinar, March 2015 ,24
Apparatus
6767
Melons Picking Up Simulator
AgRA TC Webinar, March 2015 ,24
68
20 trials In each trial, 3-7 objects Density of 2-5 objects/m2
total collecting area of 4m X 0.5m Cart velocity: 51 mm/sec manipulator velocity: 800 mm/sec
AgRA TC Webinar, March 2015 ,24
Dynamic state experiment
69
Total success rate of 84% Due to technical and
communication problems Position error 7-10 mm at
reach location Collection pace: 7-8
objects per minute
AgRA TC Webinar, March 2015 ,24
Results
70
Conclusions There has been considerable progress. Technical feasibility was shown. Agricultural modifications or human integration.
AgRA TC Webinar, March 2015 ,24
71
THANK YOU
Avital Becharavital@volcani.agri.gov.il