Dynamic Workcell for Industrial Robots

Yanfei Liu

CLEMSONCLEMSONU N I V E R S I T Y

Dynamic Workcell for Industrial Robots

Dept. of Electrical and Computer EngineeringClemson University, SC

Clemson University

Outline

• Motivation for this research– Current status of vision in industrial workcells

– A novel industrial workcell with continuous visual guidance

• Work that has been done– Our prototype: camera network based industrial workcell

– A new generic timing model for vision-based robotic systems

– Dynamic intercept and manipulation of objects under semi-structured motion

– Grasping research using a novel flexible pneumatic end-effector

Clemson University

Motivation for this research

• Current industrial workcells– No vision or a single snapshot in certain locations

– Disadvantages

• Cannot deal with flexible parts

• Cannot deal with uncertainty

Clemson University

Motivation for this research• Our novel dynamic workcell design

– Manipulation is integrated with visual sensing– Applications ( reduce fixtures, handle objects on the ship)

Clemson University

System architecture

– A set of cameras embedded into the workcell

– An industrial manipulator with its conventional controller

Clemson University

Experimental platform• Our prototype

– Staubli RX130 manipulator with its conventional controller

– Six cameras, wired to two PC-RGB framegrabbers mounted in a Compaq Proliant 8500 computer

– V+ Operating systems and language

– Alter command to accomplish real time motion

Clemson University

Tracking experiments

Clemson University

First part:

A new generic timing model for vision-based robotic system

Clemson University

Introduction

• Classical visual servoing structure

– eye-in-hand systems• Corke (1996), an eye-in-hand manipulator to fixate on a

thrown ping-pong ball

• Gangloff (2002), a 6-DOF manipulator to follow unknown but structured 3-D profiles.

– part-in-hand systems• Stanvnitzky (2000), align a metal part with another fixed part

– mobile robot systems• Kim (2000), a mobile robot system to play soccer

controlpower

amplifiers robot

camera

desired position

+-

e

Clemson University

Introduction

• Vision guided control structure– Allen (1993), a PUMA-560 tracking and grasping a moving model

train which moved around a circular railway.

– Nakai (1998), a robot system to play volleyball with human beings.

– Miyazaki (2002), a robot accomplished a ping pong task based on virtual targets

camera

desiredposition

control jointcontroller

robot encoders+-

e

Clemson University

Introduction

• Three common problems in visual systems– Maximum possible rate for complex visual sensing and

processing is much slower than the minimum required rate for mechanical control.

– Complex visual processing introduces a significant lag (processing lag) between when reality is sensed and when the result from processing a measurement of the object state is available.

– A lag (motion lag) is produced when the mechanical system takes time to complete the desired motion.

Clemson University

Previous work

• the first two of the three problems have been addressed to some extent in previous works. All of these works neglect the motion time (motion lag) of the robot.

• Corke and Kim, presented timing diagram to describe time delay, used discrete time models to model the systems and simplified these asynchronous systems to single-rate systems.

Work Image processing rate

(HZ)

Control rate

(HZ)

Processing lag

(ms)

Motion lag

(ms)

Corke, Good 50 70 48 --

Stavnitzky, Capson 30 1000 -- --

Kim et. al. 30 30 90 --

Allen et. al. 10 50 100 --

Nakai et. al. 60 500 -- --

this work 23 250 151 130

Clemson University

Timing Model: notation

processinglag

s1 s2

u1

kq

c1 … cN

motion lag

u2

f

sensing

image processing

synchronizing tracking

controlling

finishingmotion

Clemson University

Timing Model: our prototype

• Inherent values (obtained by analysis/measurement) s = 33ms u = 19+30+14 = 63ms wm = 39ms wf = (5+16+27)/3 = 16ms w = 39+16 = 55ms

l = s + u + w =151ms f = 130ms

• User-variable values c = 4ms q = 40ms

Clemson University

Experiments• Problem description

– The most recently measured position and velocity of the object is where the object was (l+k) ms before, xt-l- k, vt-l- k

– The current position, xt

– N, d?

)( klvxx kltkltt

Clemson University

Experiments

• Solutions ))1(( fincivixix ttit

dinixix qtit

ivcd

ixixivcfin

t

qttt

)(

c

divt

Constraint:

N

ixixid qtit

Clemson University

Experiments: model validation

• Setup

– A small cylindrical object is dragged by a string tied to a belt moving at a constant velocity.

– The robot will lunge and cover the object on the table with a modified end-effector, a small plastic bowl.

Clemson University

Experiments (video)

Clemson University

Experiments

q=40 q=80Velocity

range(mm/s)

Stdev

range

Catch percentage

Velocity

range(mm/s)

Stdev

range

Catch

percentage

84.4 – 97.4 1.3 –3.8 100% 85.9 – 95.1 2.5 - 3.7 100%

129.8-146.7 1.7 - 3.2 100% 126.1 – 137.7 1.7 – 3.3 100%

177.6 – 195.1 0.5 – 2.6 100% 175.8 – 192.8 1.1 – 2.7 100%

• Experiment description– We set q to two different values, 40 and 80, in these two sets of

experiments. We let the object move at three different velocities. For each velocity, we ran the experiment ten times.

• Results

Clemson University

Experiments (video)

Clemson University

Second part:

Dynamic Intercept and Manipulation of Objects under Semi-Structured Motion

Clemson University

Scooping balls (video)

Clemson University

Scooping balls: problem description

Start tracking Make prediction (t)

Open loopClosed loop

Impact (t+i)

x

y

xt , yt : object position at time tvx , vy: object velocity at time txr , yr: initial robot positionxf , yf : final impact position

robot

Unknown variables: yf , i

Clemson University

Scooping balls: solution

• Solutions

rf xx

fyt yivy

fxt xivx

x

tr

v

xxi

trx

ytf xx

v

vyy

c

fim

• Object unsensed time– Time between the last instant when reality is sensed

and the final impact time

– Delay between visual sensing and manipulation

Clemson University

Timeline description: object unsensed timet

processing lag(l) + k

synchronizing tracking

q q

controlling

finishing motion

closed loop open loop

…c1 cN…c1 cN

m 20

motion lag (f)

t = l + k + 4m+ f m < N = 10, k < 30 + 14 = 44ms

t = 151 + ( 40 + 44 ) / 2 +115 = 308ms

…

Clemson University

Impact point

20 alters

impact point

10 alters

20 alters

impact point

10 alters

y

z

Clemson University

Equations

tvxx

tvxx

if

iif

ˆ

t

xxvv

ff

i

ˆ

3082

wvvi

• Implementation– Predict the maximum acceleration of the object motion that the

robot still can achieve a successful catch

– Calculate the size of the end-effector in order to overcome the maximum acceleration of the moving objects

• Solutions

Clemson University

Experimental Validation

Bowl Scooper1 scooper2

Catch Miss Catch Miss Catch miss

Catch 88.6% 1.4% 84.3% 2.9% 95.7% 0.0%

Miss 1.4% 1.4% 2.9% 4.3% 0.0% 0.0%

Too fast 7.1% 5.1% 4.3%

Total failure 2.8% 5.8% 0.0%

• Setup– Two types of end-effector (bowl, two scoopers with different width).– Three types of interference (wind, bump, ramp)

• Results– With wind interference

Clemson University




Catch 2.9% 0.0% 2.9% 7.1% 1.4% 4.3%

Miss 0.0% 84.3% 0.0% 80.0% 2.9% 85.7%

Too fast 12.9% 10.0% 5.7%




Catch 37.1% 7.1% 31.4% 0.0% 50.0% 2.9%

Miss 1.4% 45.7% 2.9% 52.9% 0.0% 45.7%

Too fast 8.6% 12.9% 1.4%


– with bump interference, weighted corner

– with bump interference, balanced

Clemson University




Catch 0.0% 4.3% 0.0% 8.6% 1.4% 8.6%

Miss 2.9% 82.9% 4.3% 78.6% 2.9% 77.1%

Too fast 10.0% 8.6% 10.0%




Catch 60.0% 7.1% 50.0% 5.7% 68.6% 8.6%

Miss 4.3% 21.4% 14.3% 30.0% 4.3% 11.4%

Too fast 7.1% 0.0% 7.1%

Total failure 11.4% 20.0% 12.9%

– with ramp interference, weighted corner

– with ramp interference, balanced

Clemson University

Third part:

A Novel Pneumatic Three-finger Robot Hand

Clemson University

Related work• Three different types of robot hands

– Electric motor powered hands, for example:• A. Ramos et. al. Goldfinger

• C. Lovchik et. al. The robonaut hand

• J. Butterfa et. al. DLR-Hand

• Barrett hand

– Pneumatically driven hands:• S. Jacobsen et. al. UTAH/M.I.T. hand

– Hydraulically driven hands:• D. Schmidt et. al. Hydraulically actuated finger

• Vision-based robot hand research– A. Morales et. al. presented a vision-based strategy for computing three-

finger grasp on unknown planar objects

– A. Hauck et. al. Determine 3D grasps on unknown, non-polyhedral objects using a parallel jaw gripper

Clemson University

Novel pneumatic hand

• Novel hand architecture– build-in pneumatic line in

Staubli RX130

– Paper tube, music steel wire embedded inside

– Camera mount adjusting “finger” spread angle

– 120 degrees between each other

• Disadvantages of current robot hands– Most robot hands are heavy

– Even with visual guidance, the robot hand can only grasp stationary objects

Clemson University

Novel pneumatic hand

• Close position Open position

• Our research here is to demonstrate that we use a novel idea to built a flexible end effector and it can grasp semi-randomly moving objects. This is not a new type of complex research tool-type robot hands.

Clemson University

Grasping research• Problem statement

robot

ball track

x

y

initial hand position final hand

position

Clemson University

Grasping research• Position prediction

– Same as the method in the second part work of this research

• Orientation adjustment– Line fitting to get the final “roll” angle

– equations

n

i

n

i ii

n

i

n

i

n

i iiii

n

i

n

i ii

n

i

n

i

n

i

n

i iiiii

xxn

yxyxnb

xxn

yxxxya

1 1

22

1 1 1

1 1

22

1 1 1 1

2

)(

)(

bxay

)arctan(b

Clemson University

Grasping experiments (video)

Clemson University

Conclusions: timing model

• A generic timing model for a robotic system using visual sensing, where the camera provides the desired position to the robot controller.

• We demonstrate how to obtain the values of the parameters in the model, using our camera network workcell as an example.

• Implementation to let our industrial manipulator intercept a moving object.

• Experimental results indicate that our model is highly effective, and generalizable.

Clemson University

Conclusions: dynamic manipulation

• Based on the timing model, we present a novel generic and simple theory to quantify the dynamic intercept ability of vision based robotic systems.

• We validate the theory by designing 15 sets of experiments (1050 runs), using two different end effectors under three different interference.

• The experimental results demonstrate that our theory is effective.

Clemson University

Conclusions: novel pneumatic hand

• A novel pneumatic three-finger hand is designed and demonstrated.

• It is simple, light and effective.

• Experimental results demonstrate that this novel pneumatic hand can grasp semi-randomly moving objects.

• Advantages– The compliance from pneumatics will allow the three-finger hand

to manipulate more delicate and fragile objects.

– In the experiments of grasping moving objects, unlike the traditional gripper, the contact position for this continuous finger is not very critical, which leaves more room for sensing error.

Clemson University

Sponsors

• The South Carolina Commission on Higher Education

• The Staubli Corporation

• The U.S. Office of Naval Research

Clemson University

Thanks

Clemson University

Conclusions: different manipulations

Manipulation

name

Basic robot motion

Impact time Success definition

Scoop Changing position and yaw orientation

Half way of scoop motion

Scoop the object out of table

Catch Changing position Finishing whole motion

Grab the object

Trap Changing position Half way of trap motion

Cover the object under the bowl

Ensnare Changing position and roll orientation

Finishing whole motion

Trap and then grasp the object

Clemson University

bump interference ramp interference

The distribution of |vi – vavg | in the balance ramp and bump cases.

Clemson University

Determining the Values

• An external camera to observe operation

• A conveyor moving in a fixed path at a constant velocity

• A light bulb as a tracking object

• A laser mounted in the end effector of the robot

Dynamic Workcell for Industrial Robots

Documents

Transcript of Dynamic Workcell for Industrial Robots