Outline Vision-Based Control of Unmanned Aerial Vehicles · Vision-Based Control of Unmanned Aerial...

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Vision-Based Control of Unmanned AerialVehicles

Using Type-1 and Type-2 Fuzzy Logic Controllers with ROS

Erdal Kayacan, Changhong Fu

School of Mechanical and Aerospace EngineeringNanyang Technological University (NTU), Singapore

Email: [email protected] | Web: http://www.erdalkayacan.com

FUZZ-IEEE 2017 TutorialJuly 9th, 2017Naples, Italy

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Outline

1. Motivation: Visual Control of UAVs Using FLCs with ROS

2. UAV and its Dynamic Model

3. Computer Vision for UAV Control

4. Type-1 and Type-2 FLCs for UAV Control

5. Robot Operating System (ROS) for UAV Application

6. UAV Simulation and Real Flight Demo

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Outline







FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

1. Motivation: Visual Control of UAVs Using FLCs withROS

� Multiple-rotor, fixed-wing and fixed-wing vertical take-off (VTOL) UAVs

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Why UAVs?more than 300 civilian UAV applications

*saving 10x time, 10x money and 10x lives ...

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Why Vision?

Onboard sensors for autonomous flights of UAVs:

However,� GPS: unreliable in urban environment, lost in indoor� IMU: low accuracy, accumulated error� Laser: limited range, heavy, expensive, high computing cost

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Why Vision?

camera

� small size, light, more flexible to mount on UAV, lessconsumption (1 USB interface, no extra powerrequirement)

� low cost, more options (different lenses), providericher environment information for UAV

� suitable for more UAV applications, e.g., detectdifferent objects with different filters (specialwavelengths)

different lenses filters

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Why Vision?Different onboard sensors used for the UAVs

2006

-2007

2007

-2008

2008

-2009

2009

-2010

2010

-2011

2011

-2012

2012

-2013

2013

-2014

2014

-2015

2015

-2016

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Publicationnumber

UAV + Computer VisionUAV + LaserUAV + GPS + IMU

� applications of all onboard sensors for UAVs are increasing year by year� vision-based applications are obviously more than the applications with other sensors, the

vision-based application is increasing faster

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Why FLCs?

UAV Control Scheme

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Why FLCs?

Classical controllers for UAVs:� proportional-integral-derivative

(PID)� linear quadratic regulator (LQR)� sliding mode control (SMC) ...

UAV platforms

However, classical controllers:� require a precise dynamic model of the UAV

↑ achieving an accurate mathematical model for such complex aerialvehicles is often time-consuming

� work under the assumption that significant internal and external uncertainties do notsubstantially affect the UAV systems ...

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Why FLCs?

In addition:� frequently-used sensors onboard UAVs, e.g., camera, often lack precise modeling� measurements consist of numbers of uncertain, incomplete and possibly inaccurate

information� the process of conducting a UAV application also contains numerous uncertain and

challenging factors ...

For example:

Left image Right image Depth image

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Why ROS?

� Working with different platforms:

Ubuntu Ubuntu (armhf) OS X Android

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Why ROS?

� Working with different robots:

https://vimeo.com/146183080

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Why ROS?

In addition:� Working with different camera sensors:

monocular RGB-D Stereo� Working with different simulation environments, e.g., Gazebo� Seamless interface, e.g., the C/C++ codes developed in the ROS-Gazebo simulation can be

used for the real applications/systems directly

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Outline







FUZZIE

EE20

17Tu

to

FUZZIE

EE20

17Tu

t

FUZZIE

EE20

17Tu

to

FUZZIE

EE20

17Tu

to


X4 Quadcopter

ritotori

torial

orial

orial

orial

Y6 Coaxial Tricopter

� {�xW ,�yW ,�zW } is the world fixed inertial reference frame,� {�xB ,�yB ,�zB} is the body frame,� ωi is the speed of the i-th propeller,� fi is the force generated by the i-th propeller,� {τφ ,τθ ,τψ} are torques around x , y and z axes, respectively.� T is the thrust force.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


Motion along z-axis

Motion along y -axis

FFF

Motion along x-axis

Motion around z-axis

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

3. UAV and its Dynamic ModelUAV dynamic model:

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎩

x = vx vx = cosφ cosψ sinθ+sinφ sinψm T

y = vy vy = cosφ sinψ sinθ−cosψ sinφm T

z = vz vz = cosφ cosθm T −g

φ = ωφ +sinφ tanθωθ +cosφ tanθωψ ωφ = Iyy −IzzIxx

ωθ ωψ + 1Ixx

τφ

θ = cosφωθ −sinφωψ ωθ = Izz−IxxIyy

ωφ ωψ + 1Iyy

τθ

ψ = sinφcosθ ωθ + cosφ

cosθ ωψ ωψ = Ixx−IyyIzz

ωφ ωθ + 1Izz

τψ

.

UAV state:

x =[

x y z φ θ ψ vx vy vz ωφ ωθ ωψ]T

[ {position} {attitude} {linear velocity}{angular velocity}]T.

UAV control inputs:u =

[T τφ τθ τψ

]T.

FUZZIE

EE20

1

FUZZIE

EE20

17

FUZZIE

EE20

17

FUZZIE

EE20

17


17Tu

tori

7 Tutor

ial

17Tu

torial

17Tu

torial

X4 Quadcopter

The relation between u and ωi is

⎧⎪⎪⎪⎨⎪⎪⎪⎩

T = b(ω2

1 + ω22 + ω2

3 + ω24

)τφ = sin π

4 bl(ω2

1 −ω22 −ω2

3 + ω24

)τθ = cos π

4 bl(−ω2

1 −ω22 + ω2

3 + ω24

)τψ = d

(−ω21 + ω2

2 −ω23 + ω2

4

) .

FUZZIE

EE20

1

FUZZIE

EE20

17

FUZZIE

EE20

17

FUZZIE

EE20

17


177T

17T

17T7 Tu

torial

Tutor

ial

7 Tutor

ial

7 Tutor

ial

Y6 Coaxial Tricopter

The relation between u and ωi is

⎧⎪⎪⎪⎨⎪⎪⎪⎩

T = b(ω2

1 + ω22 + ω2

3 + ω24 + ω2

5 + ω26

)τφ = sin π

3 bl(ω2

3 + ω24 −ω2

5 −ω26

)τθ = bl

(−ω21 −ω2

2 +cos π3 (ω2

3 + ω24 + ω2

5 + ω26 ))

τψ = d(−ω2

1 + ω22 −ω2

3 + ω24 −ω2

5 + ω26

) .

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Outline







FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

What is an image?

It is a two dimensinal (2D) numerical representation, i.e., a 2D function f (x ,y ) or m×n matrix of athree dimensional (3D) scene or object.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

RGB image and pixel values

An RGB image has three channels: red (R), green (G), and blue (B). RGB channels roughly followthe color receptors in the human eye, and are used in computer displays and imaging systems.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

What is image processing?

� It is a series of computer operations leading to some desired results, e.g., object detection,image content-based information retrieval and security.

Object detection

What is image processing?

� Autonomous robot control

FUZZIE

EE20

17Tu

tori lllalalal

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

tori lllalalal

FUZZIE

EE20

17Tu

tori lllalalal

Visual Odometry: it is the process of incrementally estimating the pose of the vehicle, e.g., UAV, byexamining the changes that motion induces on the images of its onboard cameras, e.g., monocular,stereo and RGB-D cameras

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Visual Odometry (VO)Two types of VO systems:

� direct method: it employs

� all image pixels (dense)� all pixels with sufficiently large intensity gradient (semi-dense)� sparsely selected pixels (sparse)

minimize a photometric error obtained by direct image alignment on the selected pixels.

Image alignment:

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Visual Odometry (VO)

� feature-based approach: it extracts a sparse set of features from each image, and thenmatch them across multiple frames. Camera poses and feature depths are estimated byminimizing the reprojection errors between feature pairs.

Feature detection and matching:

Indoor Outdoor Indoor & Outdoor

� feature descriptors are invariant to some extend to illumination and view-point changes� feature-based methods are more robust than direct methods to brightness inconsistencies

and large view-point changes among consecutive frames, etc

feature-based approach!


VO computes the camera path incrementally, i.e., pose after pose:

Image sequence

Feature detection

Feature matching (tracking)

Motion estimation

2D-2D 3D-3D 3D-2D

Local optimization Tk,k-1

Tk+1,k

Ck-1

Ck

Ck+1

* Scaramuzza, D., Fraundorfer, F., Visual Odometry: Part I - The First 30 Years and Fundamentals,IEEE Robotics and Automation Magazine, 18(4), 2011.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Pixel value variation over time

1

Pixel values on the 1st location165 170 175 180 185 190 195 200 205

Entri

es

0

100

200

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


However, besides the modeling errors of UAV and sensors,� Larger illumination changes� Faster rotations and translations� Less feature detection, etc

→ More uncertainties on the vision-based pose estimation output

Examples of feature tracking:

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Flight video:

https://youtu.be/xcFqg2ZTQTI

More details:Changhong Fu, Andriy Sarabakha, Erdal Kayacan, Christian Wagner, Robert John, and Jonathan M. Garibaldi. Input UncertaintySensitivity Enhanced Non-Singleton Fuzzy Logic Controllers for Long-Term Navigation of Quadrotor VTOL UAVs. In IEEE/ASMETransactions on Mechatronics, Submitted, December, 2016.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


FUZZZZUZZZZZUFUFUZUZUUFUZZ

FUZZZFUZZUZ

FUZZZFUZZZZZFUZUZZUZZ

FUZFUZZUZ

FUZFUZZFUZZZFUZZUZ

FUZZZFU

ZZFUZZZZFUZFUZZUZ

FUReal Flight in Corridor

Captured Images(illumination changes, blur motions, floor or wall

reflections etc)

Video: Real Flight Test

https://youtu.be/RgujyKVj50o

FUZZIE

EE20

FUZZIE

EE2

FUZZIE

EE20

FUZZIE

EE20


Library Environment (Indoor&Outdoor)

−20 −15 −10 −5 0 5−5

0

5

10

15

20

25

X Position (m)

Z Po

sitio

n (m

)

Our MethodLibviso2

Estimated Trajectories

Libviso2: http://www.cvlibs.net/software/libviso/

17Tu

toorialal

2017

Tutor

iuto

riuto

rial

2017

Tutor

iuto

riuto

riaalal

2017

Tutor

iuto

riuto

riaalal

3D Mapping of Library

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Outline







FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

How perfect world do we have?

”Given, for one instant, an intelligencewhich could comprehend all the forcesby which nature is animated and therespective situation of the beings whocompose it an intelligence sufficientlyvast to submit these data to analysis itwould embrace in the same formula themovements of the greatest bodies ofthe universe and those of the lightestatom. For it, nothing would beuncertain and the future, as the past,would be present to its eyes”.

— Pierre-Simon Laplace Less uncertainty

Less uncertainty

Less uncertainty

Less uncertainty

Less uncertainty

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Model-based and model-free control

When an engineer faces with an industrial control problem, two distinct forms of solutionapproaches exist in literature:

� the use of objective knowledge: mathematical models (model-based approach)

� the use of subjective knowledge: rules, expert information, design requirements (model-free

approach)� Artificial Neural Networks� Fuzzy Logic. FUZZ

IEEE

2017

Tutor

ial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Fuzzy logic theory

What is fuzzy logic?

“Fuzzy logic is a precise conceptual system of reasoning, deduction and computation in which theobjects of discourse and analysis are, or are allowed to be, associated with imperfect information.Imperfect information is information which in one or more respects is imprecise, uncertain,incomplete, unreliable, vague or partially true.”

— Lotfi A. Zadeh

“Imprecise enough but useful”

— Anonymous

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

A Boolean membership function

Sharp-edged! A small change in an input value may cause a big change in the output values.

Height (cm)

Degr

ee o

f mem

bers

hip

()

(Short) 0.0

(Tall) 1.0

170

Crisp membership function

Figure: A possible description of the vague concept “tall” by a crisp set

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Type-1 Fuzzy Membership Functions

No uncertainty in the membership function! There exits a clear (crisp) membership value for everyinput data point.

190 150

Two different fuzzy membership functions

160 180 Height (cm)

Degr

ee o

f mem

bers

hip

()

(Short) 0.0

(Tall) 1.0

Figure: A possible description of the vague concept “tall” by a fuzzy set

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Pros and Cons of Fuzzy Logic Control

Pros:� Ability to control a system without

having its mathematical model� Close to human way of thinking� Computation time is acceptable� Suitable for adaptive structures� Can incorporate a conventional

controller (PID, state feedback) andfine tune it

Cons:� Number of design parameters (scaling

factors, number and shape of themembership function, etc) is high

� No clear relationships between theseparameters and the controller’sperformance.

� No systematic way of analyzing thestability of the system.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Western and Eastern Perspectives

Western and Eastern perspectives on fuzzy logic control

China USA Taiwan India England Canada Turkey S. Korea Japan Iran0

100

200

300

400

500

600

Pub

licat

ion

num

ber

Fuzzy logic control

(a)

(a)

USA China England Germany Canada Italy Spain Switzerland

Netherlands

Australia0

500

1000

1500

2000

2500

Pub

licat

ion

num

ber

Model predictive control

(b)

(b)

Figure: Web of Science search for ”fuzzy logic control” (a) Web of Science search for ”modelpredictive control” (b)

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

A Type-1 Fuzzy Logic System Block Diagram

Figure: A Type-1 Fuzzy Logic System Block Diagram

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Fuzzifier

Singleton fuzzifier:

μ∑j (σj ) =

{1, σj = σ ′

j

0, σj �= σ ′j

.

Non-singleton fuzzifier:

μ∑j (σj ) = exp

[− (σj −σ ′

j )2

2σ2F ,j

].

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Non-Singleton Type-1 Fuzzy Logic Control

Types of prefilters:� standard,� centroid,� similarity (Best!).

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Video

� Flight video:

https://youtu.be/NVfgz38RFuA

More details:Changhong Fu, Andriy Sarabakha, Erdal Kayacan, Christian Wagner, Robert John, and Jonathan M. Garibaldi. Similarity-BasedNon-Singleton Fuzzy Logic Control for Improved Performances in UAVs. In 2017 IEEE International Conference on Fuzzy Systems(FUZZ-IEEE), July 2017.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

A Type-2 Fuzzy Logic System Block Diagram

Although the structure in the figure below brings some advantages when dealing with uncertainties,it also will increase the computational burden!

Figure: A Type-2 Fuzzy Logic System Block Diagram

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Type-1 vs Type-2

� Type-1 fuzzy sets two dimensional� Type-2 fuzzy sets three dimensional� Membership grade in [0,1]� Membership grade is an interval� Type-1 fuzzy systems computationally cheap� Type-2 fuzzy systems computationally expensive

Interval Type-2 Membership Function

ZZZZFUZZ

I

UZZIE

UZZIE

ZZZZZZZZZZZ

EE20

17Tu

toriallalllll

ZFUZZ

IEZZ

IZZ

IZZZZZ

EE20

17Tu

toriall

ZZZZIE

ZZI

ZZI

FUZZIE

UZZIE

UZZIE

ZZI

ZI

ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZEE

2017

Tutor

iallalllalalalalaaalallall

ZZZIE

ZZZFUZZ

IE

UZZIE

UZZIE

ZZI

ZZZZZZZZZZZZZZZZZZZZZZZZZZZZEE

2017

Tutor

iallallllalalalallall

Triangular interval type-2 membership function:

⎧⎪⎨⎪⎩

μ Akj

(σj ) = max(

min( σj−ai−1

ai−ai−1,

ai+1−σjai+1−ai

),0)

μAk

j(σj ) = mk

j μ Akj

(σj ).

Symmetrical MFs are employed to simplify the design complexity:

m1j = m2

j = m3j = αj .

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Single Input Interval Type-2 Fuzzy PID

Fuzzy mapping:

ϕ(σ ) =σ2

(1

α + |σ |−α|σ | +−1+ α

−1+ α |σ |).

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Single Input Interval Type-2 Fuzzy PID

PS-1: aggressive αp , moderate αi , aggressive αdPS-3: aggressive αp , moderate αi , smooth αdPS-5: moderate αp , moderate αi , moderate αdPS-7: smooth αp , moderate αi , aggressive αdPS-9: smooth αp , moderate αi , smooth αd

x-axis tracking. y -axis tracking. z-axis tracking.

More details:Andriy Sarabakha, Changhong Fu, Erdal Kayacan, and Tufan Kumbasar. Type-2 Fuzzy Logic Controllers Made Even Simpler: From Designto Deployment in Real-Time for Quadcopter UAVs. IEEE Transactions on Industrial Electronics. Submitted for publication.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Video

Flight video (Real Flight):

https://youtu.be/Hz3f8N-30dg

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Double Input Interval Type-2 Fuzzy PD

Control surfaces:

-1

1 1

0.5 0.5

σ1σ2

0 0

0

ϕ(σ1,σ2)

-0.5-0.5

-1 -1

1

PS-1: α1 = 0.1,α2 = 0.1

-1

11

0.5 0.5

σ1σ2

00

0

ϕ(σ1,σ2)

-0.5-0.5

-1 -1

1

PS-2: α1 = 0.5,α2 = 0.5

-1

11

0.5 0.5

σ1σ2

0 0

0

ϕ(σ1,σ2)

-0.5 -0.5

-1-1

1

PS-3: α1 = 0.9,α2 = 0.9

More details:Andriy Sarabakha, Changhong Fu, and Erdal Kayacan. Double-Input Interval Type-2 Fuzzy Logic Controllers: Analysis and Design. In 2017IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), July 2017.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Double Input Interval Type-2 Fuzzy PD

x-axis tracking. y -axis tracking. z-axis tracking.

3D trajectory tracking.

DI-IT2-FPID controller PS-1 PS-2 PS-3

Euclidean MSE, [m] 0.777 0.748 0.821

Overshoot, [m] 0.405 0.069 0.031

Rise time, [s] 0.80 1.18 3.32

Settling time (5%), [s] - 1.71 2.53

Table: Control characteristics.

More details:Andriy Sarabakha, Changhong Fu, and Erdal Kayacan. Double-Input Interval Type-2 Fuzzy Logic Controllers: Analysis and Design. In 2017IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), July 2017.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Video

Flight video (Real Flight):

https://youtu.be/4U7BHQFo9boFUZZ

IEEE

2017

Tutor

ial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Outline







FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Review of ROS

� a peer-to-peer robot middleware package� a ”meta” operating system� a collection of packaging, software building tools� an architecture for distributed inter-process or

inter-machine communication and configuration� a language-independent architecture, including

C++, java, python, lisp, etc� provides development tools for system runtime

and data analysis

Groovy Fuerte Electric Hydro Indigo

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

How to install the ROS

Instruction → http://wiki.ros.org/kinetic/Installation/Ubuntu

The main steps include:

� Setup your sources.list

sudo sh -c ’echo ”deb http://packages.ros.org/ros/ubuntu $(lsb release -sc) main” >/etc/apt/sources.list.d/ros-latest.list’

� Set up your keys

sudo apt-key adv –keyserver hkp://ha.pool.sks-keyservers.net:80 –recv-key421C365BD9FF1F717815A3895523BAEEB01FA116

� make sure your Debian package index is up-to-date

sudo apt-get update� Desktop-Full Install: (Recommended)

sudo apt-get install ros-kinetic-desktop-full� Initialize rosdep

sudo rosdep initrosdep update

� Environment setup

echo ”source /opt/ros/kinetic/setup.bash” >> /.bashrcsource /.bashrc

How ROS works

ZZIE

EE20

17Tu

torial

ZIE

EE20

17Tu

torial

ZZIE

EE20

17Tu

torial

ZZIE

EE20

17Tu

torial

ROS master

UAV node

Image processing node

Camera node

Control node

Publish images on topic “image”

Receive velocities on topic “cmd_vel”

Receive position on topic “position” and publish velocities on

topic “cmd_vel”

Receive images on topic “image” and publish position on

topic “position”

Camera Onboard Computer UAV

Example of ROS-based UAV navigation using vision-based position estimation

ROS master

EE20

17Tu

torial

E20

17Tu

torial

EE20

17Tu

torial

EE20

17Tu

torial

ROS master

UAV node


Camera node

Control node




topic “cmd_vel”




ROS Master� provides naming and registration services to the rest of the nodes in the ROS system� publishers and subscribers to topics as well as services� enables individual ROS nodes to locate one another. Once these nodes have located each

other they communicate with each other peer-to-peer, etc.

ROS nodes

EE20

17Tu

torial

E20

17Tu

torial

EE20

17Tu

torial

EE20

17Tu

torial

ROS master

UAV node


Camera node

Control node




topic “cmd_vel”




Nodes� processes those perform computations� combined together into a graph and communicate with one another using streaming topics,

and the Parameter Server, etc.

ROS topics

IEEE

2017

Tutor

ial

EEE20

17Tu

torial

IEEE

2017

Tutor

ial

IEEE

2017

Tutor

ial

ROS master

UAV node


Camera node

Control node




topic “cmd_vel”




Topics:

� named buses over which nodes exchange messages

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

ROS code example: talker

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


is a convenience include that includes all the headers necessary to use the most common public pieces of the ROS system.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


This includes the std_msgs/String message, which resides in the std_msgs package.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


Initialize ROS. This allows ROS to do name remapping through the command line

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


Create a handle to this process' node.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

ROS code example: talkerTell the master that we are going to be publishing a message of type std_msgs/String on the topic chatter. This lets the master tell any nodes listening on chatter that we are going to publish data on that topic.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


A ros::Rate object allows you to specify a frequency that you would like to loop at.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


ros::ok() will return false if: • a SIGINT is received (Ctrl-C) • we have been kicked off the

network by another node with the same name

• ros::shutdown() has been called by another part of the application.

• all ros::NodeHandles have been destroyed

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


Broadcast a message on ROS FUZZ

IEEE

2017

Tutor

ial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


ROS_INFO is the replacement for printf/cout.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


ros::spinOnce() will call all the callbacks waiting to be called at that point in time.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


use the ros::Rate object to sleep for the time remaining to let us hit our 10Hz publish rate

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


• Initialize the ROS system • Advertise that we are going to be publishing std_msgs/

String messages on the chatter topic to the master • Loop while publishing messages to chatter 10 times a second

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

ROS code example: listener

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


This is the callback function that will get called when a new message has arrived on the chatter topic.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


Subscribe to the chatter topic with the master. ROS will call the chatterCallback() function whenever a new message arrives.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


ros::spin() enters a loop, calling message callbacks as fast as possible.

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Initialize the ROS system� Subscribe to the chatter topic� Spin, waiting for messages to arrive� When a message arrives, the chatterCallback() function is called

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Outline







FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Gazebo Simulator� an essential tool in every roboticist’s toolbox� well-designed simulator to rapidly test

algorithms, design robots, perform regressiontesting and train AI system using realisticscenarios

� offers the ability to accurately and efficientlysimulate populations of robots in complex indoorand outdoor environments

� a robust physics engine, high-quality graphics,and convenient programmatic and graphicalinterfaces

� free with a vibrant community, etc

� Real UAVs for Gazebo simulator

......

Different quadcopters and Y6 tricopter (more UAVs)

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Gazebo Simulator

� three different types of UAVs shown in the Gazebo simulator

Screenshot for Gazebo simulator (Quadcopters and Y6 Tricopter)

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Gazebo Simulator - rqt graph (Namespaces)

Screenshot for all communications in Gazebo simulator

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Gazebo Simulator - rqt graph (Nodes)


FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Gazebo Simulator - rqt graph (Topics)


FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Gazebo Simulator - Comparison Demo� Comparison of parameter settings in DI-IT2-FLC

Screenshot for Gazebo simulator with three bebop UAVs

Red → 0.1, Green → 0.4, Blue → 0.95

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Gazebo Simulator - Comparison Demo� 4 waypoints

1st waypoint 2nd waypoint

3rd waypoint 4th waypoint

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Gazebo Simulator - Comparison Demo

Screenshot for Rviz with three bebop UAVs

Red → 0.1, Green → 0.4, Blue → 0.95

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Online tuning using rqt reconfigure

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial


� Performance check using rqt plot

Screenshot for rqt plot with three bebop UAVs

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Real Flight Demo: Target Following

Real UAV: Bebop Target

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

For better future, vision for robotics!

School of Mechanical and Aerospace EngineeringNanyang Technological University (NTU), Singapore

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

FUZZIE

EE20

17Tu

torial

Thank you for yourattention

Your questions, suggestions and comments are welcome!

Contact:[email protected]

C++ codes:https://github.com/andriyukr/FLC

Outline Vision-Based Control of Unmanned Aerial Vehicles · Vision-Based Control of Unmanned Aerial...

Documents

Transcript of Outline Vision-Based Control of Unmanned Aerial Vehicles · Vision-Based Control of Unmanned Aerial...