Learning Ball-balancing Robot Through Deep Reinforcement Learning

Yifan Zhou1, Jianghao Lin2, Shuai Wang3, Chong Zhang3

Abstract— The ball-balancing robot (ballbot) is a good plat-form to test the effectiveness of a balancing controller. Con-sidering balancing control, conventional model-based feedbackcontrol methods have been widely used. However, contacts andcollisions are difficult to model, and often lead to failure inbalancing control, especially when the ballbot tilts a large angle.To explore the maximum initial tilting angle of the ballbot,the balancing control is interpreted as a recovery task usingReinforcement Learning (RL). RL is a powerful technique forsystems that are difficult to model, because it allows an agentto learn policy by interacting with the environment. In thispaper, by combining the conventional feedback controller withthe RL method, a compound controller is proposed. We showthe effectiveness of the compound controller by training anagent to successfully perform a recovery task involving contactsand collisions. Simulation results demonstrate that using thecompound controller, the ballbot can keep balance under alarger set of initial tilting angles, compared to the conventionalmodel-based controller.

Keywords—Ball-balancing robot; Balance control; Reinforce-ment Learning.

I. INTRODUCTION

Ballbot is a dynamically-stable mobile robot balancing ona single spherical wheel. This unique way of moving makesballbot move in omnidirectional and thus exceptionally agileand maneuverable compared to other ground vehicles. Also,continuously position adjusting allows robot to be tall enoughfor human-robot interaction and having large accelerationswithout tipping over. Balancing on a ball, apart from theimpressive look and behavior, also makes the robot moreapplicable to human environments. The underlying dynamicsof robotic system is highly nonlinear and unstable, whichrequires full system dynamics planning and control. Whileconventional feedback control method can solve task suchas balancing or velocity control in omnidirectional direction,applications that involve contacts between the wheel and theball, and between the ball and the ground are difficult toapproach with conventional control method. We usually usepure viscous damping friction model, neglecting the staticfriction and nonlinear dynamic friction. Also, the large tiltangle will lead to the no-contact, collision and additionalforce, which cannot be modeled. Identifying and modelingcontacts and friction under these circumstances is difficult,even if a reasonable contact behavior is provided by acarefully identified physical model. Hence, it is often difficult

1Y. Zhou is with Zhejiang University, Hangzhou, China (e-mail: yi-fanzhou7@gmail.com)

2J. Lin is with Shanghai Jiao Tong University, Shanghai, China (e-mail:chiangel@sjtu.edu.cn)

3S. Wang and C. Zhang are with Tencent Robotics X, Shenzhen, China(e-mail: {shawnshwang, chongzzhang}@tencent.com)

to achieve adaptable yet robust behavior as soon as theunmodelable elements are introduced.

RL algorithms have been widely used to learn policiesfor unstructured environments, because they allow agent tointeract with their environments continuously and to iteratethe policy spontaneously. Moreover, RL can handle controlproblems that are difficult to approach with conventional con-trollers because the control goal can be specified indirectlyas a term in a reward function and not explicitly as theresult of a control action [1]. However, standard RL methodscannot be solely used in the ballbot, since it requires the robotlearn through interaction, which can lead to instability. Andcollecting the amount of interaction that is needed to learna complex skill can be time-consuming.

In this paper, we attempt to combine the benefits of bothconventional feedback control and RL. We study controlproblem that the ballbot have large initial tilt angle, whichcannot be described by the model. The problems possessthe modelable structure that can be partially handled withconventional feedback control. And the unmodelable part ofthe control task, which must consider contacts and friction,is solved with RL.

The rest of the paper is organized as follows. In SectionII, related work on ballbot is reviewed and followed by theRL in control problems. Section III provide an overview ofballbot system and introduce the fundamental of RL. In Sec-tion IV, we present the control system. We give the dynamicmodeling of the ballbot and design a feedback controller,followed by the introduction of the DDPG method. SectionV shows the simulation results implemented on ballbot, whileSection VI concludes the paper.

II. RELATED WORK

A. Ballbot

Many ball-balancing robots have been built in the pastyears. The main ballbot research platform is shown in Fig.1. The CMU Ballbot [2] used an inverse mouse-ball drivemechanism, consisting of four rollers to drive the ball. BallP[3], Rezero [4], Kugle [5] use three omni-wheels to drive theball. A tall robot is built up for the convenince to interactwith people. With a small leaning angle, the projection ofthe center of mass on the horizontal plane moves out of itssupporting triangle. To increase the contact force or avoid thefact that the ball gets pushed out during large inclinations,the Kugle [5] robot is equipped with a skirt and the Rezero[4] robot is equipped with ball arrester, which hold the ballin place. These equipment reduce the unmodelable elements’influence on the ballbot.

Fig. 1. Main ballbot contributions: (a) Ballbot from CMU; (b) “BallIP”from TGU; (c) “Rezero” from ETH (d) “Kugle” from AAU.

In [2] [3] and [4], linear feedback control approachesare used to maintain balance and achieve motion. TheCMU ballbot uses an inner loop that maintains the body atdesired inclination angles and an outer loop linear quadraticregulator (LQR) controller that achieves desired ball motionsby commanding body angles to the inner controller. Later in[3] [6], CMU ballbot is introduced with arms and uses itsplanning procedure to plan in the space of both body leanangles and arm angles to achieve desired ball motions. TheKugle robot uses both linear feedback controllers [5] andnonlinear sliding mode is designed to control the body anglesto follow a given path.

B. RL in Control Problems

With RL getting increasingly popular, a large number ofnew complex robot control applications have been achieved(e.g., solving Rubik’s Cube [7], learning dexterous in-handmanipulation [8], learning agile walk for legged robots [9],and trajectory tracking of a rotorcraft aerial robot [10]).

However, the dynamically stable robot is much less in-vestigated due to the inherent instability which makes it im-possible to collect enough data. And the random initial willeasily cause robot tipping over or being destroyed. Hence,approaches that combine the convectional control methodand the RL was proposed. A residual RL is used in robot armcontrol. The dynamical system consists of a fully actuatedrobot and underactuated objects. By decomposing them intoa part that is solved efficiently by conventional feedback

Fig. 2. The model of the ballbot: (a) Top view of the ballbot; (b) Ballbotis approximated into two decoupled plant.

control methods, and the residual which is solved with RL,an agent was trained successfully to perform a real-worldblock assembly task involving contacts and unstable objects[11]. For the ballbot, a two-step algorithm is introducedwhich combines the proven performance of a model-basedcontroller with a model-free method for compensating formodel inaccuracy [12]. The authors build a linearized modelof ballbot and design an iLQG controller, which can alreadystabilize the unstable system, enabling the learning algorithmto sample useful data. For the subsequent learning, the RLalgorithm uses the designed iLQG controller as an optimizedinitial guess for the parameters for they share the same costfunction. In this method, the RL is used to adapt the model-based controller to the real environment, rather than copewith the unmodelable problems.

III. PRELIMINARIES

In this section, we provide an overview of ballbot system.Also, the RL that we build on in our approach is introduced.

A. System Overview

The ballbot system consists of three rigid bodies: the ball,the actuating wheels and the body. The ball transmits andbears all arising forces, and the actuating wheels are threeomniwheels mounted with 120◦ spacing. Additionally, thefollowing assumptions are made [2]:• Rigid Body Assumption: In this model, all the objects

are considered as rigid bodies, which means all thedeformation should be neglected.

• Friction Assumption: Friction between the wheel and thefloor and between the wheel and the body is modeledas pure viscous damping. Forces due to static frictionand nonlinear dynamic friction are neglected.

• No Slip Assumption: There is no slip between the balland floor and between the ball and the wheel.

The three contacting points on the ball form a supportingtriangle, which is essential to the system. Consider two casesof the ballbot shown in Fig. 2 (a). As the first case, the ballbot

Fig. 3. Structure of the system.

tilt a small angle and its center of gravity is directly abovethe supporting triangle, which guarantees that all wheelsare contacting with the ball. However, for some reason, theinclination angles of a ballbot increase to a large extent,leading to the center of gravity out of the supporting triangle.With the gravity torque, the contact condition and the frictionchange a lot, thus the assumptions we made when modelingthe robot are no longer satisfied. More specifically, there is apractical limit on the magnitude of the tilt angle. Exceedingthis limit induces lots of unpredictable factors not present inthe ballbot model. The unpredictable factors such as contactand friction in this model makes it impossible to identifythe model accurately. Based on the discussion above, weconsider the ballbot a system that consists of modelable partand unmodelable one.

B. Reinforcement Learning

In RL, we consider an agent interacting with an environ-ment E over discrete timesteps, which can be modeled as aMarkov Decision Process. At every timestep t, the agent is ina state st, takes an action at, and the environment providesagent with a scalar reward rt and the state st+1. An agent’sbehavior is defined by a policy π, which maps states to aprobability distribution over actions π : S → P(A). Policylearning can be subdivided into two categories: 1) Stochastic2) Deterministic. Stochastic policy learns the conditionalprobability of taking an action a, given that it is in a state s,π(a|s) = P [At = a|St = s]. A deterministic policy on theother hand learns the action as a function of state, a = π(s),which means that the agent will give a deterministic actionin each state s.

We denote the return by Rt =∑Ti=t γ

(i−t)ri, where T isthe horizon that the agent optimizes over and γ ∈ [0, 1] is adiscount factor for future rewards. In RL, the agent aims tolearn a policy ut = π(st) to maximize the expected return.

IV. BALANCE CONTROL

Based on the analysis in Section III, we introduce acontrol system that consists of two parts. The first part is

a conventional model-based linearized feedback controllerthat is able to keep the ballbot balance around the balancingpoint. It might have decreased performance on unmodelableconditions. And RL method is superposed and maximize thereward function we define. The control structure is illustratedin Fig. 3. In this way, the performance near the equilibriumwill be guaranteed, and the iteration time may be decreased.

A. Task Definition

In this paper, we aim to design a controller that performsbetter than the conventional controller in the unmodelablesection. To illustrate the ability of two controllers, we choosea balance recovery task, aiming to stabilize the unstablesystem from a tilt angle (with zero angular velocity) andkeep balance for a period of time.

Now we will give the parameters to describe the initialstate of the robot. For the ballbot standing on the ball withall three omniwheels contacting with ball, the body can beregarded revolving around the center of the ball. As shownin Fig. 4, the orientation of the ball can be described: 1)rotates α around the z-axis of the ball 2) rotates β aroundthe y-axis of the ball. Generally speaking, the body has 2DOFs: the inclination angle β and the direction α to whichthe robot tilts.

B. Conventional Feedback Controller

1) Simplified Mathematical Model: A nonlinear 3D modelof the system dynamics of ballbot has been analyticallyderived in [2], where the wheel dynamics are neglected. Inthis model, the three-dimensional system is approximated inthree decoupled planes. As shown in Fig. 2 (b), the mediansagital plane (xz-plane) and the median coronal plane (yz-plane) related to the system moving forward or backwardon the flat floor are identical and share same equations ofmotion. Wheels are simplified as one actuating wheel that fitsin a plane. The third plane (xy-plane) describes the rotationaround the z-axis in the body fixed reference frame, to whichwe pay less attention in balance control. Under these cir-cumstances, the controller for the three-dimensional systemcan be designed by analysing and designing independentcontrollers for the two separate and identical systems.

We choose the median sagital plane to derive the model. Indetail, the body angle θ represents the change of ball positioncalculated by x, which is read from sensors.

θ =x

rb, (1)

where rb is the radius of the ball. Another configuration is φ,aiming to describe the position of the robot above the ball.φ is accessed by IMU. The mass of the ballbot and the ballare given by mB and mb respectively. The distance from thecenter of mass of the ballbot to the center of mass of bodyis given by l. The Lagrangian formulation is used to derivethe nonlinear equations of motion for the simplified model.According to Euler-Lagrange equations, the energies of thesystem obeys:

d

dt

(∂L∂q

)− ∂L∂q

= Q, (2)

where L is the difference between the kinetic energy andthe potential energy, q is generalized coordinate, q is thederivative of generalized coordinate q, and Q is the externalforce, including the friction and the torque between the balland the wheel.

The kinetic energy Kb of the ball is computed withrotation energy and transnational kinetic energy:

Kb =Ibθ

2

2+mb(rbθ)

2

2, (3)

where Ib, mb, and rb are the moment of inertia, the massand the radius of the ball respectively. The potential energyof the ball is Vb = 0. The kinetic energy KB and potentialenergy VB of the body are

KB =mB

2(r2b θ

2 + 2rbl(θ2 + θφ) cos(θ + φ) + l2(θ + φ)2)

+IB2

(θ + φ)2,

VB =mBgl cos(φ+ θ),(4)

where IB is the moment of inertia of the body with respectto the center of the ball, mB is the mass of the body, and gis the gravity acceleration. The total kinetic energy is K =Kb + KB and the total potential energy is V = Vb + VB .Define the system configuration vector q =

[θ φ

]T. The

Lagrangian L is a function of q and q :

L(q, q) = K − V = KB +Kb − VB − Vb. (5)

Let τ be the component of the torque applied between theball and the body in the direction normal to the plane. Tomodel the viscous friction terms, define the vector

D(q) =

[µθ θ

µφφ

], (6)

where µθ and µφ are the viscous damping coefficients thatmodel ball-ground and ball-body friction, respectively. Usingthis notation, the Euler-Lagrange equations of motion for thesimplified ballbot model are

d

dt

(∂L∂q

)− ∂L∂q

=

[0τ

]−D(q). (7)

After computing the derivatives in the Euler-Lagrange equa-tions and rearranging terms, the equations of motion can beexpressed as

M(q)q + C(q, q) +G(q) +D(q) =

[0τ

]. (8)

The mass matrix M(q) is

M(q) =

[Γ1 + 2mBrbl cos(θ + φ) Γ2 +mBrbl cos(θ + φ)Γ2 +mBrbl cos(θ + φ) Γ2

],

(9)where

Γ1 = Ib + IB +mbr2b +mBr

2b +mBl

2, (10)

Γ2 = IB +mBl2. (11)

The vector of Coriolis and centrifugal forces is

C(q, q) =

[−mBrbl sin(θ + φ)(θ + φ)2

0

], (12)

and the vector of gravitational forces is

G(q) =

[−mBgl sin(θ + φ)−mBgl sin(θ + φ)

]. (13)

Premultiply both sides of (8) by the inverse of matrix M :

q = M(q)−1([0τ

]− C(q, q)−G(q)−D(q)

). (14)

Rewrite the expression:

q =

[θ

φ

]=

[b(Γ2+lmBrb cos(φ+θ))−aΓ2

ca(Γ2+lmBrb cos(φ+θ))−b(Γ1+2lmBrb cos(φ+θ))

c

],

(15)where

a =mBgl sin(φ+ θ)− µθ θ + lmBrb sin(φ+ θ)(φ+ θ)2,

b =τ − µφφ+ glmB sin(φ+ θ),

c =Γ22 − Γ1Γ2 + l2m2

Br2b cos2(φ+ θ).

(16)

To put these equations into standard nonlinear state spaceform, define the state vector to be x =

[qT qT

]Tand define

the input u = τ . The relationship sin(θ + φ) = θ + φ andφ = θ = 0 are satisfied when the ballbot system is at theequilibrium point. Then the linearized state space model canbe described as: { .

x = Ax+Bu

y = Cx+Du,(17)

where the inner state is

x =[θ φ θ φ

]T, (18)

and the system input is

u = τ (19)

with

A =

0 0 1 00 0 0 1

m2Bl

2grbH

m2Bl

2grbH

µθΓ2

H−µφFH

mBgl(F−G)H

mBgl(F−G)H

−µθ(Γ2+mbrbl)H

µφGH

,(20)

B =[0 0 F

H −GH

]T, (21)

where F = Γ2 + mBrbl, G = Γ + 2mBrbl and H = Γ22 −

Γ1Γ2 + m2Bl

2r2b . C and D are the identity matrix and zero

matrix with appropriate dimensions respectively.2) Design of Controller: Now a linear state feedback

controller can be used that stabilizes the system about φ = 0.We apply the state feedback control law

u = −Kx (22)

to system, and we choose the structure of K to be

K =[k1 k2 k3 k4

]. (23)

For the linearized system, a control law can be designed toensure stability by traditional linear method.

C. Balance Control Based on Deep Reinforcement Learning

In this paper, we use DDPG algorithm [13], a model-free, off-policy actor-critic algorithm using deep functionapproximators that provides us with a good frameworkto train the neural network for learning highly non-linearfunctions.

1) Deep Deterministic Policy Gradient: DDPG is anactor-critic policy learning method [14] with additional targetnetworks to stabilize the learning process. The action-valuefunction describing the expected return after taking an actionat in the state st:

Qπ(st, at) = Eri≥t,si>t∼E,ai>t∼φ[Rt|st, at], (24)

and the recursive version of (24), known as the Bellmanequation, is as follows:

Qπ(st, at) = Ert,st+1∼E [r(st, at)+γEat+1∼π[Qπ(st+1, at+1)]](25)

DDPG is based on DQN [15], a commonly used off-policy algorithm. It uses the greedy policy π(s) =arg maxaQ(s, a). We consider function approximators pa-rameterized by θQ, which we optimize by minimizing theloss:

L(θQ) = Est∼ρβ ,at∼β,rt∼E [(Q(st, at|θQ)− yt])2, (26)

where

yt = r(st, at) + γQ(st+1, µ(st+1)|θQ). (27)

In DDPG algorithm the target policy is deterministic Wecan describe it as a function µ : S → A and avoid the innerexpectation:

Qµ(st, at) = Ert,st+1∼E[r(st, at) + γQµ(st+1, µ(st+1))

](28)

The actor is updated by following the applying the chainrule to the expected return from the starting distribution Jwith respect to the actor parameters:

∇θµJ ≈Est∼ρβ [∇θµQ(s, a|θQ)|s=st,a=µ(st|θµ)]

=Est∼ρβ [∇aQ(s, a|θQ)|s=st,a=µ(st)∇θµµ(s|θµ)|s=st ](29)

To guarantee the samples’ independence and identicaldistribution, a replay buffer R is used, which also makes effi-cient use of hardware optimizations. Transitions are sampledfrom the environment according to the exploration policyand the tuple (st, at, rt, st+1) is stored in the replay buffer.When this buffer becomes full, oldest samples are discarded.The actor and critic are updated by sampling a minibatchuniformly from the buffer.

Additionally, DDPG treat the problem of exploration in-dependently from the learning algorithm to deal with the inexploration in continuous action space. So the explorationpolicy µ′ is constructed by adding noise sampled from anoise process N to the actor policy µ

µ′(st) = µ(st|θµt ) +N . (30)

When updating the target values, DDPG uses “soft” targetupdate, rather than directly copying the weights: θ′ ← τθ+(1−τ)θ′ with τ � 1, ensuring the consistently train withoutdivergence. For more details, refer to Algorithm 1.

Algorithm 1: DDPG algorithmRandomly initialize critic network Q(s, a|θQ) and actor policyµ(s|θµ) with weights θQ and θµ.Initialize target network Q′ and µ′ with weights θQ

′← θQ,

θµ′← θµ

Initialize relay buffer Rfor episode =1→M do

Initialize a random process N for action explorationReceive initial observation state s1for t = 1→ T do

Select and execute action at = µ(st|θµ) +NtObserve reward rt and new state st+1

Store transition (st, at, rt, st+1) in RSample a minibatch of transitions (si, ai, ri, si+1) from R

Set yi = ri + γQ′(si+1, µ(si+1|Qµ′)|θQ

′)

Update critic network by minimizing the loss function:

L =1

N

∑i

(yi −Q(si, ai|θµ))2

Update actor policy using the sampled policy gradient:

∇θµJ ≈1

N

∑i

∇aQ(s, a|θQ)|s=si,s=µ(si)∇θµµ(s|θµ)|si

Update the target networks:

θQ′← τθQ + (1− τ)θQ

′

θµ′← τθµ + (1− τ)θµ

′

end forend for

2) State Vector And Network Architecture: Consistentwith the conventional controller, the chosen state vectorconsists of the body angle φ and the angular velocity φ inboth the median sagital plane and the median coronal plane.As the large inclination angle will cause the problem that theball might be pushed out, only this state cannot identify thefull state of the ballbot and the ball. We introduce the theattitude of the robot z to identify the full state of the ballbot.

And the action vector is the target velocity of the virtualwheel in each plane, which can be converted into the velocityfor the real three motors. Considering the ability to improvethe performance of the conventional controller, we decide theamplitude of the action approximately equals the max speedthat the motor will reach.

The main object of our task is to provide a set of motorinput that are needed to keep the ballbot in balance. The onlyelement included in the reward function is β, the includedangle between the z-axis in the body fixed reference frameand the vertical direction. The final reward function is asfollows

rt = −||β||+ c, (31)

where c is a constant to make sure the reward function isalways positive.

Fig. 4. Parameters used to describe the initial state.

V. EXPERIMENTSWe evaluate our method in simulation. In this section,

we start by reporting how the experimental system is setup. Then the application of conventional feedback controlmethod is detailed, followed by the improvement usingDDPG algorithm. Several experiments are implemented, withresults being analyzed.

A. Experimental Setup

We use CoppeliaSim 4.1.0 shown in Fig. 5, a full-featuredsimulator for model-based optimization considering bodycontacts, to evaluate our method in simulation. This envi-ronment consists of a ballbot with three omni wheels, andwithout equipment such as skirt or ball arrester. The omni-wheel is driven by a motor controlled by a speed incrementalcontroller. Also, we assume the full robot state informationis available thanks to the virtual sensor on the body. Tospeed up the training process, we use the RL library, stable-baselines, which is an open-source library for reinforcementleaning that offers a simple interface to train, evaluate agentsand do hyperparameter tuning for a variety of applications.

The neural architectures for actor and critic network areboth two fully connected Multi-Layer Perceptron with layernormalization. The other hyperparameters we used are listedin Table I.

TABLE IHYPERPARAMETERS USED FOR DDPG

Hyperparameter ValueHardware configuration 1 CPU + 4 CPU coresOptimizer AdamActor learning rate 0.0001Critic learning rate 0.0001Weight decay regularization 0.00001Minibatch size 128Replay buffer size 50000Constant c in reward function 0.7

B. Experimental Results

In the test run shown in Fig. 6 and Fig. 8, a ballbotwas commanded to recover from different initial states, with

Fig. 5. Ballbot in CoppeliaSim.

conventional controller and the compound controller respec-tively. The radial coordinate represents the angle β, and theangular coordinate represents the angle α. All angles areexpressed in degrees. Yellow points indicate that the ballbotcan recover at those points, and purple points otherwise. Thethree lines mounted with 120◦ spacing represent the wheeldirection. Boundaries between yellow and purple points inboth figures are roughly triangular. Define the yellow partas recovery area. Compared with the conventional con-troller, the compound controller has larger recoverable area.However, the ballbot has the least tilt angle β with eitherconventional controller or the compound controller in thedirection of the three wheels (α = 90◦, 210◦ or 330◦). Thecompound controller has very limited expending capacity atthese directions. When the ballbot tilts a large angle in thedirection α = 90◦, only one omniwheel contacts with theball. The force between the ball and the wheel is in thetangential direction. The motor rotation contributes less to thebalancing. When α = 30◦, 150◦ or 270◦, the ballbot has twowheels contacting with the ball when α = 30◦, which allowsthe ball moving under two DOFs. Under these α angles, βcan be up to 12◦ ∼ 13◦. Although the RL-based controllerlearns efficiently through contacting with the environment, itcannot break physical limitations of the system.

Detailed states and inputs of the ballbot with respect totime are shown in Fig. 7 and Fig. 9. When α = 210◦,β = 7.5◦, the ballbot with conventional controller fallsover but the one with compound controller recovers. Bodyangles φx, φy and the angular velocities of three motorsare depicted. Oscillation exists during the adjusting time.Under this situation, the three wheels change their rotatingdirections rapidly and the ballbot “kicks” the ball to keepbalance. Collision, bounce and slipping exist in this system,which are unpredictable and unmodelable.

VI. CONCLUSION

In this paper, RL algorithm is superposed to the conven-tional controller, making up a compound controller. Boththe conventional feedback controller and the compound con-troller can stabilize the ballbot system, while the compoundcontroller has larger recovery area. When the compoundcontroller tries to balance the ballbot from an initial state that

Fig. 6. The recovery area of the conventional controller.

Fig. 7. The body angle φx ,φy and the rotation velocity of three motorswith conventional controller in 1.25s.

the conventional controller fails, the ballbot “kicks” the ball,which is an unmodelable but interesting phenomenon. Thiscontrol policy can only be designed by RL. It is demonstratedthat under certain circumstances, RL-based approach couldgive better or even unpredictable policies that can rarely bederived by conventional control framework.

REFERENCES

[1] Wang, Cong, et al. “Learning Mobile Manipulation through DeepReinforcement Learning.” Sensors 20.3 (2020): 939.

[2] Lauwers, Tom B., George A. Kantor, and Ralph L. Hollis. “A dy-namically stable single-wheeled mobile robot with inverse mouse-balldrive.” Proceedings 2006 IEEE International Conference on Roboticsand Automation, 2006. ICRA 2006.. IEEE, 2006.

[3] Kumagai, Masaaki, and Takaya Ochiai. “Development of a robotbalancing on a ball.” 2008 International Conference on Control,Automation and Systems. IEEE, 2008.

[4] Fankhauser, Peter; Gwerder, Corsin (2010). Modeling and control ofa ballbot. Bachelor Thesis, ETH.

[5] Thomas Kølbæk Jespersen (April 2019). Kugle - Modelling andControl of a Ball-balancing. Master Thesis, AAU.

[6] Sonnleitner, Fabian, Roberto Shu, and Ralph L. Hollis. “The mechan-ics and control of leaning to lift heavy objects with a dynamicallystable mobile robot.” 2019 International Conference on Robotics andAutomation (ICRA). IEEE, 2019.

Fig. 8. The recovery area of the compound controller.

Fig. 9. The body angle φx, φy and the rotation velocity of three motorswith compound controller in 1.25s.

[7] Akkaya, Ilge, et al. “Solving rubik’s cube with a robot hand.” arXivpreprint arXiv:1910.07113 (2019).

[8] Andrychowicz, OpenAI: Marcin, et al. “Learning dexterous in-handmanipulation.” The International Journal of Robotics Research 39.1(2020): 3-20.

[9] Hwangbo, Jemin, et al. “Learning agile and dynamic motor skills forlegged robots.” Science Robotics 4.26 (2019).

[10] Palunko, Ivana, et al. “A reinforcement learning approach towardsautonomous suspended load manipulation using aerial robots.” 2013IEEE International Conference on Robotics and Automation. IEEE,2013.

[11] Johannink, Tobias, et al. “Residual reinforcement learning for robotcontrol.” 2019 International Conference on Robotics and Automation(ICRA). IEEE, 2019.

[12] Farshidian, Farbod, Michael Neunert, and Jonas Buchli. “Learning ofclosed-loop motion control.” 2014 IEEE/RSJ International Conferenceon Intelligent Robots and Systems. IEEE, 2014.

[13] Lillicrap, Timothy P., et al. “Continuous control with deep reinforce-ment learning.” arXiv preprint arXiv:1509.02971 (2015).

[14] Rosenstein, Michael T., et al. “Supervised actor-critic reinforcementlearning.” Learning and Approximate Dynamic Programming: ScalingUp to the Real World (2004): 359-380.

[15] Mnih, Volodymyr, et al.“Human-level control through deep reinforce-ment learning.” nature 518.7540 (2015): 529-533.