Pohl, 1

A LANE KEEPING ASSIST SYSTEM FOR PASSENGER CARS

-DESIGN ASPECTS OF THE USER INTERFACE

Jochen PohlJonas EkmarkVolvo Car CorporationDept. 96260, PVH 34SE-405 31 GothenburgSwedenPaper No. 529

ABSTRACT

Many road departure accidents are caused by thedrivers’ lack of attention. This is in many cases due todistraction, drowsiness or intoxication. The LaneKeeping Aid System (LKAS) described in the paperintends to be used as an active safety system, thusaiming at decreasing the amount of unwanted lanedepartures. The challenge in the development of suchkinds of functions lies in the determination ofdangerous situations and the design of appropriatewarning/ intervention strategies. The system isintended to go unnoticed with the driver andintervenes only in instances where the drivermismanages steering control. Unlike systems whichissue an audible sound, the type of warning is a tactilefeedback via the steering wheel. The torque isdesigned in a way that it communicates to the driverthe appropriate steering wheel angle required in orderto return in lane. In this paper, both the underlyingideas for such a lane departure warning system andtest track measurements will be presented anddiscussed.

1. INTRODUCTION

Surprisingly many Single Vehicle RoadwayDepartures (SVRD) accidents take place incomparable uncritical traffic situations and goodweather conditions, see ADAC (2001). Such accidentsare often road departures due to drivers' lack ofattention. SVRD accidents are often caused by thedriver’s distraction, drowsiness, intoxication or illnesswhich leads to the relinquishment of lane keeping(LK) control.The lane keeping assist system described here isintended to be an active safety system, thus aiming atdecreasing the amount of unwanted lane departuresand preventing a possible subsequent accident. In thismanner, active safety functions collaborate with thedriver-vehicle system in order to increase the overallsystem performance in critical driving situations. Theprincipal system layout is shown in figure 1. Besidestechnical issues such as sensor and actuatorperformance, the challenge in the development is to

find the "correct code of communication" betweenhuman and machine. Product ergonomists call thisscenaric transparency, i.e. the understanding of anadvice is an essential prerequisite of its following.An overall goal with the Human Machine Interface(HMI) is besides enhancing transparency, toincrease system reliability. Here the behavioralscience definition of reliability is more importantthan the engineering definition. The higher thedriver’s confidence in the system is, the moreactive safety issues can the car manufacturerassociated with it. In these terms systemtransparency is a contributing factor for reliability,i.e. if the driver fully understands the systemsfunction his/her confidence in the system increasesthus increasing reliability.Lane keeping assist systems have been in focusduring the last years, leading to an abundance ofdifferent system layouts in terms of sensors,actuation and HMI. Before entering details aboutthe system described here, a classification schemeis briefly discussed. According to Pilutti it is usefulto distinguish between warning, intervention andcontrol functions. Warnings do not directly alterthe vehicle trajectory and require that the drivermust choose to act on the warning in order for thewarning to have any effect. Intervention andcontrol have the ability to directly affect the vehicletrajectory. Intervention has limited authority and ismeant to augment driver commands – not replacethem. Control is defined here as automatic controlhaving full authority to steer the vehicle, whicheffectively removes the driver from the loop. Thisdefinition gives rise to the question of theappropriate actuator for lane keeping. There are inprinciple two ways to intervene with the vehicle interms of steering, namely to provide a torque signalin the steering wheel, or to provide a differentialwheel angle (which could be accomplished on thefront or rear axle). The size of the angle or torquedetermines whether such a system is classed as anintervention system or a control system, i.e.whether it is possible for the driver to override thesystem.

Pohl, 2

In terms of this definition the system presented herefalls in the category of intervention systems, since thedriver perceives a torque feeling in the steering wheelwhich mediates the correct lane position but still canbe overridden by the driver. However, the type ofintervention provided is of importance as well, i.e.how the torque is built up. In Pohl et al (2003) anintervention system for unintended roadway departureis discussed, where the principle of operation is asteering wheel torque which is related to the vehiclelateral position and speed in relation to the laneboundaries. This approach is extended here takinglateral vehicle dynamics into account, so that thetorque signal guides the driver back in lane.

In this paper, both the underlying theory for such alane keeping system as well as test trackmeasurements with test participants will be presentedand discussed.

2. LANE KEEPING MODULE

Figure 2 shows the principle layout of the requiredhard and software modules as they have beenimplemented in the test vehicle. A CCD camera isused in order to detect the current lane position, whilean Electric Power Assisted Steering (EPAS) actuatoris used in order to generate the required offset torquein the steering column. As the EPAS is primarily usedfor power assisted steering, it provides an assist torquewhich is determined by the boost curve and thecurrent driving conditions. The lane keeping torque is

therefore an offset torque to the base assist torque.In contrast to the Lane Keeping Aid Systemspresented by Naab et al.(1994) and by Renault,where an extra actuator on the steering column isused in order to produce the additional torquerequested by the system, an Electric PowerAssisted Steering (EPAS) gear is used.An important issue is how the steering wheeltorque relates to lane position and how this torqueis perceived by the driver. The Lane Keeping Aidmodule shown in figure 1 consists of several submodules, namely a vehicle state observer, steeringangle calculation, torque calculation as well as anintervention module.

The task of the vehicle state observer is to calculatethe required vehicle states as well as road geometrydata from camera and vehicle data. From thesesignals the optimal steering angle is computed,which is translated into a steering wheel offsettorque by the torque calculation module. Thistorque is superposed on the basic power steeringassist characteristic. The intervention modulefinally determines whether all required conditionsfor a steering intervention are fulfilled, as well asduration of this intervention. Intervention criteriaare for instance the current lane position, indicatorand brake pedal activity as well as whether thedriver has his/her hands on the steering wheel ornot. The individual modules are discussed in detailin the following sections.

Lateral Position

Road

Lane Tracker

Lane Keeping- Module

Driver

EPAS

Figure 1. Principal System Layout for a Lane Keeping System

Pohl, 3

2.1 Vehicle State Observer

In order to establish the vehicle state observer, whereeven state variables of the road are included, severalsets of equations describing vehicle motion, roadgeometry and the discrete state observer equationsitself are required.

2.1.1 Vehicle Equations of Motion. The vehiclemodel used here is a so called bicycle or one-trackmodel, where the vehicles roll motion (rotation roundx-axis, see figure 3), pitch motion (rotation roundy-axis) and bounce (z-axis) are neglected. Thesesimplifications reduce the order of the model to fourdegrees, namely yaw rate, sideslip angle and sideforce on both front and rear axle.

δ

β

ψ b

aαf

αr

v f

v

v r

F f

F r

Figure 3. Bicycle Model

The vehicles front and rear axle track is thus set tozero and the front and rear tires are lumped to onesingle tire respectively. Both wheels on one axle thusshare the same steering angles and each wheelproduces the same side force. These simplificationsare reasonable for driving scenarios with moderateside accelerations (< 0.4 g on normal dry asphaltroads, (Milliken and Milliken, 1995)) and constantvehicle speed. However, when it comes to test caseswith higher side accelerations and/or acceleration andbraking, a more complete vehicle model that eventakes the vehicles roll and pitch characteristic into

account, should be used. Figure 3 depicts the

bicycle model with state variables yaw rate Ψ& ,sideslip angle β and lateral force on both front Ff

and rear axle Fr.Using the equations of motion round the vehiclescenter of gravity yields:

nrrff maFF =+ )()( αα (1)

ψαα &&JbFaF rrff =⋅−⋅ )()( (2)

The lateral acceleration aL is related to yaw rateand sideslip angle time derivative by equation (3):

( )βψ && +⋅= tL va (3)

The lateral forces Ff and Fr are functions of thesideslip angle

fff CF α⋅= , rrr CF α⋅= (4)

In equation (3) Cf and Cr are the corneringstiffnesses of front and rear tires respectively. Thetire sideslip angel at front and rear can becalculated from yaw rate, vehicle sideslip angle andwheel angle:

va

f

⋅−−=

ψβδα

&,

vb

r

⋅+−=

ψβα

&(5)

Equations (1) to (5) lead finally to the two stateequations for sideslip angle and yaw rate:

Hfrf

rf

Cmv

bCaCvmmv

CCmv

δψ

ββ

⋅⋅⋅+⋅+⋅

+=

1-)(

1-

)(1

-

2 &

&

(6)

Hfrf

rf

aCJ

bCaCJ

bCaCJ

δψ

βψ

⋅⋅

=

1-)-(

1-

)-(1

22 &

&&

(7)

2.1.2 Lateral Vehicle Motion Relative LaneBoundaries. In the following required relations forlateral vehicle control are reviewed, see figure 4.

Vehicle StateObserver

Steering Angle Calculation

Intervenion

Lane Keeping Module Camera

EPAS Steering Gear

Driver

Torque Calculation

Figure 2. Components and Sub-functions of the Lane Keeping System

Pohl, 4

OC

Ry

0

yR

dp

y0R

δθ∆

v

ψ

Right line

Middle of lane

β

Figure 4. Lateral Vehicle Motion Relative LaneBoundaries

The motion of the vehicle relative to a given path,here prescribed by the road geometry, can be

described by the vehicle heading angle ∆Θ and thelateral distance to the lane markings at the vehiclescenter of gravity, here denoted as y0r and y0r for theright and left lane marking respectively.

Rv

+−=∆ ψθ && (8)

∆∆

∆

Θ−=+Θ−≈−−Θ−==

vvv

vyy LR

βββ

)(

))(sin(00 && (9)

In order to determine the rate of the heading angleform equation (8), the road radius must be measured.This can be done directly by the camera system incase the video image is processed at several previewdistances. However, this requires a camera with acomparably long look ahead distance. If the roadradius or curvature (the inverse of the radius) cannotdirectly be obtained from the camera system, it can becalculated from the following equation, taking thevehicle motion into account.

RDp

yDpy LL 2

2

0 ++⋅= ∆θ (10)

RDp

yDpy RR 2

2

0 ++⋅= ∆θ (11)

Equations (6)-(11) describe the required relations forthe estimation of vehicle and road state variables fromlane position and vehicle yaw rate measurements.

2.1.3 State Observer. Kalman filtering is a wellknown estimation technique. When the systemdynamics vary with time then the system matricesneeds to be updated in every time step whichrequires a time dependant Kalmanfilter. A discretestate space can be described as:

)()()()1( 1 tvtGutFxtx ++=+)()()( 2 tvtHxty += (12)

where x are the system state variables (namelybody sideslip angle, yaw rate, lateral distance tolane boundary, heading angle and road curvature),u the input signal and y is the output signal (herethe state vector). The matrices F and G areobtained from the previous derived equations in astraightforward manner. These are time dependentdue to the time dependency of the vehicle velocity.Here T is the sampling time.The measurement update of the states is given byfeedback of the innovation, i.e. the differencebetween the estimated and the measured signal.

))1|(ˆ)()(()1|(ˆ

)|(ˆ

−−+−=

ttxHtytLttx

ttx (13)

The measurement update of the error covariancematrix P is given by:

)1|())1|((

)1|()1|()1|(1

2 −+−

−−−=+− ttPHRHttHP

HttPttPttPT

T (14)

The calculation of the Kalman gain L takes intoaccount the uncertainty of the estimates andmeasurement noise R2.

12 ))1|((

)1|(−+−

−=

RHttHP

HttPLT

T (15)

The time update of the P-matrix is given by:

2)|()|1( RFttPFttP T +=+ (16)

Finally, the time update of the state estimate, i.e.the new state vector, is given by:

)()|(ˆ)|1(ˆ tGuttxFttx +=+ (17)From the estimated state vector the optimal steeringangle and steering wheel offset torque arecalculated as explained in the following. Athorough treatment of Kalman filter techniques andapplications can be found in Gustafsson et. al.(2001).

2.2 Steering Angle and Torque CalculationModules

The steering angle module calculates the optimalsteering angle in order to guide the vehicle backinto lane using the observed states from theobserver.

Pohl, 5

( ) ( )

00

2

)()( yvKvK

bavba

m

C

a

C

b

y

steerrf

opt

⋅+Θ⋅+

⋅

++

+

−

=

∆Θ∆

κβ

δ

(18)

The optimal steering angle optδ is calculated using

curvature, heading angle and vehicle lateraldisplacement at the center of gravity. The curvaturecan be seen as an outer disturbance thus being feedforward, while heading angle and lateral displacementare "feedback" signals. The gains for the latter arelinearly dependent on the vehicle speed, and thecurvature gain is obtained from the well known steadystate cornering equation.The torque calculation module consists of a feedforward and feedback path, with two vehicle speeddependent coefficients:

)()()( δδδ −⋅+⋅= optoptff vKvKT (19)

2.3 Intervention Module

The task of the intervention module is to makedecision whether an intervention should occur and theduration of the same. The basic intervention strategyis illustrated in figure 5. Steering intervention shouldoccur, when a set of conditions are fulfilled. Theseconditions can with advantage be arranged in differentclusters, such as driver, vehicle and road/environmentclusters.The driver cluster contains information regarding thedriver status in terms of drowsiness and distraction, aswell as whether the driver has his/her hands on thesteering wheel or not. A method for deriving thissignal making use of the steering column torqueprovided by the EPAS system is discussed in thefollowing section.The vehicle cluster contains conditions about thevehicle states, such as vehicle speed, acceleration,braking, indicator operation, lateral acceleration and

applied LK-torque. In the road environment clusterconditions regarding curvature, distance to lanemarking and heading angle are included. If allconditions from the driver and vehicle cluster arefulfilled, intervention shall occur when the vehicleenters the intervention zone indicated in figure 4,and end when the heading angle is below a certainthreshold. The time history of the applied torque isshown as well. This torque will guide the driverback in lane outside the control zone.

2.3.1 "Hands On" Detection. An integralcomponent of lane keeping systems where thesteering characteristics are changed due to laneposition is a device which prevents the driver frommisusing the system as an auto-pilot as stated byBalzer. The driver must always be kept “in theloop” and is thereby ultimately responsible for thecontrol of the vehicle. Such a device could be asteering wheel which is sensitive to pressure on thesteering wheel rim, in order to detect whether thedriver has his/her hands on the steering wheel ornot. Thereby an extra sensor is added to the system,which increases system costs.Here the existing steering column torque sensor ofthe EPAS system is used in order to estimate theamount of torque which is applied on the steeringwheel by the driver. The correct estimation of thedriver’s torque is essential for the properfunctioning of the “Hands on” detection. If thetorque applied on the steering wheel is understoodas an outer disturbance, a Kalman filter whichincludes a simple model of the column assemblycan be used. Such a Kalman filter requires a set ofobservable measurements, here we use the piniontorque (as the torque sensor of the EPAS steeringgear is used) and the steering wheel angle, seefigure 3.In figure 5, TS represents the sensor torque, TH

represents the driver’s torque, HT̂ represents theestimated driver’s torque, L the Kalman gain(calculated from the covariance matrices from

Figure 5. Intervention Strategy for the Road/Environment Cluster

Pohl, 6

process and measurement noise), BV the viscousfriction coefficient, α the steering wheel angle and Jthe steering wheel inertia. For real time applicationsthe filter needs to be transformed into the discretetime domain which is straightforward using forinstance the zero-order hold transform.

Figure 5. Disturbance Observer Kalman Filter

In instances where the driver does not apply anytorque on the steering wheel, i.e. “Hands-off”, the

estimated driver torque HT̂ should be zero, if themodel perfectly represents the dynamic characteristicsof the steering column assembly. As for examplefrictional effects are only modeled by simple viscousfriction and stick slip issues were neglected, theKalman filter “confuses” these with a torque from thedriver. The same counts for acceleration effects (aswell as gravity effects) due to non-centric center ofgravity (CG) where the lateral acceleration (orgravity) times the steering wheel mass and CGeccentricity acts as a torque on the wheel.Acceleration effects are easy to implement into thefilter, but proved to have only minor influence on theestimated steering wheel torque in contrast to non-linear friction.

0 2 4 6 8 10-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Time /[s]

To

rqu

e /

[Nm

]

Measured Sensor and Estimated Driver Torque

“Hands On”

“Hands Off”

Sensor Torque

Estimated Driver Torque

Figure 6. Torque Signals TH and TS of theSteering Column Assembly from Figure 5

However, thresholds can be used for decidingwhether “Hands-On” or “Hands-Off” cures theproblem in a practical manner. If the observeddriver’s torque is below a certain threshold over acertain time, the lane keeping torque is deactivated.This is interpreted as “Hands-off”, meaning that thedriver does not have the hands on the steeringwheel. As soon as the driver applies a torque againthe system will be activated.In figure 6 the measured and Estimated Sensortorque signals are shown, as well as two zones for“Hands-On” and “Hands-Off” determination.

3. MEASUREMENT RESULTS

Figure 7 to 10 show a test case where the driverdeliberately forces the lane marking withoutentirely leaving the lane. The vehicle lane position,steering wheel angle, lane keeping torque as well asestimated drivers torque have been recorded.

Figure 7. Measured Steering Wheel Angle

Pohl, 7

Figure 8. Measured Lane Position, i.e. Distance toLeft and Right Lane Marking

Figure 9. Applied Lane Keeping Torque

A close inspection of the lane position in figure 8produces the impression of an oscillatory systembehavior. However, it has to be kept in mind that therecorded sequence covers 15 seconds at a vehiclespeed of 70km/h. The peak value of the lane keepingtorque is 1.5 Nm and the main duration about 10seconds. It has to be added that it is hard to assess thesystem performance from simulation or measurementsonly. Test and tuning has to be an integral part of thedesign procedure. However, the final judgement is tobe made by the vehicle driver that the system isintended for. Concequently, human-machine aspectsin terms of system transparency are of majorimportance. Tests with different drivers have beencarried out in order to investigate these issues.

4. TEST DRIVE RESULTS

A decisive question in terms of customer acceptance,and whether a lane keeping system like the onedescribed here truly reduces the amount of singlevehicle accidents is if the driver intuitivelyunderstands the systems function, without knowingthat a LKAS is active. Twelve participants have

therefore been asked to drive the vehicle on a testtrack. The test participants have not been informedabout the real intend of the experiment, they haverather been informed that their driving capabilityduring distraction is measured. The experiment hasbeen conducted on a closed track with only fewother vehicles, and without oncoming traffic. Incontrast to situation in real traffic, where severedistraction can result in dangerous situations, thedrivers felt safe during the test, despite of largerlane deviations.In order to produce lane deviation the testparticipants were severely distracted by severaltasks they were asked to perform. These tasksstarted with changing radio channel, storing newchannels in the radios memory, answering a cellphone and finally sending text messages from thesame. A few test participants refused to executecertain tasks, as that might result in dangerousdriving situations. This is important to have inmind when it comes to the evaluation of thequestionnaire filled in by the test persons after thestudy. The fact that the usefulness of the systemhas been judged from proving ground driving only,influences surely the answers of the testparticipants on the test protocol.It has to be mentioned that the test leader observedboth the amount of LK torque and the testparticipant during the distraction tasks. It isinteresting to note that the participants were highlyoccupied with the distraction tasks, that nonecommented on the steering intervention. Directlyafter the individual test programs were carried outthe participants were asked if they liked thehandling properties of the vehicle. Several driversfound the steering to behave strangely (see below),but could not relate this behavior to the drivingsituation.After the driving test the drivers were asked toanswer the following six questions, in aquestionnaire: table 1.1) How was your first impression of the system? Since the test participants were not used to suchkinds of systems, they ascribed the vehiclebehavior to a number of different reasons, as forinstance power steering malfunction or miss-aligned front wheels.2) Was the functionality of the system easy tounderstand?All test participants claimed that the systemoperation was far from being intuitive, andsuggested an additional HMI in form of adesignated lamp in the visual field.3) Did the force given in the steering wheel botheryou?The majority of the test participants were notdisturbed by the additional torque, however, twocompared the function with lecturing the driver.4) Did the system support you in your driving?

Pohl, 8

The drivers stated that the system lulls the driver in asense of false security.5) Did the system perform any incorrect maneuvers? The test participants found the amount o torqueapplied on the steering wheel both to high and toolow, which points out the need for a personalizedsetting of the LK-function.6) Would you like to have a system like this in yourcar?

The questions were answered by ranking them on ascale from zero to ten, where ten is the perceivedhighest satisfaction. The mean value of the ranking,by question, is shown in table 1.

Table 1. Mean Values by Question

Question Mean Value1 5.12 7.13 6.34 5.75 7.26 6.6

The result indicates a promising satisfaction with thesystem. The majority of the test participants wouldlike to have such a system installed in their ownvehicle, if it is reasonably priced. However, it has tobe remembered that the small size of the test groupdoes not represent any statistical result.

5. CONCLUSIONS

In this paper design issues for a Lane Keeping AidSystem are discussed, covering basic systemrequirements, algorithm modules and human machineinterface aspects. The presented system has beentested on twelve drivers, who did not know the realintend of the test. In that manner, systemtransparency, i.e. whether the system function isintuitive or not, could be investigated.The result of this investigation is twofold:• The haptic interface as HMI in the steering wheel

is in itself not sufficient. It is suggested that amulti-modal HMI is integrated to increase thedriver’s awareness about system operation. TheHMI could incorporate a Head-up-Display orsimply a standard symbol in an appropriateposition together with the haptic steering wheel.

• In order to reduce the amount of nuisancewarnings, the system needs additionalinformation about the drivers' state in terms ofdistraction and drowsiness.

The next step towards a full-fledged Lane KeepingAid System therefore includes detection of drivers’state, by e.g. integrating vision based driver monitors,

which today unfortunately still lack the requiredrobustness in terms of varying lightning conditionsand varying drivers.

6. NOMENCLATURE

β sideslip angle [rad]κ road curvature [1/m]θ∆ heading error [rad]

fα ,rα front/ rear wheel angle [rad]

δH wheel angle [rad]βsteer gearing ratio [-]a, b position of CG [m]aL lateral acceleration [m/s2]Cf, Cr cornering stiffness front, rear [N/rad]Dp camera measurement distance [m]F various forces [N]F, G, H system, input, output matrixJ vehicle inertia aroiund z-axis [Nm·rad/s2]K various gainsL observer gainm vehicle mass [kg]P covariance matrixR road radius [m]T various torques [Nm]v vehicle speed [m/s]v1, v2 noise vectorsx, y, u state, output, input vectory0 lane position [m]

Ψ& yaw rate [rad/s]

7. REFERENCES

ADAC (2001), Verunglückte nach Unfallart,http://www.adac.de.Balzer, D, (2002), Safety enhancement by laneobservation of Advanced Driver AssistanceSystems (ADAS), VDA, Technical Congress 2002,Germany.Gustafsson F, Lung L, Millnert M, (2001)Signalbehandling, Studentlitteratur, page 289-332.Naab K, Reichart G, (1994), Driver AssistanceSystems for Lateral and Longitudinal VehicleGuidance - Heading Control and Active CruiseSupport, Paper No 78, AVEC 94.Pilutti, T E, (1997), Lateral Vehicle Co-Pilot toavoid Unintended Roadway Departure, PhD thesis,Dept. Mechanical Engineering, The University ofMichigan.Pohl J, Ekmark J, (2003), Development of aHaptic Intervention System for Unintended LaneDeparture, SAE Paper 2003-01-0282, SAE WorldCongress, SAE international.Renault safety site, www.conceptsecurite.renault.fr/demain/einxc5.htm.