Towards Target-Level Testing and Debugging Tools for Embedded Software

Harry Koehnemann, Arizona State University

Dr. Timothy Lindquist, Arizona State University

AbstractThe current process for testing and debugging

embedded sojware is ine~ective at revealing errors. Thereare currently huge costs associated with the validation ofembedded applications. Despite the huge costs, the mostdl~cult errors to reveal and locate are found extremely latein the testing process, making them even more costly torepm”r. This paper first presents a discussion of embeddedtesting research andpractice. This discussion raises a needto improve the existing process and tools for embe&@itesting as well as enable better processes and tools for thejWure. To fmilitate this improvement, architectural andsoftware capabilities which support testing and &buggingwith minimal intrusion on the executing system must bedeveloped. Execution visibility and control must come@om the underlying system, which should ofJer interjbcesto testing and debugging tools in the same numner it offersthem to a compiler. Finally we propose txtenswns to theunderlying system, which consists of adiiitions to both thearchitecture and run-time system that will help reulizetarget-level tools.

1. Introduction

Software validation involves many activities thattake place throughout the lifecycle of soft w aredevelopment. A substantial portion of the validationprocess is software testing, which is the development oftest procedures and the generation and execution of test

eases. Notice we are not only concerned with thegeneration of a test case, but are also concerned with howthat test is executed. Therefore, a test case is not simplycomposed of inputs to a system, but rdso includes anyenvironmental factors. Other research has examined theissues behind test case selection, but few are addressing theproblems that surround the execution of those test cases.The goal of this paper is to identify the problemsassociated with test case execution for embedded systemsand to propose solutions for making embedded testing moreeffective at revealing errors.

1.1 Testing and Debugging ProcessMany of the activities, tools, and methods used

during software testing are shared by software debugging.Software testing is concerned with executing a piece ofsoftware in order to reveal errors, while software debuggingis concerned with locating and correcting the cause of anerror once it has been revealed. Though these twoactivities are often referenced separately, their ac$ivi$ies aretightly coupled and share many common features.

Permission to copy without fee all or part of this material is grantedprovided that the copies are not made or distributed for direct commercialadvantage, the ACM copyright notice and the title of the publication and itsdate appear, and notice is given that copying is by permission of theAssociation for Computing Machinery. To copy otherwise or republish,requires a fee and/or specific permiss~m.

During debugging, a developer must recreate theexact execution scenario that revealed the fault duringtesting. Not only must the code execute the sameinstruction sequences, but all environmental variants mustbe accounted for during the debugging session. In addition,the tools assisting in the debugging process mustproviding a &veloper with a certain degree of executionvisibility and control while not impacting the executionbehavior of the program.

1.2 Embedded SystemsThe testing and debugging process is greatly

restricted by embedded systems. Embedded applications areamong the most complex software systems beingdeveloped today. Such software is often constrained by

● Concaumnt designs● RcaI-time constraints● l%lbedded target Imvilrmments● Distributed hardware ambitectures● Device control dependencies

Each of these properties of embedded software severelyrestrict execution visibility and control, whichconseqmdy restricts the testing and debugging process.

Our current methods and tools for software testing anddebugging require a great deal of computing resources.Such resources are not available on the target environment.Therefore, a large gap exists between the methods and toolsused during evaluation on the host and those used on thetarget. Unfortunately, mauy errors are only revealed duringtesting in the target environment.

Because of the above issues, concerns are raisedover the effectiveness of software validation for embeddedsystems. Embedded applications are responsible forcontrolling physical devices and their correct execution iscritical in avoiding and/or recovetig from device failure.Often these physical devices control life-critical processes,making the embedded software a key element of a life-critical system. Software failure can lead to system failure,which in turn could lead to loss of life.

In addition, designers are increasing their use ofembedded software to conuol the physical elements of largesystems. This rate of increase is likely to increase as thecost for embedded controllers becomes cheaper and moreattractive when compared with other mechanical techniques.Computer networks are fast replacing point-to-pointwiring, due to the networks light weight, easycotilgurability and expansibility, and lower designcomplexity. The advancement in the complexity ofproblems addressed by software in these types of

01993 ACM 0-89791-621 -2/93/0009--0288 1.50

applications may soon be limited by our inability tosatisfy reliability needs and concerns.

2. Software Testing

The software testing phase is concerned withexecuting a software program in order to reveal errors.Software testing for embedded systems takes place in fourbasic stages:

1) Module Level Testing2) Integration Testing3) System Testing4) Hardware/Software Integmtion Testing

The first three testing stages are typical of any softwareproduct. Testing begins with exercising each soft waremodule and concludes when the software is shown to meetsystem specifications by passing some rigorous set ofsystem tests. The fourth phase is unique to embeddedsystems. The software must not only be correct, but mustalso interface properly with the devices it is controlling.

Testing literature contains countlessmethodologies, techniques, and tools that support thesoftware testing process. They range from softwareverification and program proving to random test case

selection. All testing methods indirectly apply toembedded systems, as they do all software. Of particularinterest to this paper are those techniques that address theproblems identified for embedded software - concurrency,real-time constraints, embedded environment, etc.Unfortunately, there exists little research into the uniqueproblems SSSOCilltd with testing embedded software.

2.1 Testing Concurrent SystemsConcurrency increases the difficulty of software

testing, Given a concurrent program and some set ofinput, there exists an unmanageably large set of legalexecution sequences the program will take. Furthermore,subsequent execution with the same input may yielddifferent, yet correct results due to differences in theoperating environment. This is all complicated by Ada’snondetermins tic select construct. Therefore, when testingconcurrent software, we are not only concerned with a validresuk but must also be concerned with how the programarrived at that result.

Since multiple executions of a concurrent programmay yield different results, it is not enough to ensure thatthe system produces the correct output for a given input.One must also ensure that the system always produces anacceptable output for each execution sequence that is legalunder the language definition. Without sufficient controlover program execution, there is no way of ensuring agiven test is exerasing the code it was intended to test.

Taylor and Osterweil ~ay180] examined staticanalysis of concument programs, However, this researchconsidered processes in isolation and does not considerinterprocess cmrummication. Taylor later extended thiswork to Ada and a subset of the Ada rendezvous mechanism~ay183]. Through this static aualysis technique, one coulddetermine aIl parallel actions and states that could block atask from executing. This method, as with most static

techniques, must examine a large set of statw and thereforemust constrain itself to small, simple programs.

Research in dynamic testing of concurrent Adaprograms has largely focused on the detection of deadlocks~emb85], the saving of event histories KeDo85,Maug85], and other tec.huiques that passively watch aprogram execute then allow the execution sequences to bereplayed after a failure has been detected.

Hanson Eans78] was among the first to discussrun-time control of concment programs. In order toregulate the sequences of events, he assigned eachconcurrent event in the test program a unique time value.He then introduced a test clock that regulated the systemduring execution. A given event could only execute if it’stime was greater than that of the clock.

Tai ~ai86, Tai91] extended Hanson’s work to theAda programnn “ng language. His method takes an Adaprogram P and a rendezvous ordering R and produces a newAda program P’ such that the intertask communication inP’ is always R. A similar approach was used in Koeh89]to apply these techniques to testing and debugging tools.This work addressed the facl that in order to,testa specificprogram state, values in a program may need to bemodified during run-time. Modification o~f the programstate is a capability provided by any debugging tool and isa required property of a tool debugging tasked programs. Itis important to note that both techniques explicitly performrendezvous scheduling, removing those decisions from therun-time system and placing control in the hands of thetool.

2.2 Non-intrusive testingIntrusion plays a significant role in the testing and

debugging of embedded software. Any technique used toraise execution visibility or provide for program controlmust not interfere with the behavior of the teat program.Embedded applications have strict timing requirements andany intrusion on a test execution will likely make that testvoid. Intrusion is typical for host-based testing, but

becomes a large problem for target-level testing anddebugging activities.

The above approaches address the need forvisibility, control, and predictability for testing concurrentsoftware. However, they are all intrusive and useinstrumentation (inserting probes into a usem program andrewriting certain constructs before submission to thecompiler) to gather run-time information and to control

~gram exmtion. After the probes are added, the user’sobject code is linked with the rest of the debugging systemand then executed under test. This additional code has aserious impact on the execution behavior of the program.Instrumentation is not appropriate for testing real-time,embedded applications.

A non-intrusive debugger for Ada is proposed in[Gil188]. A separate processor executes the testing systemand communicates with the target processor through somespecial purpose hardware. Lyttle and Ford ~ytt90] havealso implemented a non-intrusive embedded debugger forAda. Their tool provides monitoring, breakpoints, and

289

display facilities for executing embedded applications.While these efforts provide an excellent start towards target-level tools, they do have severe limitations. Theseimplementation do not deal with high level activities suchas task interactions and are only concerned with items thatcan be translated from monitoring system bus activity. Asdiscussed later in Chapter 5, techniques dependent on busactivity will likely fail for future architecture designs. Inaddition, many of the error detwted in the targetenvironment are indeed concerned with high-level activities(process scheduling and interactions, fault handling,interrupt response, etc.).

Other real-time, embedded tools have beenproposed for crossdevelopment environments. They cantypically be classified into one of the following threecategories: 1) ROM monitors, 2) Emulators, and 3) Busmonitors. These types of tools will be further discussedlater in this paper.

2.3 Impact of the Underlying SystemOne of the large problems with testing concurrent

systems is dealing with abstraction. The Adaprogramming language abstracts concurrent activitiesthrough task objects ~D83]. Tasks allow a developer toabstract the concepts of concurrency and interprocesscxmmmnication and discuss them at a high level. Theburden of implementation is then placed on the compiler,and typically the run-time system.

While abstraction is a powerful design tool, itleads to significant prthlems during the testing phase ofsoftware development. Implementation details becomeburied in the underlying system. At the development level,this high degree of abstraction is appropriate. However,abstraction complicates the testing process. Not only arewe concerned with implementation details, but we mustaIso control them to demonstrate that certain propertiesabout a program will hold for every legal executionscenario. Without sufficient control over programexecution, there is no way of ensuring that a spedc test isexerasing the code it was intended to evaluate. In addition,cmmxt operation in one environment (host) does notnecessarily imply comet operation in another (target) dueto implementation difference in the underlying system.

The underlying system is composed of two parts,the features of the hardware architecture and the operationsprovided by the run-time system. As language constructsbecome more abstract, compilers are required to generatemore code to implement them. There is no longer a trivialmapping from language construct to machine instruction.Rather, the compiler must provide an algorithmic solutionin order to implement these high level constructs. Thosesolutions exist as operations in the run-time system.Rather than generate code for these constructs, the compilergenerates a call to a run-time system operation or servim.

As the constmcts become more abstract,compilers develop an increasing dependency on theunderlying system. This increase in shown in figure 2.1.As new constructs are introduced to programminglanguages, their increase in abstraction is greater than that

of hardware and the run-time system is called upon tobridge the impending gap. No argument is made as to therate of increase identified by the line slopes; nor is anargument ma& that these increases are even linear.

co

L=Language

constructsm

P1

e Hardware

x

i~t

Y Time

Figure 2.1

Embedded systems raise many problems forsoftware testing and debugging. Such software typicallymust deal with concurrency, real-time constraints, auembedded target environment, distributed hardwarearchitectures, and a great deal of hardware-softwareinterfaces for controlling externaI devices. These issuestdone do not provide a complete view of the problemsextcotttttered by embedded testing. bbedded SyStetUS are

typically developed on custom hardware configurationsm@ng that each system introduces it’s own uniqueproblems. Tools and techniques that apply to one are notgenerally applicable on another, which leads to ad hocapproaches to integration and system testing of embeddedsoftware. The program is executed for some length of timeand continual y bombarded with inputs in an attempt toshow it adheres to some speeifkation,

3.1 Current state of embedded testingAs described earlier, the testing process for

embedded systems consists of 4 phases that conclude withHardware/Software (H/S) Integration During H/Sintegration testing, device and timing related errors arereveakd. These errors eneompass problems such as:

● incorrect handling of interrupts● distributed communication problems● incorrect ordering of concumen t events● resource contention● incorrect use of device protocols and timing“ incomect response to failures or transients

These errors are often extremely difficult problems to fixand often require significant modifications to the softwaresystem. In addition, software is for~d to conform tocustom hardware that may itself have errors. As statedabove, H/S integration is the last phase of testing for anembedded system. Since errors are much cheaper to fix theearlier they are revealed,. why would one wait until the lastphase of product development to reveal the most diftlcult to

290

locate, costly errors to fix? Our goal should be to revealthese errors as early as possible. Unfortunately, targetlevel testing tools have yet to become a reality.

The target processor of au embedded computer istypically minimal in function and size. It is only a smallportion of a larger system, whose goals are to minimizecost and space. Therefore, target hardware of au embeddedsystems will not support software development nor anydevelopment tools. To resolve this problem, the source isdeveloped on a larger host platform and cross axnpilers andlinkers are used to generate code and download it to thetarget processor.

Consequently, two environments exist in ourdevelopment process, the host environment and the targetenvironment, each having completely differentfunctionality and interface to a user. Tools that run on thehost provide a high level interface and give users detailedinformation on and control over their program execution.However, little is provided on the target. Typically, thebest information obtainable is a low-level execution traceprovided by an in-circuit emulator.

3.2 Current SolutionsApproaches to dealing with the above problems

can be divided into hardware solution and softwaresolutions. The hardware solutions are attempts at gainingexecution visibility and program control and include thebus monitors, ROM monitors, and in-circuit emulators. Abus monitor gains visibility of an executing program byobserving data and instructions transferred across thesystem bus. With a ROM monitor, debugger code isplaced into ROM on the target board. When a break point

is encountered, control is transfered to the debug codewhich can accept commands from the user to examine andchange the program’s state. Finally, an in-circuit emulatorconnects with a host system across an ethernet connection.At the other end, a probe replaces the processor on thetarget board. The emulator then simulates the behavior ofthe processor in (ideally) real-time, which allows theemulator to tell the outaide world what it’s doing while it’sdoing it.

The hardware solutions have mini m al

effectiveness for software development. They can onlygather information based on low-level machine data. Thedeveloper must then create the mapping between low-levelsystem eventa and the entities defiied in the program. That

-ping is the implementation strategy chosen by a givencompilation system and becomes severely complicated forabstractions such as tasks Maintaining an understanding ofthe mapping is extremely difficult and cumbersome.

The software solutions cart be viewed as attemptsto reduce the tremendous costs of testing on the target.Several factors determine how a pitxe of software is tested

1) Level of criticality of software moduleEach software module is assigned a different level ofcriticality based on it’s importance to the overalloperation of the system.

2) Test platform availability

Typically, there will exist several test environmentsavailable to test a piece of soft ware, each providing acloser approximation to the actual target environment:

● Host-baaed sours level debugger● Host-based instruction set simulator● Target emulator● Integrated validation faality

3) Test ClassificationThe tests to be performed can be categorized todetermine what they are attempting to demonstrate.The goal of a test plays a large role in determining theplatform on which it will execute. Some examples areshown below.

● Inter-module ● Algorithmic● Intra-module ● Performance● InttX-cabinet ● HE integration

Each of these factors play a role in assigning programmodules to the various test platforms based on somecriteria that might contain the following:

● Type of software● Hardware requirements● Test chssifkation● Platform availability● Coverage requirements● Test support software availability (drivers, stubs)● Certification Requirements

● Level of effort required for testThis criteria takes into account the 3 factors discussedabove as well as additional ones.

The software solutions are an attempt to minimizethe time spent testing in the target environment.Validation facilities are expensive to build and time utilimd

for testing is expensive. This is due to the f;act that targetlevel testing occurs extremely late in the developmentlifecycle and only a small window is allocated for HASintegration testing. However, the target is the onlylocation that can reveal tin errors. It is ironic that ourcurrent solutions attempt to reduce the anmunt of targettesting, but will likely lead to extensive modifications andthercfom extensive retesting.

4. Problems with Embedded IrestinqThe solutions proposedaboveare not effective at

revealing errors. Effective implies that a technique revealsa high percentage of the errors and that it does so in a cost-ef!iaent manner. Instead, what the above tools provide is aminimal, low-level view of the execution of a program andthose tools become available at a very late stage indevelopment. Below is a list of problems associated withcurrent approaches to embedded testing

4.1 Expense of Testing ProcessTarget testing requires expensive, custom

validation facilities. The expense of these target facilitiesis incurred for every project, since little reuse acrossprojects is ever realized. The effort required to build thesevalidation facilities means that every test execution isexpensive, making retests extremely costly. Yet, hardwareoften arrives late and full of errors, forcing software to be

291

modified and subsequently retested. This late arrival ofhardware also impacts the cost of an error, since certainerrom are only revealed during I-IN integrations testing.

Perhaps the largest factor associated with the highcosts of testing will be the questions and concerns thatcertification processes are beginning to raise about sofhvaretools. Typically, development tools have not been requiredto meet any validation criteria and certairdy not the strictcriteria imposed on the development system. This luxurymay soon disappear as the role tools play in thedevelopment process comes under tighter scrutiny. Thehuge expense of validation facilities will increasedmmaticauy.

4.2 Level of Functionality on TargetThe level of functionality found on a target

machine is minimal and does not support tools. This lackof functionality greatly limits the effectiveness of testing,since more time and effort is required to locate an error.While a host system provides a high-level interface anddiscussed software in terms of the high-level language, thetarget typically deals in machine instructions and physicaladdresses. Translating these low-level entities requires timeand a great deal of tedious, error-prone activities.

4.3 Errora revealed late in developmentlifecycle

Embedded system designs often incorporatecustom ASIC parta that are typicaIly not available untilvery late in the development process, delaying theavailability of any target validation facility. In addition,errors designed into the ASICa are extremely expensive tofix, requiring new masks be created and completerefabrication. InsteaA errors in ASICs and other hardwareproblems are resolved by modifying the software. Asstated before, this greatly delays the time which errors arerevealed, which in turn increasing the cost of softwaretesting.

4.4 Poor teat selection criteriaAIl to often, tool availability diclates the quality

of a testing process. Tests cases and scenarios aredetermined by what will work on available platforms andwhich test are achedulable rather than being determined bysome theoretical test cxitcria. A prime example is theFAA’s requirements ~AAS51 that 1) all testing be done inthe target environment and 2) testing include statementcoverage. Of course, test coverage is not currentlymeasured on the target.

Unfortunately, it is cheaper for a company tospend it’s resources preparing an argument to obtain someform of ‘waiver” than to actually perform a test. In time,the argument approach will no longer be accepted and thesolutions for embedded testing must be in place toaccommodate this change. It will only take oneimplementation that performs statement coverage on thetarget to force every embedded, real-time software developerto perform statement coverage on the target to meet such acertification requirement.

4.S Potential use in advancing architecturesPerhaps the largest problem facing embedded

testing is that the current solutions cannot be applied tofuture h~dware architectures. Future architectures are

Proposing● wider addreas Spaces● higher -Sor speeds● huge numbers of pins● internal pipes● multiple execution units● large internal caches● multi-dip modules

Such complexities cast a dark shadow over the hardwaresolutions previously discussed. With internal caching andparallel activity being done on the chip, one will no longerbe able to gain processor state information from simplymonitoring the system bus. And as on-chip functionsbecome more complex, emulator vendors will no longer beable to see into the chip through the pins making themobsolete as well.

In [Chi191] an even stronger claim is made thatthe debugging capabilities provided by the chip will need tobecome more sophisticated. In future architectures, perhapsthe only possibility to view and control the execution ofhardware is to gain that information from the hardwareitself.

The previous sections raised issues about theeffectiveness of our testing process and claimed that tcatingis currently being limited by tool functionality. l%e goalof this paper is to identify shortcomings in the embeddedtesting proccas and propose a solution to those problems.The view taken by the authors is that tool support forembedded systems is lacking. Further, those approachescurrently used for gaining execution visibility and controlwill soon be obsolete for future architectures. We proposeadding facilities to the underlying system to better supporttesting and debugging tools for embedded software.

As stated previously, the underlying system iscomposed of the hardware architecture and the run-timesystem (RTS). Both are composed of data structures andoperations that implement common system abstractionssuch as processes, semaphores, ports, timers, memoryheaps, and faultdexceptions. It should be noted that thereis no distinct line between features of hardware and featuresof the RTS. In fact, as these features and abstractionsbecome more standardized, newer architectures areattempting to incorporate them into their instruction sets~Nl%92]. In addition, the implementation,of a featuremay span parts of the architecture, RTS, and compilergenerated code (i.e. fauhdexceptions).

5.1 Model Debugging SystemBelow is an illustration of a debugging system

(Figure 5.1). The data path from the debugging/testingtool represents symbol table information that allows thetool to map machine level information to source level

292

Test/Debug

Tool

Compiler\ 1

Generated

Code e

x

t

e Ada

3 “s-Compilation

A

sw-tions describe ‘tie architecture additions and RTSinterfaces in more detail.

1 I I

Figure 5.1constmcts. The ASIS toolkit provides easy required for this physical connection. The next twoimplementation for this facility. ASIS is a proposedstandard interface between an Ada library and any toolrequiring compilation information.

Of more interest is the communication pathbetween the target processor and the testing tool. A toolsits external to the rest of the embedded system, while theRTS resides internally on the target board. At frost glance,this conceptual path seems rather difficult to realize.However, the implementation becomes easier if thoughtabout as a typical host debugging system. Any debuggingsystem has a least two processes executing, one runningthe test program and one running the &bugga. These two

p= S- a common physical machine, which allowsone process to gain information about the other. Thedebugger procem simple requires data and computation

time, which it shares with the test program.This same scenario is rcqnired for embedded

debugging, except that the debugger process is split. Partof the debugger process runs on the target machine and partruns on the host. The goal is to minimize the portion thatmust be run on the target so that it does not intrude onexecution of the test program. To realize this non-intrusive execution of the debug software, the target

-= -1) Execute debug code only at a break poinL2) Run the debugger as a separate process, or3) Provide a separate execution unit to execute thedebugger.

The details of these options are explored in depth later inthis paper.

The problem now lies with the interfa= betweenthe embedded part of the debugger (intermddebuggcr) andthe portion that lies on the host (external-debugger). Thesolution requires hardware additions that will be discussedlater in this paper. A high level view is given in figure5.2. In this figure, the tool makes logical calls to servicesprovided by the RTS. These calls are actually implementedby the debugging system through data passed between theinternal and external debuggers. Hardware additions arc

Figure 5.2

The past decade has seen hug{: advances inmicroprocessor designs. Several of these advancementswere listed previously and include pipelining and separatefunctional units. The concept of partitioning amicroprocessor in order to perform parallel activities is ofgreat interest to this work.

It was noted earlier that these parallelamputations severely restrict current methods for testingand debugging embedded systems, since on{e must simulatea great amount of computations. However, debuggingtools can also use architectural parallelism to theiradvantage. If a hardware design is partitioned successfullyto allow certain activities to occur concurrently, then thetesting and debugging methodologies might wish to add

293

their own computational requirements to the list of parallelactivities.

This section will explore additions to hardwarearchitectures. No claim is made as to the costs associatedwith these features. They assured y will require space(transistors) and possibly even add to the execution cyclesrequired to implement certain instructions.6.1 Hardware Partitioning of Memory

One primary concern for industry is reducing thehuge volume of retests associated with development. Thecurrent testing process ensures that errors are revealed late,which forces retesting large portions of the system.Despite correcting these problems, industry will still befaced with software that is constantly changing. Softwareis deceivingly easy to change and often the element of asystem assigned to unknown or “risky” aspects duringdesign.

Changing software is extremely expensive late inthe development for critical systems. Such systemstypically have requirement that an error raised in oneportion of the system won’t interfere with the correctoperation of the rest of the system. Current softwarecertification agencies ~AA85] have several softwarerestricdons including

● Any modikation made to a software module forces theretesting of all other modules operating on that samephysical device.

. All software on a device must be developed under thehighest level of criticality of any module that willexecute on the same device.

Without the ability of hardware to guarantee softwareboundaries, such requirements must be enforced. However,these requirements add a great deal of costs to softwaredevelopment. Consequently, software is often physicallypartitioned based on critical level, rather than designfactors.

Partitioning software modules based on criticallevels greatly interferes with the design process. Onewould rather partition modules based on factors such asprocessor utilization and inter-module exmnmnicationrequirements. In fact, load balancing and p17XXsSmigrationare techniques that would not be usable by embeddedsystem developers unless all software is developed at thehighest critical level.

The solution to these issues is hardware

partitioning. Each process should have it’s own protectedaddress space that is not accessible by any other process.In addition, sets of processes may wish to share memory.The processor should tdso provide the capability to restrictaccess to segments of memory based on some criteria.6.2 Computational Facilities for Debugger.

The debugging system is partitioned into aninternal debugger and au external debugger. The internaldebugger must physically exist on the target board andcommunicate with the external debugger through somededicated medium. The internal debugger will also requireexecution from the target without interfering with the

operation of the application program. There are twopossible scenarios

● The internal debugger runs as a regular process on the

-Or● The architecture provides separate facilities to execute

the internal debugger codeIn either case, control is transfered to the debugger when abreakpoint is encountered

In the fiit scemuio, the debugger is executed bythe processor as any other process. If the debuggerexecutes as a low-level process, it would not interfere withthe operation of the rest of the system. However, this isnot a feasible approach. Most intern-sting errors occurduring peak system loads, which would mean that thedebugger could only execute when the probability of anerror occurring was low. Another approach wouId be toexecute the internal debugger as a periodic process of high

priority and design the entire system to take this processinto account when determining issues such as scheduling.

The second scenario requires the target processorto provide some form of computational facilities. Thisextra execution will certainly require some amount ofutilization of architecture resources such as internal

registers and bus accesses. The simplest example ofarchitecture facilities would be a machine that contirtuaU ydumps some representation of the instruction it is currently

executing. This would require a dedicated bus to theexternal world (proposed later in this section) and thatadditional circuitry be attached to the computation units togain access to the current instruction.

The problem with fis approach is that theprocessor is not aware of what data is required by the toolsat the other end. Therefore, it must dump everything. Athigh processor speeds, the amount of information beingsent could become overwhelming. However, the data couldbe faltered and then captured so that a tool could parse itlater and recreate an execution history of the program. Thehardware required for filtering is not trivial and requiresgreat speed and storage capaaty to maintain pace with thetarget processor.

lle next step is to allow software to dictate theinformation sent by the processor. The functional unit ofthe hardware sed.ing messagea could be implemented as astate machine, emitting different messages based on itscurrent state. The default state would be all processortransactions. Basically, in this eontiguration, the processoris performing the filtering rather than the externaldebugger. This addition should not add much incomplexity to the hardware architecture and would greatlyreduee the wmplexity of the external debugging hardware.

The final step is to take the (now stateful)functional unit and make it programmable. Instead of astate machine, it now becomes a complete functional unitwithin the processor itself. The internal debugger codewould then be loaded into this portion of the prowssor atboot time and reside there for the entire execution,

transmitting and receiving messages to and from theexternal debugger.

294

6.3 Hardware Break PointsSoftware break points are intrusive and require

instructions be inserted into the code of the test program.Conditioned break points present a more significantproblem, since they require a computation every time theyare encountered to determine if the proper conditions aremet to halt execution. Such breakpoints are unacceptablefor d-time programs.

To resolve this issue, architectures need to providethe capability to set breakpointa in hardware. A set ofregisters would be classified as BreakPoint Registers(BPR), which the processor would check against theoperands for each instruction. Two types of breakpointsare required, data and instruction. Each data BPRs insidethe processor would be compared with the address of everydata operand for each instruction. Instruction BPRs wouldbe compared with instruction addresses or type. When amatch occurs, a breakpoint fault would be raised andcontrol trsnafered to the internal debugger

Upon returning from a break, the processor isrequired to restart execution precisely where it hadterminated. The state of the processor consists of all it’sinternal registers, including any pipeline information andcache memory. These values must be saved automaticallywhen a break is encountered.

Another issues is that of conditional breakpoints.Such breakpoints require computations by the processorthat run in the background behind the program under test.The evaluation of the conditional expression must begin farenough in advance so that it may complete before theprocessor has passed the breakpoint location. Thisevaluation will require memory accesses, raising additionalproblems. The current value of operands in the expressionsmust be available to the processor, which might involveaccessing it from memory or cache. Any accesses tomemory must be scheduled in such a manner that they donot block any resources required by the program under test.F@dly, the value used must be valid and not in danger of

-g before the breakpoint.While the problems raised above seem difficult,

they are not insurmountable. The extend debugger mustcompile the conditional expression and download the code.At that point it can determine the scheduktbili~ of thisevaluation by comparing the @e for the conditional to theother code that will occur in parallel. The user could thenbe notified of problems with their additional breakpoint.The hardware is responsible for detecting any collisions inparallel activity and must not assume the debugger isalways accurate. Any debugger activity intruding on thebehavior of the test program is important information andmust be flagged by the processor.

The primary additions required for hardware breakpoints are additional registers from the architecture and thelogic necessary to compare them with the operands of thecurrent instruction. To support conditional breakpoints,the processor must provide background computationalsupport. This support could come from a portion of theprocessor dedicated to conditional breakpoints, or the code

could be downloaded to the internal debugger, given theinternal debugger support described previously.6.4 Architectural Support for Abstractions

As common programming paradigms becomemore refined, architectures will begin to inax-porate theminto their instruction sets. It would be unlikely that theonly abstractions supported by architectures would remainsimple data types (integer, real) and their associatedoperations (add, subtract, convert). Other abstractions suchas processes, semaphores, ports, timers, memorymanagement, and faults that are found in typicalapplications should be supported as well, along withassociated operations on those abstractions.

M&ing hardware to another level of abstractionprovides huge advantages for testing tools. As statedearlier, the architecture must be the basis for emulationcapabilities and providing execution visibility. As thehardware becomes more aware of programming elements, itgains the abdity to send more meaningful messages to theexternal world. A context switch between processes couldbe sent with a single message, rather than the hundreds ofmachine instructions it takes to implement the switch.

As the processor becom-es the single point ofvisibility, awareness of the progrdng environment

becomes important. A processor with a high-levelunderstanding of program ,entities can emit fewer, moremeaningful - messag-es than a processor that onlycomprehends low-level instructions.6.5 Dedicated Bus

Embedded testing and debugging require aninterface that aliows the processor to communicate with theexternal world without interfering with the behavior of thesystem under test. This physical connection should resideon the target and interface extemall y tlhrough somedetachable mechanism. The separation technique isimportant, since the external debugging system will bedetached from this connection once the system is placedinto operation. The execution behavior alf the programshould be independent of whether or not any external toolis attached.

Assuming an adequate physical connection, thenext detmminah “on is the protocol across it. ‘Ile followingissues must be addressed1) At what rate will messrwes need to be sent?

‘Processor speed raises i~teresting problems, since futurespeeds might be too quick for external processingtechniques. A solution to this problem was discussedpreviously where the processor became aware of high-level program elements. The goal is to decrease thenumber of messages required relative to the number ofmachine cycles. -

2) How much data is associated with a message?If an architecture is required to emit large volumes of datafor messages, there may be instanms where the processormust be suspended to allow the internal debugginghardware to catch up to the current processor state.Higher level messages may compound the problem,since more maningful messages might require more

295

information. There is likely a tradeoff between messagelevel and data volume.

3) Is the connection bidirectional?Visibility concerns dictate that state information travelout of the processor. However, methods requiring

control of the executing program require that stateinformation travel the other direction. Protoczds must bein place to handle contention across the bus and thosemust be extremely well defined, due to the extreme datarate that could will be emmuntered across the bus.

4) Who is the active element in sending message9?Either the processor or the RTS must determine theinformation sent from the processor. The processorcannot provide all the state information needed, while theRTS will likely not be able to maintain adequate speedsfor sending messages.

These questions play a role in determining the interface

between the internal and external debuggers. A likelysolution would be a master-slave relation, where either theinternal or external debugger regulated the other. Thisscenario does not seem likely, since each has such criticalprocessing concerns. Therefore, each will likely executeindependently, while communication is handled via somebus and protocol.

There does exist a master-slave relationship inrespect to the bus, howevex. During program executiw,the internal debugger must ‘ownn the bus, since it’sprocessing concerns are the greatest. It must meet themessage sending deadlines without altering computationsin other parts of the system. There are points dtingexecution where the extend debugger must aeiz cmtrol.If the internal debugger cannot allocate the bus to meet thedemands of the extcmal debuggm, the user must be notifkdthat their requested operation cannot be accomplishedduring a real-time execution.

The final determination is that of the activeelement within the processor. There are two basicapproaches to detemnining control of the internal debugging

activities. In the fiit the processor is active and becomesresponsible for sending messages to the extend debugger.‘fhesecondappmachuse aaspecialdebugge rportionoftheRTS to emit messagea, which is loaded into a dedicatedfunctional unit within the architecture. Tools requireinformation maintained by both the architecture and theRTS. Perhaps the solution lies between the two whereboth the RTS and architecture have the ability to dumpmessages, depending on the cmrcnt mquiremcnts dictated bythe external tool.

7. Run-Time Svs tern Additions

The RTS requirements deseribe an interfacebetween a tool and the underlying system. This is alogical interface requiring substantial hardware support asoutlined above. An obvious goal is to minimize therequired data and computational requhements of the internaldebugger as well as the required communications betweenthe internal and external debuggers.

This paper does not address the question of howthese interfaces should be UtdiZSd. Such SllSWerS should begiven by methodologies and techniques for detecting andlocating errors in embedded, real-time systems. Asdiscussed earlier, the lack of these methods has led todifficulties for determining adequate RTS services fortesting and debugging tools, which has forced a differentapproach to determine the required operations. Since theRTS is in essence offering au implementation of high-levelabstractions, services that provide visibility into theimplementation of RTS abstractions should adequatelyfidfdl the needs of most testing and debugging techniques.

A standard currently exists for implementing theseabstractions in the MRTSI [ARTE89] and CIFO

[ARTE91]. In addition, most of the needs for testing anddebugging can be fulfiiled by these standards. This is notsurprising, since our solution is based on implementationvisibility, and the MRTSI and CEO are providing an

implementation interface. However, it is important to notethat this approach also indicates that implementations thatsupport these staudards should require minimal additions toakw SUppCWt testing and debugging tools as prOpOSed bythis paper. Below is a small discussion surrounding eachof these abstractions and a list of shortcomings in theMRTSI and CIFO for testing and debugging.

7.1 ProcessesConcurrency is a common abstraction used in

embedded systems. A design can be decomposed withoutconcern for computational resources, which can then bedetermined by a scheduler during run-time. A&irqplements concurren cy through tasks and task types. TheCIFO and MRTSI provide extensive tasking supportincludlng identifieation, creation and activation,communication through rendezvous, concurmat access toshared entities, and support for scheduling control.Elements of interest that are not provided by the CIFO orMRTSI include

● Task State - A developer must have the ability to queryand modify the task state for each task in their system.However, a modification could leave the RTS in aninanaistent state. For example, changing a task’sstate from “delaying” to %unning” without removingit from the &lay queue would place the RTS into astate that could not be achieved through normalexecution. However, the same modification ability isavailable on typical debugging systems and should beoffered by embeddcddebwrgera as Wd.

● Commm&ation and Synchronization - A developer

●

7.2

must have the ability tb view and modify eaeh entryqueue to determine the concurrent state of the system.Again, modifications could leave the RTS in anunobtainable state.Scheduling Control - In addition to the extensiveoperations provided by the CIFO for concurrencycontrol, a developer must have awess to the dispatchport (or ports for muhiprqxssor systems).

Interrupt Management

296

One of our criticism of the current approach toembedded testing is that timing errors are revealed late inthe development process. Interrupts are very related totiming issues and their correctness is an important elementin embedded testing. Therefore, support for interrupts isextremely important to target testing and debugging.Faalities provided through the CIFO and MRTSI wouldallow developers to bind various interrupt handlingroutines, enable and disable certain interrupts, mask andunmask interrupts, and generate software interrupts allcontrolled dynamically duting program a program test.

7.3 Time ManagementAs stated earlier, timing issues are extremely

important” to target testing and debugging. Therefore,target tools require sfilaertt control over issues relating totime. Tools must be allowed to view and modify the clock(although such modifications might produce undefinedresults) and the delay Iist of waiting processes maintainedby the RTS.

7.4 Memory Management

Dynamic memory is not typically used byembedded ti@iC4itiOliS due to diffldtk% in dcmonstradngreliability. However, future systems will likelyincorporate algorithms that requite dynamic storage. Inaddition, memory ~agemcnt for dynamic allocations ispart of a RTS and should therefore be included in RTSvisibility and control discussions. A tool will likely

require that ability to demonstrate an application programsbehavior when memory is exhausted.

The MRTSI would need to be extended to provideoperations that mim a collection making it smaller toshow execution behavior when memory is exhausted or

larger to demonstrate correct execution should a collectionbe expanded by the developer. Resizing is not cheap andcould require a gnat deal of computation and data transfers,depending on an implementation.

7.5 Exception/Fault HandlingProper handling of exceptional events is evaluated

during hardwaresoftware integration testing. Therefore,

tools require a great deal of cattrol over exceptions andrecovery mechanisms. One must be able to raise an

exception or fault during program execution and alsomodify handler binding during execution.

Another question of interest might be to locate thehandler for a given fault or exception at a given programlocation. Such information is not easily gained from theunderlying system. The compiler is responsible forhandling exception propagation [ARTE89], so &k “ “ gthe handler from only RTS information might be animpossibility and is at best resolved uniquely for eachcompilation system.

s conclusionsThe goal of this paper is two fold. The first goal

is to identify defkienaes in embedded system testing andraise questions about the future of current tools. Thesecond is to propose a solution to these problems through

architectural and RTS additions. The architectural additionswill certainly be costly in both time and space, requiringspace (transistors) on the chip and access to internalregisters and busses that could cause contention and slowthe execution of other instructions provided by thearchitecture. However, the RTS additions are minimal.We defined the needs of testing as making theimplementation details of common system abstractionsvisible and then determined the functionality required toview and control them. The ARTEWG’S MRTSI andCIFO provided an outstanding basis for this approach.

The RTS additions are admittedly weak. Ourinitial goal was to have the methodologies and techniquesused for testing embedded, real-time systems drive theoperations required by the RTS. Unfort.tmatcl y, suchmethods do not yet exist. As stated earlier, testing anddebugging of embedded, real-time software remains a blackart, with ad hoc methods and techniques. While there hasbeen much research into the concurrency and distributionissues, none has examined real-time constraints, embeddedenvironments, and other issues relating to embeddedsystems Perhaps the MRTSI and CIFO are sufficient forimplementing target level testing and debugging tools.However, this question cannot fully be resolved until moreformal methods exist.

Our next step is to evaluate the additions anddetermine their feasibility. Questions relating the cost ofthese additions to au architecture and RTS in terms of timeand space must be answered. Also, a more completemapping should exist between the added feattues and theimpact they have on the desired features. One can thenmake a valid comparison between a feature and the costsassociated with it.

‘l’he embedded contmllermark~ is currently huge,but has only begun to require the computational powers~SOCiKltti With lUiCrOpKXXWOrS. Embedded i@k.i3tiOttS

have traditional been event driven rather than computationdependent. Due to their light weigh~ easy con@mbilityand expansibility, and lower design complexity, computersare quickly being chosen over mechanical techniques forcontrolling devices. As this transition continues, the sizeand complexity of embedded programs will grow.Controllers will not only have strict timing requirements,but also have significant computational needs as well.This combination requires new approaches to our currenttesting process for embedded systems and therefore, moreeffective tools to aid in testing and debugging embeddedapplications.

References

[ARTES9] Ada Run-time Environment WorkingOroup, “A Model Run-Time SystemInterface for A&m Ada Letters, January,1989.

[ARTE91] Ada Run-time Environment WorkingGroup, “Catslogue of Interface Features

297

and Options for the Ada RuntimeEnvironment,” Special Edition of AdaLetters, Fall 1991 (fI).

[CHIL91] Child, Jeffrey, “32-bit EmulatorsStruggle with Processor Complexities,nComputer Design, May 1,1991.

DD83] Department of Defense, ReferenceManual for the Ada ProgrammingLanguage, ANSI/MIL-STD- 1815a,United States DoD, 1983.

Federal Aviation Association, SoftwareConsideration in Airlx)me Systems andEquipment Certification, RTCA/DO-178A, 1985.

[GILL88] Gilles, Jeff aud Ford, Ray, “A GuidedTour Through a Window OrientedDebugging Environment for EmbeddedReal Time Ada Systems,” IEEETransactions on Software Engineering,1988.

&IAm78] Hansen, B., ~eproduable Testing ofMonitors,” Software-practice andExperience, Volume 8,1978.

Hembold, D. and Luckham, D.,“Debugging Ada Tasking Rograms,”IEEE software, March, 1985.

Iw’fJ=l Intel Corporation, i960 ExtendedArchitecture programmer’s ReferenceManual, 1993.

KOEH91] Koehnemann, H.E. and LindquisL T.E.,“Runtime Control of Ada Rendezvous forTesting and ~U@llg,” Procedm - gs ofthe 24th Hawaii International Conferenceon System Sciences, Volume II, 1991.

LeDoux, C, and Parker, D.S., “SavingTraces for Ada Debugging,” Ada in UseProceedings of the Paris Conference,1985.

Km] Lyttle, D. and Ford, R., “A SymbolicDebugger for Red-Time Embedded AdaSoftware,” Software - Practice andExperience, May 1990.

lyxn6J Tai, K.C., “&producing Testing of AdaTasking Programs,” IEEB Transactionson Software Engineering, 1986.

~A191] Tai, K.C., Carver, R.H., and Obaid,E.E., “Debugging Concurrent AdaPrograms by Deterministic Execution,”IEEE Transactions on SoftwareEngineering, January, 1991.

~AYL80] Taylor, R.N. and Osterweil, L. J.,“Anomaly Detection in ConcurrentSoftware by Static Data Flow Analysis,”IEEE Transactions on SoftwareEngineering, May, 1980.

~AYIJ33] Taylor, R.N., “A General PurposeAlgorithm for Analyzing ConcurrentPrograms,” Communications of theACM, ~y, 1983.

@fAUG851 Mauger, C. and Pammett K., “An Event-Driven Debugger for Ati” Ada in Use:Proceedings of the Paris Conference,1985.

298