Acquiring and Integrating Knowledge from Images for aLarge Scale Hypermedia System

John H. Boose, Larry S. Baum, Randy J. Kelley

The Boeing Company, 7L-44, PO Box 24346, Seattle, WA 98124john@redwood.rt.es.boeing.com, lbaum@redwood.rt.cs.boeing.com, rkelley@redwood.rt.es.boeing.com

Boeing’s Intelligent Graphics team also includes Mike Carter, Susan Chew, Chuck Gebhardt, Brent Hadley,Ed Hall, Tom Jenkins, Cathy Kitto, Steve Pruitt, and Dave Shema

AbstractThe Boeing Company maintains tens of millions ofpages of information associated with the manufactureand delivery of its products. Much of this informationmust be made available electronically. We havedeveloped tools to automatically convert and integrateelectronic data into industry standard formats. Someof the technical challenges include I) handling a widevariety of source formats, 2) making sure that thetools scale up to handle millions of pages ofinformation, and 3) adding functionality to graphics.We have processed over four million pages of textcontaining tens of thousands of graphics. In this paperwe describe on our tools that recognize and useinformation within vector and raster images.

Four Million Pages and Growing

The Boeing Company is committed to Air TransportAssociation Specification 2100 (ATA, 1995) whichrequires the use of the Standard Generalized MarkupLanguage (SGML) (Goldfarb, 1990) To comply these specifications Boeing has converted millions ofdocument pages to SGML. Document sources includecommercial tools (e.g., Microsoft Word, InterLeaf,FrameMaker), internal proprietary tools, and data fromexternal vendors.

Manually converting such documents would beprohibitively expensive and error-prone. To meet thischallenge our team built a series of text autotaggers thatautomatically convert documents into SGML. From 1990to the present we developed autotaggers to create a testbedhypermedia system containing data from over four millionpages. We use Electronic Book Technologies’ DynaTextsystem as the browser, including custom extensions forviewing graphics. We will report our text autotaggingmethods and results elsewhere.

Graphic Navigation and Functionality

The autotaggers receive text in various formats. Theyreceive graphics in vector formats (usually CGM) d

raster formats (usually TIFF) and originally did little morethan pass the graphics to viewers in DynaText. Thegraphics lacked information for navigation and retrieval.Users often needed to examine many images to find theright one, including multiple panning and zoomingoperations. Hyperlinking from a text reference in a graphicwas not possible, nor was full text search. Graphics lackthe basic functionality of the SGML text. The graphicsalso lacked useful functional information -- What is thelogic in this troubleshooting diagram? What are therelationships between items in this table? What lines inthis wiring diagram represent continuous circuits?

We built a series of graphic recognition tools that usevector and raster graphics to produce navigational andfunctional information represented in SGML. We thenintegrated this information into the testbed.

Graphics Recognition Tools

We used a similar approach for each graphic recognitionproblem:

¯ Develop a tool to reason about vector or rastergraphics.

¯ Tune and evaluate the tool for accuracy and the abilityto scale up.

¯ Integrate acquired graphic information with text in thetestbed.

¯ Evaluate the usability of the results.¯ Transfer the results to products and services.The image counts given below are for a recent Boeing

747 airplane maintenance manual set. They illustrate thelarge scale of the problems.

Troubleshooting charts are vector drawings thatprovide mechanics with decision trees for fault isolation.One drawing can consist of as many as 30 individual sheetswith intersheet connections linked by references. Oursoftware analyzes the layout of the boxes and the arrowsconnecting them and builds an internal representation ofthe decision tree. It then generates hot spots so that userscan easily navigate within a sheet and among sheets as wellas follow links to other information. This manual setcontains over 750 charts containing over 16,000 hot spots.

37

From: AAAI Technical Report SS-97-03. Compilation copyright © 1997, AAAI (www.aaai.org). All rights reserved.

-- EE~OF:T ANtAPPLICABLE CBCUIT BREAKERS I I R1 ~;

11(:33 11R10

ILP|4 ODNT RAN FLT DK J 11R24

11PZ5 ~ ] 11R2SILP20 ~J 11R28

ILP27 [ZONE TEMP CONT VLV CLOSE AFT 11R27

ZONE CNTLR JZONETEMP IND [ZONE ~OVHTFLT DK [

lZONE DUCT O’V’HT FINDZONE 13.1L"T OVHT MIDZON E DI.IL’T OVHT AFT

AIR DISTRIBUTION AND TEMPERATURE CONTROL (SHEET 1 ) - FAULT CODES

EFFECTE/rrYI AIR CONDITIONING

AL L 2 7 P,0DE 4FEB 10188

Figure 1. The fault diagram recognizer finds arrows, diamonds, and other information, then uses knowledgeabout diagram layout to produce the application’s flow of logic.

Fault reporting diagrams are vector images that pilotsuse to determine what messages to record in the flightlogbook when a malfunction occurs. The variations inlayout are more complex than the troubleshooting charts;still we’ve been able to accurately determine the decisionnetworks in these diagrams and generate over 20,000 hotspots in over 1,600 diagrams (Figure 1).

Component location diagrams provide exploded viewsof aircraft and their components. Typically one diagramconsists of multiple sheets, each of which has one or moresubpictures (insets) with internal callout references. Oursoftware must reliably subdivide each image intosubpictures and generate the hot spots that link the calloutsto the correct subpicture, including references to other

sheets. In addition, the software generates hot spots aroundeach equipment number, so that mechanics can easilynavigate from the picture to information about that piece ofequipment. This manual set contains over 650 vectorcomponent location diagrams; the system produced over3200 subpictures containing over 25,000 hot spots.

Component index tables are vector drawings that tellthe mechanic where to go to find the maintenanceprocedures for the equipment illustrated in the componentlocation drawings. The index table recognizer determinesthe individual table cells and relationships among cellcontents. It generates over 14,000 hot spots in over 300tables linking the drawings to component location

38

diagrams, other component index tables and tomaintenance procedures (Figure 2).

Wiring diagrams are vector drawings extracted fromcomputer-aided design data sets. The intelligent graphicssoftware analyzes both the vector image and the data set todetermine how wires are laid out in the drawings so thatcircuit tracing is enabled in the hypermedia viewer. Inaddition, there are hot spots providing hyperlinks toequipment lists and wire lists as well as other diagrams.This manual set contains over 1400 diagrams with over135,000 hot spots (Figures 3, 4).

System Schematics are similar in functionality andrecognition requirements to wiring diagrams. This manualset contains over 1300 system schematics with over461,000 hot spots.

Parts illustrations are raster images that provideexploded views of the airplane much like componentlocation diagrams. They contain callouts and item

numbers that must be linked to other pages. Because theyare raster images we must perform more complex analysisthan with the vector images. We have developed routinesto quickly process callouts and item numbers with highaccuracy. This manual set contains over 13,000 sheetswith more than 241,000 hot spots (Figure 5).

Standards tables contain information about Boeingengineering practices. The sources for many of thesetables are paper copies that were several generations old,made with typewriters and ruled with ball point pens. Theconversion team scanned these images for on-line use andwas planning to manually re-enter the information. Wedeveloped recognition software to extract the logic in thetables. Together with a commercial optical characterrecognition package we automatically converted over 6600tables to InterLeaf format (Figures 6, 7). Later these willalso be converted to SGML.

Figure 2.

IG0olln~l o Corrponent tooation IiR~urt 101 (Sheet 4) I

The component location recognizer uses page layout logic to induce groups and attach caliouts.

39

-- 811Ll?.

Figure 3. A wiring diagram recognizer finds connected circuits and links references together.

iLL

Equipment Nmaber: AE0418Part Number: 284U2415Part Description: POWER DIST-RIGHT/P415

Vendor Number: V81205Quantity: 0000

Station: 440HL: 150ET,: R B5

Figure 4. Users follow hotlinks to retrieve equipment information. The systemstores wiring diagram links and other references in the DynaText database.

4O

DO

Figure 5. A raster parts illustration recognizer finds caliouts and reference numbers,using symbol finding and optical character recognition.

P~=L,,ementiii i

WP-r ~GJt.I ~tfimti~J,ii lul ill ii luu

A~ Tr~ N~ ~Pd’~mm ~lDr

=,

A~ Yellow omwiii

E~.Z~A ~A A~IA AWS AS,19 Yellmv Nm~g

NB~IAQE>

Y=II~ ~ak ~T~tB

~AZlOIA A21OIA Yallc~,,,, ,,,,,,,,,

i b~A Axo , ou7 Yellow ’Rlj~kill i J i iiii i imlllll ill

Figure 6. This original standards table was made using a typewriter and ball-point pen.

Figure 7. After scanning the table in Figure 6, the table recognizer automatically finds cells, performs characterrecognition, then produces standard mark-up files.

41

Evaluating Recognition Results

To evaluate the results of text tagging our team commercialSGML verifiers and specialized custom systems. Wedeveloped tools that analyze the text output for consistencyand completeness, including statistical measures and linkverification. Often these tools find authoring errors as wellas tagging errors.

It is more difficult to build such tools to check graphicrecognition accuracy. We built a custom graphic analysistool that simultaneously shows recognized objects and thecorresponding SGML (Figure 8). For example, if the

recognizer finds a reference, the viewer displays the hotspot along with the information as to what type ofreference it is and to what it refers. Using this viewer wemanually sampled thousands of images to assessrecognition accuracy. Recognition accuracy ranged fromabout 97% for component location graphics to 99.9% fortroubleshooting charts. These results exceeded ourperformance goals.

We can also find some types of graphics authoringerrors. For example we can automatically detect incorrectreferences that point to non-existent locations. Previously,such errors could only be found by manual inspection.

Text 2415 3558 final "DOWNLOCK SENSOR’:UneWtdth 8;Une 608e 7~,~J~ 6158 7287 (i153 7283 (1066 7233 7583 7073 (1874 7000 6843 7027 ~ 7073 (1874 7016

6064;Une ~ e247 5351 620g 5251 6188 52531233 5351S208:CharOrl 0 97 80 0;CharHelght 97;TextFontlndex 2;Text 558o 48e7 final "BODY GEAR WHEEL WELL";CharOrl 0 (m (14 CherHelght 68"TextFontlndex 14;iText 5161 5721 final "LH, RH,":!Text 51 (11 5583 final "ALTERNATE DOOR":Text 51el 5837 final "(Sqge.S427)";Tm(t 511ll ~ final "WARNING SFNSCJR":

,~HSUST hld-’HS2">,~HOTSPOT>,=PT x-g18 y-7261 ~.cPT x..glB y-4"/gS-,oPT x~ y,-4"/88~.cPT x-3885 y-7261 ~.,oPT x-g18 y-72111 ¯~TIOTSPOT~,c/HSUST-

,cCOMP old=’COMP_UNLABELED2" hsllsts-’HS2"~~HSUST hld-’HS3"-cHOTSPOT*oPT X-2375 y-7311 ̄oPT x-2375 y’6727~¯ PT x-~>29 y’¢727¯¯ PT x-3528 y-7311 ̄¯ PT x-2375 y-7311 >c/HOTSPOT>c/HSLIST>-TEXTAREA old-’O~" hallsts-"HS3">,¢PARA>

Figure 8. We assess recognition accuracy using our custom graphics analysis tool.

42

Future Challenges - Work in Progress

We are currently working with Boeing’s library of millionsof structural and tooling raster drawings. These drawingshave been scanned from aperture cards and placed on-line.We would like to recognize objects such as title blocks,part numbers, dimension lines, materials lists, and partstables. Much of the text is hand written and many of thescanned images are skewed or noisy. This is challengingwork (Figure 9).

We are also working on visual information retrievalstrategies, including content-based and similarity-basedmethods (Figure 10).

with text in on-line navigation systems, addressing theproblems of scale and accuracy. We hope to continue toadd functionality to graphics to enhance the overall utilityof Boeing’s digital data.

References

ATA (1995). Air Transport Association Specification 2100- Digital Data Standards for Aircraft Support, 1995, Ordercode A090

Goldfarb, Charles F. (1990). The SGML Handbook, OxfordUniversity Press, ISBN 0-19-853737-9

Summary

We have successfully added functionality to tens ofthousands of vector and raster graphics, integrating them

Figure 9. The structural and tooling drawing recognizer must find part numbers, title blocks, and dimensions linesin raster images.

43

Figure 10. Our prototype visual search engine finds similar vector images in a database of over 6,000 drawings.