Intelligent Shipwreck Search Using Autonomous Underwater Vehicles

Jeffrey Rutledge∗1, Wentao Yuan∗2, Jane Wu1,Sam Freed3, Amy Lewis3, Zoe Wood3, Timmy Gambin4, Christopher Clark1

Abstract— This paper presents an autonomous robot systemthat is designed to autonomously search for and geo-localizepotential underwater archaeological sites. The system, based onAutonomous Underwater Vehicles, invokes a multi-step pipeline.First, the AUV constructs a high altitude scan over a large areato collect low-resolution side scan sonar data. Second, imageprocessing software is employed to automatically detect andidentify potential sites of interest. Third, a ranking algorithmassigns importance scores to each site. Fourth, an AUV pathplanner is used to plan a time-limited path that visits siteswith a high importance at a low altitude to acquire high-resolution sonar data. Last, the AUV is deployed to follow thispath. This system was implemented and evaluated during anarchaeological survey located along the coast of Malta. Theseexperiments demonstrated that the system is able to identifyvaluable archaeological sites accurately and efficiently in a largepreviously unsurveyed area. Also, the planned missions led tothe discovery of a historical plane wreck whose location waspreviously unknown.

I. INTRODUCTION

Current methods of searching underwater areas for archae-ological sites involve many steps with expensive equipmentand time consuming analysis. First, a large survey areais selected taking into account the likelihood it containsvaluable sites and the risk those sites have of being damaged(e.g., dredging, pipeline construction, and fishery install-ment). Second, a high altitude survey is performed usingside scan sonar, often deployed with a tow fish or morerecently with an autonomous underwater vehicle (AUV).Next, experienced humans closely analyze images producedfrom the sonar to find and rank potential sites. This rankingis important because there typically is not enough time orresources to revisit all potential sites identified. Furthermore,sites usually need to be revisited, up close, to accuratelyassess their value. This typically is accomplished with humandivers or remotely operated vehicles (ROVs). Confirming thevalue of a site is especially important for its preservation.

Meanwhile, Autonomous Underwater Vehicles (AUVs) arehaving an increasingly significant impact on underwater ar-

∗Equal contribution.1Department of Engineering, Harvey Mudd College, Claremont, CA

91711, USA {jrutledge,jwu,clark}@hmc.edu2Robotics Institute, Carnegie Mellon University, USA. The work

was done while Wentao was working at Harvey Mudd College.wyuan1@cs.cmu.edu

3Department of Computer Science, California Polytechnic State Uni-versity, San Luis Obispo, CA, USA {stfreed, alewis19,zwood}@csc.calpoly.edu

4Department of Classics and Archaeology, University of Malta, Msida,Malta timmy.gambin@um.edu.mt

This material is based upon work supported by the National ScienceFoundation under Grant No. 1460153.

(a) OceanServer Iver 3 AUV

(b) Fairey Swordfish wreck

Fig. 1: The OceanServer Iver3 AUV and a sidescan sonarimage of the Fairey Swordfish dive bomber wreck discoveredin the deployments.

chaeology ([1], [2]). Unlike human divers, who are typicallydepth limited to 100 meters, AUVs can operate for longhours at depths of thousands of meters. The advance insonar imaging technology also enables AUVs to acquirehigh-resolution images of the seafloor (e.g., [3]), and theincreasing usage of AUVs in archaeological surveys has ledto discoveries of deep water shipwrecks that are inaccessibleby humans (e.g., [4]).

This paper proposes an AUV-based system to automatethe process of searching for underwater archaeological sites.The system invokes a multi-step pipeline. First, the AUVconducts a high altitude scan over a large area to collect low-resolution side scan sonar data. Second, image processingsoftware is employed to automatically detect and identifypotential sites of interest. Third, these sites are assignedimportance values using a ranking algorithm. Fourth, anAUV path planner is used to plan a time-limited path thatvisits the identified sites (of highest value) at low altitudeto acquire high-resolution sonar data. Last, the AUV isdeployed to follow this path.

This process requires two key steps of automation: Thefirst is an image processing pipeline which locates and rankspotential sites in the large survey area. The second is aplanner that uses these ranked sites to create missions foran AUV to revisit sites. This automation decreases the timebetween the large survey and site revisiting—allowing formore efficient use of limited time with expensive AUVresources and quicker evaluation of targets at risk of damage.In addition to efficiency gains, this method can reduce therisk of overlooking important archaeological sites becauseexamining large scans from the high level survey is anexhaustive task even for human experts.

The complete system was implemented and tested usingan OceanServer Iver3 AUV (Fig. 1a), in an unexplored areaon the coast of Malta. The AUV was able to obtain high-level scans of the entire area and low-level, high-resolutionscans of some important sites it identified. Fig. 1b showsan example of the high-resolution scans. The experimentsdemonstrated that the proposed system has the potential ofsurveying large areas of seafloor and identifying valuableunderwater archaeological sites with a very low level ofhuman supervision.

The contributions of this paper is summarized as follows:• An approach to automatically detect and explore under-

water archaeological sites using a combination of robotvision and planning.

• A demonstration that the system is able to work in realworld deployments and find new archaeological sites.

• An evaluation of the pipeline that highlights the chal-lenges in detecting underwater archaeological sites andprovides insight for future improvements.

II. BACKGROUND

a) Underwater Archaeology Using Side Scan Sonar: Inan underwater environment, acoustic imaging devices suchas sonars have a significant advantage over cameras. Lightsignals can be attenuated over short distances underwater andrequire sufficient lighting conditions. Sonars can operate ata much longer distance with no light. Among the varioustypes of sonars, Side Scan Sonar (SSS) is especially popularfor underwater archaeological surveys ([5], [6]) because ofits large coverage and bathymetric capabilities.

Manual analysis of SSS data requires searching largeimages for which many square kilometers of area are repre-sented as millions of pixels. Typically, a marine archaeologistperforms a quick scan looking for the echo-shadow patternproduced by objects protruding from the seafloor [7]. Ifan echo-shadow pattern is found, closer inspection will beconducted. The surrounding terrain is evaluated to determineif the pattern a) is in a field of rocks, b) has texture that is flatwith sharp corners, c) has the shape of a recognizable object,and d) is of reasonable size. If these observations pointtowards it not being a rock, the patch will be recognized asa potential site. Once the first scan is completed, the markedsites will be ranked on a numerical scale. This ranking allowsthem to pick sites to visit using the limited resources theyhave [7].

b) Object Detection in Sonar Images: Previous workson object detection from side scan sonar images make use ofthe echo-shadow pattern characteristic of objects protrudingfrom a surface([8], [9], [10]). Unlike normal camera images,which are colored and illuminated from a multitude ofsources and reflections, side scan sonar produces imagesilluminated by only one source. The lack of color preventsthe use of colors to separate objects from their background,but the single source of illumination makes objects create abright echo followed by a shadow in the direction away fromthe sonar. This echo-shadow pattern can be identified usingrectangular features searching for adjacent abnormally brightand dark regions. [9] uses three rectangular features—themean value of pixels in a rectangle—to separately identifythe background, shadow, and echo for each pixel. [10] uses amultitude of Haar-like features (rectangles with positive andnegative sub regions). As can be seen in Fig. 3, Haar-likefeatures can be shaped to match the echo-shadow pattern.

The echo-shadow pattern of objects can also be caused bynatural terrain (rocks, ridges, ripples, etc.), which is a mainsource of false detections. To avoid these false detections,[8] incorporates a sand ripple filtering algorithm which fitsellipses to potential shadow detections. If it finds a cluster ofdetections with similar orientation and location it will assumethey are ripples and filter them out.

The work in [9] improves on previous algorithms byeliminating hyper parameters, allowing the algorithm tobetter generalize across sea floor environments. For example,the size parameter of the rectangular feature used to create ashadow map is defined by the height of the object beingsearched for. Also, instead of using a static threshold todetermine abnormally dark shadow regions, the regions arecompared to a background map. These qualities allow theunsupervised method to better adapt to differing sea floorenvironments.

c) Learning to Rank with CNN: Learning to rank is awell-studied problem in the information retrieval community.The problem can be cast as a machine learning task in whicha ranking model is constructed using training data, and thismodel can automatically sort new objects according to theirdegrees of importance [11]. One particular approach to theproblem is called Ranking SVM. It was proposed by [12]based on a variant of support vector machines (SVMs). Bytransforming examples into their pairwise differences, theranking problem can be formulated as a binary classificationproblem that can be solved by a linear SVM.

The performance of linear classifiers like SVM dependheavily on the availability of good features. Recent advancesin deep learning have shown that deep convolutional net-works are capable of extracting high-level features that workwell for tasks such as image classification and object recogni-tion ([13], [14]). Training a deep CNN from scratch requireslarge sets of labeled data. However, [15] has shown that thefeatures learned by these deep networks are generalizable toother tasks even without further training. In particular, [16]has shown that features generated by CNN can be used incombination with SVM to achieve good classification results.

Fig. 2: System pipeline

III. INTELLIGENT SHIPWRECK SEARCH

The proposed system for intelligent search invokes themulti-stage pipeline illustrated in Fig. 2. In the first stage,an AUV Deployment is conducted to perform a high altitudesurvey of the sea floor. The path for this initial survey, P0, isa lawnmower pattern that covers the entire area of interest,(e.g., 4 km2). The AUV collects a set of side scan sonarimages Iss which are then passed to the Image Processingsoftware. The output of the Image Processing software is aranked list O∗ of site proposals, which are raster locations ofbounding boxes of potential archaeological sites in the sonarimages (see Fig. 5). The raster locations are then convertedto projected coordinates (of earth) and passed to the PathPlanner, which will construct a path P that visits a subsetof proposals in O∗ for low-altitude, up-close inspection of thesites. This can in turn generate additional side scan sonar datathat can be fed back into the system for continual replanningand site revisits.

Described below are key functions of the Image Process-ing and Path Planner software blocks.

A. Shadow Feature Convolution

A shadow map Ish is created by convolving the inputtedside scan sonar image Iss with the Haar-like feature kernel[17] shown in Fig. 3. This kernel is defined by the parametervector W = [we ws], where we is the width of echoregion and ws is the width of shadow region. This feature isdesigned to look for the echo-shadow pattern characteristic ofobjects protruding from the sea floor. This feature essentiallysubtracts the sum of the shadow region from the sum of theecho region. Note, to prevent the relative area of the regionsfrom affecting the feature value, the regions are weighted bytheir area so a constant image would produce a feature valueof zero.

Fig. 3: Haar-like feature kernel (left) and a horizontal cross-section of the kernel (right)

B. Contour Extraction

The shadow map Ish is thresholded at α · σ, where σ isthe standard deviation of pixel values in the shadow map andα is a tunable parameter where smaller values allow lightershadows to be detected. This thresholded shadow map Ic is abinary image in which shadow contour pixel locations havevalues of 1, and otherwise 0.

C. Clustering Algorithm

This algorithm clusters the shadow contour locations fromIc into a set Oc of detection boxes using an agglomerativealgorithm. The clustering algorithm is essentially one itera-tion of bottom-up hierarchical clustering [18]. The algorithmappends contour pixels to clusters and joins clusters thatare within a distance threshold, ρ. Bounding boxes are thenfit around the clusters of contours to produce detectionboxes. The bounding boxes are the smallest upright rectanglecontaining all pixels in the cluster.

D. Area Thresholding

The set of detection boxes Oc is then filtered based onarea to become a set of proposals OA ⊆ Oc. Unlike desireddetections that typically have one to three large contours,rock fields usually produce many small shadow contoursthat are close enough to get clustered. For such rock fields,the area covered by shadow contours Acontour is much lessthan the total area of the bounding box Abox. In this case, atuneable threshold Aprop is compared with the ratio of thesetwo areas, (see Eq. 1).

Seafloor ridges, another source of false positives, areusually long (hundreds of meters) when compared to actualsites of interest. This makes their area Acontour abnormallylarge, promoting the idea of a filter that removes potentialsites with areas greater than Amax. Similarly, potential sitesthat are too small can be filtered using a minimum area Amin.

Hence, only sites in the input set O that meet the followingcriteria are added to the function’s output set OA:

Acontour

Abox> Aprop

Abox < Amax

Abox > Amin

(1)

(a) Scan (b) Convolution

(c) Shadow Map (d) Contours

Fig. 4: Site Proposal Processing Examples on Detected Site

Notably, these thresholds may be set based on the size ofdesired objects.

Fig. 5: Site proposals generated from a scan in data set B, andevaluated against human labels. Green: Labeled Site Found;Red: Labeled Site Missed; Blue: True Positive; Yellow: FalsePositive;

E. Hyperparameter Tuning

Most parameters involved in the image processing stepscan be set based on the characteristics of the desired target.Those that cannot are α, ρ, Aprop. To tune these hyperparam-eters a mix of intuition based tuning and random searchingwere utilized. [19]

F. Proposal Ranking

Next, each site proposal in OA is assigned an importancescore to produce the ranked list of site proposals O∗. Theranking algorithm consists of three steps. First, each siteproposal p ∈ OA is grown to a subimage of size 224× 224.Next, a 50 layer residual network trained on ImageNet isused to extract a 1×2048 feature vector x from the subimage,using average pooling over the output of the last convolution

block (see [14]). Finally, x is fed to a trained ranking SVMwhich outputs the importance score s = w·x, where w is theSVM’s learned weights. The output of the ranking algorithmis a list of tuples (p, s) ordered decreasingly by the score s.

To train the ranking SVM, data sets were created by havingexperienced users label site proposals with a ranking numbery ∈ {0, 1, 2}, (see example rankings in Fig. 6). This yieldsa set of data tuples S = (x,y), consisting of feature vectorsand their associated labels. Following the practice in [12], Sis then transformed into a new set of data tuples S′. Eachpair of data tuples (x1,y1), (x2,y2) ∈ S where y1 6= y2

is transformed into a new tuple (x1 − x2, sgn(y1 − y2)) ∈S′, where sgn is the signum function. The ranking SVM istrained on S′ as a linear SVM [20].

Note that the typical dimension of site proposals is 60 to150 pixels. Expanding the to 224 × 224 subimages allowsthe ranking algorithm to incorporate surrounding seafloorinformation. Further, 224 × 224 is the standard input sizefor many pre-trained deep CNNs (e.g., ResNet [14], VGG[13]), making it convenient for feature extraction.

G. Path Planner

Given a ranked list of site proposals O∗, the objective ofthe Path Planner is to return an optimal path P that visits asubset of the sites in O∗.

The search space consists of N site proposals in O∗ andthe AUV, which form a complete undirected graph withvertices V = {v0, . . . , vN} and edges E = {eij | 0 ≤ i <j ≤ N}. The coordinates of v1, . . . , vN are given by thelocation of the center of the corresponding site proposal inUniversal Transvers Mercator (UTM) coordinates, and thecoordinate of v0 is given by the start location of the AUV.Each vertex vi is associated with a non-negative reward riproportional to the probability of site i containing a target(0 ≤ ri ≤ 1), as well as a non-negative cost ci, given by thedistance that the AUV needs to travel to explore the site usinga lawnmower pattern. Each edge eij also has an associatedcost defined as cij = dij + 0.5(ci + cj), where dij is theEuclidean distance from vi to vj .

The planning algorithm is a two-step approach that firstselects a subset S of O∗, then returns an optimal tour of Susing the Lin-Kernighan-Helsgaun (LKH) heuristic for theTraveling Salesman Problem (TSP) [21]. The subset S isselected as described in [22], and sites are chosen based onthe ”Reward-to-Connection-Cost Ratio” (RCCR), defined asrici

for each vertex of the graph.After a subset of the vertices S is selected, the LKH

algorithm is executed to obtain an optimal tour for the TSPproblem.

IV. EXPERIMENTS AND RESULTS

In this section, the different components of the pipelineare evaluated, as well as the complete pipeline, using ex-periments and real world deployments. Three sets of sonarscans were collected for the evaluation, denoted by A, B andC. Each of these data sets were examined by experienced

(a) Not target(label 0)

(b) Maybe target(label 1)

(c) Definite target(label 2)

Fig. 6: Proposals of different importance levels

users who applied labels for all the sites they thought hadthe potential to be archaeological sites.

Data set A is from a survey completed using a BF12-1006Bluefin AUV equipped with Edgetech 2205 side scan sonars.The AUV flew at 15m altitude and the sonars were set touse 410kHz frequency and 100m range. The survey coveredan area of 4 square km.

Data set B was collected from a multi-day survey com-pleted using an OceanServer Iver3 AUV equipped with anEdgetech 2205 side scan sonar. The AUV flew at 14maltitude and the sonars were set to a frequency of 600kHzand range of 80m. The survey was conducted between thedates of June 12, 2017 to June 19, 2017 (the top 6 rows inTABLE III) which covered a 2.7km×0.8km area in total.

Data set C was also collected using the same equipmentand settings as in data set B, but from a different area (seethe 7-th row in TABLE III).

A. Evaluation of the Site Proposal Generator

In this section the performance of the site proposal gener-ator of the pipeline is analyzed. The goal is to evaluate howwell the site proposal algorithm can propose sites of archae-ological interest. To accomplish this, experienced humanslabeled sites they thought had potential to be archaeologicalsites. These labels are compared against the site proposalgenerator’s output to calculate recall, precision, and adjustedprecision. Additionally, an adjusted precision is calculatedfrom having humans relabel the site proposal generator’sproposals after they are generated. This yielded an extra 58targets in data set A and 24 targets in data set B.

Data Set Recall Prec. Adj. Prec. Sites ProposedA 93.75% 20.71% 62.8% 140B 18.18% 10.43% 31.3% 115

TABLE I: Site proposals results

The motivation for calculating the adjusted precision is thedifficulty human labelers have in finding all the targets in a

survey. The resolution of the scans makes the labels of smalltargets ambiguous to humans and hard to find. This difficultyis exhibited by the site proposal algorithm proposing truesites that were missed by human labelers.

The performance of the algorithm on data set B is worsethan on data set A because data set A contains fewer definitetargets. There are 8 label 2 sites in data set B, but 43 label2 sites in data set A. Despite having a low overall recallon data set B, all of the label 2 sites were proposed by thealgorithm.

The two difficulties mentioned above, finding all the sitesand accurately determining site values, are key parts of thissite proposal problem. These difficulties not only make ithard to label accurate test data sets, but show the task isdifficult for even experienced humans.

Despite the poor metrics, the site proposal generator isable to capture all the sites that labelers are certain to havevalue, while also filtering more than 95% of the area in thehigh level scans.

B. Evaluation of Proposal Ranking

The proposal ranking algorithm is evaluated by three setsof experiments. The first experiment compares the featuresthat are extracted from site proposals and passed to theranking SVM. The second experiment investigates how thecomposition of training data affects the ranking result. Thethird experiment evaluates the effectiveness of the ranking inthe context of planned missions. For all the experiments, thetraining and testing data was drawn from the 101 proposalsoutputted from the generator when processing data set A,and the 124 proposals from processing data set B.

1) Choice of features: This set of experiments looks intothe choice of feature vectors extracted from site proposalsand passed to the ranking SVM. Specifically, it comparesthe features generated by the convolutional neural networkswith a hand-crafted feature set. Below is a description of thehand-crafted features and a bar plot showing the distributionof some of these features. It can be seen that some features

(e.g., 10 percentile of the pixel value) are quite informativefor distinguishing the positive and negative examples.

1) log(w/h), w and h are detection box width and height2) Total area of the shadow contours3) Area of the largest shadow contour4) Ratio of total contour area to detection box area5) Ratio of largest contour area to total contour area6) 10 percentile of the pixel values in the detection box7) 90 percentile of the pixel values in the detection box8) Mean of the pixel values in the detection box9) Standard deviation of the pixel values in detection box

10) 10 percent of shadow map values in the detection box11) 90 percent of shadow map values in the detection box12) Mean of the shadow map values in the detection box13) Standard deviation of the shadow map values in the

detection box

Fig. 7: Bar plots of some hand-picked features on data set B

In the experiments, each set of proposals is randomlysplit 50 by 50 into training and test sets. Then, a rankingSVM is trained on the training set using the features intest, and two metrics—Normalized Discounted CumulativeGain (NDCG) [23] and recall of the definite targets at rank10—are calculated on the test set. This train and test cyclewas repeated 500 times to compute an average and standarddeviation of metrics (TABLE II).

Feature set Data set NDCG at rank 10 Recall at rank 10Hand-pick A 0.73± 0.09 0.62± 0.12

ResNet A 0.85± 0.07 0.72± 0.13Hand-pick B 0.32± 0.14 0.52± 0.36

ResNet B 0.71± 0.14 0.89± 0.20

TABLE II: Ranking performance using hand-picked andResNet-generated features

To interpret the results, note that the NDCG at rank 10 isgiven by the total gain of the top 10 proposals divided bythat of an ideal ranking, where the gain of a proposal rankedi with label li is given by

2li − 1

log2(i+ 1). (2)

In other words, the gain measures the importance of theproposal (indicated by its label), discounted by its positionin the ranking (lower ranked proposals are discounted more).

The recall at rank 10 is the percentage of proposals labeled2 that are ranked in the top 10. While NDCG focuses on the10 top ranked proposals, recall gives us a sense of what ismissing from the top 10.

The results show that features generated by ResNet gaveconsistently good performance on both data sets, whereas thehand-crafted features performed acceptably on data set A, butvery poorly on data set B. This suggests features learned byconvolutional neural nets generalize better than hand-craftedfeatures. The reason could be that the hand-crafted featuresare designed by manually inspecting the statistics of dataset A, but the ResNet-generated features are produced by aneural network trained on large amount of general data. Inaddition, the large number of features that ResNet generatescan also make it more robust to variations in the data. It isvirtually impossible to come up with a hand-crafted featureset with 2048 features.

2) Mixed training: A particularly challenging aspect ofthe proposal ranking problem is the scarcity of labeled data.In the previous experiments, the training and test sets areof the same size and from the same set of proposals, but inpractice, given a set of unlabeled proposals, often there is notan equal amount of labeled proposals collected from the samearea using the same equipment for training. Thus, this set ofexperiments investigates the effect of incorporating trainingdata from another source on the ranking performance.

Specifically, the ranking results of ranking SVMs trainedon a portion of proposals from B together with all 101proposals from A are compared against the results of rankingSVMs trained on the portion of proposals from B only, asshown in Fig. 8. The NDCG score is calculated on a fixed setof 62 proposals from data set B, which is not included in anyof the training sets. Note that for the bottom left yellow point,since no training data is available, the NDCG is calculatedusing a uniform ranking (i.e., everything is given the samescore) as a baseline.

Fig. 8: Comparison of models trained on a mixture ofproposals from A and B and those trained from B only.

Fig. 9: Left: percentage of all targets (proposals labeled 1 or 2) visited in limited time. Right: percentage of definite targets(proposals labeled 2) visited in limited time. The percentage is averaged over 20 runs with random start locations.

Two observations can be made from the results. First, ifthere is a lack of labeled proposals from the same sourceas the proposals to be ranked, bringing training data fromanother source, even if the distribution of the additional datais different, can improve the ranking results. In these exper-iments, the additional data source is labeled proposals froman older deployment (data set A), but other less expensivedata sources like synthetic data could also be useful.

Second, given an unlabeled set of proposals from a newdeployment, labeling a small portion of it for training canimprove the ranking results. The ranking algorithm showssome ability to generalize across data sets (the SVM trainedsolely on proposals from A scored an NDCG of 0.4 onproposals from B). However, the additional labeling stilloffers an improvement because factors like the equipment,sonar settings and the environment of the survey area havea heavy influence on the distribution of the data.

3) Evaluation with planning: This set of experimentsevaluates the effectiveness of the ranking algorithm in thecontext of the entire pipeline. Since the purpose of theranking is to tell the planner how important each site proposalis, these experiments investigate whether the ranking resultshelp the planner generate plans that visit more important siteproposals in a limited amount of time.

In the experiments, a ranking SVM is trained on 101proposals from A and 24 proposals from B, and is used toproduce importance scores for the remaining 100 proposalsin B. The scores are then fed to the planner to plan a missionwith a certain time limit. The planner takes into accountimportance scores as well as proximity in determining whichsite proposals to visit.

Fig. 9 shows the percentage of targets visited in a sim-ulated mission with time limit. The yellow curve is theresult of missions generated using ground truth rankinglabeled by human experts; the green curve is the result ofmissions generated using automatic ranking learned fromdata; the blue curve is the result of missions generatedwith uniform/no ranking. The gap between the green andblue curves shows the ranking algorithm helps the plannergenerate missions that prioritize more valuable targets. For

definite targets (Fig. 9), the performance of automatic rank-ing quickly converges to that of the ground truth ranking.However, the gap between the green and blue curve in theleft figure indicates that the learned ranking model is able todistinguish definite targets from non-targets but struggles torecognize maybe targets. As discussed above, these targetsare quite hard to distinguish from false detections even forhuman experts. How to recognize and handle maybe targetsis a challenging task for future work.

C. Field Deployments

Field deployments were conducted that validated the siteproposal generator. At the time of deployment the proposalranking algorithm was not trained and hence human oper-ators were used to rank proposals. Despite not validatingthe entire pipeline, these deployments still demonstrated animprovement in the overall search process by proposingobjects not considered by humans and by planning efficientreturn missions.

During these deployments, the planner generated lowaltitude paths to revisit site proposals that included threenotable archaeological sites. In following the paths, the AUVgathered high frequency sonar data (for increased resolution),and video data. From this data, human experts hypothesizedthe three sites were remnants of a plane wreck, a shipwreck, and a plane debris field. Of greatest interest is theplane wreck, identified as a WWII era Fairey Swordfish divebomber (show in Fig. 1b), later confirmed from a scuba dive.

Although the system was not fully implemented duringthe deployments, i.e., it was without the proposal rankingalgorithm, the success of these deployments demonstrate 1)the benefit of using the system in tandem with experiencedhumans, and 2) potential of the entire system for use withreal underwater archaeological site search expeditions.

V. CONCLUSIONS

Proposed in this paper is a system for automaticallysearching and exploring underwater archaeological sites us-ing AUVs. Within this system, the image processing pipelineis able to efficiently detect and rank potential archaeological

Date Data Set Area Purpose Results06/12/17 B Sliema High Level Survey 4 scans06/13/17 B Sliema High Level Survey Connection Problems06/14/17 B Sliema High Level Survey 3 scans06/15/17 B Sliema High Level Survey 5 scans06/16/17 B Sliema High Level Survey 5 scans06/19/17 B Sliema High Level Survey 1 scan06/21/17 C Xemxija High Level Survey 6 scans06/22/17 NA Sliema Planner test high-res scans of the Swordfish wreck06/23/17 NA Sliema Planner test06/26/17 NA St Elmo Planner test high-res scans of the HMS Maori wreck06/27/17 NA Sliema Planner test06/28/17 NA Sliema Planner test06/29/17 NA Sliema Planner test

TABLE III: Field Deployments in Malta

sites in side scan sonar images and its performance iscomparable to that of a human expert. Furthermore, field de-ployments provide evidence that the pipeline can be appliedto the real world and contribute to the discovery of novelsites of archaeological interest. Ideally, this demonstrates atransformational approach to conducting underwater archae-ological surveys.

This work is a first attempt in automating underwaterexploration, and there are several improvements that can bemade. First, the pipeline is not fully automated in that imageprocessing and path construction is not done on the AUV.Second, a larger data set is needed for better evaluationof the proposed algorithms. The scarcity of data leads toclassification algorithm training that is overly sensitive tomislabeled data. New data sources, or perhaps synthetic data,may be a potential solution to this problem in the future.

ACKNOWLEDGMENT

We would like to thank Christian Dalton, Karly Ogden,Smaranda Oaie, and Jason for helping us deploy the robot.

This work was performed in part at the Claremont Col-leges Robert J. Bernard Biological Field Station.

REFERENCES

[1] B. Bingham, B. Foley, H. Singh, R. Camilli, K. Delaporta, R. Eustice,A. Mallios, D. Mindell, C. Roman, and D. Sakellariou, “Robotic toolsfor deep water archaeology: Surveying an ancient shipwreck with anautonomous underwater vehicle,” Journal of Field Robotics, vol. 27,no. 6, pp. 702–717, 2010.

[2] D. Mindell and B. Bingham, “New archaeological uses of autonomousunderwater vehicles,” in OCEANS, 2001. MTS/IEEE Conference andExhibition, vol. 1. IEEE, 2001, pp. 555–558.

[3] M. Klein, “Side scan sonar,” in International handbook of underwaterarchaeology. Springer, 2002, pp. 667–678.

[4] A. Jacobs. (2014) Archaeologists discover 13,800-year-old underwater site at haida gwaii. [Online]. Available:https://indiancountrymedianetwork.com/history/genealogy/archaeologists-discover-13800-year-old-underwater-site-at-haida-gwaii/

[5] H. Singh, J. Adams, D. Mindell, and B. Foley, “Imaging underwaterfor archaeology,” Journal of Field Archaeology, vol. 27, no. 3, pp.319–328, 2000.

[6] R. Quinn, W. Forsythe, C. Breen, M. Dean, M. Lawrence, andS. Liscoe, “Comparison of the maritime sites and monuments recordwith side-scan sonar and diver surveys: A case study from rathlinisland, ireland,” Geoarchaeology, vol. 17, no. 5, pp. 441–451, 2002.

[7] T. Gambin, “Side scan sonar and the management of underwatercultural heritage,” in Future Preparedness: Thematic and SpatialIssues for the Enivronment and Sustainability, S. Formosa, Ed. Msida,Malta: The Department of Criminology, Faculty for Social Wellbeing,University of Malta, 2014, ch. 15, pp. 259–270.

[8] D. P. Williams, “On adaptive underwater object detection,” in 2011IEEE/RSJ International Conference on Intelligent Robots and Systems,Sept 2011, pp. 4741–4748.

[9] D. P. Williams and J. Groen, “A fast physics-based, environmentallyadaptive underwater object detection algorithm,” in OCEANS 2011IEEE - Spain, June 2011, pp. 1–7.

[10] J. Sawas and Y. Petillot, “Cascade of boosted classifiers for automatictarget recognition in synthetic aperture sonar imagery,” Proceedingsof Meetings on Acoustics, vol. 17, no. 1, p. 070074, 2012. [Online].Available: http://asa.scitation.org/doi/abs/10.1121/1.4788639

[11] T.-Y. Liu et al., “Learning to rank for information retrieval,” Founda-tions and Trends R© in Information Retrieval, vol. 3, no. 3, pp. 225–331, 2009.

[12] T. Joachims, “Optimizing search engines using clickthrough data,” inProceedings of the eighth ACM SIGKDD international conference onKnowledge discovery and data mining. ACM, 2002, pp. 133–142.

[13] K. Simonyan and A. Zisserman, “Very deep convolutional networksfor large-scale image recognition,” arXiv preprint arXiv:1409.1556,2014.

[14] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for imagerecognition,” in Proceedings of the IEEE conference on computervision and pattern recognition, 2016, pp. 770–778.

[15] A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, “Cnnfeatures off-the-shelf: an astounding baseline for recognition,” inProceedings of the IEEE conference on computer vision and patternrecognition workshops, 2014, pp. 806–813.

[16] F. J. Huang and Y. LeCun, “Large-scale learning with svm andconvolutional for generic object categorization,” in Computer Visionand Pattern Recognition, 2006 IEEE Computer Society Conference on,vol. 1. IEEE, 2006, pp. 284–291.

[17] P. Viola and M. Jones, “Rapid object detection using a boosted cascadeof simple features,” in Proceedings of the 2001 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition. CVPR 2001,vol. 1, 2001, pp. I–511–I–518 vol.1.

[18] C. D. Manning, P. Raghavan, and H. Schtze, Introduction to Informa-tion Retrieval. Cambridge University Press, 2008.

[19] J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” J. Mach. Learn. Res., vol. 13,no. 1, pp. 281–305, Feb. 2012. [Online]. Available:http://dl.acm.org/citation.cfm?id=2503308.2188395

[20] C. Cortes and V. Vapnik, “Support-vector networks,” Machine learn-ing, vol. 20, no. 3, pp. 273–297, 1995.

[21] K. Helsgaun, “An effective implementation of the Lin-Kernighan trav-eling salesman heuristic,” European Journal of Operational Research,vol. 126, no. 1, pp. 106–130, 2000.

[22] H. Ding, E. Cristofalo, and J. Wang, “A Multi-Resolution Approach forDiscovery and 3-D Modeling of Archaeological Sites Using SatelliteImagery and a UAV-borne Camera,” in American Control Conference(ACC). ACC, 2016, pp. 1359–1365.

[23] K. Jarvelin and J. Kekalainen, “Cumulated gain-based evaluation ofir techniques,” ACM Transactions on Information Systems (TOIS),vol. 20, no. 4, pp. 422–446, 2002.