Learning to Navigate the Energy Landscape

Julien Valentin1,3 Angela Dai2 Matthias Nießner2

Pushmeet Kohli1 Philip Torr3 Shahram Izadi1 Cem Keskin1

1Microsoft Research 2Stanford University 3University of Oxford

Abstract

In this paper1, we present a novel, general, and efficientarchitecture for addressing computer vision problems thatare approached from an ‘Analysis by Synthesis’ standpoint.Analysis by synthesis involves the minimization of recon-struction error, which is typically a non-convex function ofthe latent target variables. State-of-the-art methods adopta hybrid scheme where discriminatively trained predictorslike Random Forests or Convolutional Neural Networks areused to initialize local search algorithms. While these hy-brid methods have been shown to produce promising re-sults, they often get stuck in local optima. Our method goesbeyond the conventional hybrid architecture by not onlyproposing multiple accurate initial solutions but by alsodefining a navigational structure over the solution spacethat can be used for extremely efficient gradient-free localsearch. We demonstrate the efficacy and generalizability ofour approach by on tasks as diverse as Hand Pose Estima-tion, RGB Camera Relocalization, and Image Retrieval.

1. Introduction

Recent years have seen the re-emergence of the ‘anal-ysis by synthesis’ or ’inverse graphics’ approach to Com-puter Vision problems [3, 41, 38, 26, 20, 42, 43]. This ele-gant technique works by finding the parameters of the syn-thesis model that minimizes the reconstruction error; e.g.,the distance between synthesized and query images. It hasbeen explored by researchers many times [16, 4, 45, 49]but repeatedly fell out of favor due to the perception of be-ing computationally expensive and brittle due to reconstruc-tion error being a non-convex function of the target vari-ables [25].

More lately, however, the availability of large datasetsof training data along with the ability to train high capacitymodels like Convolutional Networks (CNN) [44, 26] andRandom Forests (RF) [41, 38] have given the approach anew breath of life. These models have led to the develop-ment of hybrid architectures, where discriminatively trainedfeed-forward predictors (RFs or CNNs) are used to initial-

1All of our datasets are publicly available (for details, see Sec. 7.3.):http://www.graphics.stanford.edu/projects/reloc/

ize continuous local search algorithms. While these hybridtechniques have been shown to produce impressive results,it is hard to make them work robustly in real time. This isin large part due to the cost of computing the derivative ofthe reconstruction error with respect to the model parame-ters, as it is a function of the rendering process. Althoughgradient-free optimization methods such as Particle SwarmOptimization (PSO) exist, they suffer from the problem thatgenerative methods often require a significant amount of er-ror evaluations to converge to good solutions. This can bevery expensive, as each error evaluation requires perform-ing a rendering operation on-the-fly2. Another major obsta-cle is that it is non-trivial to extract good search seeds insuper real-time. This is a requirement for any continuous orstochastic optimization operating on a non-convex surfaceto converge to good minima while allowing the entire opti-mization pipeline to run in real-time. There is then a needfor flexible and general optimization frameworks mimick-ing the generative optimization procedure, but orders ofmagnitude faster than the current methods.

In this paper, we propose a new general framework forminimizing the reconstruction error emanating from theanalysis by synthesis approach. Our approach is based onintegration of two search structures: retrieval forests and anavigation graph. The retrieval forest is composed of a col-lection of trees, each of which operates like a learned, con-ditional, locality sensitive hash function that maps to a hashtable bucket (leaf) where a subset of the instances from thetraining set are stored which can be used as candidates forlocal search. However, instead of computing the derivativeof the rendering pipeline or doing a blind search like PSO,our local search method operates by traversing a hierarchi-cal navigation graph defined on the parameter space. Eachinstance in the training set has a corresponding vertex in thegraph which is connected with a set of neighboring vertices(ideally i.i.d. distributed in parameter space around that ver-tex) according to a distance measure. A simple choice forthe distance for defining the neighborhood is the L1- or L2-distance in descriptor-space, as it is involved in the com-putation of the reconstruction error, but L2-distance in tar-get parameter space can also be used to build the graph.

2It is worth noting that it is sometimes possible to alleviate those limi-tations by heavy code engineering [38] to render small images.

1

arX

iv:1

603.

0577

2v1

[cs

.CV

] 1

8 M

ar 2

016

Traversal of the edges in this navigational graph can be seenas making updates in the parameter space in the rough di-rection of the gradient minimizing the reconstruction error.We build this navigational graph with multiple levels, wherevertices in higher levels are more sparse, allowing for largerjumps across the parameter space.

During test time, we use a collection of discriminativelytrained retrieval trees to propose candidate vertices fromwhere the local search should be initiated. We then re-peatedly consider all the neighbors of the candidate ver-tices, compute the distance between them and the queryimage (reconstruction error or surrogate error), and selectthose having lowest error as candidates for future explo-ration. We consider a Navigation Graph to be convergedwhen no new vertices with a better energy can be reached.Then, we initiate a new search using the next Graph in thehierarchy, and proceed until the convergence of all the Nav-igation Graphs in the hierarchy. At this stage, the best pre-diction can be directly used as a prediction. Alternatively,the most promising predictions can be used to seed a contin-uous (e.g., Levenberg-Marquardt) or stochastic procedure(e.g., Particle Swarm Optimization) for a few steps of localand final refinement.

We first demonstrate the efficacy and generalizability ofour approach on Hand Pose Estimation and Image Retrievaltasks. We also demonstrate the efficacy of our approachon the problem of RGB Camera Relocalization. To makethis problem particularly challenging, we introduce a newdataset of 3D environments that are significantly larger thanthose found in other publicly-available datasets. Our ex-perimental results indicate that the composite retrieval tree-navigation graph architecture not only leads to dramatic im-provements in computation time but also results in more ac-curate solutions.

To summarize, the main contributions of this work are(i) a new gradient-free heuristic optimization method basedon navigational graphs that is fast and accurate, (ii) theintroduction of a new public dataset for RGB and RGB-D relocalization that is significantly larger that those cur-rently available, (iii) the demonstration that our approachis generic and reaches state-of-the-art results on three verydistinct vision problems, namely RGB camera relocaliza-tion, hand pose estimation, and image retrieval.

2. Related work

We briefly introduce popular methods for approximatenearest neighbor search in the next four paragraphs. We re-fer the interested reader to [15, 14, 24, 10, 48, 36, 47] and[41, 38] for relevant litterature on RGB camera relocaliza-tion and Hand Pose Estimation, respectively.

LSH A number of papers in the literature have consid-ered the use of hash functions for solving regression prob-

lems such as human pose estimation [37]. These are basedon the concept of locality sensitive hashing (LSH) [19], us-ing several random projection functions to hash each pointof the dataset so that similar items map to the same bucketwith high probability. Our approach is related to the pose-sensitive hashing method [37], as they focus on retrievingneighbors in the latent low-dimensional manifold where thedata lives (articulated human pose) rather than the observedspace of images. In this paper, the main target is to mini-mize the camera pose distance between the predictions andthe actual poses of the ‘lost’ cameras.

In their influential work, the authors of [34] show howto use simple approximations that allow better probing se-quences for LSH. Their multiprobe LSH led to improve-ments in both space and time. Significant work on buildingefficient index that can be used by LSH include [50] wherethe authors propose a compounds representation by decom-posing images into background and image-specific informa-tion. For additional details on LSH, see [33, 1, 34].

KD-tree Contrary to LSH, a KD-tree [5, 12] recursivelypartitions the space with axis-aligned hyperplanes that splitthe data in two halves. Notable work in this area include[32] where the authors build a hierarchical k-means cluster-ing and the widely used library based on [28] More detailsabout different variants of KD-tree can be found in [7].

Derivative-free optimization Particle Swarm Optim-mization [23], Cuckoo Search, and Genetic Algorithms aresearch heuristics. The common idea behind these meth-ods is to perturbate the current solution set (random i.i.d.jumps, mutations, etc.) in the hope of finding a new solu-tion set that better explains the target. In the context of handpose estimation, such methods can be used [18] to itera-tively refine the pose parameters that best explain the inputimage. We refer the reader to [8] for extensive discussionson derivative-free optimization.

k-NN graphs There are a number of works that focuson hill-climbing strategies or k-NN graphs, but to the bestof our knowledge there has been only one attempt at per-forming hill-climbing on k-NN graphs [17]. This workis the most related to ours; however, we introduce severalsignificant differences. First, [17] starts to search from apoint drawn randomly from a uniform distribution over thedatabase. Following such a strategy is prone to reachingpoor minima, especially for large and diverse databases. Tocircumvent this problem, [17] allows the system to restartthe entire search with a new seed a pre-defined number oftimes, which is relatively computationally wasteful. Wethen propose to train a Random Forest adapted for retrievalthat will very quickly generate seeds that are in generalmuch closer to the minima reached after optimization (i.e.less edges are traversed). Thus we are able to reach bet-

ter results, faster. Second, [17] only explores one path ata time, which is extremely greedy. Similar to bio-inspiredtechniques, we explore and ‘optimize’ multiple hypothesisat once. Again, this make the search less prone to gettingstuck in bad local minima. All these elements contribute toa method that significantly outperforms [17].

3. Method OverviewThe proposed method is essentially a discrete optimiza-

tion technique that uses a hierarchical navigation graph anda retrieval forest. We build the graph using multiple distancemeasures in order to decrease the number of local minima,and use the reconstruction error (or a surrogate) when nav-igating through the graph. We discriminatively train treesto generate seeds to initiate the navigation in the first graphin the hierarchy. The search starts from multiple seeds, tra-verses the hierarchy of navigation graphs, and finally out-puts a number of solutions which are used to initiate thecontinuous optimization step if refinement is needed.

A retrieval forest [13] outputs a list of samples for eachtree, which are ranked based on how many times they wereselected. The top-ranking vertices are used to initiate thesearch in the top level of the navigation graph.

At each iteration of the search, a candidate solution listis expanded by including all neighbors of the current esti-mates. The query image is compared against this list usingthe reconstruction error (or a surrogate measure), and theresulting energies used to rank the candidates. A few top-ranking vertices are then used to initiate the next iterationof the search in the same level of the graph. If the candi-date solution set does not change throughout the iteration,or if a maximum number of iterations are met, we continuethe search in the next level of the hierarchy, using the finallist of vertices from the previous level as seeds. Once thesearch concludes in the bottom level, the solution is finaland can optionally be sent to a continuous optimizer (e.g.PSO, gradient descent).

4. Retrieval ForestsA retrieval forest [13] is a randomized decision forest

(RDF) [9], which acts as a hash function that assigns eachquery to a leaf. The leaves of a typical RDF store an em-pirical distribution over the samples. In retrieval forests, theleaves act as the entries of a lookup table and store the in-dices of dataset elements. During inference, each tree votesfor a set of elements present in the training set, and we countthe number of votes per element in order to rank them.

We use the standard greedy tree training algorithm togrow the trees. Each tree is trained with some randomnessfrom the random generation of pairs of feature indices φand thresholds τ , and optionally also from bagging. Wedenote each pair (φ,τ ) as θ, and the set of candidate ran-

dom parameters θ in node n as Θn. Each node n uses therandomly-generated Θn to greedily optimize

θ∗n = argmaxθ∈Θn

In(θ), (1)

where In is the information gain:

In(θ) = E(Sn)−∑

i∈{L,R}

|Sin||Sn|

E(Sin), (2)

Here, E(S) is a measure of the differential entropy of theset S in descriptor space. Note that the left and right subsetsSin are implicitly conditioned on the candidate parameters θ.As in [46], we use the determinant of the covariance matrixas the entropy measure.

Hash functions typically use dense projections, while weperform very sparse projections to traverse our trees moreefficiently. The results obtained using the retrieval forestare then further refined in the next phases of the pipeline.

5. Multiscale Navigation GraphThe core of our discrete optimization method is the mul-

tiscale navigation graph, which allows for rapid refinementof a set of initial estimates in solution space. The initialseeds are provided by the retrieval forest (cf. Section 4),which are too crude to be used directly in a gradient-basedcontinuous optimizer without a robust refinement proce-dure. The graph search rapidly refines these solutions whileavoiding most local minima. Fig. 1 offers an intuition aboutthe optimization procedure.

We start by constructing a multiscale graph G = (V,E)from a set of samples, where the vertices V are the individ-ual samples and the directed edges E are formed betweenvertices p and q, if q is among the k nearest neighbors ofp. The graph consists of multiple levels corresponding todifferent scales, each described with an adjacency list Gi,much like a pyramid. While top level holds a fraction of allsamples, chosen uniformly, the bottom level keeps all thesamples in the dataset. Every vertex in a higher level of thegraph, must appear in every level below it. k neighbors arecalculated separately for each level.

Note that E(p, q) does not necessarily imply E(q, p)since nearest neighbor relationships are not symmetric,even if the distance metric is. The distance Dm betweensamples p and q is denoted by Dm(p, q), where m denotesthe metric used (e.g., Euclidean distance in a certain space).

The graph search algorithm is explained in Algo. 1.Given a query t, the search starts in the top level graph GTfrom an initial set of seeds C, which is provided by the re-trieval forest as described in the previous section. A candi-date list C ′ is then formed from samples in C and all theirneighbors. For each candidate in C ′, we measure their dis-tance Dt to the query. Note that the distance measure Dm

used to build G is not necessarily the same as Dt. We rankthese candidates based on their distances and choose n sam-ples to replace C. If C does not change in a given iteration,or if a maximum number of iterations is reached, we switchto the next graph in the hierarchy, initiating a new searchwith the set C where the previous graph converged to. TheC from the final level in the hierarchy is the output of thediscrete optimization method.

Figure 1. Intuition for Navigation Graphs. Left: query; right:reconstruction error as a function of the pose parameters. Thecurrent candidate pose is highlighted in orange; the optimizationprocess consists of estimating whether any of its neighbor mini-mizes the reconstruction error any further. If this is the case, theoptimization may consider them as potential descent directions.Orange arrows mark connections between candidates, and blue ar-rows denote their positions on the solution manifold.

6. Continuous Pose RefinementThe previous section has described how good camera hy-

potheses can be formed. Using these hypothesis, we follow[30, 11] and minimize the photo-consistency error in orderto make accurate pose predictions. Given an initial pose hy-pothesis p and a query image Cq , we optimize for the rigidbody transform between p and Cq by minimizing the pro-jection of the model of the scene M to Cq . At any stage ofthe optimization, we raycastM from the current pose of hy-pothesis p and produce a color imageCm and a depth imageDm. We then optimize the rigid transform T to maximizethe photo-consistency when projecting Cm into Cq usingDm. Let G denote the intensity image of Cm, πd the depthcamera projection, and πc the color camera projection (πdand πc are defined by the respective camera intrinsics), thenthe photo-consistency error is given as

Ec(T ) =

#pixels∑

k

∥∥∥Gq(πc(T π−1d (Dm(k))))−Gm(k)

∥∥∥2

2.

Note that we discard pixels with invalid depth. The non-linear least-squares objective Ec is solved using Gauss-Newton optimization. We use a hierarchical coarse-to-fineoptimization strategy, with the finest hierarchy level con-taining the full color resolution, and each subsequent hier-archy level sub-sampled by a factor of 2.

Then, for query image Cq and its top k pose proposalsp1, ..., pk, we minimize Ec for each pose proposal, produc-

Algorithm 1: Graph traversalInputs: A new test sample t. M adjacency listsdenoted Gm, defining each level of the multiscalegraph structure G. A function D that estimates somedistance between two images. A set S of seeds. Anumber of maximum iterations itmax. A desirednumber of predictions K. A maximum number ofvertices to keep after each iteration n.

Output: K approximate nearest neighbors of t

Algorithm:

C = S

for lv = M to 1 dofor it = 1 to itmax doC′ = Cforeach c ∈ C do

Using the neighborhood structure definedin Glv , add all unvisited neighbors of c into C′ without repetition, and mark themas visited

endSort C′ according to D computed between t

and each element of C′.C = first n elements of C′if C has not changed this iteration then

breakend

endend

return first K elements of C

ing potential refined poses p′1, ..., p′k. The final refined pose

p∗ is then given by the pose p′i with i = arg minj Ec(p′j).

As this optimization is straightforward to parallelize, we caneasily compute T on a GPU within only a few millisecondsfor a given image pair.

7. ResultsWe evaluate our method on three important computer vi-

sion tasks. First, we show results on Hand Pose Estimation;second, we provide an evaluation on approximate nearestneighbor queries for Image Retrieval; third, we show resultson RGB relocalization. Our method outperforms the base-lines in all experiments, both in accuracy and speed. Open-source code for retrieval forests and Navigation Graphs willbe released upon acceptance.

7.1. Experiments on Hand Pose Estimation

In this section, we first evaluate our method on the taskof hand pose estimation using the synthetic dataset from[38]. This dataset comprises 100k training and 10k test

Figure 2. Precision of hand pose predictions. The abscissa de-fines how many seeds are used for any given hand image, and theordinate represents the closest of those seeds in average joint error.These results are generated by considering the top K samples fromwhich we determine the lowest average joint pose error using anoracle. Note that our method not only provides more precise seedsthan [41] but also extracts them an order of magnitude faster.

hand poses that have been generated by rendering depth im-ages of a synthetic hand model. For more details regardingthe dataset generation, we refer the reader to [38].

In our experiments, we compare our results to the lateststate of the art [41]. We follow [40] and use the average jointpose as an error metric. In Fig. 2, we show that regardless ofthe number of seeds we predict, our method is consistentlybetter at predicting hand poses of lower average joint poseerror than [41]. Furthermore, we extract our seeds an orderof magnitude faster than [41].

We further compared our method on Dexter [40], NYU[44] and FingerPaint datasets [38] which only contain realimages. In Fig. 3 we show that the proposed method sig-nificantly outperforms Retrieval Forest only and the reini-tializer presented in [38]. We also demonstrate the powerof our method under the presence of a powerful continu-ous LM based optimizer, where our method generates 10seeds per frame to initiate the gradient descent. Each frameis treated independently (i.e. without tracking) to properlyshow the effect of using our reinitializer instead of the onepresented in [38]. Our method shows significant gains onDexter and NYU, and shows on par results for FingerPaint.The continuous optimization method and more experimen-tal results can be found in [2].

7.2. Experiments on Image Retrieval

In this section, we demonstrate the performance of theproposed NN-search method on a well-established imageretrieval dataset, GIST-1M [21].

In the experiments presented in Fig. 4, we compare ourwork against all the baseline methods implemented in the

Figure 4. Results on the GIST-1M dataset [21]. In these experi-ments, the task is to retrieve 100 neighbors, with recall computedagainst the actual 100 nearest neighbors. For our Multiscale Nav-igation Graph, we use a hierarchy of 2 graphs where the first con-tains 1/10th of the dataset and the second contains all of it. Ourmethod outperforms all the baselines in FLANN in almost anyregime. Note that the Retrieval Forest is not plotted on this fig-ure, but reaches a speed-up over linear search of around 6500 fora recall of 2.07%.

FLANN library [28] using the code provided on the authors’website as well as our own implementation of [17] (referredto as GNNS). Notably, our proposed method significantlyoutperforms all baselines from FLANN with a comparablecompute budget. For instance, at 450× speedup over linearsearch, the hierarchical, k-means, and composite baselinesfrom FLANN only attain half the recall of our method. Ad-ditionally, our method is capable of reaching even higherrecall rates if more compute budget is available. With a 5msbudget (on a laptop computer), we achieve recall rates over70%, corresponding to a 60× speedup compared to linearsearch. Note that to provide a fair comparison against [17],we set the number of restarts to be the same as the numberof seeds we extract from the retrieval forest.

7.3. RGB Relocalization

Dataset The main dataset currently used for RGB andRGB-D relocalization is the 7-Scenes dataset [39, 15]. Thisdataset contains several scenes recorded from a KinectV1.These scenes are all very limited environments (at most6m3), for which extremely precise results have already beenreached [46]. A major increase in the volume in which torelocalize is a clear way to make the relocalization prob-lem more challenging. Additionally, the 7-Scenes record-ings suffer from low color quality (VGA resolution, motionblur artifacts, auto-white balance, auto-exposure) and no ex-ternal calibration between the depth and RGB sensors. Asthese issues are all easily solvable hardware or calibrationproblems, they should not be part of benchmark evaluations

Dexter NYU FingerPaint

Figure 3. Hand pose estimation on real data. First row: pure pose regression. Second row: regressed poses optimized using the frameworkpresented in [2]

of relocalization techniques. Fig. 6 illustrates these effectson the scene model quality, which is a major obstacle tosynthesizing RGB frames consistent with the real world.

We thus introduce a new dataset which we believe willallow the community to push the boundaries of RGB andRGB-D relocalization further. Using a Structure.io depthsensor coupled with an iPad color camera, we capture vari-ous environments of significant physical extent (volumes anorder of magnitude larger than those of 7-Scenes), as can beobserved in Tab. 1. Note that both cameras have been cal-ibrated and temporally synced. RGB image sequences arerecorded at a resolution of 1296×968 pixels; the depth res-olution is 640× 480 and of similar quality to the KinectV1.

We capture four large RGB-D scenes, each of whichis composed of several rooms as shown in Fig. 5. Foreach scene, two independent captures are performed. Thefirst capture is used to reconstruct a scene model, with re-construction performed using the VoxelHashing frameworkof [31] in combination with global bundle adjustment. Thisreconstruction provides the ground truth camera poses (6d.o.f.) for each frame. We then uniformly sample a fixednumber of poses around those ground truth camera posesand synthetically render the corresponding RGB images.This procedure is used to form the training set. The secondrecording is also reconstructed and bundle adjusted. Then,manual alignment between the two models is performed.The RGB images and camera poses of this second record-ing correspond to our test set.

Note that the dataset (RGB-D frames, ground-truthposes, and 3D models) will be made publicly available tothe community.

Parameter settings The parameters of the system are

Sequence Volume # training frames # test framesKitchen 33m3 7160 357Living 30m3 10554 493Bath 24m3 22500 230Bed 14m3 17800 244

Kitchen 21m3 14665 230Living 42m3 34280 359Luke 53m3 38382 624

Floor 5a 38m3 10680 497Floor 5b 79m3 10360 415

Copy 26m3 13640 649Gates362 29m3 18641 386Gates381 44m3 20240 1053Lounge 38m3 3160 327Manolis 50m3 17940 807TOTAL 521m3 240002 6671

Table 1. Physical description of our new dataset. The recordedenvironments are significantly larger than the largest environmentof the 7-Scenes dataset (6m3).

held constant for all relocalization scenarios. The retrievalforest is comprised of 64 trees with a maximum depth of13, where each decision node only uses 2 dimensions ofthe input vector to route the samples. For the MultiscaleNavigation Graph, at any stage, the 20 closest neighbors ofeach candidate are considered and the 10 best candidates arepassed to the next iteration over a maximum of 5 iterations.The 4 top candidates are then passed to a pose refinementstep.

Continuous Pose Refinement Similarly to [46, 39]which perform RGB-D relocalization by first generating hy-

Figure 5. A new public dataset for RGB and RGB-D camera relocalization. Our new dataset comprises two apartments and two officescenes. There is a total of 14 rooms, each of which are individually larger than the largest scene of [39].

Our dataset 7-Scenes [39]

Figure 6. Qualitative difference between our dataset modelsand those of [39]. Both images are raycast from their respective3D models. Note that we are able to synthesize RGB frames ofmuch higher quality.

pothesis and then optimizing them in a continuous fashionfor high precision, we run a continuous pose optimizer ontop of the top prediction from our method. To do so, wefollow [30, 11] and minimize the photo-consistency error.It is worth noting that the energy surface is non-convex and

hence good initial candidates are essential to sustain goodrelocalization rates. We refer the interested reader to thesupplementary materials for more details. Note that pre-cise camera relocalization is required in applications suchas SLAM where it allows the system to recover from track-ing failures.

Speed On a laptop equipped with an i7-4720HQ, passinga new sample down the entire test pipeline (retrieval forestand Multiscale Graph Navigation) takes less than 1ms. Werun the pose refinement step on the GPU using an NVIDIAGTX Titan, which takes less than 10ms to fully optimizeeach pose proposal. For each query, we only optimize thetop 4 pose proposals from the Multiscale Graph Navigation.

Baseline comparisons We compare our approach againsttwo RGB relocalization baseline based upon matchingsparse features, following mainstream feature-based relo-calization approaches such as [47, 10, 29]. Those ap-proaches either use the SIFT or the ORB algorithm for

sparse feature extraction and description. First, sparse fea-tures are detected in the RGB-D training images, which arestored with their corresponding 3D positions and descrip-tors. At test time, descriptors are computed per query frameand matched. Triplets of strong feature matches are thenused to generate candidate camera poses using the perspec-tive 3-point method. These hypotheses are refined using aRANSAC optimization. Note that during training and test,at most 100 SIFT features or 1000 ORB features are ex-tracted per image. In order to maximize the relocalizationperformacne of the baselines, we use brute force featurematching followed by RANSAC PnP from OpenCV’s [6]implementation.

It is worth noting that we do not compare againstPoseNet [22] as the only dataset for which they report re-sults and we could potentially use is 7-Scenes [39]. As de-scribed earlier in the paper, the test relocalization pipelinefor precise estimates requires a good calibration betweenthe color and depth sensors, which is not the case in the 7-Scenes dataset where no external calibration has been pro-vided. Nevertheless, it is worth noting that the results ob-tained by PoseNet are only slightly better than those of near-est neighbors. The best results from PoseNet requires 95mson a GPU. Using this budget, we could fully optimize 115cameras using the method presented in this paper. To geta feel for where the proposed method stands, we ran onlythe Retrieval Forest and Navigation Graphs on the 7-Scenesdataset. On average, we retrieve the nearest neighbor 97%of the time with an average run-time of 1ms (CPU).

Main relocalization results Fig. 7 shows qualitative re-sults obtained by the Multiscale Navigation Graph when fedwith seeds generated by the retrieval forest. The first pre-dictions are visually very similar to the queries. The predic-tions from Multiscale Navigation Graph often contain thesame objects as the query, but sometimes from slightly dif-ferent viewpoints. However, Fig. 8 shows that these pre-dictions still lie in the convergence basin of the continuouspose refinement.

Table 2 lists the main quantitative results on our relocal-ization dataset. On average, the proposed system is able torelocalize 67.4% of the queries with error below 5cm and5◦. For applications that don’t require this level of preci-sion, our method scores 88% within 30cm and 10◦.

Comparatively, the sparse relocalization baseline ofSIFT matching and RANSAC PnP pose refinement resultsin an average relocalization rate of 51.8% within 5cm and5◦. Those results are obtained in ≈ 1500ms, whereas ourMultiscale Navigation Graph followed by continuous poseoptimization runs in ≈ 40ms. By replacing SIFT [27] withORB [35], speed increases 3×, but accuracy falls 5-10%.Thus, we provide both better camera pose estimates andfast performance. We believe that these results would bevery beneficial to many applications.

Sequence ORB + PnP SIFT + PnP Our methodKitchen 66.39% 71.43% 85.7%Living 41.99% 56.19% 71.6%Bath 53.91% 48.70% 92.2%Bed 71.72% 72.95% 66.4%

Kitchen 63.91% 71.74% 76.7%Living 45.40% 56.19% 66.6%Luke 54.65% 70.99% 83.3%

Floor 5a 28.97% 38.43% 66.2%Floor 5b 56.87% 45.78% 71.1%

Copy 43.45% 62.40% 51.0%Gates362 49.48% 67.88% 51.8%Gates381 43.87% 62.77% 52.3%Lounge 61.16% 58.72% 64.2%Manolis 60.10% 72.86% 76.0%Average 52.99% 61.56% 67.4%

Table 2. Main quantitative results for the task of RGB relocal-ization. For each sequence, we show the percentage of refinedposes within 5cm/5◦ for our method and generic baselines. Notethat the baselines with ORB and SIFT are first optimized using aPnP optimization over RanSaC rounds. For SIFT + PnP, note thatwe use 100 features per image. For ORB + PnP, we use 1000 fea-tures per image. Also, note that the feature matching for SIFT andORB is done exactly via brute-force matching.

However, despite the relatively good quality reconstruc-tions which we obtain, all 3D models still remain incon-sistent with the real world due to tracking errors in the re-construction, as well as other artifacts such as motion blur.In fact, our method would be able to provide much moreaccurate results if the training and test images were consis-tent with each other; e.g., generated from the same model.We set up such an experiment, generating query images byraycasting the scanned scenes rather than using the origi-nal color frames captured by the iPad camera. In this case,we obtain significantly higher accuracy, as shown in Fig. 9.This suggests that there is room for improvement by usingmore accurate models; however, it is quite challenging toreconstruct large scenes at millimeter-level accuracy.

Memory For all sequences, we require less than 25MBmemory. Note that loading everything into RAM is not re-quired and that lazy data access is possible if necessary.

8. ConclusionIn this work, we presented a novel discrete optimization

method and demonstrated its applicability to several com-puter vision tasks, including RGB relocalization, hand poseestimation, and nearest neighbor queries for image retrieval.At the core of the pipeline lies our multiscale navigationalgraph. Given a list of seeds predicted by a random for-est, the graph search quickly traverses the discrete manifold

Query 1st pred. 2nd pred. 3rd pred.

Figure 7. Qualitative results from the Multiscale NavigationGraph. Given a query image (left column) and seeds from theretrieval forest, the graph search produces viewpoints to thequery. Our method is relatively robust even when some datais missing from the synthetic views or when the illuminationchanges.

Query Top pred. Refined pose

Figure 8. Continuous pose refinement for predicted cam-era hypotheses. From a query image (left column), the graphsearch searches for the most similar viewpoints. From the top4 predicted viewpoints (the top prediction shown in the middlecolumn), we optimize for the final pose by minimizing photo-metric error.

of solutions in order to make fast and accurate predictions.These predictions can then be further refined; for instance,in the case of RGB relocalization, we run a continuous poseoptimization for precise 6DOF camera pose inference. Thesame applies to the results we have shown on hand poseestimation and image retrieval tasks.

References[1] A. Andoni, P. Indyk, H. L. Nguyen, and I. Razenshteyn.

Beyond locality-sensitive hashing. In Proceedings of theTwenty-Fifth Annual ACM-SIAM Symposium on Discrete Al-

Figure 9. Precision of the relocalization averaged across allscenes. The value of the plot at (x, y) represents the % of pre-dicted cameras with translational and rotational error lower x andy. Left: precision on real world test data. Right: precision onsynthetic test data.

gorithms, pages 1018–1028. SIAM, 2014. 2[2] Anonymous. Concerto: Efficient and precise interactive

hand tracking through joint, continuous optimization of poseand correspondences. In Proc. ACM SIGGRAPH. ACM,2016. 5, 6

[3] J. T. Barron and J. Malik. Shape, illumination, and re-flectance from shading. TPAMI, 2015. 1

[4] B. G. Baumgart. Geometric modeling for computer vision.Technical report, DTIC Document, 1974. 1

[5] J. L. Bentley. Multidimensional divide-and-conquer. Com-mun. ACM, 1980. 2

[6] G. Bradski. Dr. Dobb’s Journal of Software Tools, 2000. 8[7] R. A. Brown. Building a balanced k-d tree in o(kn log n)

time. CoRR, abs/1410.5420, 2014. 2[8] A. R. Conn, K. Scheinberg, and L. N. Vicente. Introduction

to derivative-free optimization, volume 8. Siam, 2009. 2[9] A. Criminisi and J. Shotton. Decision forests for computer

vision and medical image analysis. Springer, 2013. 3[10] E. Eade and T. Drummond. Unified loop closing and recov-

ery for real time monocular slam. In BMVC, volume 13, page136. Citeseer, 2008. 2, 7

[11] J. Engel, T. Schops, and D. Cremers. Lsd-slam: Large-scaledirect monocular slam. In European Conference on Com-puter Vision. Springer, 2014. 4, 7

[12] J. H. Friedman, J. L. Bentley, and R. A. Finkel. An algorithmfor finding best matches in logarithmic expected time. ACMTrans. Math. Softw., 1977. 2

[13] H. Fu, Q. Zhang, and G. Qiu. Random forest for imageannotation. In Computer Vision–ECCV 2012, pages 86–99.Springer, 2012. 3

[14] A. P. Gee and W. W. Mayol-Cuevas. 6d relocalisation forrgbd cameras using synthetic view regression. In BMVC,pages 1–11, 2012. 2

[15] B. Glocker, S. Izadi, J. Shotton, and A. Criminisi. Real-time rgb-d camera relocalization. In Mixed and AugmentedReality (ISMAR), 2013 IEEE International Symposium on,pages 173–179. IEEE, 2013. 2, 5

[16] U. Grenander. Pattern synthesis: Lectures in pattern theory,volume 18. Springer Science & Business Media, 2012. 1

[17] K. Hajebi, Y. Abbasi-Yadkori, H. Shahbazi, and H. Zhang.Fast approximate nearest-neighbor search with k-nearestneighbor graph. IJCAI, 2011. 2, 3, 5

[18] A. A. A. Iason Oikonomidis, Nikolaos Kyriazis. Efficientmodel-based 3d tracking of hand articulations using kinect.In BMVC, 2011. 2

[19] P. Indyk and R. Motwani. Approximate nearest neighbors:Towards removing the curse of dimensionality. In Proceed-ings of the Thirtieth Annual ACM Symposium on Theory ofComputing, STOC, 1998. 2

[20] V. Jampani, S. Nowozin, M. Loper, and P. V. Gehler. Theinformed sampler: A discriminative approach to bayesianinference in generative computer vision models. ComputerVision and Image Understanding, 136:32–44, 2015. 1

[21] H. Jegou, M. Douze, and C. Schmid. Searching with quan-tization: approximate nearest neighbor search using shortcodes and distance estimators. 2009. 5

[22] A. Kendall, M. Grimes, and R. Cipolla. Posenet: A convolu-tional network for real-time 6-dof camera relocalization. InProceedings of the IEEE International Conference on Com-puter Vision, pages 2938–2946, 2015. 8

[23] J. Kennedy and R. Eberhart. Particle swarm optimization.Internal Conference on Neural Networks, 1995. 2

[24] G. Klein and D. Murray. Improving the agility of keyframe-based slam. In Computer Vision–ECCV 2008, pages 802–815. Springer, 2008. 2

[25] P. Kohli, J. Rihan, M. Bray, and P. H. Torr. Simultane-ous segmentation and pose estimation of humans using dy-namic graph cuts. International Journal of Computer Vision,79(3):285–298, 2008. 1

[26] T. Kulkarni, P. Kohli, J. Tenenbaum, and V. Mansinghka.Picture: A probabilistic programming language for sceneperception. In Proc. CVPR, 2015. 1

[27] D. G. Lowe. Object recognition from local scale-invariantfeatures. In Proc. ICCV, 1999. 8

[28] M. Muja and D. G. Lowe. Scalable nearest neighbor algo-rithms for high dimensional data. Pattern Analysis and Ma-chine Intelligence, IEEE Transactions on, 2014. 2, 5

[29] R. Mur-Artal, J. Montiel, and J. D. Tardos. Orb-slam: a ver-satile and accurate monocular slam system. Robotics, IEEETransactions on, 31(5):1147–1163, 2015. 7

[30] R. Newcombe, S. Lovegrove, and A. Davison. Dtam: Densetracking and mapping in real-time. In ICCV, 2011. 4, 7

[31] M. Nießner, M. Zollhofer, S. Izadi, and M. Stamminger.Real-time 3D reconstruction at scale using voxel hashing.ACM TOG, 32(6), 2013. 6

[32] D. Nistr and H. Stewnius. Scalable recognition with a vo-cabulary tree. In Computer Vision and Pattern Recognition,2006. 2

[33] R. ODonnell, Y. Wu, and Y. Zhou. Optimal lower boundsfor locality-sensitive hashing (except when q is tiny). ACMTransactions on Computation Theory (TOCT), 6(1):5, 2014.2

[34] L. Qin, W. Josephson, Z. Wang, M. Charikar, and K. Li. Mul-tiProbe LSH: Efficient Indexing for High-Dimensional Sim-ilarity Search. In Very Large Data Bases, 2007. 2

[35] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. Orb:an efficient alternative to sift or surf. In Computer Vision(ICCV), 2011 IEEE International Conference on, 2011. 8

[36] S. Se, D. G. Lowe, and J. J. Little. Vision-based global lo-calization and mapping for mobile robots. Robotics, IEEETransactions on, 21(3):364–375, 2005. 2

[37] G. Shakhnarovich, P. Viola, and T. Darrell. Fast pose estima-tion with parameter-sensitive hashing. In Computer Vision,2003. Proceedings. Ninth IEEE International Conference on,pages 750–757. IEEE, 2003. 2

[38] T. Sharp, C. Keskin, D. Robertson, J. Taylor, J. Shotton,D. Kim, C. Rhemann, I. Leichter, A. Vinnikov, Y. Wei,D. Freedman, P. Kohli, E. Krupka, A. Fitzgibbon, andS. Izadi. Accurate, robust, and flexible real-time hand track-ing, 2015. CHI. 1, 2, 4, 5

[39] J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, andA. Fitzgibbon. Scene coordinate regression forests for cam-era relocalization in rgb-d images. In Computer Vision andPattern Recognition. IEEE, 2013. 5, 6, 7, 8

[40] S. Sridhar, A. Oulasvirta, and C. Theobalt. Interactive mark-erless articulated hand motion tracking using rgb and depthdata, 2013. ICCV. 5

[41] D. Tang, J. Taylor, P. Kohli, C. Keskin, T.-K. Kim, andJ. Shotton. Opening the black box: Hierarchical samplingoptimization for estimating human hand pose. In Interna-tional Conference on Computer Vision. IEEE, 2015. 1, 2,5

[42] J. Thies, M. Zollhofer, M. Nießner, L. Valgaerts, M. Stam-minger, and C. Theobalt. Real-time expression transfer forfacial reenactment. ACM Transactions on Graphics (TOG),34(6), 2015. 1

[43] J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, andM. Nießner. Face2face: Real-time face capture and reen-actment of rgb videos. In Proc. Computer Vision and PatternRecognition (CVPR), IEEE, June 2016, 2016. 1

[44] J. Tompson, M. Stein, Y. Lecun, and K. Perlin. Real-timecontinuous pose recovery of human hands using convolu-tional networks. ACM Transactions on Graphics, 2014. 1,5

[45] Z. Tu and S. C. Zhu. Image segmentation by data-drivenmarkov chain monte carlo. PAMI, 24(5):657–673, 2002. 1

[46] J. Valentin, M. Nießner, J. Shotton, A. Fitzgibbon, S. Izadi,and P. Torr. Exploiting uncertainty in regression forests foraccurate camera relocalization. In Proc. CVPR, 2015. 3, 5, 6

[47] B. Williams, G. Klein, and I. Reid. Real-time slam relocal-isation. In Computer Vision, 2007. ICCV 2007. IEEE 11thInternational Conference on, pages 1–8. IEEE, 2007. 2, 7

[48] B. Williams, G. Klein, and I. Reid. Automatic relocaliza-tion and loop closing for real-time monocular slam. PatternAnalysis and Machine Intelligence, IEEE Transactions on,33(9):1699–1712, 2011. 2

[49] A. Yuille and D. Kersten. Vision as bayesian inference: anal-ysis by synthesis? Trends in cognitive sciences, 10(7):301–308, 2006. 1

[50] X. Zhang, Z. Li, L. Zhang, W.-Y. Ma, and H.-Y. Shum. Effi-cient indexing for large scale visual search. In InternationalConference on Computer Vision, 2009. 2