BlurTags: spatially varying PSF estimation without-of-focus patterns

Alexander ReuterGoethe University Frankfurt,

Germanyreuter@vsi.cs.uni-frankfurt.de

Hans-Peter SeidelMPI Informatik, Germanyhpseidel@mpi-inf.mpg.de

Ivo IhrkeSaarland University/ MPI

Informatik, Germanyihrke@mmci.uni-saarland.de

ABSTRACTCurrent research is targeting the estimation and correction of lens imperfections. often modeled as a set of spatiallyvarying point spread functions (PSFs). One way to measure these PSFs is their calibration with checkerboardpatterns. Previous work, however, does not fully exploit all benefits of using a checkerboard. In particular, we showin this paper that the pose of the checkerboard with respect to the camera can be exploited to yield informationon the circle of confusion, and thus the image blur of an ideal camera. By removing this expected blur, we canestimate residual PSFs that are due to the deviation of the optical system from a thin-lens model. The residualPSFs can then be used to sharpen images at comparable lens settings.Practical side effects of our method are the design of a self-identifying pattern that can be robustly detected evenin the case of image blur, and a corresponding algorithm for its detection.

Keywordsspatially varying PSF, self-identifying markers, deconvolution

1 INTRODUCTION2012June 25 – June 28

The optical resolution of camera images is often lim-ited by the quality of the lens system used for taking thepicture. Even expensive lens systems show aberrations,especially at high aperture settings (low f-number). Al-ternatively, cheap plastic lenses are mass-produced andshow less then ideal imaging capabilities. The devia-tion from perfect imaging is usually expressed by thesystem’s point spread function (PSF). Though it is of-ten assumed to be invariant with respect to shifts on thesensor and the settings of the camera, this is in gen-eral an invalid assumption. The space of PSFs is high-dimensional, varying with position on the sensor, aper-ture, focus and, potentially, zoom-setting of the camera.Knowing the distribution of the PSFs enables the com-putational removal of aberrations from the image viadeconvolution.

It is therefore important to calibrate the PSFs, ide-ally for all possible combinations of camera settings.Using the EXIF information of an image taken withsuch a calibrated lens, it can be sharpened in a com-

Permission to make digital or hard copies of all or part ofthis work for personal or classroom use is granted withoutfee provided that copies are not made or distributed for profitor commercial advantage and that copies bear this notice andthe full citation on the first page. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requiresprior specific permission and/or a fee.

putational post-process. Recently, the work of Kee etal. [KPCW11] has introduced a method for estimatingthe PSF variation for aperture and zoom changes. How-ever, they do not address the focus-setting of the cam-era. This leads to the problem that the calibration pat-tern has to be kept in-focus during the calibration pro-cess and thus, has to be moved.

In our approach, we take a different route. We exploitthe planarity of the checkerboard to estimate its posewith respect to the camera. This, in turn, allows forestimating the circle of confusion, the PSF of an idealthin lens. The observed calibration pattern is convolvedwith a different PSF, which we consider to be a combi-nation of the ideal PSF and the in-focus PSF due to lensaberrations. Our goal is the estimation and removal ofthe latter. The resulting images are approximations toimages that would be taken by a perfect lens systemwith a finite aperture.

Our algorithm relies on the ability to detect blurred pat-terns. For this reason, we design a checkerboard patternthat is well suited to estimate PSFs and which can berobustly detected in the presence of blur. We call theresulting pattern and detection routines BlurTag.

2 RELATED WORKThe correction of image blur is content of muchresearch today. Usually, the goal is to correct for scenerelated degradations like motion blur [SJA08], out-of-focus blur [VBC+02], or camera shake [FSH+06].Blur removal is a deconvolution problem which comes

in two flavors: blind deconvolution e.g. [AD88, Car01]and deconvolution based on a known kernel e.g.[Ric72, Wie49]. Joshi et al. [JSK08] determine thekernel by edge prediction using the rationale thatpart of the blur kernel can be observed in a directionorthogonal to the edge. Based on this approach, Keeet al. [KPCW11] propose a non-blind approach toestimate and remove lens aberrations. They employcheckerboard-like patterns including circles that pro-vide edges in all orientations to estimate the kernel.As mentioned earlier, the pattern has to be in focus forthe method to yield unbiased estimates. Point spreadfunctions may change when the focus changes. If thecalibration image has been taken with a different focussetting from the one that is to be corrected the resultswill be inaccurate. Our work aims at relaxing thisrequirement.

Our approach is based on pose estimation with re-spect to a planar target, a standard problem in com-puter vision [Zha00]. An implementation is availablein Bouguet’s calibration toolbox [Bou02]. The toolboxestimates the common parameters related to a pinholecamera such as focal length, camera orientation, as wellas radial distortion. In this area, it is common to useuncoded checkerboards where the user has to define theouter checkerboard corners manually.

In the field of augmented reality, tag-based systemssuch as [GGSC96, ZFN02, dIMH02] are often em-ployed. These methods address the automated poseestimation problem, but do not consider the aspect ofcalibration. The focus is on the automatic detectionand identification of markers. In general, the tags ofaugmented reality can be adapted to checkerboards byusing them as codes. The approach of Atcheson etal. [AHH10] places two-dimensional codes into thesquares of the checkerboard. To enhance the detectionquality in blurry regions, the authors introduce a newdesign where codes are rotated and separated from thefeature points. Due to this, the resolution of the codesis reduced. In addition, the pattern employs straightedges which are not suitable to sample general two-dimensional blur kernels densely. We therefore opt foran alternative design.

Our goal is the automatic estimation of spatially vary-ing PSFs which are distributed over a captured image.We capture an image which provides a checkerboardpattern. This checkerboard pattern is optimized for

1. automatic detection as in the CALTag implementa-tion [AHH10],

2. accurate PSF estimation as in [KPCW11], and

3. blur robust detection for out-of-focus checker-boards.

focusedcheckerboard

unfocusedcheckerboardcovering thewhole image

lens imagesensor

aperturecircle of confusion

Figure 1: In our setup a calibration pattern is imagedat an out-of-focus position. The circle of confusion isblurring our image in addition to the lens PSF at thein-focus setting.

Further, we want to estimate the pose of the checker-board. Knowing the distance of the focal plane as wellas the aperture and focal length, we are able to estimatethe circle of confusion of the image regions covered bythe checkerboard pattern. These circles of confusionoffer the ability to compute in-focus PSFs from out-of-focus PSFs.In summary, we combine and extend different aspectsof previous work in order to provide a practical solutionto spatially varying PSF estimation for lens-inducedaberrations.

3 OVERVIEWThe goal of our approach is the calibration of the spa-tially varying PSFs for different focus settings of an op-tical system using a static calibration pattern at a fixedposition. The main motivation for this setting is that itis difficult to move a single calibration pattern to differ-ent focal planes while maintaining a full coverage of theimage and a comparable apparent resolution of the tar-get at different distances without changing the patternon the target.We therefore choose to image the calibration pattern inan out-of-focus position, see Fig. 1. Consider first thecheckerboard to be placed in the focal plane. In thiscase, an ideal, finite aperture, thin-lens imaging sys-tem would produce a perfect image since the systemresponse of the thin lens is Dirac. In a real system,however, the system response is different and the ob-served image IIF

obs is given by the object space patternIob j, convolved with the system PSF p:

IIFobs = Iob j⊗ p, (1)

where IF stands for “in-focus”. In general, the convo-lution has a spatially varying kernel p(x,y). If the pat-tern is placed in a different position, a finite aperture,thin-lens system will produce out-of-focus blur whichis usually described by the circle of confusion, a spe-cial PSF which is a scaled version of the aperture shape.The ideal image IT LOF

obs under these circumstances isgiven by

IT LOFobs = Iob j⊗ pcoc. (2)

Here, pcoc is the circle of confusion PSF and TLOFstands for “thin-lens out of focus”. Again, as in the in-focus case, a real system does not produce IT LOF

obs sinceits PSF is not an ideal circle of confusion which wouldhave a pill-box shape (assuming a circular aperture). In-stead, the out-of-focus PSF of the real system is givenby pcoc⊗ p. The observed out-of-focus image IOF

obs in areal system is thus given by

IOFobs = Iob j⊗ pcoc⊗ p. (3)

Our method aims at estimating p, given the observedout-of-focus image IOF

obs and the pattern definition Iob j.In order to make this problem tractable we need a re-liable way of estimating pcoc. We also aim at relaxingthe condition that the pattern has to be placed parallelto the image plane.The above requirements can be fulfilled if the pose ofa (planar) target with respect to the camera can be es-timated from the calibration image IOF

obs . Knowing thepose, and the aperture and focus settings of the camera,pcoc(x,y) can be computed at every position in the im-age. This implies, that we can recover the PSF of thein-focus plane p(x,y) by inverting Eq. 3.In practice, we compute the orientation of the pla-nar target by tracking a checkerboard coded with self-identifying markers (tags). This automates the calibra-tion process which is important if several focus set-tings are to be processed, potentially for several aper-ture/zoom settings of the camera. In this article, weonly discuss the computation of the PSFs from a singleimage. Calibrating an arrangement of settings wouldapply our method to each image independently.The processing pipeline is shown in Fig. 2. First, wedetect the individual squares of the checkerboard pat-tern and verify the contained tags. Then we estimatethe PSFs, which are blurred by the circle of confusion.In parallel, we estimate the pose of the checkerboard,which can also be regarded as the extrinsic parametersof the camera with respect to the checkerboard. Wecan extract the focal plane parameters from the EXIFheader provided by the captured image. The combi-nation of pose information and focal plane parametersallows us to compute the circle of confusion for eachposition covered by the checkerboard pattern. Finally,we deconvolve each PSF with the related circle of con-fusion. The result is a full image covering set of PSFsfor an image in the focal plane of the camera.In order for this process to work well, we have to regardtwo aspects: First, a checkerboard design that is wellsuited to PSF estimation. Second, a detection algorithmthat is robust to blur.

4 CHECKERBOARD DESIGNEstimating the PSF distribution based on a capturedcheckerboard pattern requires its initial detection. In

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

1 2

3 4

5

1. Square detection3. PSF estimation

2. Pose estimation4. CoC estimation

5. PSF sharpeningFigure 2: Overview of our approach

our approach, the checkerboard detection fullfills twogoals at once: First, we want to estimate one PSF foreach checkerboard square, and second, we want to es-timate the pose of the checkerboard for estimating thecircle of confusion. Based on these two goals, we canderive criteria, which have to be met by our checker-board design:

1. We seek to estimate accurate PSFs,

2. the detection must be robust to blur, and

3. the process should be automatic.

Based on these criteria, we derive a checkerboardpattern (Fig. 5, upper left) providing a code for eachcheckerboard square, for an example see Fig. 3.

For our initial tests we use a simple binary code. Thiscode could be made error-correcting as e.g. in [AHH10]by sacrificing some bits. The codes, however, are notonly used to identify a marker, but also to serve as apattern for PSF estimation, i.e. Iob j in Eq. 3. It has beenobserved in previous work that it is important to usepatterns that provide edges in every direction [JSK08].We therefore choose circles as the basic elements ofour codes. Instead of coding a bit as circle present (1)vs. circle absent (0) in a particular position, we use aninner-/outer-circle hierarchy, where the presence of aninner circle indicates a logical ’one’, see Fig. 3. Thismeasure ensures the presence of edges even for codesthat contain many zeros and thus improves the stabilityof the PSF estimation.

1 4

32

256

16384

Figure 3: Sample code with active bits 20, 22, 25, 28

and 214 leading to the ID 16677. The bits are using alexicographical order. Codes are unique under rotationand mirroring.

Further, we decided to make the codes unique in termsof rotation and mirroring by removing symmetric onesfrom the list of available codes. This property is sim-plifying the detection process. Knowing the orientationhelps when assembling a complete checkerboard pat-tern from single detected squares. We are dealing withtwo dimensional codes providing w horicontal and hvertical code elements. For two reasons, we decided toset w = h = 4. First, we are dealing with squares so weset w = h to distribute the corresponding objects homo-geneously. Second, regarding uniqueness with respectto symmetries and robustness to errors, 3×3 codes of-fer too few candidates to fill a reasonably sized checker-board pattern with unique codes. On the other hand,5× 5 codes become too dense for small resolutions inthe captured image.

5 IMPLEMENTATIONThe implementation of our algorithm is split into sev-eral parts. At first, we detect the checkerboard and iden-tify the markers, Sect. 5.1. The important aspect in thisstep is the robustness of the detection routine to blur.Next, we compute the combined PSFs pcoc ⊗ p fromthe image and the pre-defined pattern, Sect. 5.2. In or-der to remove the influence of the circle of confusionpcoc, we compute the pose of the checkerboard and es-timate pcoc, Sect. 5.3. Finally, we compute the in-focusPSFs p by deconvolving the combined PSFs pcoc⊗ pwith the circle of confusion PSF pcoc, Sect. 5.4.

5.1 Detection processThe critical aspect in the detection process is theidentification of the individual checkerboard squaresin the presence of blur. The main difficulty is theapparent merging of neighboring regions of the pat-tern [AHH10]. The main differentiating aspect of ourdetection scheme is that we interpret the thresholdedimage as a hierarchical tree structure: a connected

detect squares

identify squares sample and check codes

validate squares#code objects

#shape corners

isolate squares split merged squares

remove codes

add codes

Figure 4: Overview of the detection process.

component is a child of another if it is completelyincluded in the other. This feature lets us removeparts of the image (the codes) for the early parts of thedetection routine. This way, we can focus on separatingthe checkerboard squares without fearing to damagethe codes inside, see Fig. 4.

An overview of the processing pipeline is shown inFig. 5. The number of the processing step refers to thenumber of the subfigures in the pipeline overview. Inthe following, we describe the individual steps in moredetail.

1. Thresholding the image

Checkerboard patterns are usually black and white tosimplify the detection process. During capture, ambi-ent light, shades and light gradients lead to a gray-scalecheckerboard pattern. Our first step is to binarize theimage via thresholding. We use a median based adap-tive thresholding algorithm [Jai89]. The definition ofthe window size is the most critical parameter withinthe process. Too small windows lead to noisy resultswhile too large windows may lead to shrunk, open, orvanishing circles. In our implementation, we use a win-dow width of about 4 to 5 percent of the image width.After that we remove noise from the thresholded image.

2. Removing circles

In this step, we remove the circles that represent thecode of a square from the binary image. The result isa binary image without code structures as in Fig 5 (2).This leads to significant improvements in the followingsplitting step because the fine detail of the code struc-tures can be ignored. As discussed above, we interpretthe thresholded image as a tree of connected compo-nents, Fig. 6. The tree construction is performed intwo steps. First, the binary image is decomposed intoconnected components. Second, the connected compo-nents are organized in the component tree [NC06]. Thecomponent tree encodes the complete interrelations be-tween the nested structures. Using it, we can removethe circles by removing the corresponding nodes. Inour pattern, we remove the nodes which provide exactlyone leaf and the leafs themselves. After that, we drawthe tree again as a thresholded image, this time withoutthe circles.

3. Splitting adjacent squares

2.Remove codes

3.Separate into positive and

negative image versionand separate merged areas

4.Add codes to white

areas again

5.Filter by number of

children

7.Sample and verify codes

x

x x

xx x

xx

6.Filter by number of corners

1.Threshold image

Figure 5: The tracking pipeline.

Conversion totree structure

...

Tree of connected components

root node

code objectnodes

leafs

Figure 6: Conversion from pattern to tree structure. Thehierarchy of the connected components is encoded asparent/children relations in the tree.

At this step we have a binary image of the checkerboardwithout the circles. However, due to image blur theinitial thresholding step results in merged squares, i.e.squares of equal color meeting at one corner appear asa single connected component. The goal in this step isto split these structures.

First, we determine the dominant points for each struc-ture in the image by using the IPAN algorithm [CV98].We use an angle threshold of 160◦. Now we search

for pairs of dominant points that only have a small dis-tance from each other. To specify this distance we usea threshold Tx. The goal is to identify point pairs onopposite sides of a “bridge”, i.e. an area where squaresare merged. If Tx is chosen too large, squares can besplit diagonally which has to be avoided to preservetheir quadrilateral shape. For this reason we set Tx = 20pixels which is related to the minimal diagonal size ofusable squares. A square is usable if each circle can bedetected. Consider the minimal case: the cross-sectionalong the diagonal of a square provides 4×2 outer cir-cle edges + 4 inner circles + 3 spaces between the cir-cles and 2× 2 parts between the outer circles and thesquare border. Assuming that each part consists of atleast one pixel, the resulting minimal diagonal length is2×4+4+3+2×2 = 19 pixels.After having identified the candidates for splitting weperform this operation by drawing lines between neigh-boring dominant points. The resulting binary imagecontains what our algorithm believes to be squares. Inorder to avoid separate implementations for the whiteand the black squares we work with two copies of theimage, one being the inverse of the other. In the follow-ing, we describe the further processing for only one ofthose images.4. Adding circlesAt this point, we have a superset of square candidates.Our goal is to filter out all false positives. We achieve

this by observing the inner structures of each squarecandidate. For this, we add the circles that were re-moved in the second step back into the image. Thiscan be done by taking the original thresholded image(step 1) and the negated result of step 3 and performinga logical ’and’ operation between the two images.

5. Filtering by number of circles

Our checkerboard pattern provides only squares with16 circles. We take advantage of this fact by filteringout all candidates which do not meet this requirement.In our implementation, we convert the square candidateimage to a component tree as in step 2. Now, we cantraverse the tree and check the number of children ofall connected components. All squares which do notprovide 16 children are filtered from the tree. After that,the filtered tree is converted back to an image.

6. Filtering squares by number of corners

In this step we filter out all shapes that do not havefour corners. They tend to be no squares. We com-pute the potential corners by a variant of the IPAN al-gorithm. We then identify the point with the steepestangle less than 135◦. This threshold is used to avoidtreating edges that have accidently been split by a dom-inant point. These points usually make angles close to180◦ with their neighbors. In addition, if a perspectiveprojection is so strong as for the 90◦ corner to projectto an angle of larger than 135◦, then the internal codeof the square is likely to be of very low resolution andthus unusable. The output of this step is an image thatcontains square candidates with exactly 4 corners and16 code objects.

7. Sampling the codes and verification of the squares

Next, we estimate the homography between the de-tected quadrilateral and an ideal two-dimensional [0,1]2

space, [HZ10] which is used for sampling the code.

However, the corners computed so far are of low qualitysince they have been computed from the binary image.We increase the quality of the estimated points by merg-ing close points of different squares to their mean posi-tion. It is important to ensure, that these points share thesame real point. In addition we apply the Harris cornerdetector [HS88] on the gray value image.

5.2 PSF estimationAs outlined in Sect. 3, we can compute the combinedPSF pcoc ⊗ p by deconvolving the observed imageIOFobs with the definition of the pattern Iob j. For this

it is necessary to remove the perspective projectionof the recorded pattern, or, alternatively to warp thepre-defined pattern into the pose of the checkerboard.We choose the latter option to compute the PSFsdirectly in screen space. We do this by applying theinverse of the homographies computed in step 7 of the

detection scheme to the pre-defined pattern. This pro-cess is performed independently for each square of thecheckerboard. We use Wiener deconvolution [Wie49]to solve for pcoc⊗ p.

5.3 Pose and CoC estimationAs outlined above, the pose of the checkerboard al-lows for the estimation of the circle of confusion, thePSF of an ideal finite aperture camera system. Comput-ing the checkerboard pose is equivalent to determiningthe camera’s extrinsic parameters. We re-use the es-timated homographies for this purpose. Coupled withthe known real world size of each checkerboard squareand the camera intrinsics, we derive the position andorientation of the checkerboard with respect to the cam-era [HZ10].

Based on the pose of the checkerboard, we compute thedistance d between a selected point on the pattern andthe camera. With the known distance of the focal planeS, the aperture diameter A and the focal length f we cancompute the circle of confusion as

c = A|d−S|

df

S− f(4)

The circle of confusion diameter c needs to be con-verted to pixel size. Therefore, we need to regard thepixel dimensions of the camera sensor (e.g., CanonEOS 5D Mark II: 6.41 µm). The circle of confusion es-timation can be applied at each position within the im-age, which is covered by the captured checkerboard pat-tern. The information on aperture size and focal length,and focal distance can be extracted from the EXIF tagsaccompanying the captured images.

5.4 In-focus PSF estimationOnce the circle of confusion pcoc and the combinedPSF pcoc⊗ p are known, the in-focus PSF p can be re-covered by deconvolution. We again employ Wienerdeconvolution for this task. A validation example ofour procedure is shown in Fig. 7. In the first row weshow ground truth PSFs obtained by applying Eq. 1.The checkerboard is placed in the focal plane which isplaced at a distance of about 2.5m. Due to the largefield-of-view, the checkerboard cannot cover the com-plete image. In the second row we have moved thecheckerboard to a much closer distance of about 30cm.Here, the checkerboard covers the full image but ap-pears blurry. The insets show the same region on thecheckerboard. Correspondingly, the estimated com-bined PSFs are much larger than in the first row. Thethird row of the figure shows the estimated circles ofconfusion as well as the in-focus PSFs computed us-ing our method. They agree quite well with the groundtruth of the first row.

1

3

5

2

4

6

Figure 7: Estimation process of in-focus PSFs1. Well focused, checkerboard in focal plane (~250cm)2. Estimated PSFs from 1. incl. close-up3. Same focus as in 1., but differently positioned checkerboard (~30cm)4. Estimated PSFs of 3.5. Estimated circles of confusion of 3.6. In-focus PSF estimated from 3.

0

10

20

30

40

50

60

70

80

0 2 4 6 8 10

# d

ete

cted

sq

uare

s

blur kernel size(sigma)

BlurTagCALTag

Figure 8: Comparison of BlurTag and CALTag

6 RESULTS AND DISCUSSIONWe perform a number of tests to evaluate our algo-rithm. First we verify the detection process. Eachsquare, which we fail to detect leads to a missing PSF.So the quality of our detection algorithm is critical. Fur-ther discussion about interpolating missing PSFs can befound in [KPCW11]. As mentioned in the overviewsection, we tend to capture blurry images which arechallenging to process. For this reason, we implementtests to compare its stability with recent work.

6.1 Robustness to blurOur first test compares our BlurTag with CALTag,the recent approach of [AHH10]. We generate syn-thetic images using our pattern and the one proposed

Figure 9: Detected squares in CALTag vs. BlurTag.Left: Detected squares of CALTag-based checkerboard.Right: Result of BlurTag-based checkerboard.

in [AHH10]. Both synthesized images provide thesame amount of squares and the same perspectivetransformation. During the test, we successivelyincrease the blur. The number of detected squaresversus the size of the blur kernel (sigma) is shown inFigure 8.

We see that CALTag and BlurTag have compareabledetection results when the kernel size is small. Ini-tially, BlurTag’s performance decreases earlier thanCALTag’s due to the increase complexity of the circlepattern as compared to the squares of CALTag. Circlesrequire a higher resolution to work properly comparedto square based codes. With larger blur kernels,however, we see that CALTag’s results deterioratemore quickly as compared to BlurTag. The conclusionof this test is that CALTag is preferrable when thecamera resolution is limited and the amount of blur issmall. BlurTag shows its strengths with an increasingamount of blur at sufficient image resolution.

This observation is also manifest in a real-world test,Fig. 9. Large variations in square resolution, combinedwith significant amounts of blur lead to a reduced per-formance. However, The impact on BlurTag’s perfor-mance is smaller than for CALTag. Whereas CALTagis able to detect 29/70≈ 41% squares, BlurTag extracts52/70≈ 74% squares.

6.2 PSF quality for blurred capture vs.ground truth

The major goal of our approach is the estimation of cir-cles of confusion to avoid their impact on the estimatedPSFs. In Fig. 7 we show the final, in-focus PSFs (6) ascompared to PSFs estimated from a pattern recorded inthe focal plane (2). We see that the in-focus PSFs are inclose agreement with those, which we estimated with awell focused checkerboard. It is important to keep inmind, that the focus was not changed during the exper-iment.

For numerical comparison, we compute the RMS er-ror between the ground truth PSFs p of the in-focusrecorded checkerboard (computed via Eq. 1) againstthe combined PSFs pcoc ⊗ p as well as the estimatesin-focus PSFs using our method, Fig, 10. We see thatthe quality increase obtained from our method becomesbetter with an increasing diameter of the circle of con-fusion, i.e. the larger the blur, the more benefit to using

0

0.02

0.04

0.06

0.08

0.1

4 6 8 10 12 14 16 18 20

RM

SE

Diameter of circles of confusion in pixels

CorrectedUncorrected

Figure 10: Root mean square error test:Error between corrected or uncorrected PSFs and in-focus PSFs

our method. Another insight is that our computed in-focus PSFs have an approximately constant quality withrespect to the ground truth over the tested blur sizes.

6.3 Video ratesWhile our main focus is on improved and more flexi-ble PSF estimation, our method is fast enough to run atinteractive rates and could thus be of interest for blur-resistant tracking in applications where AR-markers aretraditionally used. We run on a test on a video with size640× 480 pixels. We achieve a mean speed of 13.04fps. Our unoptimized implementation runs in a single-threaded process on an AMD Athlon(tm) 64 X2 DualCore Processor 4600+. It should be possible to paral-lelize the algorithm on a GPU to achieve real time per-formance.

6.4 ApplicationFinally, we show results for sharpening a given targetimage using our estimated PSFs. Fig. 11 shows theapplication to a 22 Mpixel image taken with a Canon5D mark II camera and a 50mm f/1.4 lens. We seethat blur can be removed and the final image becomessharper. Especially structures with high gradients likethe handrails and the flowers improve in sharpness.

7 CONCLUSIONS AND FUTUREWORK

We have introduced a novel design for checkerboardpatterns that is well suited to PSF estimation and detec-tion in the presence of blur. Further, we have developeda new tracking algorithm, which is robust to resolutionvariations and large amounts of blur. In addition, we ex-ploited the planarity of checkerboards to estimate out-of-focus blur of the calibration pattern. This allowed usto generate “in-focus” PSFs independent of the focussetting of the camera.

Our tracking approach is directly tailored to the de-signed checkerboard pattern. Therefore, changes in thedesign require different detection algorithms as well. In

Figure 11: Results after deconvolutionTop: Target image with marked regions ( best viewed in color )Left: Blurred captured image regionsRight: Corresponding deconvolved image regions

general, it would be possible to analyze the checker-board patterns automatically to derive an appropriatetracking algorithm by analyzing its component tree, asdiscussed in the second step of our pipeline.

For the future, we would like to explore a real-time im-plementation. Further, it would be interesting to ana-lyze the space of a PSF shapes occuring under varyingfocus settings. Due to the expected low intrinsic dimen-sionality of this space this could lead to simpler modelsfor their estimation.

ACKNOWLEDGEMENTSThis work was supported by the DFG through the Clus-ter of Excellence “Multimodal Computing and Interac-tion” and the Max-Planck-Center for Visual Comput-ing.

8 REFERENCES[AD88] G.R. Ayers and J.C. Dainty. Iterative blind de-

convolution method and its applications. Opticsletters, 13(7):547–549, 1988.

[AHH10] Bradley Atcheson, Felix Heide, and WolfgangHeidrich. Caltag: High precision fiducial mark-ers for camera calibration. In VMV’10, pages41–48, 2010.

[Bou02] J.Y. Bouguet. Complete camera calibra-tion toolbox for Matlab. Retrieved from theWorld Wide Web: http://www. vision. caltech.edu/bouguetj/calib doc/index. html, 2002.

[Bra00] G. Bradski. The OpenCV Library. Dr. Dobb’sJournal of Software Tools, 2000.

[Car01] Alfred S. Carasso. Direct blind deconvolu-tion. SIAM Journal on Applied Mathematics,61(6):pp. 1980–2007, 2001.

[CV98] D. Chetverikov and J. Verestoy. Tracking featurepoints: a new algorithm. In Pattern Recog-nition, 1998. Proceedings. Fourteenth Interna-tional Conference on, volume 2, pages 1436–1438. IEEE, 1998.

[dIMH02] D.L. de Ipiña, P.R.S. Mendonça, and A. Hopper.Trip: A low-cost vision-based location systemfor ubiquitous computing. Personal and Ubiqui-tous Computing, 6(3):206–219, 2002.

[FSH+06] R. Fergus, B. Singh, A. Hertzmann, S.T. Roweis,and W.T. Freeman. Removing camera shakefrom a single photograph. ACM Transactions onGraphics (TOG), 25(3):787–794, 2006.

[GGSC96] S.J. Gortler, R. Grzeszczuk, R. Szeliski, and M.F.Cohen. The lumigraph. In Proceedings of the23rd annual conference on Computer graphicsand interactive techniques, pages 43–54. ACM,1996.

[HS88] C. Harris and M. Stephens. A combined cornerand edge detector. In Alvey vision conference,volume 15, page 50. Manchester, UK, 1988.

[HZ10] R. Hartley and A. Zisserman. Multiple view ge-ometry in computer vision 2nd ed. CambridgeUniv Press, 2010.

[Jai89] A.K. Jain. Fundamentals of digital image pro-cessing. Prentice-Hall, Inc., 1989.

[JSK08] Neel Joshi, Richard Szeliski, and David J. Krieg-man. PSF estimation using sharp edge predic-tion. Computer Vision and Pattern Recognition,IEEE Computer Society Conference on, 0:1–8,2008.

[KPCW11] E. Kee, S. Paris, S. Chen, and J. Wang. Model-ing and removing spatially-varying optical blur.

In Computational Photography (ICCP), 2011IEEE International Conference on, pages 1–8.IEEE, 2011.

[NC06] Laurent Najman and Michel Couprie. Build-ing the Component Tree in Quasi-LinearTime. Transactions on Image Processing,15(11):3531–3539, 20006.

[Ric72] W.H. Richardson. Bayesian-based iterativemethod of image restoration. JOSA, 62(1):55–59, 1972.

[SJA08] Q. Shan, J. Jia, and A. Agarwala. High-qualitymotion deblurring from a single image. In ACMSIGGRAPH 2008, pages 1–10. ACM, 2008.

[VBC+02] P. Vivirito, S. Battiato, S. Curti, M. La Cascia,and R. Pirrone. Restoration of out of focus im-ages based on circle of confusion estimate. InProceedings of SPIE 47th Annual Meeting, vol-ume 4790, pages 408–416, 2002.

[Wie49] N Wiener. Extrapolation, interpolation, andsmoothing of stationary time series: with engi-neering applications, volume 47. Wiley, 1949.

[ZFN02] X. Zhang, S. Fronz, and N. Navab. Visual markerdetection and decoding in AR systems: A com-parative study. In Proceedings of the 1st Inter-national Symposium on Mixed and AugmentedReality, page 97. IEEE Computer Society, 2002.

[Zha00] Z. Zhang. A flexible new technique for cameracalibration. Pattern Analysis and Machine In-telligence, IEEE Transactions on, 22(11):1330–1334, 2000.