Text detection and recognition in natural images

Steven BellStanford Universitysebell@stanford.edu

Abstract

Natural scenes pose a major challenge to traditionaloptical character recognition methods because they oftencontain noise, occlusions, distortions, or relatively smallamounts of highly styled text. In this work, we build aprobabilistic system which unifies the tasks of text detectionand recognition with a language model. We use an efficientmulti-scale character detector to locate characters withinan image without performing segmentation. This is fol-lowed by a graph-based search which groups the detectionsinto words and evaluates their relative probabilities, avoid-ing binary decisions except where computationally neces-sary.

1. IntroductionThe modern world is filled with text, which humans use

effortlessly to identify objects, navigate, and make deci-sions. Although the problem of reading text with a com-puter has been largely solved within limited domains – suchas identifying license plate numbers, digitizing books anddocuments, or reading handwritten mailing addresses – themore general question of how to detect and read text in nat-ural, unstructured images remains an open challenge.

The applications for such a system are numerous. Forexample, buildings photographed in Google’s Street Viewcould be automatically tagged with their address if a com-puter could detect and read the building numbers. Likewise,robots could be made to more intelligently act in a world an-notated for humans, and translation systems for blind per-sons or foreign travelers would be improved.

Several factors make reading text in natural scenes par-ticularly difficult. In unstructured scenes, such as a pho-tograph of a storefront, clutter dominates the image. Textregions are of varying size and orientation, scattered, andpossibly occluded. Letters may appear in multiple fonts andsome letters may be ambiguous based on their shape alone.

Many approaches to this problem treat the recognitionproblem as two separate tasks, first locating text and draw-ing a bounding box, and then binarizing and reading it

with traditional OCR methods. By forcing binary decisionsat several points, this approach forfeits information whichcould be used to produce a more accurate result.

This project uses a probabilistic framework where im-age regions are assigned probabilities of containing differ-ent characters, and the result is read with the help of lan-guage characteristics, specifically the co-appearance of let-ters.

To limit the scope of the project, we considered onlyhorizontal strings of upright lowercase letters. We workedprimarily on synthetic images with automatically generatedground truth, which reduced the time spent creating trainingdata.

2. Prior workOptical character recognition for printed documents and

handwritten digits has a long history. Many methods havebeen proposed and used successfully, but most of these as-sume that the input image has been cleanly binarized andthat characters can be segmented easily, which is rarely thecase in unstructured images. Because of the complexity ofthe input images, text recognition in natural scenes is moreclosely related to object detection than it is to traditionalOCR.

A large body of past work has focused purely on thechallenge of locating text within scenes, spurred primarilyby the ICDAR text detection challenges of 2003 and 2011.These works can be roughly categorized into connected-component based methods, which segment the image andattempt to pick out which blobs are characters, and patch-based methods, which use convolutional filters, image patchcorrelations, wavelets, or other features to label the proba-bility that an image patch contains text [1, 2].

Another domain has focused on the task of reading scenetext given a bounding box, centered around the ICDAR ro-bust reading challenge. Weinman et al. use a Bayesianmodel to unify text segmentation and reading, and showthat reading accuracy can be improved by incorporating ad-ditional context such as language characteristics and the vi-sual similarity of letters [3].

More recently, several groups have created end-to-end

detection and recognition systems. These use a variety offeatures for detecting and classifying characters, includingGabor filters, and connected-component dimensions anddensities [4]. A support vector machine or linear discrim-inant analysis is used to perform classification based onthe features. This work is an extension of these, wherehigher-level information is used to aid text detection, notjust recognition.

3. Method

3.1. Character detection

The core of our algorithm is a multi-scale Histogram ofOriented Gradients (HOG) detector. The HOG descriptor isa patch-based algorithm which provides a dense estimate ofshape over an entire image, using histograms of quantizedgradient orientations for small image patches. It was origi-nally introduced for pedestrian detection [5], but has sincebeen applied to a wide variety of recognition tasks, includ-ing character classification [1].

Because it works on a images patches rather than onpre-segmented components, the descriptor is robust againstcharacters which are accidentally split apart or joined to-gether, which are difficult for a connected-component rec-ognizer to handle. By using gradients rather than raw pixelvalues and by normalizing the resulting histogram, HOGis invariant to illumination and brightness changes. HOGinherently encodes position information, but also allows adegree of variance by virtue of its coarse spatial binning.

However, for HOG to work correctly, the sizes of theletters must roughly match the dimensions of the trainingexamples, so the descriptor must be run at multiple scales.Additionally, characters have an extreme range of aspect ra-tios, which means that the detector must also run across arange of widths.

To detect characters, we use logistic regression on blocksof the HOG descriptor. Logistic regression was chosensince it is efficient to evaluate and provides a direct estimateof probability. We train and save a unique detector for eachcharacter, then the HOG descriptor is computed on the inputimage and each detector is run in a sliding window fashionacross it. This produces a 2-D matrix of detection probabil-ities for each character. Points with probabilities meeting athreshold - generally far more than the number of charac-ters actually present in the image - are carried forward tothe next stage.

3.2. Line detection

Given a set of detections, the next step is to find the mostlikely lines of characters. We do this by taking weightedvotes across all character detections. The vote V can be

written

V (v) =

n∑i=1

p(di)w(v − yi)

where v is the pixel height, xi and yi are the detection posi-tion, p(d) is the probability of a detection d (i.e, the result oflogistic regression), and w is a window function weightedtoward the center of the detection. We currently use a Ham-ming window, but a triangular or Gaussian window wouldalso be appropriate.

An example of a detection and the corresponding line isshown in Figure 1.

0

20

40

60

80

100

120

140

160

180

200

Figure 1. Detection of lines in an image. The left image shows thedetections and the resulting line. The right shows the voting result.

3.3. Recognition and reading

Given a set of lines, we can determine the probabilitythat each detected character belongs to that line based onits vertical position. Using a straightforward application ofBayes’ rule we can write

p(lij |yi) =p(yi|lj)p(lj)

p(yi)

where lij represents the probability that detection i belongsto line j, and p(yi) is the probability of detection i appearingat vertical position yi.

Assuming that p(yi), and p(lj) are constants (i.e, thatthe positions of the characters and lines are uniformly dis-tributed in the test images) the equation simplifies to

p(lij |yi) ∝ p(yi|lj)

The distribution on the right can easily be calculated us-ing detected lines and ground truth bounding boxes for eachcharacter.

To read words, we scan horizontally along each line andbuild a directed graph of word possibilities. An empty rootnode begins the graph, and letters are successively added asthe algorithm scans left to right.

Overlapping detections of the same letter are merged to-gether, while overlapping detections of differing letters cre-ate multiple paths in the graph. Each node is connected toall of the possible letters immediately preceding it, that is,the nodes in the graph closest to the leaves which do notoverlap. An example is shown in Figure 2.

2

spurn

spurn

s p ur n

mFigure 2. Completed word graph for a simulated example of theword ‘spurn’. A false detection of the letter ‘m’ causes the graphto have multiple paths.

With the graph completed, we can can assign edgeweights based on the probability of the letter combinationappearing in English text, and node weights using the prob-ability of detection and the probability that the characterbelongs to the line. By taking negative log-likelihoods ofthe probabilities, finding the most likely word becomes aminimum-cost graph traversal, which can be solved withDjikstra’s algorithm or A*.

Using a complete dictionary of English words would al-low more aggressive and accurate guessing of words. How-ever, the corresponding disadvantage is that a large propor-tion of words in natural images are non-dictionary words,such as proper names.

4. ImplementationWe implemented our own HOG descriptor in MATLAB,

which efficiently calculates the descriptor at multiple scalesby computing the gradients once, building a multidimen-sional equivalent of an integral image, and then computingthe descriptor at each scale. While our code is theoreticallymore efficient than recomputing the HOG descriptor fromscratch at each scale, we found that in practice our MATLABimplementation runs significantly slower than the C imple-mentation of HOG included in the open-source VLFeat li-brary.

4.1. Detectors

The detectors are trained by computing the HOG de-scriptor for all input images and using logistic regressionto obtain a linear classifier. Input images are scaled so thatthe resulting descriptor is 8 × n × 31, where n varies withthe aspect ratio between 3 and 12. This gives feature vectorswith between one and three thousand elements.

We initially normalized all of the characters to a constantwidth and height based on their character bounding boxes.However, this vastly increases the range of scales which thedetectors must run at, since ’m’ and ’w’ are several times

wider than ’t’ or ’f’ in many fonts. Additionally, the char-acters ’i’ and ’l’, are extremely distorted (stretched to fillthe whole width) and thus the training images bear little re-semblance to the actual characters which must be detected.Instead, we computed the median width for each characterusing the training set bounding boxes, and then normalizedeach character to a single height and the median aspect ratio.This preserves the relative shape, but means that a separatetraining set must be generated for each character.

We experimented with a variety of training set sizes andrelative ratios of positive and negative examples. Furtherdetails of the training data are described in Section 5.1.

We used the liblinear package to perform logistic regres-sion in MATLAB. Training on 15,000 images takes approx-imately an hour on a 2 GHz Intel Core 2 Duo laptop with3.2 GB of RAM. 1

To run the detector, we take the dot product between thedetector and a flattened block of the image HOG features ina sliding window across the image. By keeping the detectorand the image HOG features as 3-dimensional matrices, thisbecomes a series of cross-correlations between the corre-sponding planes of the descriptors, which can be computedefficiently as a 3-dimensional convolution in MATLAB.

4.2. Reading

Character position statistics were calculated by takingthe mean and variance of ground truth character centers.The vertical position standard deviation is on the order of2-4 pixels for a text height of 72 pixels, somewhat smallerthan we expected. Character coappearance statistics werecalculated using 18 popular documents from Project Guten-berg, comprising approximately 12 million characters.

5. Testing5.1. Synthetic dataset generation

Because obtaining a large dataset with ground truth isdifficult and time consuming, we relied on a syntheticallygenerated dataset, which labels ground truth for severalstages of the pipeline. This approach was used successfullyby Neumann et al. in [6] to train character classifiers for nat-ural images using only computer-generated data. Despiteusing synthetic data, building a good training set turned outto be the most difficult and time-consuming challenge of theproject.

The input to our data generation program is a dictionaryof words and a list of fonts installed on the computer. Foreach requested training sample, the program selects a wordand font at random, and generates a black-and-white imagecontaining the word. This image can easily be segmented to

1I attempted to run my code on the corn cluster, but the network/disklatency from AFS made it far slower than running locally, since the trainingincludes reading tens of thousands of tiny images.

3

provide character bounding boxes and other ground truth. Asecond color image is generated by selecting random colorsfor the background and text. Noise is added, and the finalresult is saved. Compression artifacts add a small amountof additional noise.

For character detector training, it is important to includea large number of examples with many small variations.Since the HOG cells form a coarse grid and are run at adiscrete set of scales, it is important for the detector to findmatches which may have small scale and position differ-ences.

To achieve this, we took each of 1000 input examplesand created 15 new images for training. Approximatelyhalf of the images contain small position offsets and scal-ing, while the other half contain large offsets or scales. Theformer are kept as positive examples, while the latter aremarked as negative examples, along with images of all theother letters. This helps prevent the detector from misfir-ing on parts of letters which might otherwise be consideredmatches.

Figure 3. Top: Sample input image. Middle: Positive training ex-amples which exhibit small variations. Bottom: Negative trainingexamples which exhibit large variations.

We found that it was important to provide a smallpadding space around each training example. If the outsideedge of a letter is cut off, the gradient along that edge nolonger contributes to the descriptor and the result is muchweaker. This is similar to the finding in [5] that spacearound the pedestrian is important, and that increasing thesize of the pedestrian but removing the border space causesa lower accuracy.

After training a set of detectors with this dataset, we cre-ated an augmented dataset to reduce false positives. Thedetectors were run on the original 1000 input images, andtraining examples were pulled from all false positives.

5.2. Evaluation

To provide a quantitative metric on the accuracy of ourdetector, we ran it on a separate test set of 100 images. Theprecision-recall curves are shown in Figure 4, and confusion

matrices for the test set are shown in Figure 5.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cis

ion

Original dataset

Augmented dataset

Figure 4. Precision-recall curves for the original and augmenteddatasets. Most of the images contained multiple detections of atleast one character; these were combined into one for calculatingprecision and recall.

Detected

Tru

th

a b c d e f g h i j k l mn o p q r s t u v w x y z

abcdefghij

kl

mnopqrstuvwxyz

DetectedT

ruth

a b c d e f g h i j k l mn o p q r s t u v w x y z

abcdefghij

kl

mnopqrstuvwxyz

Figure 5. Confusion matrices without (left) and with (right) aug-mented training set, tested on a separate testing set of 100 images.

While this classifier performs fairly well on the test setwhich contains single characters, it fares poorly on wordsand sentences, often misfiring on combinations of lettersand spaces between letters. Additionally, it fails badly onthe letters ‘i’ and ‘l’, and has some difficulty with ‘f’ and‘t’, because the training set for these letters contains manynegative examples which appear identical to positive exam-ples. Figure 6 shows several negative examples for the letter‘i’, which could easily be considered positive examples.

Figure 6. Negative training examples for the letter ‘i’ which arevery similar to positive examples. Because of these, the ‘i’ detectordoes a very poor job.

In order to mitigate these problems, we created a new

4

training set based on images with complete words. As be-fore, positive examples are taken from the character bound-ing box with small perturbations. We used a larger bor-der space to allow better differentiation between characterssuch as ‘a’, ‘d’, and ‘q’, which can look identical if theascender or descender is ignored. Negative examples aredrawn from random patches of the image. We used an in-put set of 2000 images with an average of 20 letters each,which produced approximately 46,000 training images perletter, split 40% positive/60% negative.

Subjectively, the resulting detector performs better onthe problematic letters above. However, it performed poorlyon the single-character test set, with precision less than 10%across all recall values. Most of the false positives were onflat patches of noise. Possible solutions for this might be toinclude a higher percentage of flat examples in the trainingset, or to replace HOG’s local normalization with a wide orglobal normalization.

The detection algorithm is not accurate enough to pro-vide reliable input to the graph construction and evaluationalgorithm, so we did not get to the point of testing it. Run-ning the graph code on ground truth bounding boxes is notparticularly enlightening, since it consists of a single truepath.

Example results for some images in the ICDAR datasetare shown in Figure 7. On real photographs, our detectorfrequently reports hundreds of false positives, particularlytextured regions of the image such as brick. Using texturedbackgrounds or image patches as training examples wouldprobably produce better results here than flat colors.

6. Future workSome letters, particularly the lowercase letter ‘a’, can be

written in multiple ways. It would therefore be more appro-priate to split the character into two classes with separatedetectors but the same label.

Due to the computational load, we did not have timeto experiment with as many parameter permutations as weoriginally hoped. Further work should test larger descrip-tors (10 or 12 blocks high) and a range of padding widths.

References[1] K. Wang and S. Belongie, “Word spotting in the wild,”

in Computer Vision – ECCV 2010, ser. Lecture Notesin Computer Science, K. Daniilidis, P. Maragos, andN. Paragios, Eds. Springer Berlin / Heidelberg, 2010,vol. 6311, pp. 591–604.

[2] A. Coates, B. Carpenter, C. Case, S. Satheesh,B. Suresh, T. Wang, D. Wu, and A. Ng, “Text detectionand character recognition in scene images with unsu-pervised feature learning,” in International Conference

Figure 7. Sample detection results from the ICDAR dataset. Thedetector finds most of the characters, but includes many false pos-itives. In the bottom image, only the strongest results are shown.

on Document Analysis and Recognition (ICDAR), Sep.2011, pp. 440–445.

[3] J. Weinman and E. Learned-Miller, “Improving recog-nition of novel input with similarity,” in IEEE Com-puter Society Conference on Computer Vision and Pat-tern Recognition, vol. 1, Jun. 2006, pp. 308–315.

[4] L. Neumann and J. Matas, “Real-time scene text local-ization and recognition,” in Computer Vision and Pat-tern Recognition (CVPR), 2012 IEEE Conference on,june 2012, pp. 3538–3545.

[5] N. Dalal and B. Triggs, “Histograms of oriented gradi-ents for human detection,” in Computer Vision and Pat-tern Recognition, 2005. CVPR 2005., vol. 1, Jun. 2005,pp. 886 –893 vol. 1.

[6] L. Neumann and J. Matas, “A method for text local-ization and recognition in real-world images,” in Com-puter Vision ACCV 2010, ser. Lecture Notes in Com-puter Science, R. Kimmel, R. Klette, and A. Sugimoto,Eds. Springer Berlin / Heidelberg, 2011, vol. 6494,pp. 770–783.

A. Project sharingThis project was done in combination with CS 231A

(Computer Vision). I was the only student working on theproject in either class.

5