What’s New in Apache Lucene 3.0

ProgrammerOpen Source SearchWhat’s New in Apache LuceneA Lucid Imagination Technical White Paper

Programmer’s Guide to Open Source Search:

Apache Lucene 3.0

Programmer’s Guide: What’s New in Lucene 2.9 / 3.0 A Lucid Imagination Technical White Paper • June 2010 Page i

© 2010 by Lucid Imagination, Inc. under the terms of Creative Commons license, as detailed at http://www.lucidimagination.com/Copyrights-and-Disclaimers/. Version 1.02, published 6 June 2010. Solr, Lucene, Apachecon and their logos are trademarks of the Apache Software Foundation.

Programmer’s Guide: What’s New in Lucene 2.9 / 3.0 A Lucid Imagination Technical White Paper • June 2010 Page ii

Abstract

Apache Lucene is a high-performance, cross-platform, full-featured Information Retrieval library in open source, suitable for nearly every application that requires full-text search features.

Since its introduction nearly 10 years ago, Apache Lucene has become a competitive player for developing extensible, high-performance full-text search solutions. The experience accumulated over time by the community of Lucene committers and contributors and the innovations they have engineered have delivered significant ongoing advances in Lucene’s capabilities.

This white paper describes the new features and improvements in the latest versions, Apache Lucene 2.9 and 3.0. It is intended mainly for programmers familiar with the broad base of Lucene’s capabilities, though those new to Lucene should also find it a useful exploration of the newest features. Key topics such as how to upgrade from 2.9 to 3.0, as well as considerations for migrating from Lucene to Solr, are also addressed.

In the simplest terms, Lucene is now faster and more flexible than before. Historic weak points have been improved to open the way for innovative new features like near-real-time search, flexible indexing, and high-performance numerical range queries. Many new features have been added, new APIs introduced, and critical bugs have been fixed—all with the same goal: improving Lucene’s state-of-the-art search capabilities.

Programmer’s Guide: What’s New in Lucene 2.9 / 3.0 A Lucid Imagination Technical White Paper • June 2010 Page iii

Table of Contents

Introduction ............................................................................................................................................................ 1

Core Features and Improvements .................................................................................................................. 4

Numeric Capabilities and Numeric Range Queries .............................................................................. 4

New TokenStream API .................................................................................................................................... 8

Per-Segment Search ...................................................................................................................................... 12

Near Realtime Search (NRS) ...................................................................................................................... 14

MultiTermQuery-Related Improvements ............................................................................................. 15

Payloads ............................................................................................................................................................. 16

Additions to Lucene Contrib .......................................................................................................................... 18

New Contrib Analyzers ................................................................................................................................ 18

Lucene Spatial (formerly known as LocalLucene) ............................................................................ 18

Lucene Remote and Java RMI .................................................................................................................... 20

New Flexible QueryParser .......................................................................................................................... 20

Minor Changes and Improvements in Lucene 2.9 ............................................................................. 21

Changes and Improvements in Lucene 3.0 .......................................................................................... 23

Lucene Version by Version Compatibility since 2.9 ......................................................................... 24

Strategies for Upgrading to Lucene 2.9 / 3.0........................................................................................... 25

Upgrade to 2.9—Recommended Actions .............................................................................................. 26

Upgrade to 2.9—Optional Actions ........................................................................................................... 26

Migrating from Lucene to Solr? .................................................................................................................... 27

References ............................................................................................................................................................ 29

Next Steps ............................................................................................................................................................. 30

APPENDIX: Choosing Lucene or Solr .......................................................................................................... 31

Programmer’s Guide: What’s New in Lucene 2.9 / 3.0 A Lucid Imagination Technical White Paper • June 2010 Page 1

Introduction

Apache Lucene is a high-performance, cross-platform, full-featured Information Retrieval library, in open source, suitable for nearly every application that requires full-text search features. Lucene currently ranks among the top 15 open source projects and is one of the top 5 Apache projects, with installations at over 4,000 companies. Downloads of Lucene, and its server implementation Solr, have grown nearly tenfold over the past three years; Solr is the fastest-growing Lucene subproject. Lucene and Solr offer an attractive alternative to proprietary licensed search and discovery software vendors.1 With the release of versions 2.9 and 3.0 (September and November 2009), the Apache Lucene community delivered the latest upgrades of Lucene.

This white paper aims to address key issues for you if you have an Apache Lucene-based application, and need to upgrade existing code to work well with these latest versions, so that you may take advantage of the various improvements and prepare for future releases and application maintainability. If you do not have a Lucene application, the paper should also give you a good overview of the innovations in this release.

Unlike the previous 2.4.1 release (March 2009), Lucene 2.9 and 3.0 go well beyond just a bug-fix release. They introduce multiple performance improvements, new features, better runtime behavior, API changes, and bug-fixes at a variety of levels. Importantly, 2.9 deprecates a number of legacy interfaces, and 3.0 is in the main a reimplemented version of 2.9, but without those deprecated interfaces.

The 2.9 release improves Lucene in several key aspects, which make it an even more compelling alternative to other solutions. Most notably:

� Improvements for Near-Realtime Search capabilities make documents searchable almost instantaneously.

� A new, straightforward API for handling Numeric Ranges both simplifies development and virtually wipes out performance overhead.

� Analysis API has been replaced for more streamlined, flexible text handling.

1 See the Appendix for a discussion of when to choose Lucene or Solr.


And, behind the scenes, the groundwork has been laid for yet more indexing flexibility in future releases.

Lucene Contrib also adds new utility packages, introduced with this release:

� An extremely flexible query parser framework opens new possibilities for programmers to more easily create their own query parsing syntax.

� Local-Lucene and its geo-search capabilities, now donated to Apache, provide this near-mandatory functionality for state-of-the-art search.

� Various contributions have markedly improved support for languages like Arabic, Persian, and Chinese.

Version 3.0 is again a cleanup release and considered feature equivalent to its predecessor. 3.0 is the first Apache Lucene release requiring Java 5 at runtime, enabling Lucene to make use of new language features such as Generics Enumerations, Variable Arguments, along with Java 5’s concurrent utilities.

2.9 release improves Lucene in several key aspects and 2.9 deprecates a number

of legacy interfaces. 3.0 is in the main a reimplemented version of 2.9, but without

those deprecated interfaces.

While the majority of programmers are already running on either version 1.5 or 1.6 platforms (1.6 is the recommended JVM), Java 1.4 reached its end of service life in October 2008. With the new major Lucene 3.0 release, all legacy issues marked as deprecated have now been removed, enforcing their replacement.

Some important notes on compatibility: because previous minor releases also contained performance improvements and bug fixes, programmers have been accustomed to upgrading to a new Lucene version just by replacing the JAR file in their classpath. And, in those past cases, Lucene-based apps could be upgraded flawlessly without recompiling the software components accessing or extending Apache Lucene. However, this may not be so with Lucene 2.9/3.0.

Programmer’s Guide: What’s New in A Lucid Imagination Technical White Paper

Lucene 2.9 introduces several backchanges in your code that uses the library. A to be successful; at a minimum, it is not likely to be flawless. As a result, we recommend that if you are upgrading from a previous Lucene release, you should at least recompile any software components directly accessing or extending the library. In the latter case, recompilation alone will most likely not be sufficient. More details on these dependencies are discussed in the “Upgrading Lucene” section of the paper.

We’ve also noted any significant compatibility issues, labeling them with this flag:

This document is not intended to be a comprehensive overview 2.9/3.0, but rather of new key features and capabilities. Imagination Certified distribution Lucene Website (lucene.apache.org

What’s New in Lucene 2.9 / 3.0 White Paper • June 2010

Lucene 2.9 introduces several back-compatibility-breaking changes that may well require that uses the library. A drop-in library replacement is not guaranteed

to be successful; at a minimum, it is not likely to be flawless. As a result, we recommend that if you are upgrading from a previous Lucene release, you should at least recompile any

ftware components directly accessing or extending the library. In the latter case, recompilation alone will most likely not be sufficient. More details on these dependencies are discussed in the “Upgrading Lucene” section of the paper.

ignificant compatibility issues, labeling them

This document is not intended to be a comprehensive overview of all functions of new key features and capabilities. Always check the Lucid

Certified distribution (www.lucidimagination.com/downloadslucene.apache.org) for the most up-to-date release information.

Page 3

breaking changes that may well require library replacement is not guaranteed

to be successful; at a minimum, it is not likely to be flawless. As a result, we recommend that if you are upgrading from a previous Lucene release, you should at least recompile any

ftware components directly accessing or extending the library. In the latter case, recompilation alone will most likely not be sufficient. More details on these dependencies

all functions of Lucene Always check the Lucid

www.lucidimagination.com/downloads) and the official date release information.


Core Features and Improvements

Numeric Capabilities and Numeric Range Queries One of Apache Lucene's basic properties is its representation of internal searchable values (terms) as UTF-8 encoded characters. Every value passed to Lucene must be converted into a string in order to be searchable. At the same time, Lucene is frequently applied to search numeric values and ranges, such as prices, dates, or other numeric field attributes. Historically, searching over numeric ranges has been a weak point of the library. However, the 2.9 release comes with a tremendous improvement for searching numeric values, especially for range queries.

Prior to Lucene 2.9, numeric values were encoded with leading zeros, essentially as a full-precision value. Values stored with full precision ended up creating many unique terms in the index. Thus, if you needed to retrieve all documents in a certain range (e.g., from $1.50 to $1500.00), Lucene had to iterate through a lot of terms whenever many documents with unique values were indexed. Consequently, execution of queries with large ranges and lots of unique terms could be extremely slow as a result of this overhead.

Many workaround techniques have evolved over the years to improve the performance of ranges, such as encoding date ranges in multiple fields with separate fields for year, month, and day. But at the end of the day, every programmer had to roll his or her own way of searching ranges efficiently.

In Lucene 2.9, NumericUtils and its relatives (NumericRangeQuery / NumericRangeFilter ) introduce native numeric encoding and search capabilities. Numeric Java primitives (long , int , float, and double ) are transformed into prefix-

encoded representations with increasing precision. Internally each prefix precision is generated by stripping off the least significant bits indicated by the precisionStep. Each value is subsequently converted to a sequence of 7-bit ASCII characters (due to the UTF-8 term encoding in the index, 8 or more bits would split into two or more bytes) resulting in a predictable number of prefix-terms that can be calculated ahead of time. The figure below illustrates such a Prefix Tree.


Example of a Prefix Tree, where the leaves of the tree hold the actual term values and all the descendants of a

node have a common prefix associated with the node. Bold circles mark all relevant nodes to retrieve a range

from 215 to 977.

The generated terms are indexed just like any other string values passed to Lucene. Under the hood, Lucene associates distinct terms with all documents containing the term, so that all documents containing a numeric value with the same prefix are “grouped” together, meaning the number of terms that need to be searched is reduced tremendously. This stands in contrast to the relatively less efficient encoding scheme in previous releases, where each unique numeric value was indexed as a distinct term based on the number of terms in the index.


Directory directory = new RAMDirectory();

Analyzer analyzer = new WhitespaceAnalyzer();

IndexWriter writer = new IndexWriter(directory, ana lyzer,

IndexWriter.MaxFieldLength.UNLIMITED);

for (int i = 0; i < 20000; i++) {

Document doc = new Document();

doc.add(new Field("id", String.valueOf(i), Field.St ore.YES, Field.Index.NOT_ANALYZED_NO_NORMS));

String num = Integer.toString(i);

String paddedValue = "00000".substring(0, 5 - num.l ength()) + num;

doc.add(new Field("oldNumeric", paddedValue, Field. Store.YES, Field.Index.NOT_ANALYZED_NO_NORMS));

writer.addDocument(doc);

}

writer.close();

Indexing a zero-padded numeric value for use with an ordinary RangeQuery.

You can also use the native encoding of numeric values beyond range searches. Numeric fields can be loaded in the internal FieldCache, where they are used for sorting. Zero-padding of numeric primitives (see code example above) is no longer needed as the trie-encoding guarantees the correct ordering without requiring execution overhead or extra coding.

The code listing below instead uses the new NumericField to index a numeric Java primitive using 4-bit precision. Like the straightforward NumericField , querying numeric ranges also provides a type-safe API. NumericRangeQuery instances are created using one of the provided static constructors for the corresponding Java primitive.


Directory directory = new RAMDirectory();

Analyzer analyzer = new WhitespaceAnalyzer();

IndexWriter writer = new IndexWriter(directory, ana lyzer,

IndexWriter.MaxFieldLength.UNLIMITED);

for (int i = 0; i < 20000; i++) {

Document doc = new Document();

doc.add(new Field("id", String.valueOf(i), Field.St ore.YES, Field.Index.NOT_ANALYZED_NO_NORMS));

doc.add(new NumericField("newNumeric", 4,

Field.Store.YES, true).setIntValue(i));

writer.addDocument(doc);

}

writer.close();

Indexing numeric values with the new NumericField type

The example below shows a numeric range query using an int primitive with the same precision used in the indexing example. If different precision values are used at index or search time, numeric queries can yield unexpected behavior.


IndexSearcher searcher = new IndexSearcher(director y, true);

Query query = NumericRangeQuery.newIntRange("newNum eric", 4, 10, 10000, true, false);

TopDocs docs = searcher.search(query, null, 10);

assertNotNull("Docs is null", docs);

assertEquals(9990, docs.totalHits);

for (int i = 0; i < docs.scoreDocs.length; i++) {

ScoreDocs d= docs.scoreDocs[i];

assertTrue(sd.doc >= 10 && sd.doc < 10000);

}

Searching numeric values with the new NumericRangeQuery

Improvements resulting from new Lucene numeric capabilities are equally significant in versatility and performance. Now, Lucene can cover almost every use-case related to numeric values. Moreover, range searches or sorting on float or double values up to fast date searches (dates converted to time stamps) will execute in less than 100 milliseconds in most cases. By comparison, the old approach using padded full-precision values could take up to 30 seconds or more depending on the underlying index.

New TokenStream API Almost every programmer who has extended Lucene has worked with its analysis function. Text analysis is common to almost every use-case, and is among the best known Lucene APIs.

Since its early days, Lucene has used a “Decorator Pattern” to provide a pluggable and flexible analysis API, allowing a combination of existing and customized analysis implementations. The central analysis class TokenStream enumerates a sequence of tokens from either a document's fields or from a query. Commonly, multiple TokenStream instances are chained, each applying a separate analysis step to text terms represented by a Token class that encodes all relevant information about a term.

Prior to Lucene 2.9, TokenStream operated exclusively on Token instances transporting term information through the analysis chain. With this release, the token-based API has been marked as deprecated. It is completely replaced by an attribute-based API.


Here’s how it has changed. Rather than receiving a Token instance from one of the two TokenStream.next() methods, the new API follows a stateful approach instead. To advance in the stream, consumers call TokenStream.incrementToken() , which returns a Boolean result indicating if the end of the stream has been reached. Information gathered during the analysis process is encoded in attributes accessible via the new TokenStream base class AttributeSource . In contrast to the older Token class, the Attribute- based approach separates specific term characteristics from others not necessarily related. Each TokenStream adds the attributes it is specifically targeting at construction time (see code listing below) and keeps a reference to it throughout its lifetime. This provides type-safe access to all attributes relevant for a particular TokenStream instance.

protected CharReplacementTokenStream(TokenStream in put) {

super(input);

termAtt = (TermAttribute) addAttribute(TermAttribut e.class);

}

Adding a TermAttribute at construction time

Inside TokenStream.incrementToken() , a token stream only operates on attributes that have been declared in the constructor. For instance, if you have Lucene replacing a character like a German umlaut in a term, only the TermAttribute (declared at construction time in the code listing above) is used. (Other attributes like PositionIncrementAttribute or PayloadAttribute are ignored by this TokenStream as they might not be needed in this particular use-case.)


public boolean incrementToken() throws IOException {

if (input.incrementToken()) {

final char[] termBuffer = termAtt.termBuffer();

final int termLength = termAtt.termLength();

if (replaceChar(termBuffer, termLength)) {

termAtt.setTermBuffer(output, 0, outputPos);

}

return true;

}

return false;

}

Replacing characters using the new attribute-based API.

What the above example does not demonstrate is the full power of the new token API. There, we replaced one or more characters in the token and discarded the original one. Yet, in many use-cases, the original token should be preserved in addition to the modified one. Using the old API required a fair bit of work and logic to handle such a common use-case.

In contrast, the new attribute-based approach allows capture and restoration of the state of attributes, which makes such use-cases almost trivial. The example below shows a version of the previous example improved for Lucene 2.9/3.0, in which the original term attribute is restored once the stream is advanced.


public boolean incrementToken() throws IOException {

if (state != null) {

restoreState(state);

state = null;

return true;

}

if (input.incrementToken()) {

final char[] termBuffer = termAtt.termBuffer();

final int termLength = termAtt.termLength();

if (replaceChar(termBuffer, termLength)) {

state = captureState();

termAtt.setTermBuffer(output, 0, outputPos);

}

return true;

}

return false;

}

Replacing characters and additionally emitting the original term text using the new attribute-based API (position

increments are omitted).

The separation of attributes makes it possible to add arbitrary properties to the analysis chain without using a customized Token class. Attributes are then made type-safely accessible by all subsequent TokenStream instances, and can eventually be used by the consumer. This way, you get a generic way to add various kind of custom information, such as part-of-speech tags, payloads, or average document length to the token stream. Unfortunately, Lucene 2.9 and 3.0 don't yet provide functionality to persist a custom Attribute implementation to the underlying index. This improvement, part of what is often referred to as "flexible indexing," is under active development and is proposed for one of the upcoming Lucene releases.

Beyond the generalizability of this API, one of its most significant improvements is its effective reuse of Attribute instances across multiple iterations of analysis. Attribute implementations are created during TokenStream instantiation and are reused each time the stream advances to a successive increment. Even if a stream is used for another


analysis, the same AttributeThis greatly reduces the rate of object creation, streamlining execution and minimizing any required garbage collection.

While Lucene 2.9 provides full backTokenStream implementations, it is strongly recommended to update any existing custom TokenStreamexclusively use incrementTokenheavy next() methods. Lucene 3.0 removed this compatibility layer and enforces the new attribute based API.

If you are trying to update your custom (TokenFilter and Tokenizer)abstract BaseTokenStreamTestCasetesting against the new and old API. The testsource distribution of Apache Lucene 2.9

Per-Segment Search Since the early days of Apache Lucene, documents have been stored at the lowest level in a segment—a small but entirely independent index. On the highest abstraction level, Lucene combines segments into one large indexAs more and more documents are added to an index, Lucene buffers RAM and flushes them to disk periodically. Depending on a variety of factors, Lucene either incrementally adds documents reduce the negative impact of Lucene tries to combine/merge multiple segments into larger ones. For optimal search performance, Lucene can optimizeinto a single segment.

Prior to Lucene 2.9, search logic resided at the highest abstraction level, accessing a single IndexReader no matter how many segments the index was composed of. SimilarFieldCache was associated with the topinvalidated each time an index was reopened. With Lucene 2.9FieldCache have moved to a perinternal complexity, the benefit of the tradeoff is yields a rich variety of performance improvements for unoptimized indexes.


Attribute instances may be used, provided the stream is reusable. This greatly reduces the rate of object creation, streamlining execution and minimizing any

n.

While Lucene 2.9 provides full back-compatibility for old-style implementations, it is strongly recommended to

TokenStream implementations to incrementToken instead of one of the overhead-methods. Lucene 3.0 removed this compatibility layer

and enforces the new attribute based API.

If you are trying to update your custom TokenStream or one of its subclass Tokenizer) implementations, it is recommended that you use the

BaseTokenStreamTestCase class, which provides various utility functions for testing against the new and old API. The test case is freely available for download in the

he Lucene 2.9/3.0.

Since the early days of Apache Lucene, documents have been stored at the lowest level in a a small but entirely independent index. On the highest abstraction level, Lucene

combines segments into one large index and executes searches across all visible segments. As more and more documents are added to an index, Lucene buffers your RAM and flushes them to disk periodically. Depending on a variety of factors, Lucene either incrementally adds documents to an existing segment, or creates entirely new segments. To reduce the negative impact of an increasing number of segments on search performanceLucene tries to combine/merge multiple segments into larger ones. For optimal search

optimize an index that essentially merges all existing segments

Prior to Lucene 2.9, search logic resided at the highest abstraction level, accessing a single no matter how many segments the index was composed of. Similar

was associated with the top-level IndexReader, and then had to be invalidated each time an index was reopened. With Lucene 2.9, the search logic and the

moved to a per-segment level. While this has introduced a little morethe benefit of the tradeoff is a new per-segment index behavior

yields a rich variety of performance improvements for unoptimized indexes.

Page 12

instances may be used, provided the stream is reusable. This greatly reduces the rate of object creation, streamlining execution and minimizing any

or one of its subclass implementations, it is recommended that you use the

which provides various utility functions for case is freely available for download in the

Since the early days of Apache Lucene, documents have been stored at the lowest level in a a small but entirely independent index. On the highest abstraction level, Lucene

and executes searches across all visible segments. your documents in

RAM and flushes them to disk periodically. Depending on a variety of factors, Lucene either to an existing segment, or creates entirely new segments. To

increasing number of segments on search performance, Lucene tries to combine/merge multiple segments into larger ones. For optimal search

essentially merges all existing segments

Prior to Lucene 2.9, search logic resided at the highest abstraction level, accessing a single no matter how many segments the index was composed of. Similarly the

and then had to be the search logic and the

his has introduced a little more segment index behavior that

yields a rich variety of performance improvements for unoptimized indexes.


In most applications, existing segments rarely change internally, and this property had not been effectively utilized in previous versions of Lucene. IndexReader.reopen() , first added in Lucene 2.4, now has the ability to add new or changed segments to an already existing top-level IndexReader instead of reloading all existing segments. The FieldCache also takes advantage of rarely changing segments. Cache instances of unchanged or updated segments can remain in memory or need only be rebuilt instead of invalidating the FieldCache entirely. Depending on the number of changed index segments, this can heavily reduce I/O as well as garbage collection costs, compared to reopening the entire index.

Previous versions of Lucene also suffered from long warming time for sorting and function queries. Those use-cases have been improved as the warm-up of reopened searchers is now much faster.

It's worth mentioning that Per-Segment Search doesn't yield improvements in all situations. If an IndexReader is opened on an optimized index, all pre-existing segments are merged into a single one, which then loads in its entirety. In other situations, perhaps more common, where some changes have been committed to the index and a new IndexReader instance is obtained by calling IndexReader.reopen() on a previously opened reader, the new per-segment capabilities can dramatically speed up reopening. But in this case, opening a new IndexReader using one of the overloaded static IndexReader.open() methods will create an entirely new reader instance and therefore can't take advantage of any per-segment capabilities.


IndexReader reader = indexWriter.getReader();

…

IndexReader newReader = indexWriter.reopen();

if (reader != newReader) {

reader.close();

reader = newReader;

}

Obtaining and reopening a Near-Realtime Reader from an

The majority of Lucene users won’t touchSegment Search during their working on low-level code implementing

Collector classes. Both classes directly expose the per

model like Collector#setNextReader()for each segment during search. The

immediately yield its relation to perlots of confusion in the past. Filter#getDocIdSet(IndexReader)relative Filter#bits(IndexReader)segment instead of once per index. The document IDs set by the Filter must be relative to the current segment rathe

Near Realtime Search (NRS)More and more, Lucene programmersrequirements with their search applications. Previous Lucene versions did a decent job with the incremental changes characteristic of this scenario, capturing those changes and making them available for searching. Lucene 2.9 adds significant new addressing the requirements of high

First of all, the IndexWriterand flushing documents to diskdirectly from the writer. The newly obtained reader then not only reflects the documents


IndexReader reader = indexWriter.getReader();

IndexReader newReader = indexWriter.reopen();

if (reader != newReader) {

reader = newReader;

Realtime Reader from an IndexWriter instance

The majority of Lucene users won’t touch the changes related to Per-ment Search during their day-to-day business unless there are

level code implementing Filters or Custom –

classes. Both classes directly expose the per – segment

Collector#setNextReader() , which is called once

for each segment during search. The Filter API instead doesn’t

immediately yield its relation to per-segment search and has caused lots of confusion in the past. Filter#getDocIdSet(IndexReader) and its deprecated

Filter#bits(IndexReader) are also called once per

segment instead of once per index. The document IDs set by the relative to the current segment rather than absolute.

Near Realtime Search (NRS) programmers are pursuing real-time or near-real

requirements with their search applications. Previous Lucene versions did a decent job with the incremental changes characteristic of this scenario, capturing those changes and making them available for searching. Lucene 2.9 adds significant new capabilities for addressing the requirements of high-change document environments.

IndexWriter —in general responsible for modifying the underlying index and flushing documents to disk—now offers a way to obtain an IndexReaderdirectly from the writer. The newly obtained reader then not only reflects the documents

Page 14

real-time requirements with their search applications. Previous Lucene versions did a decent job with the incremental changes characteristic of this scenario, capturing those changes and

capabilities for change document environments.

in general responsible for modifying the underlying index IndexReader instance

directly from the writer. The newly obtained reader then not only reflects the documents


already flushed to disk, but also makes all uncommitted documents still residing in memory almost instantly searchable.

The reader instance returned by reader as long as the writer releasing the reader has not been committed. Once it is committed, reopening the reader will result in an

It is important to understand why this feature is referred to as “real-time.” When IndexWriter.getReader()needs to consume a reasonable amount of additional resources (file descriptors) to make uncommitted documents searchuncommitted documents will not always be available instantaneously. Nonetheless, in most cases, the performance gained with this feature will be better than just reopening the index, or the traditional simpler approach of

To keep the latency as low as possible, the functionality, by which newly merged segments can be prepared for realare new to this feature, you should be aexperimental and might change in future releases.

MultiTermQuery-Related In Lucene 2.4, many standard queries, such as WildcardQuery, and PrefixQuerysubclassed under MultiTermQueryimprovements under the hood, resulting in much better performance for those queries.2

In Lucene 2.9/3.0, multiterm queries now use a constant score internally, based on the assumption that most programmersresulting from the term expansion that takes place during query rewriting.

2 This could be a back-compatibility issue if one of those classes has been subclassed.


already flushed to disk, but also makes all uncommitted documents still residing in memory almost instantly searchable.

The reader instance returned by IndexWriter.getReader() supports reopening the reader as long as the writer releasing the reader has not been committed. Once it is

reopening the reader will result in an AlreadyClosedExecption

It is important to understand why this feature is referred to as “near realIndexWriter.getReader() is called for the very first time, Lucene

needs to consume a reasonable amount of additional resources (i.e., RAM, CPUdescriptors) to make uncommitted documents searchable. Due to this additional work,

uncommitted documents will not always be available instantaneously. Nonetheless, in most cases, the performance gained with this feature will be better than just reopening the index, or the traditional simpler approach of opening a brand new reader instance.

To keep the latency as low as possible, the IndexWriter offers an optional “prewarmup” functionality, by which newly merged segments can be prepared for real

should be aware that the pre-warmup API is still marked experimental and might change in future releases.

elated Improvements many standard queries, such as FuzzyQuery ,

PrefixQuery were refactored and MultiTermQuery . Lucene 2.9 adds some

improvements under the hood, resulting in much better performance

, multiterm queries now use a constant score internally, based on the programmers don't care about the interim score of the queries

resulting from the term expansion that takes place during query rewriting.

compatibility issue if one of those classes has been subclassed.

Page 15

already flushed to disk, but also makes all uncommitted documents still residing in

supports reopening the reader as long as the writer releasing the reader has not been committed. Once it is

AlreadyClosedExecption .

“near real-time” rather than is called for the very first time, Lucene

RAM, CPU-cycles, and able. Due to this additional work,

uncommitted documents will not always be available instantaneously. Nonetheless, in most cases, the performance gained with this feature will be better than just reopening the

opening a brand new reader instance.

offers an optional “prewarmup” functionality, by which newly merged segments can be prepared for real-time search. If you

warmup API is still marked

, multiterm queries now use a constant score internally, based on the score of the queries

resulting from the term expansion that takes place during query rewriting.

compatibility issue if one of those classes has been subclassed.


Although constant-scoring is now the default behavior, the older scoring mode is still available for multiterm queries in 2.9/3.0. Beyond that, you can choose one of the following scoring modes:

� Filtered constant score: rewrites the multiterm query into a ConstantScoreQuery in combination with a filter to match all relevant documents.

� BooleanQuery constant score: rewrites the multiterm query into a ConstantScoreQuery based on a BooleanQuery by translating each term into an optional Boolean clause. This mode still has a limitation of maxClauseCount and might raise an exception if the query has too many Boolean clauses.

� Conventional scoring (not recommended): rewrites the multiterm query into an ordinary BooleanQuery .

� Automatic constant score (default): tries to choose the best constant score mode (Filter or BooleanQuery ) based on term and document counts from the query. If the number of terms and documents is small enough, BooleanQuery is chosen, otherwise the query rewrites to a filter-backed ConstantScoreQuery .

You can change the scoring mode by passing an implementation of RewriteMethod to MultiTermQuery.setRewriteMethod() as shown in the code example below.

PrefixQuery prefixQuery = new PrefixQuery(new Term( "aField", "luc"));

prefixQuery.setRewriteMethod(

MultiTermQuery.CONSTANT_SCORE_FILTER_REWRITE);

Explicitly setting a filtered constant-score RewriteMethod on a PrefixQuery

Payloads The Payloads feature, though originally added in a previous version of Lucene, remains pretty new to most programmers. A payload is essentially a byte array that is associated with a particular term in the index. Payloads can be associated with a single term during text analysis and subsequently committed directly to the index. On the search side, these byte arrays are accessible to influence the scoring for a particular term, or even to filter entire documents.


For instance, if your Lucene application is analyzing the phrase “Gangs of New York,” payloads can encode information about the terms “New” and “York” together, so that they are treated as a paired term for the name of a city, or can specify that “Gangs” is a noun rather than a verb. Prior to 2.9, payloads were exposed via a query called BoostingTermQuery, which has now been renamed to PayloadTermQuery . By using this query type, you can query Lucene to find all occurrences where “New” is a part of a city name like “New York” or “New Orleans”.

In comparison with previous versions, Lucene 2.9/3.0 also provides more control and flexibility for payload scoring. You can pass a custom PayloadFunction to the constructor of a payload-aware query. Each payload is fed back to the custom function, which calculates the score based on the cumulative outcomes of payload occurrences.

This improvement becomes even more useful when payloads are used in combination with span queries. Spans represent a range of term positions in a document, whereas in turn, payloads can help scoring based on the distance between terms. For instance, using a PayloadNearQuery , documents can be scored differently if terms are in the same sentence or paragraph if that information is encoded in the payload.

At a higher abstraction level, another payload aware TokenFilter has been added. DelimitedPayloadTokenFilter splits tokens separated by a predefined character delimiter, where the first part of the token is the token itself and the second part after the delimiter represents the payload. For example, it can parse an e-mail address, for example [email protected], by making “carol.smith” the token, and creating a payload to represent the domain name, “apache.org”. A customizable payload-encoder takes care of encoding the values while everything else magically happens inside the filter. Besides being a convenient way to add payloads to existing search functionality, this class also serves as a working example of how to use payloads during the analysis process.3

3 See www.lucidimagination.com/blog/2009/08/05/getting-started-with-payloads for more information.


Additions to Lucene Contrib So far, we’ve reviewed key new features and improvements introduced in the Apache Lucene core API. This section outlines the major additions and improvements to Lucene Contrib packages. Contrib packages are parts of Lucene that do not necessarily belong to the API core but are often helpful in building Lucene applications.

New Contrib Analyzers The Analysis package in Lucene Contrib has always been a valuable source for almost every Lucene programmer. The latest release brings several noteworthy improvements especially in terms of language support.

� Better support for Chinese: Chinese, like many Asian languages, does not use white spaces to delimit one word from another, nor is punctuation used at all. Smart-CN

provides an analyzer with improved tokenization and capabilities in splitting individual characters. While Smart-CN is part of the analyzers contrib module, it is distributed in its own JAR file because of the large (6MB) file resources it depends on.

� “Light10”-based Arabic analysis: a new Analyzer based on a high-performance stemming algorithm (Light10) applying lightweight prefix and suffix removal to Arabic text.

� Persian analyzer: applying character normalization and Persian stopword removal to Persian-only or mixed language text.

� Reverse String filter, as in leading wildcards: to support a search feature like leading wildcards efficiently, one of the common tricks/approaches is to index terms in reverse order. A leading wildcard effectively becomes a trailing wildcard if searched against a field with reversed tokens.

Lucene Spatial (formerly known as LocalLucene) Geospatial search has become a very common use-case, especially with the advent of mobile devices. Almost every new mobile platform supports a “nearby” search feature. End users seeking data on something near their current location (restaurants, movie theatres,


plumbers, etc.) expect both that results are limited to within a certain range, and that results can be ranked by distance from the end user’s location.

In early 2009, an open source project formerly known as LocalLucene was donated to Apache Lucene and integrated as a contrib package. Lucene Spatial extends Lucene capabilities with support for geographical and location-based search.

While Lucene Spatial doesn't have any distance scoring capabilities, it can effectively help to filter and sort based on geographical information like longitude and latitude values. Filtering is an especially common use-case, when combined with a full-text query. In searching for “French restaurant” within 5 miles from a specific location, the filter restricts down the search space to documents with location fields within 5 miles; the rest of the search operation is implemented in core Lucene.

Lucene Spatial has a couple of different ways to encode geographic information:

� GeoHash: a hierarchical spatial data structure that subdivides space into buckets in a grid shape. GeoHash takes the even bits from the longitude value while the odd bits are taken from the latitude value. The result is an arbitrary precision base 32-encoded string that offers the property of gradually removing characters from the end of the string to reduce the size and precision of the code. Nearby places are likely to have similar prefixes due to this property.

� Cartesian Tiers: projects the world on a flat surface. Overlays to this projection are created as grids (Cartesian Tiers), with each tier having an increasing number (always by a power of two) of grid boxes on it dividing up the projection. Location data can be placed within one of the grid boxes with different precision depending on the number of grid boxes on the tier.

Both of the above allow efficient storage of geo-information in a Lucene index. In contrast to plain latitude and longitude values indexed in separate fields, GeoHash and Cartesian

Tiers encode in a single field.

Note that despite its previous releases under a different name (LocalLucene), the Lucene Spatial API still isn't considered stable and might change in future releases.


Lucene Remote and Java RMI The historic dependency on Java RMI from the Lucene core has now been removed: Lucene Remote is now partitioned into an optional contrib package. While the package itself doesn't add any functionality to Lucene it introduces a critical back-compatibility issue likely to be relevant for many programmers. In prior versions, the core-interface Searchable extended java.rmi.Remote to enable searches on remote indexes. If you had taken advantage of this convenience, you will now have to add the new Lucene-remote JAR file to the classpath and change their code to use the new remote base interface RMIRemoteSearchable as shown below.

final RMIRemoteSearchable remoteObject = ...;

final String remoteObjectName = ...;

Naming.rebind (remoteObjectName, remoteObject);

Searchable searchable = (Searchable)Naming.lookup(r emoteObjectName);

Using RemoteSearchable with Lucene 2.9

New Flexible QueryParser Lucene’s built-in query parser has been a burden on developers trying to extend the default query syntax. While changing certain parts of it, such as query instantiation, could be readily achieved by subclassing the parser, changing the actual syntax required deep knowledge of the JavaCC parser-generator.

The new contrib package QueryParser provides a complete query parser framework,

which is fully compliant with the core parser but enables flexible customization by using a modular architecture.

The basic idea of the new query parser is to separate the syntax from the semantics of a query, internally represented as a tree. Ultimately the parser splits up in three stages:

1. Parsing stage: transforms the query text (syntax) into a QueryNode tree. This stage is exploited by a single interface (SyntaxParser ) mandatory for custom implementation of this stage.


2. Query-Node processing stage: once the QueryNode tree is created, a chain of processors start working on the tree. While walking down the tree, a processor can apply query optimizations, child reordering, or term tokenization even before the query is actually executed.

3. Building stage: the final stage builds the actual Lucene Query object by mapping QueryNode types to associated builders. Each builder subsequently applies the actual conversion into a Lucene query.

The snippet below, taken from the new standard QueryParser implementation, shows how the stages are exposed at the API's top level.

QueryNode queryTree = this.syntaxParser.parse(query, getField());

queryTree = this.processorPipeline.process(queryTree);

return (Query) this.builder.build(queryTree);

To provide a smooth transition from the existing core parser to the new API, this contrib package also contains an implementation fully compliant with the standard query syntax. This not only helps the switch to the new query parser but it also serves as an example of how to use and extend the API. That said, the standard implementation is based on the new query parser API and therefore it can't simply replace a core parser as is. If you have been replacing Lucene's current query parser, you can use QueryParserWrapper instead, which preserves the old query parser interface but calls the new parser framework. One final caveat: the QueryParserWrapper is marked as deprecated, as the new query parser will be moved to the core in the upcoming release and eventually replace the old API.

Minor Changes and Improvements in Lucene 2.9 Beside the improvements and entirely new features, Lucene 2.9 contains several minor improvements worth mentioning. The following points are a partial outline of minor changes.


Term vector-based highlighter

vectors (essentially a view of terms, offsetsfeatures like n-Gram fields andperformance on large documents. The downside is that it requires a lot more disk sdue to stored term vectors.

� Collector replaces

HitCollector was deprecated and replaced with a new

Collector class.

to collect hits across sequential

The most significant improvement here is that score calculation is now decoupled from collecting hits or skipped entirely if not needed

� Improved String “interning”

cache instead of Java's default implementation yields

� New n-gram distance

contrib spellcheck package

� Weight is now an abstract

refactored to an abstract class including minor method signature changes.

� ExtendedFieldCache marked deprecated:

interface ExtendedFieldCacheExtendedFieldCachedeclarations for binary backwards compatibility.

� MergePolicy interface changed:

requires an IndexWriterinstantiation. As a result, a method argument from all

For a complete list of improvemchanges you should consult the (lucene.apache.org/java/2_9_0/changes/Changes.html


based highlighter: a new term highlighter implementation based on term s (essentially a view of terms, offsets, and positions in a documents field). It supports

and phrase-unit highlighting with slops and yields good performance on large documents. The downside is that it requires a lot more disk s

replaces HitCollector : the low-level

was deprecated and replaced with a new

class. Collector offers a more efficient API

to collect hits across sequential IndexReader instances.

nificant improvement here is that score calculation is now decoupled from collecting hits or skipped entirely if not needed—a nice new efficiency.

Improved String “interning”: Lucene 2.9 internally uses a custom cache instead of Java's default String.intern() . The lockless implementation yields minor internal performance improvements.

istance: a new n-gram-based distance measure was added to the spellcheck package.

Weight is now an abstract class: the Weight interface was

refactored to an abstract class including minor method signature changes.

ExtendedFieldCache marked deprecated: All methods and parsers from the ExtendedFieldCache have been moved into FieldCache

ExtendedFieldCache is now deprecated and contains only a few declarations for binary backwards compatibility.

MergePolicy interface changed: MergePolicy now IndexWriter instance to be passed upon

instantiation. As a result, IndexWriter was removed as a method argument from all MergePolicy methods.

For a complete list of improvements, bug-fixes, compatibility, and runtime behavishould consult the CHANGES.txt file included in the Lucene distribution

lucene.apache.org/java/2_9_0/changes/Changes.html ).

Page 22

: a new term highlighter implementation based on term and positions in a documents field). It supports

unit highlighting with slops and yields good performance on large documents. The downside is that it requires a lot more disk space

: Lucene 2.9 internally uses a custom String intern . The lockless

internal performance improvements.

based distance measure was added to the

All methods and parsers from the FieldCache .

is now deprecated and contains only a few

and runtime behavior included in the Lucene distribution


Changes and Improvements in Lucene 3.0Lucene 3.0 provides a clean transitionfeatures have been introduceregarding backwards compatibility, API Changes

� Removed Compressed Fields:

deprecated in Lucene 2.9 have been removed without a direct replacement. While Lucene 3.0 is still able to read indexes with compressed fields, index merges or optimizations will decompress and store such fields transparently. Given this behaviorcompressed fields might suddenly become larger during a segment merge or optimization.

� Removed deprecated

methods and classes have been removed in 3.0. A full list can be found at lucene.apache.org/java/3_0_0/changes/Changes.html#3.0.0.api_changes

� Generics & Java 5 Features:

as an underlying execution environment. classes with their impr

StringBuildernow make heavy use of Generic Types and Variable Arguments. Underneath the hood, this move flattened the way to introduce improvements using Java Concurrent Utilities.

� Scorer Deprecation

Lucene’s lowest level.

DocIdSetIterationwhile equivalents for old APIs are provided. Custom Query , Scorer or

tations must be ported to the new API in order to be compatible with Lucene 3.0.


Changes and Improvements in Lucene 3.0 Lucene 3.0 provides a clean transition to a new major version of the library. Since no new features have been introduced, this section will give a short overview of important changes regarding backwards compatibility, API Changes, and removed features.

Removed Compressed Fields: Compressed fields already deprecated in Lucene 2.9 have been removed without a direct replacement. While Lucene 3.0 is still able to read indexes with compressed fields, index merges or optimizations will decompress and store such fields

ven this behavior, indexes built with compressed fields might suddenly become larger during a segment merge or optimization.

Removed deprecated Classes and Methods: Deprecated methods and classes have been removed in 3.0. A full list

lucene.apache.org/java /3_0_0/changes/Changes.html#3.0.0.api_changes.

Generics & Java 5 Features: Lucene 3.0 became the first release requiring Java 5 as an underlying execution environment. In addition to various replacements of classes with their improved equivalents like StringBufferStringBuilder , many new language features were introduced. Public APIs

heavy use of Generic Types and Variable Arguments. Underneath the this move flattened the way to introduce improvements using Java

Concurrent Utilities.

Scorer Deprecations: 3.0 refactors several methods on Lucene’s lowest level. Scorer and its abstract super-class

DocIdSetIteration have incompatible API changes

while equivalents for old APIs are provided. Custom or DocIdSetIterator implemen-

tations must be ported to the new API in order to be compatible with Lucene 3.0.

Page 23

a new major version of the library. Since no new this section will give a short overview of important changes

and removed features.

Lucene 3.0 became the first release requiring Java 5 various replacements of

StringBuffer and

many new language features were introduced. Public APIs

heavy use of Generic Types and Variable Arguments. Underneath the this move flattened the way to introduce improvements using Java 5


� Made core TokenStreams final:

Decorator-based analysis model several core TokenStream implementations have been declared final

without any replacement and can therefore not be subclassed anymore. Users subclassing streams like KeywordTokenizerrequired to rebuild the functionality.

To gain a comprehensive under2.9, programmers should consult Lucene issue tracker.

Lucene Version by Version compatibility since 2.9In Lucene 2.9, a Version constant was first introduced to help Version backwards compatibility. The initial purpose of

Lucene contributors to eventually fix longwithout breaking their own backwards compatibility policy.

Lucene’s StandardAnalyzerVersion to change its runtime beha

number. Its constructor requires an instance of Version its internal behavior accordingly:

� As of 2.4, Tokens incorrectly identified as acronyms were corrected

� As of 2.9, StopFilter

You might ask why this old, and in these cases incorrectwhy it is the user’s responsibility to decide which is correct. Yet the answer isn’t as obvious as expected. Since Lucene preserves backwards compatibility for inprevious versions of the libraryindices have been build and how they are queried.

Changes like runtime behavior of

easily break backwards compatibility, trigger unexpected behaviorexperience if queries return different documents than before.


Made core TokenStreams final: To enforce Lucene’s based analysis model several core

implementations have been declared final

without any replacement and can therefore not be subclassed anymore. Users subclassing streams like KeywordTokenizer or StandardTokenizer are

required to rebuild the functionality.

mprehensive understanding of what has changed in 3.0 in contrast to Lucene programmers should consult the CHANGES.txt file and the corresponding issues on the

y Version compatibility since 2.9 a Version constant was first introduced to help in preserving a Version by

Version backwards compatibility. The initial purpose of Version was to enable the

Lucene contributors to eventually fix long-time known bugs and limitation in Lucene without breaking their own backwards compatibility policy.

StandardAnalyzer was the first class making use of

to change its runtime behavior based on the given version

number. Its constructor requires an instance of Version that changed its internal behavior accordingly:

As of 2.4, Tokens incorrectly identified as acronyms were

StopFilter preserves position increment

and in these cases incorrect, behavior is preserved at alls responsibility to decide which is correct. Yet the answer isn’t as obvious

as expected. Since Lucene preserves backwards compatibility for indices created with of the library, it also has to preserve compatibility with

indices have been build and how they are queried.

Changes like runtime behavior of Analyzers , TokenFilters, and

easily break backwards compatibility, trigger unexpected behavior, and can cause bad user experience if queries return different documents than before.

Page 24

what has changed in 3.0 in contrast to Lucene file and the corresponding issues on the

preserving a Version by was to enable the

time known bugs and limitation in Lucene

behavior is preserved at all, and s responsibility to decide which is correct. Yet the answer isn’t as obvious

dices created with with how those

and Tokenizers can

and can cause bad user


The Version constant has been introduced to make the upgrade process easier for users

who cannot afford to rebuild their index or need to support “old” indices in production environments. In such cases, it is recommended that you pass the Version constant you are upgrading from to all constructors expecting a Version constant. Once the latest behavior is desired, the version you are upgraded to should be used in favor of Version#LUCENE_CURRENT, which has been deprecated in Lucene trunk due to the

dangers it could introduce in a subsequent upgrade.

Strategies for Upgrading to Lucene 2.9/3.0 In the main, a Lucene-based application will benefit from the improvements in 2.9, even as its new features, such as numeric capabilities and the new TokenStream API, do require code modifications and may require reindexing in order to take full advantage. That said, compared to previous version changes, an upgrade to version 2.9 requires a more sophisticated upgrade procedure.

True, there are many cases in which an upgrade won't require code changes, as changes limited to “expert” APIs won't affect applications only using high-level functionality. All the same, even if an application complies with Lucene 2.9, it is likely that some of the changes in runtime characteristics can introduce unexpected behaviors. In the sections below, we’ll offer some brief suggestions for making the transition.

An upgrade to version 3.0 requires a more sophisticated upgrade procedure. First, upgrade to 2.9; then remove the deprecation warnings. Only then should you upgrade to 3.0x.

Should you move to 2.9 or 3.0? Whichever you do, first bear in mind that going to 3.0 will require a migration first to 2.9; it is a prerequisite. Only once that 2.9 transition is completed, will you be ready to work through the deprecation warnings in order to move. Because 3.0 is a deprecation release, all deprecated-marked code in Lucene 2.9 will be removed. Some parts of the API might be modified in order to make use of Java Generics,


but in general the upgrade from 2.9 to 3.0 should be as seamless as earlier upgrades have been. Once you have replaced the usage of any deprecated API(s) in your code, you should then be able to upgrade the next time simply by replacing the Lucene JAR file.

Upgrade to 2.9—Recommended Actions At a minimum, if you plan an upgrade of your search application to Lucene 2.9, you should recompile your application against the new version before the application is rolled out in a production environment. The most critical issues will immediately raise a compile-time error once the JAR is in the classpath.

For those of you using Lucene from a single location, for example, in the JRE's ext directory, you should make sure that 2.9 is the only Lucene version accessible. In cases where an application relies on extending Lucene in any particular way and the upgrade doesn't raise a compile-time error, it is recommended that you add a test-case for the extension based on the behavior executed against the older version of Lucene.

It is also extremely important that you backup and archive your index before opening it with Lucene 2.9, as it will make changes to the index that may not be readable by previous versions.

Again, we strongly recommend a careful reading of the CHANGES.txt file included in every Lucene distribution, especially the sections on back-compatibility policy and on changes in runtime behavior. Careful study followed by proper planning and testing should prevent you from running into any surprises once the new Lucene 2.9-based application goes into production.

Upgrade to 2.9—Optional Actions Lucene 2.9 includes many new features that are not required for use of the new release. Nevertheless, 2.9 has numerous parts of the API marked as deprecated, since they are to be removed in the next release. To prepare for the next release and further improvements in this direction, it is strongly recommended that you replace any deprecated API during the upgrade process.

Applications using any kind of numeric searches can improve their performance heavily by replacing a custom solution with Lucene's Numeric Capabilities described earlier in this white paper.


Last but not least, the new TokenStream API will replace the older API entirely in the next release. Custom TokenStream , TokenFilter, and Tokenizer implementations

should be updated to the attribute-based API. Here, the source distribution contains basic test cases that can help you safely upgrade.

Finally, to reiterate, you would do best to write new added test cases against their current Lucene version, and upgrade the test and your code once you have gained enough confidence in the stability of the upgrade.

Migrating from Lucene to Solr? Many Lucene implementations, of a variety of vintages, date back to a time where Solr lacked core capabilities that were available only by building from scratch all underlying services needed by the Lucene search libraries. Happily, Solr today offers a very complete (with some small, but meaningful exceptions) implementation of Lucene functionality. While a complete, robust approach to migrating from Lucene to Solr is beyond the scope of this paper, here are some thoughts on the advantages of doing so. A slightly longer comparison of Solr and Lucene is available in the Appendix to this document.

� Using Standards by Default

As a server application running in Jetty, Tomcat, or any other Servlet Container, Solr easily installs, integrates and runs in existing production environments. Solr’s RESTful APIs and its XML-driven implementation simplify configuration, operation, and search application development. With a rich array of client libraries, from standard SolrJ for Java through JSON, Python, and many others, the base of programming skills needed for the search application is much narrower.

� Makes Lucene Best-Practices ready to use

From caching Filters, Queries, or Documents via Spell-Checking to warming Searchers in the background, Solr offers an enormous set of search features that would require lots of Lucene experience and expertise to implement without it. Solr lets you immediately benefit from these low-level developments, simplifying creation and development of your search environment.

� Reuse some of your Lucene Libraries and indexes

Because Lucene is at the core of Solr, your implementation of Lucene can reuse many of the same libraries; just plug in details for your handlers or analyzers into


Solr’s config.xml and start testing. Likely as not, you’ll find you can move much of your code as is from Lucene or even use your existing indexes with Solr directly.

� Lower maintenance and revision costs

As this transition from Lucene 2.4.1 to Lucene 2.9/3.0 demonstrates, many of the low-level, high-control advantages of implementing directly with Lucene are negated once anything changes in your environment. As a server, Solr insulates you from much of that, and helps remove the temptation to hard code optimistically or to skip abstractions.

� Cloud-readiness

As large data sets grow in scope and distribution, search services will necessary rely on a much higher level of abstraction, above not only data and I/O, but for more elastic distribution of search resources and operations, i.e., shards and insert/updates. If your business is at a place where hardware might scale via a transition into some kind of cloud environment, you will benefit by taking advantage of the forthcoming Solr cloud-enabling capabilities, including distributed node management, relevancy calculations across multiple collections, etc.

It’s important to note that there are many good reasons not to migrate to Solr from Lucene, whether they have to do with the cost of a new abstraction model in your general application implementation, or with no real need for serving search over HTTP. With the merger of the Lucene and Solr development projects, you won’t be shortchanged on any of the underlying functionality. But at the same time, the stronger functional affinity between the two means you’ll have to give careful thought to your long-term deployment goals in order to pick the right one.


References http://lucene.apache.org/java/2_9_0/index.html

http://lucene.apache.org/java/2_9_0/changes/Changes.html

http://lucene.apache.org/java/2_9_0/changes/Contrib-Changes.html

http://www.lucidimagination.com/Community/Hear-from-the-Experts/Podcasts-and-Videos/Interview-Uwe-Schindler

http://wiki.apache.org/lucene-java/NearRealtimeSearch

http://wiki.apache.org/lucene-java/Payloads

http://www.lucidimagination.com/blog/2009/08/05/getting-started-with-payloads/

http://wiki.apache.org/lucene-java/ConceptsAndDefinitions

http://wiki.apache.org/lucene-java/FlexibleIndexing

http://wiki.apache.org/lucene-java/Java_1.5_Migration

http://www.lucidimagination.com/How-We-Can-Help/webinar-Lucene-29

http://www.nsshutdown.com/projects/lucene/whitepaper/locallucene_v2.html

http://www.lucidimagination.com/Community/Hear-from-the-Experts/Podcasts-and-Videos/Interview-Ryan-McKinley

http://ocw.kfupm.edu.sa/user062/ICS48201/NLLight%20Stemming%20for%20Arabic%20Information%20Retrieval.pdf

https://javacc.dev.java.net/


Next Steps For more information on how Lucid Imagination can help search application developers, employees, customers, and partners find the information they need, please visit www.lucidimagination.com to access blog posts, articles, and reviews of dozens of successful implementations.

Certified Distributions from Lucid Imagination are complete, supported bundles of software that include additional bug fixes, performance enhancements, along with our free 30-day Get Started program. Coupled with one of our support subscriptions, a Certified Distribution can provide a complete environment to develop, deploy, and maintain commercial-grade search applications. Certified Distributions are available at www.lucidimagination.com/Downloads.

Please e-mail specific questions to:

Support and Service: [email protected]

Sales and Commercial: [email protected]

Consulting: [email protected]

Or call: 1.650.353.4057


APPENDIX: Choosing Lucene or Solr The great improvements in the capabilities of Lucene and Solr open source search technology have created rapidly growing interest in using them as alternatives for their search applications. As is often the case with open source technology, online community documentation provides rich details on features and variations, but does little to provide explicit direction on which technologies would be the best choice. So when is Lucene preferable to Solr and vice versa?

There is in fact no single answer, as Lucene and Solr bring very similar underlying technology to bear on somewhat distinct problems. Solr is versatile and powerful, a full-featured, production-ready search application server requiring little formal software programming. Lucene presents a collection of directly callable Java libraries, with fine-grained control of machine functions and independence from higher-level protocols.

In choosing which might be best for your search solution, the key questions to consider are application scope, deployment environment, and software development preferences.

If you are new to developing search applications, you should start with Solr. Solr provides scalable search power out of the box, whereas Lucene requires solid information retrieval experience and some meaningful heavy lifting in Java to take advantage of its capabilities. In many instances, Solr doesn’t even require any real programming.

Solr is essentially the “serverization” of Lucene, and many of its abstract functions are highly similar, if not the just the same. If you are building an app for the enterprise sector, for instance, you will find Solr almost a 100% match to your business requirements: it comes ready to run in a servlet container such as Tomcat or Jetty, and ready to scale in a production Java environment. Its RESTful interfaces and XML-based configuration files can greatly accelerate application development and maintenance. In fact, Lucene programmers have often reported that they find Solr to contain “the same features I was going to build myself as a framework for Lucene, but already very well implemented.” Once you start with Solr, and you find yourself using a lot of the features Solr provides out of the box, you will likely be better off using Solr’s well organized extension mechanisms instead of starting from scratch using Apache Lucene.


If, on the other hand, you do not wish to make any calls via HTTP, and wish to have all of your resources controlled exclusively by Java API calls that you write, Lucene may be a better choice. Lucene can work best when constructing and embedding a state-of-the-art search engine, by allowing programmers to assemble and compile inside a native Java application. Some programmers set aside the convenience of Solr in order to more directly control the large set of sophisticated features with low-level access, data, or state manipulation, and choose Lucene instead, for example, with byte-level manipulation of segments or intervention in data I/O. Investment at the low level enables development of extremely sophisticated, cutting edge text search and retrieval capabilities.

As for features, the latest version of Solr generally encapsulates the latest version of Lucene. As the two are in many ways functional siblings, spending time gaining a solid understanding how Lucene works internally can help you understand Apache Solr and its extension of Lucene's workings.

No matter which you choose, the power of open source search is yours to harness. More information on both Lucene and Solr can be found at www.lucidimagination.com.

What’s New in Apache Lucene 3.0

Documents

Transcript of What’s New in Apache Lucene 3.0