Open Source Search FTW

Click here to load reader

  • date post

    27-Jan-2015
  • Category

    Technology

  • view

    107
  • download

    0

Embed Size (px)

description

http://sigir2013.ie/industry_track.html#GrantIngersoll Abstract: Apache Lucene and Solr are the most widely deployed search technology on the planet, powering sites like Twitter, Wikipedia, Zappos and countless applications across a large array of domains. They are also free, open source, extensible and extremely scalable. Lucene and Solr also contain a large number of features for solving common information retrieval problems ranging from pluggable posting list compression and scoring algorithms to faceting and spell checking. Increasingly, Lucene and Solr also are being (ab)used to power applications going way beyond the search box. In this talk, we'll explore the features and capabilities of Lucene and Solr 4.x, as well as look at how to (ab)use your search engine technology for fun and profit.

Transcript of Open Source Search FTW

  • 1. Copyright 2013 Open Source Search FTW! Grant Ingersoll Lucene/Solr Committer, Apache Soft. Found. CTO, LucidWorks @gsingers

2. 2013 LucidWorks 2 Preaching to the Converted! Embrace fuzziness! Search is a system building block If the algorithms fit, use them! Search use leads to search abuse Scoring features are everywhere http://cheezburger.com/5243950080 3. 2013 LucidWorks 3 Topics Quick Intro to Lucene and Solr Whats new in Lucene and Solr 4.x? - Lucene/Solr for Info Retrieval (Ab)Using Search Engine Tech. for Fun and Profit 4. 2013 LucidWorks 4 Quick Intro to Lucene and Solr 5. 2013 LucidWorks Relax, Youre Among Friends Large, diverse search community with many non-traditional search engine usages - Object stores, Record linkage, Social, mobile -> web Open Dev. > Open Source The Apache Way - Meritocracy Those who do, decide! Always Be Testing - Randomized system tests are all the rage - http://vimeo.com/32087114 Patches Welcome! 6. 2013 LucidWorks 7. Copyright 2013 8. 2013 LucidWorks Lucene: Speed and Memory Native Near Real Time (NRT) support - Per segment - FieldCache can be controlled to only load new segments - Soft commit -- faster without fsync, allows quicker update visibility DWPT (Document Writer per Thread) - Faster more consistent index speed Faster fuzzy & wildcard query processing String -> BytesRef - Much improved data structure - means less memory and less garbage collection effort 9. 2013 LucidWorks Up and to the Right http://people.apache.org/~mikemccand/lucenebench/in dexing.html 9 10. 2013 LucidWorks Lucene: Flexibility Flexible Index Formats - New posting list codecs: Block, Simple Text, Append (HDFS..), etc - Pulsing codec: improves performance of primary key searches, inlining docs, positions, and payloads, saves disk seeks Pluggable Scoring - Decoupled from TF/IDF - Built in alternatives include BM25 & DFR, and others http://en.wikipedia.org/wiki/Okapi_BM25 http://terrier.org/docs/v3.5/dfr_description.html - Add your own 11. 2013 LucidWorks FS(A|T) Keys: - byte[] write-once - Linear time build of min. automata (nlogn if not sorted) - Compression - Reverse lookups - Weights (used for auto-suggest) - Pluggable Algebra Uses: - Term Dictionary, TokenStreams, Japanese, synonyms, spelling, others - FuzzyQuery is 100x faster -- http://bit.ly/hgO65c More: - http://slidesha.re/vKtpVA - http://bit.ly/Pkjyu0 - Smaller Representation of Finite State Automata Proc. of the 16th Inter. Conf. on Implementation and Application of Automata, CIAA'2011, vol. 6807, 2011, pp. 118192. 12. Copyright 2013 13. 2013 LucidWorks Solr 4: New Features Search/Faceting/Relevance - New Relevance Function Queries (tf, df, others) - Pivot Faceting - Pseudo-join - Improved Spatial (more later) - Full support for Lucene Codecs, pluggable scoring Indexing - New Update Processors, including scripting option - Near real time Codec/Similarity support from Lucene 4 Other - New Admin UI 14. 2013 LucidWorks Geospatial improvements Index shapes other than points (circles, polygons, etc) More complex interactions than point in a circle Indexing: - "geo:43.17614,-90.57341 - geo:Circle(4.56,1.23 d=0.0710) - geo:POLYGON((-10 30, -40 40, -10 -20, 40 20, 0 0, -10 30)) Searching: - fq=geo:"Intersects(-74.093 41.042 -69.347 44.558)" - fq=geo:"Intersects(POLYGON((-10 30, -40 40, -10 -20, 40 20, 0 0, -10 30))) 15. 2013 LucidWorks Scaling Solr Distributed/sharded indexing & search - Auto distributes updates and queries to appropriate shards - Near Real Time (NRT) indexing capable Dynamically scalable - New SolrCloud instances add indexing and query capacity - Supports re-balancing Reliable - No single point of failure - Transactions logged - Robust, automatic recover http://wiki.apache.org/solr/SolrCloud 16. 2013 LucidWorks 16 New in 4.4 (just released) HDFS backed directory for storing index and transaction logs in Apache Hadoop New Core discovery capabilities Schemaless/External Schema/Field Guessing Schema APIs Add documents from the Admin UI 17. Copyright 2013 Hacking Search Engines for Fun and Profit 17 18. 2013 LucidWorks Find your Keys, Store Your Content Lucene/Solr is a fast key-value store - Bonus: search your values! NoSQL before NoSQL was cool Solr: distributed key/value - Durable, Isolated, Redundant, Fast, Real-time - Joins, Column Storage Solr or Tika + Lucene can index popular office formats Solr can backup/replicate and scale as content grows 19. 2013 LucidWorks Find Love! Upsell! Cross-sell! Cross recommendation as search - with search used to build cross recommendation! Recommend content to people who exhibit certain behaviors (clicks, query terms, other) (Ab)use of a search engine - but not as a search engine for content - more like a search engine for behavior See Ted Dunnings talk from Berlin Buzzwords on Multi- modal Recommendation Algorithms - http://berlinbuzzwords.com/sessions/multi-modal-recommendation- algorithms Go get Mahout/Myrrix or just do it in y(our) search engine 20. 2013 LucidWorks 20 Avoid Delays 21. 2013 LucidWorks 21 Time travel? Leverage Solrs new spatial capabilities to index non-spatial data, such as time ranges - Useful for Open Hours, Shifts, etc. Query using rectangle intersections - q = shift:"Intersects(0 19 23 365) https://people.apache.org/~hossman/spatial-for-non-spatial-meetup-20130117/ 22. 2013 LucidWorks 22 Boldly go forth and rank! Faster More Flexible Easier than ever scaling More reliable than ever Reduced cost of experimentation 23. 2013 LucidWorks Lucene/Solr EU Conference: - Dublin, IE, November 4-7: http://lucenerevolution.org/ - CFP Open Now Where to Next? Lucene/Solr - http://lucene.apache.org - {java-user|solr-user}@lucene.apache.org - SIGIR 12 Open Source Workshop http://opensearchlab.otago.ac.nz/paper_ 10.pdf LucidWorks - http://www.lucidworks.com - Commercial support, products, etc. for Lucene/Solr Me - [email protected] - @gsingers on Twitter - Taming Text Engineers guide to open source search and NLP http:///www.manning.com/ingersoll 23 24. 2013 LucidWorks 24 Credits All of the Lucene/Solr committers and contributors Polar bear: http://gaijinexplorer.blogspot.ie/2012/12/its-all-just-relaxing.html Volunteers: http://www.poconohealthsystem.org/?id=228&sid=1 Not Hiring: http://naijaguardianjobs.com/wp-content/uploads/2013/03/Not-Hiring- The-American.jpg Keys: http://www.flickr.com/photos/crazyneighborlady/355232758/ Love: http://www.msruntheus.com/above-all-love-each-other-deeply/ TARDIS: http://2.bp.blogspot.com/- ysN8JskY4WM/UEZNhBywQKI/AAAAAAAABdg/gXE0A9OO6Mk/s1600/13881_do ctor_who.jpg