Solr on Windows Does it work, does it scale?

Funda Real Estate (www.funda.nl) teun@funda.nl, October 20th, 2011

Contents §  Who we are §  Why we chose for Windows, why we chose for Solr §  Our worst fears, maybe yours §  Share our experiences

•  What went spectacularly well •  What we would have loved to know before

production §  Our approach, migrating the data and the people

3

Who we are §  Teun Duynstee, software architect §  funda.nl is a household name in The

Netherlands (real estate search) §  We serve around 10M pageviews a day §  Only in NL (17M inh.) we have 3M unique

visitors eacht month

§  Strongly relies on faceted search §  Used to have homegrown solution, this year

migrated to Solr

4

5

Screenshots

6

7

8

9

10

Our architecture §  So why did we use Windows in the 1st place?

•  Great tool support •  Performance is pretty good •  Not that expensive •  Historic reasons

§  Technology stack •  Windows Server 2003 OS (upgrading) •  MS SQL Server 2005 •  ASP.NET MVC 2 •  + MSMQ + memcached + jquery + S3 + ...

11

Old implementation

12

House ID place date price 1 Amsterdam 2011-09-14 450000 2 … … …

SELECT TOP 15 * FROM Houses where place = 'Amsterdam' order by date desc

SELECT SUM(facet1), SUM(facet2) FROM Houses * FROM Houses where place = 'Amsterdam'

House ID place date price facet1 facet2 1 Amsterdam 2011-09-14 450000 1 0 2 … … …

Enters Solr §  Had been looking at commercial search

technology, but too expensive and too risky

§  Enthousiastic, good fit to our functionality §  But will it scale? §  We already had a scale out solution to search

§  Solr 1.2 relied on rsync for replication §  As soon as 1.3 came out, we were sold

13

Run anywhere? §  Not reassured by reactions of experts on Solr

on Windows

§  So what to test at scale/load? •  Index size is small, we’re talking scaling search

volume •  Real user behavior is very hard to emulate using

testing tools

14

How did we plan

15

-  CPU for reading

data

-  CPU for creating

HTML

-  Threading

-  Disk IO for

getting at data

-  Memory for

caching results

-  CPU mostly for

indexing

-  Only

memory

? ü ü

ü

? ? ?

How did we plan §  Experience with the resource requirements of

old solution

§  By placing Solr on all servers where we would stop searching in the database •  The same load should be easily manageable by

“superior technology” •  Network topology and server roles could remain

the same •  No new servers

16

How we migrated §  Testing:

•  Stress tested Solr with representative searches and 100 concurrent threads

•  Stress tested the parsing code: no client library, but lean custom XML parsing

§  Careful adoption •  We selected a part of the functionality that we

could move over to Solr with low risk •  Always made sure that we could rollback to

previous version. Database data kept intact.

17

Libraries §  SolrSharp/SolrNet §  Both looked not like a great fit

§  We rolled a very lightweight XML parsing client §  And a very light fluid URL building API

18

How we migrated §  Choice: no new servers or server roles §  Choice: not running on Linux §  Choice: Tomcat, running as Windows Service §  Choice: Topology master/slave, no sharding §  Choice: Introducing Solr first on non-mission

critical part §  Choice: not using available client libraries, but

roll our own

19

So what happened? §  Initially we only moved a small part of the

search functionality (“recently sold”) to Solr

§  Complete success •  Less code for comparable functionality •  Operations very satisfied with stability •  Fast and low resource use

20

So what happened? §  When we moved all of the core functionality

over to Solr, hell broke loose

§  Everything became very slow, CPU and memory use looked normal

21

So what happened?

22

-  Bandwidth for responses

-  Switch had some 100Mbit

ports

-  No compression enabled

-  Memcached over same ports,

for clients & partially servers

http compression §  The .NET HttpWebRequest class supports

(de)compression, but you must turn it on: webRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

23

Numbers, schema §  138 fields (127 indexed, 12 multivalued) §  Mostly using StrField, BoolField,

SortableIntField, SortableFloatField

§  No TrieFields (we need sortMissingLast) §  No full text, stemming, tokenizing (yet)

24

Numbers, queries §  Our typical query has:

•  All fields returned •  15 resulting docs •  13 field facets •  37 query facets •  Only filters (fq=), no text searches, almost no

scoring, boosting, etc.

§  Main core has: •  1.4M docs •  1.7 GB index (avg. 1300 bytes per doc)

25

Numbers, searching At peak times §  300 Solr searches per sec. (400 on peak days) §  Caches

•  Result cache hit ratio < 1% •  Document cache hit ratio 31% •  Field value cache hit ratio 99% •  Filter cache hit ratio 99%

26

Numbers, indexing §  Incremental updates every 10 min. §  With only 1.3M documents, we can afford

regular full indexes •  Nightly •  1.5 hr

§  Replication in master-slave mode appears very stable under load. •  Memory use during replication is high

27

Numbers, resources §  4 active Solr searchers, running Solr 1.3

•  dual quadcore CPU •  1.2 GB memory for Tomcat

28

Numbers, resources

29

Caching §  We have a memcached distributed cache for

results (cf. Solr result cache)

§  Here we have hit ratios of ca. 25% §  Actually, the time for serializing, deserializing,

network latency and memcached is slower than letting Solr handle everything

§  Network traffic of memcached protocol is lower

30

Wrap Up §  Solr works fine on Windows, we have had no

stability issues in Solr whatsoever §  Pick your client libraries carefully (no obvious

choice) or use raw HTTP/XML §  Think about bandwidth when planning capacity

31

Contact Teun Duynstee

•  teun@funda.nl •  www.funda.nl •  @teuntostring

Photo credits: http://www.sxc.hu/profile/lusi Microsoft Windows 7

32