Practical Search in the Cloud - By Marc Krellenstein
-
Upload
lucenerevolution -
Category
Technology
-
view
521 -
download
3
description
Transcript of Practical Search in the Cloud - By Marc Krellenstein
![Page 1: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/1.jpg)
Practical Searchin the Cloud
(sneak preview)
Martijn Koster, Lucid [email protected]
Wednesday, 25 May 2011
![Page 2: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/2.jpg)
My Background Martijn Koster
Sr. Software Engineer at Lucid Imagination• Search in early Web – WebCrawler• Customer web services – Excite Mail• Proxies for Mobile – Danger
2
Wednesday, 25 May 2011
![Page 3: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/3.jpg)
What I Will Cover A Solr based Cloud Service The need How Demo Wrap up
3
Wednesday, 25 May 2011
![Page 4: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/4.jpg)
Why? – Getting Started is Hard Many new concepts
Complex software configuration
System Administration overhead
4
Wednesday, 25 May 2011
![Page 5: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/5.jpg)
Search as a Service A few clicks to get started
Full Power of LucidWorks and Solr
Offload operations
Expertise, tools and processes are transferable to your own deployments.
5
Wednesday, 25 May 2011
![Page 6: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/6.jpg)
Usage Model
6
Web UI
Direct Access
LWE JVMLWE JVM
Users
UserCode
Lucidworks
Solr
Wednesday, 25 May 2011
![Page 7: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/7.jpg)
Architecture
7
Web UI(Rails)
Direct Access(Proxy with URL
rewriting)
Control(Chef)
LWE JVMLWE JVM
LWE JVM
Provisioner (Ruby)
Monitoring (Zabbix)
3rd party billingUsers
UserCode
Provisioning Service (Rails)
Heroku
Heroku
SQL(RDS)
Wednesday, 25 May 2011
![Page 8: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/8.jpg)
Deployment Automation AMI
Chef
Custom
8
Wednesday, 25 May 2011
![Page 9: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/9.jpg)
Scaling Heroku
Direct Access
Search Servers
Provisioning/DB
Monitoring and Control
9
Wednesday, 25 May 2011
![Page 10: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/10.jpg)
Direct Access Example The Cloudy Direct Access Solr interface
• Simple submission• Simple search
First, copy SOLR access URL:
SOLR="https://access.cloudy.io/…/solr"
11
Wednesday, 25 May 2011
![Page 11: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/11.jpg)
Example - Add$ cat > /tmp/doc.xml <<EOM<add><doc><field name="id">testdoc_LR2011</field><field name="text_en">LR rocks!</field></doc></add>EOMcurl $SOLR/my_news/update?commit=true \-H "Content-Type: text/xml" \--data-binary @/tmp/doc.xml
12
Wednesday, 25 May 2011
![Page 12: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/12.jpg)
Example - Searchcurl $SOLR'/my_news/lucid?
q=news&role=DEFAULT&wt=json'
curl $SOLR'/my_news/select?wt=json&q=news&start=0&rows=10'
13
Wednesday, 25 May 2011
![Page 13: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/13.jpg)
Example – Lucidworks API• The Cloudy Direct Access Lucid API interface
API="http://access.demo.cloudy.io/…/api”
curl $API/collections[
{"name":"lwelogs","instance_dir":"lwelogs”{"name":"collection1","instance_dir":"collection1_0"},{"name":"my_news","instance_dir":"my_news_1"}
]
14
Wednesday, 25 May 2011
![Page 14: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/14.jpg)
Live Demo Sam Pierson, Director of Product Development
The Web User Interface• Administration UI improvements• Log indexing and graphing
API access• Ruby bindings for the LucidWorks REST API• http://rubygems.org/gems/lucid_works
10
Wednesday, 25 May 2011
![Page 15: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/15.jpg)
Lessons Cloud technologies speed up development
Automate everything
Avoid lock-in
Dogfooding
16
Wednesday, 25 May 2011
![Page 16: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/16.jpg)
Wrap Up
Search as a Service
Built in the Cloud, with open source
Full Lucidworks Enterprise and Solr
Powered by Lucid Imagination
17
Wednesday, 25 May 2011
![Page 18: Practical Search in the Cloud - By Marc Krellenstein](https://reader031.fdocuments.net/reader031/viewer/2022020306/55502eaeb4c9058f2f8b4d19/html5/thumbnails/18.jpg)
Contact
Martijn Koster
• http://www.lucidimagination.com/
• twitter.com/makuk66
19
Wednesday, 25 May 2011