Riak Search 2: Yokozuna
-
Upload
pdx-web-design -
Category
Technology
-
view
1.629 -
download
3
description
Transcript of Riak Search 2: Yokozuna
Eric Redmond@coderoshi
Tuesday, November 12, 13
[Slide Intentionally Left Blank]
Tuesday, November 12, 13
A million values isn’t cool.You know what’s cool?...
Tuesday, November 12, 13
A trillion values spread acrosslots of nodes that auto-index and you canalso search in cool ways like geographicallyor maybe you could extract from a pdf on the server side if you used the extractor API
Tuesday, November 12, 13
┬─┬ ( ^_^ノ)
Setup
ノ
Tuesday, November 12, 13
## The enabled Yokozuna set this 'on'.yokozuna = off
## The port number which Solr binds to.yokozuna.solr_port = 10014
## The port number which Solr JMX binds to.yokozuna.solr_jmx_port = 10013
## The arguments to pass to the Solr JVM. Non-standard## arguments, i.e. -XX, may not be portable across JVM## implementations. E.g. -XX:+UseCompressedStrings.yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX:+UseStringCache -XX:+UseCompressedOops
## The data under which to store all Yokozuna related data.## Including the Solr index data.yokozuna.data_dir = ./data/yz
riak.conf
Tuesday, November 12, 13
## The enabled Yokozuna set this 'on'.yokozuna = on
## The port number which Solr binds to.yokozuna.solr_port = 10014
## The port number which Solr JMX binds to.yokozuna.solr_jmx_port = 10013
## The arguments to pass to the Solr JVM. Non-standard## arguments, i.e. -XX, may not be portable across JVM## implementations. E.g. -XX:+UseCompressedStrings.yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX:+UseStringCache -XX:+UseCompressedOops
## The data under which to store all Yokozuna related data.## Including the Solr index data.yokozuna.data_dir = ./data/yz
riak.conf
Tuesday, November 12, 13
riak-admin cluster join [email protected]... and so on...riak-admin cluster planriak-admin cluster commit
Tuesday, November 12, 13
riak-admin cluster join [email protected]... and so on...riak-admin cluster planriak-admin cluster commit
riak-admin security add-user eric 12345riak-admin security add-user admin 123456riak-admin security grant yokozuna.search ON index simple TO ericriak-admin security grant yokozuna.admin ON index TO admin
Tuesday, November 12, 13
require 'riak'client = Riak::Client.new
# create indexclient.create_search_index('simple')
# tie the index to bucket 'cats'bucket = Riak::Bucket.new(client, 'cats')bucket.props = { yz_index: 'simple' }
Tuesday, November 12, 13
Indexing DatatypesTuesday, November 12, 13
UTF8中搜索UTF8חיפוש בΑναζήτηση σε UTF8
Искать в UTF8
Search in UTF8
Tuesday, November 12, 13
Advanced Searchaka. Next Level Search
Tuesday, November 12, 13
solr = RSolr.connect(url: 'http://yokozuna01.bos1:8098/solr/docs')
resp = solr.get('select', params: {q: '*:*'})
Tuesday, November 12, 13
Tuesday, November 12, 13
Facets, Stats, and stuff
Tuesday, November 12, 13
eDisMax
defType=edismax
Tuesday, November 12, 13
Highlighting
Tuesday, November 12, 13
Languages
Tuesday, November 12, 13
Map/ReduceTuesday, November 12, 13
{"inputs": { "module":"yokozuna", "function":"mapred_search", "arg":["docs","title_s:Key* AND language_s:en"] }, "query":[ {"map":{ "language":"javascript", "keep":false, "source":"function(v) { return [1]; }"}}, {"reduce"{ "language":"javascript", "keep":true, "name":"Riak.reduceSum" }} ]}
Tuesday, November 12, 13
Tuesday, November 12, 13
client.create_search_index('simple')
bucket = Riak::Bucket.new(client, 'people')bucket.props = { yz_index: 'simple' }
bucket = Riak::Bucket.new(client, 'cats')bucket.props = { yz_index: 'simple' }
bucket = Riak::Bucket.new(client, 'mountains')bucket.props = { yz_index: 'simple' }
Tuesday, November 12, 13
Schemas
Tuesday, November 12, 13
<schema name="default" version="1.5"> <fields> <field name="kinds" type="string" indexed="true" stored="false" multiValued="true" /> <field name="name" type="string" indexed="true" stored="true" />
<dynamicField name="*_ss" type="string" indexed="true" stored="true" multiValued="true"/>
<field name="_yz_ed" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_pn" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_fpn" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_vtag" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_node" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_rk" type="_yz_str" indexed="true" stored="true"/> <field name="_yz_rb" type="_yz_str" indexed="true" stored="true"/> </fields></schema>
Tuesday, November 12, 13
Fallen Nodes
Tuesday, November 12, 13
Why?
Tuesday, November 12, 13
ThermoclineTuesday, November 12, 13
Thermocline of ReplicationTuesday, November 12, 13
GoalsTuesday, November 12, 13
But... why?Tuesday, November 12, 13
Tuesday, November 12, 13
How
Tuesday, November 12, 13
Yokozuna = GlueTuesday, November 12, 13
Extractorsyz_extractor.erlyz_json_extractor.erlyz_noop_extractor.erlyz_text_extractor.erlyz_xml_extractor.erlyz_doc.erl
Tuesday, November 12, 13
Interfaces
yz_pb_admin.erlyz_pb_search.erlyz_schema.erlyz_wm_extract.erlyz_wm_index.erlyz_wm_schema.erlyz_wm_search.erl
Tuesday, November 12, 13
SecurityTuesday, November 12, 13
Permissions
riak-admin security grant yokozuna.admin ON index TO admin
riak-admin security grant yokozuna.search ON index TO user
riak-admin security grant yokozuna.admin ON schema TO admin
riak-admin security grant yokozuna.search ON index wiki TO user
Tuesday, November 12, 13
Administrationyz_pb_admin.erlyz_schema.erlyz_wm_index.erlyz_wm_schema.erl
Tuesday, November 12, 13
Searchesyz_pb_search.erlyz_wm_search.erl
Tuesday, November 12, 13
Backend Magicyokozuna.erlyz_app.erlyz_general_sup.erlyz_kv.erlyz_misc.erlyz_sup.erl
Tuesday, November 12, 13
Distribution
yz_cover.erlyz_events.erl
Tuesday, November 12, 13
AAEyz_entropy.erlyz_entropy_mgr.erlyz_exchange_fsm.erlyz_index.erlyz_index_hashtree.erlyz_index_hashtree_sup.erl
Tuesday, November 12, 13
Solr code
EntropyData.javaMonitor.javayz_solr.erlyz_solr_proc.erlyz_solr_sup.erl
Tuesday, November 12, 13
Stats
yz_stat.erlyz_stat_worker.erl
Tuesday, November 12, 13
“Testing only showsthe presence, not
the absence,of bugs”
aae_test.erlyokozuna_essential.erlyz_errors.erlyz_fallback.erlyz_flag_transitions.erlyz_index_admin.erlyz_languages.erlyz_mapreduce.erlyz_monitor_solr.erlyz_pb.erlyz_rs_migration.erlyz_rt.erlyz_schema_admin.erlyz_siblings.erlyz_solr_start_timeout.erlyz_stat_test.erlyz_wm_extract_test.erlyz_component_tests.erlyz_json_extractor_tests.erlyz_kv_tests.erlyz_misc_tests.erlyz_text_extractor_tests.erlyz_xml_extractor_tests.erlyz_driver.erlyz_file_terms.erl
Dijkstra
Tuesday, November 12, 13
We Want More
• CRDT Support
• Simplify custom extractors
• Bucket Types (?)
• Users
Tuesday, November 12, 13
Thanks
Eric Redmond@coderoshi
Tuesday, November 12, 13