Riak Search 2: Yokozuna

48
Eric Redmond @coderoshi Tuesday, November 12, 13

description

Ricon|West talk about Yokozuna

Transcript of Riak Search 2: Yokozuna

Page 1: Riak Search 2: Yokozuna

Eric Redmond@coderoshi

Tuesday, November 12, 13

Page 2: Riak Search 2: Yokozuna

[Slide Intentionally Left Blank]

Tuesday, November 12, 13

Page 3: Riak Search 2: Yokozuna

A million values isn’t cool.You know what’s cool?...

Tuesday, November 12, 13

Page 4: Riak Search 2: Yokozuna

A trillion values spread acrosslots of nodes that auto-index and you canalso search in cool ways like geographicallyor maybe you could extract from a pdf on the server side if you used the extractor API

Tuesday, November 12, 13

Page 5: Riak Search 2: Yokozuna

┬─┬ ( ^_^ノ)

Setup

Tuesday, November 12, 13

Page 6: Riak Search 2: Yokozuna

## The enabled Yokozuna set this 'on'.yokozuna = off

## The port number which Solr binds to.yokozuna.solr_port = 10014

## The port number which Solr JMX binds to.yokozuna.solr_jmx_port = 10013

## The arguments to pass to the Solr JVM. Non-standard## arguments, i.e. -XX, may not be portable across JVM## implementations. E.g. -XX:+UseCompressedStrings.yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX:+UseStringCache -XX:+UseCompressedOops

## The data under which to store all Yokozuna related data.## Including the Solr index data.yokozuna.data_dir = ./data/yz

riak.conf

Tuesday, November 12, 13

Page 7: Riak Search 2: Yokozuna

## The enabled Yokozuna set this 'on'.yokozuna = on

## The port number which Solr binds to.yokozuna.solr_port = 10014

## The port number which Solr JMX binds to.yokozuna.solr_jmx_port = 10013

## The arguments to pass to the Solr JVM. Non-standard## arguments, i.e. -XX, may not be portable across JVM## implementations. E.g. -XX:+UseCompressedStrings.yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX:+UseStringCache -XX:+UseCompressedOops

## The data under which to store all Yokozuna related data.## Including the Solr index data.yokozuna.data_dir = ./data/yz

riak.conf

Tuesday, November 12, 13

Page 8: Riak Search 2: Yokozuna

riak-admin cluster join [email protected]... and so on...riak-admin cluster planriak-admin cluster commit

Tuesday, November 12, 13

Page 9: Riak Search 2: Yokozuna

riak-admin cluster join [email protected]... and so on...riak-admin cluster planriak-admin cluster commit

riak-admin security add-user eric 12345riak-admin security add-user admin 123456riak-admin security grant yokozuna.search ON index simple TO ericriak-admin security grant yokozuna.admin ON index TO admin

Tuesday, November 12, 13

Page 10: Riak Search 2: Yokozuna

require 'riak'client = Riak::Client.new

# create indexclient.create_search_index('simple')

# tie the index to bucket 'cats'bucket = Riak::Bucket.new(client, 'cats')bucket.props = { yz_index: 'simple' }

Tuesday, November 12, 13

Page 11: Riak Search 2: Yokozuna

Indexing DatatypesTuesday, November 12, 13

Page 12: Riak Search 2: Yokozuna

UTF8中搜索UTF8חיפוש בΑναζήτηση σε UTF8

Искать в UTF8

Search in UTF8

Tuesday, November 12, 13

Page 13: Riak Search 2: Yokozuna

Advanced Searchaka. Next Level Search

Tuesday, November 12, 13

Page 14: Riak Search 2: Yokozuna

solr = RSolr.connect(url: 'http://yokozuna01.bos1:8098/solr/docs')

resp = solr.get('select', params: {q: '*:*'})

Tuesday, November 12, 13

Page 15: Riak Search 2: Yokozuna

Tuesday, November 12, 13

Page 16: Riak Search 2: Yokozuna

Facets, Stats, and stuff

Tuesday, November 12, 13

Page 17: Riak Search 2: Yokozuna

eDisMax

defType=edismax

Tuesday, November 12, 13

Page 18: Riak Search 2: Yokozuna

Highlighting

Tuesday, November 12, 13

Page 19: Riak Search 2: Yokozuna

Languages

Tuesday, November 12, 13

Page 20: Riak Search 2: Yokozuna

Map/ReduceTuesday, November 12, 13

Page 21: Riak Search 2: Yokozuna

{"inputs": { "module":"yokozuna", "function":"mapred_search", "arg":["docs","title_s:Key* AND language_s:en"] }, "query":[ {"map":{ "language":"javascript", "keep":false, "source":"function(v) { return [1]; }"}}, {"reduce"{ "language":"javascript", "keep":true, "name":"Riak.reduceSum" }} ]}

Tuesday, November 12, 13

Page 22: Riak Search 2: Yokozuna

Tuesday, November 12, 13

Page 23: Riak Search 2: Yokozuna

client.create_search_index('simple')

bucket = Riak::Bucket.new(client, 'people')bucket.props = { yz_index: 'simple' }

bucket = Riak::Bucket.new(client, 'cats')bucket.props = { yz_index: 'simple' }

bucket = Riak::Bucket.new(client, 'mountains')bucket.props = { yz_index: 'simple' }

Tuesday, November 12, 13

Page 24: Riak Search 2: Yokozuna

Schemas

Tuesday, November 12, 13

Page 25: Riak Search 2: Yokozuna

<schema name="default" version="1.5"> <fields> <field name="kinds" type="string" indexed="true" stored="false" multiValued="true" /> <field name="name" type="string" indexed="true" stored="true" />

<dynamicField name="*_ss" type="string" indexed="true" stored="true" multiValued="true"/>

<field name="_yz_ed" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_pn" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_fpn" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_vtag" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_node" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_rk" type="_yz_str" indexed="true" stored="true"/> <field name="_yz_rb" type="_yz_str" indexed="true" stored="true"/> </fields></schema>

Tuesday, November 12, 13

Page 26: Riak Search 2: Yokozuna

Fallen Nodes

Tuesday, November 12, 13

Page 27: Riak Search 2: Yokozuna

Why?

Tuesday, November 12, 13

Page 28: Riak Search 2: Yokozuna

ThermoclineTuesday, November 12, 13

Page 29: Riak Search 2: Yokozuna

Thermocline of ReplicationTuesday, November 12, 13

Page 30: Riak Search 2: Yokozuna

GoalsTuesday, November 12, 13

Page 31: Riak Search 2: Yokozuna

But... why?Tuesday, November 12, 13

Page 32: Riak Search 2: Yokozuna

Tuesday, November 12, 13

Page 33: Riak Search 2: Yokozuna

How

Tuesday, November 12, 13

Page 34: Riak Search 2: Yokozuna

Yokozuna = GlueTuesday, November 12, 13

Page 35: Riak Search 2: Yokozuna

Extractorsyz_extractor.erlyz_json_extractor.erlyz_noop_extractor.erlyz_text_extractor.erlyz_xml_extractor.erlyz_doc.erl

Tuesday, November 12, 13

Page 36: Riak Search 2: Yokozuna

Interfaces

yz_pb_admin.erlyz_pb_search.erlyz_schema.erlyz_wm_extract.erlyz_wm_index.erlyz_wm_schema.erlyz_wm_search.erl

Tuesday, November 12, 13

Page 37: Riak Search 2: Yokozuna

SecurityTuesday, November 12, 13

Page 38: Riak Search 2: Yokozuna

Permissions

riak-admin security grant yokozuna.admin ON index TO admin

riak-admin security grant yokozuna.search ON index TO user

riak-admin security grant yokozuna.admin ON schema TO admin

riak-admin security grant yokozuna.search ON index wiki TO user

Tuesday, November 12, 13

Page 39: Riak Search 2: Yokozuna

Administrationyz_pb_admin.erlyz_schema.erlyz_wm_index.erlyz_wm_schema.erl

Tuesday, November 12, 13

Page 40: Riak Search 2: Yokozuna

Searchesyz_pb_search.erlyz_wm_search.erl

Tuesday, November 12, 13

Page 41: Riak Search 2: Yokozuna

Backend Magicyokozuna.erlyz_app.erlyz_general_sup.erlyz_kv.erlyz_misc.erlyz_sup.erl

Tuesday, November 12, 13

Page 42: Riak Search 2: Yokozuna

Distribution

yz_cover.erlyz_events.erl

Tuesday, November 12, 13

Page 43: Riak Search 2: Yokozuna

AAEyz_entropy.erlyz_entropy_mgr.erlyz_exchange_fsm.erlyz_index.erlyz_index_hashtree.erlyz_index_hashtree_sup.erl

Tuesday, November 12, 13

Page 44: Riak Search 2: Yokozuna

Solr code

EntropyData.javaMonitor.javayz_solr.erlyz_solr_proc.erlyz_solr_sup.erl

Tuesday, November 12, 13

Page 45: Riak Search 2: Yokozuna

Stats

yz_stat.erlyz_stat_worker.erl

Tuesday, November 12, 13

Page 46: Riak Search 2: Yokozuna

“Testing only showsthe presence, not

the absence,of bugs”

aae_test.erlyokozuna_essential.erlyz_errors.erlyz_fallback.erlyz_flag_transitions.erlyz_index_admin.erlyz_languages.erlyz_mapreduce.erlyz_monitor_solr.erlyz_pb.erlyz_rs_migration.erlyz_rt.erlyz_schema_admin.erlyz_siblings.erlyz_solr_start_timeout.erlyz_stat_test.erlyz_wm_extract_test.erlyz_component_tests.erlyz_json_extractor_tests.erlyz_kv_tests.erlyz_misc_tests.erlyz_text_extractor_tests.erlyz_xml_extractor_tests.erlyz_driver.erlyz_file_terms.erl

Dijkstra

Tuesday, November 12, 13

Page 47: Riak Search 2: Yokozuna

We Want More

• CRDT Support

• Simplify custom extractors

• Bucket Types (?)

• Users

Tuesday, November 12, 13

Page 48: Riak Search 2: Yokozuna

Thanks

Eric Redmond@coderoshi

Tuesday, November 12, 13