Riak Search 2: Yokozuna

Click here to load reader

  • date post

    30-Jun-2015
  • Category

    Technology

  • view

    1.620
  • download

    3

Embed Size (px)

description

Ricon|West talk about Yokozuna

Transcript of Riak Search 2: Yokozuna

  • 1. Eric Redmond @coderoshiTuesday, November 12, 13

2. [Slide Intentionally Left Blank]Tuesday, November 12, 13 3. A million values isnt cool.You know whats cool?...Tuesday, November 12, 13 4. A trillion values spread across lots of nodes that auto-index and you can also search in cool ways like geographically or maybe you could extract from a pdf on the server side if you used the extractor APITuesday, November 12, 13 5. ^_^) (Setup Tuesday, November 12, 13 6. ## The enabled Yokozuna set this 'on'. yokozuna = offriak.conf## The port number which Solr binds to. yokozuna.solr_port = 10014 ## The port number which Solr JMX binds to. yokozuna.solr_jmx_port = 10013 ## The arguments to pass to the Solr JVM. Non-standard ## arguments, i.e. -XX, may not be portable across JVM ## implementations. E.g. -XX:+UseCompressedStrings. yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX: +UseStringCache -XX:+UseCompressedOops ## The data under which to store all Yokozuna related data. ## Including the Solr index data. yokozuna.data_dir = ./data/yz Tuesday, November 12, 13 7. ## The enabled Yokozuna set this 'on'. yokozuna = onriak.conf## The port number which Solr binds to. yokozuna.solr_port = 10014 ## The port number which Solr JMX binds to. yokozuna.solr_jmx_port = 10013 ## The arguments to pass to the Solr JVM. Non-standard ## arguments, i.e. -XX, may not be portable across JVM ## implementations. E.g. -XX:+UseCompressedStrings. yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX: +UseStringCache -XX:+UseCompressedOops ## The data under which to store all Yokozuna related data. ## Including the Solr index data. yokozuna.data_dir = ./data/yz Tuesday, November 12, 13 8. riak-admin cluster join [email protected] ... and so on... riak-admin cluster plan riak-admin cluster commitTuesday, November 12, 13 9. riak-admin cluster join [email protected] ... and so on... riak-admin cluster plan riak-admin cluster commit riak-admin security add-user eric 12345 riak-admin security add-user admin 123456 riak-admin security grant yokozuna.search ON index simple TO eric riak-admin security grant yokozuna.admin ON index TO admin Tuesday, November 12, 13 10. require 'riak' client = Riak::Client.new # create index client.create_search_index('simple') # tie the index to bucket 'cats' bucket = Riak::Bucket.new(client, 'cats') bucket.props = { yz_index: 'simple' }Tuesday, November 12, 13 11. Indexing Datatypes Tuesday, November 12, 13 12. UTF8 UTF8 UTF8 UTF8 Search in UTF8 Tuesday, November 12, 13 13. Advanced Searchaka. Next Level SearchTuesday, November 12, 13 14. solr = RSolr.connect(url: 'http://yokozuna01.bos1:8098/solr/docs')resp = solr.get('select', params: {q: '*:*'})Tuesday, November 12, 13 15. Tuesday, November 12, 13 16. Facets, Stats, and stuffTuesday, November 12, 13 17. eDisMax defType=edismaxTuesday, November 12, 13 18. HighlightingTuesday, November 12, 13 19. LanguagesTuesday, November 12, 13 20. Map/Reduce Tuesday, November 12, 13 21. {"inputs": { "module":"yokozuna", "function":"mapred_search", "arg":["docs","title_s:Key* AND language_s:en"] }, "query":[ {"map":{ "language":"javascript", "keep":false, "source":"function(v) { return [1]; }"}}, {"reduce"{ "language":"javascript", "keep":true, "name":"Riak.reduceSum" }} ] } Tuesday, November 12, 13 22. Tuesday, November 12, 13 23. client.create_search_index('simple') bucket = Riak::Bucket.new(client, 'people') bucket.props = { yz_index: 'simple' } bucket = Riak::Bucket.new(client, 'cats') bucket.props = { yz_index: 'simple' } bucket = Riak::Bucket.new(client, 'mountains') bucket.props = { yz_index: 'simple' }Tuesday, November 12, 13 24. SchemasTuesday, November 12, 13 25. Tuesday, November 12, 13 26. Fallen NodesTuesday, November 12, 13 27. Why?Tuesday, November 12, 13 28. Thermocline Tuesday, November 12, 13 29. Thermocline of Replication Tuesday, November 12, 13 30. Goals Tuesday, November 12, 13 31. But... why? Tuesday, November 12, 13 32. Tuesday, November 12, 13 33. HowTuesday, November 12, 13 34. Yokozuna = Glue Tuesday, November 12, 13 35. yz_extractor.erl yz_json_extractor.erl yz_noop_extractor.erl yz_text_extractor.erl yz_xml_extractor.erl yz_doc.erl Tuesday, November 12, 13Extractors 36. yz_pb_admin.erl yz_pb_search.erl yz_schema.erl yz_wm_extract.erl yz_wm_index.erl yz_wm_schema.erl yz_wm_search.erl Tuesday, November 12, 13Interfaces 37. Security Tuesday, November 12, 13 38. Permissions riak-admin security grant yokozuna.admin ON schema TO adminriak-admin security grant yokozuna.admin ON index TO adminriak-admin security grant yokozuna.search ON index TO userriak-admin security grant yokozuna.search ON index wiki TO userTuesday, November 12, 13 39. yz_pb_admin.erl yz_schema.erl yz_wm_index.erl yz_wm_schema.erl Tuesday, November 12, 13Administration 40. yz_pb_search.erl yz_wm_search.erl Tuesday, November 12, 13Searches 41. yokozuna.erl yz_app.erl yz_general_sup.erl yz_kv.erl yz_misc.erl yz_sup.erl Tuesday, November 12, 13Backend Magic 42. Distributionyz_cover.erl yz_events.erl Tuesday, November 12, 13 43. yz_entropy.erl yz_entropy_mgr.erl yz_exchange_fsm.erl yz_index.erl yz_index_hashtree.erl yz_index_hashtree_sup.erl Tuesday, November 12, 13AAE 44. Solr code EntropyData.java Monitor.java yz_solr.erl yz_solr_proc.erl yz_solr_sup.erl Tuesday, November 12, 13 45. Statsyz_stat.erl yz_stat_worker.erl Tuesday, November 12, 13 46. aae_test.erl yokozuna_essential.erl yz_errors.erl yz_fallback.erl yz_ag_transitions.erl yz_index_admin.erl yz_languages.erl yz_mapreduce.erl yz_monitor_solr.erl yz_pb.erl yz_rs_migration.erl yz_rt.erl yz_schema_admin.erl yz_siblings.erl yz_solr_start_timeout.erl yz_stat_test.erl yz_wm_extract_test.erl yz_component_tests.erl yz_json_extractor_tests.erl yz_kv_tests.erl yz_misc_tests.erl yz_text_extractor_tests.erl yz_xml_extractor_tests.erl yz_driver.erl yz_le_terms.erl Tuesday, November 12, 13Testing only shows the presence, not the absence, of bugs Dijkstra 47. We Want More Tuesday, November 12, 13CRDT Support Simplify custom extractors Bucket Types (?) Users 48. Thanks Eric Redmond @coderoshiTuesday, November 12, 13