Confessions of a Newbie Elasticsearch Application Developer

Shaunak Kashyap May 16, 2015

www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

2

Agenda

• Filters and queries • Searching multivalue fields for exact equality • Making moving-window filters efficient • Index aliases

www.elastic.co3

Filters and queries

Example: Show me all recipes that mention "lamb", originating in Colorado

www.elastic.co4

GET /recipes/_search { "query": { "filtered": { "query": { "match": { "description": "lamb" } }, "filter": { "term": { "state": "TX" } } } } }

Filters and queries

Example: Show me all recipes that mention "lamb", originating in Colorado

www.elastic.co5

GET /recipes/_search { "query": { "filtered": { "query": { "match": { "description": "bbq" } }, "filter": { "term": { "state": "CO" } } } } }

Filters and queries

Example: Show me all recipes that mention "lamb", originating in Colorado

www.elastic.co6

GET /recipes/_search { "query": { "filtered": { "query": { "match": { "description": "lamb" } }, "filter": { "term": { "state": "CO" } } } } }

Filters and queries

Example: Show me all recipes that mention "lamb", originating in Colorado

www.elastic.co7

Filters and queries

• In a filtered query, filters are run before queries • By default, most filters are cached but queries are not • When filters are used each hit (result) gets the same score, so no

sorting needs to be done

When searching, try to filter as much as possible

In future versions of Elasticsearch, filters and queries

might be combined in the query DSL to simplify usage

www.elastic.co8

Searching multivalue fields for exact equality

Example: Show me all t-shirts in our catalog that are available only in size XL

{ "sku": "12345-8", "size": [ "S", "M", "XL" ] }

{ "sku": "999-2", "size": [ "XL" ] }

www.elastic.co9

Searching multivalue fields for exact equality

GET /catalog/_search { "query": { "filtered": { "filter": { "term": { "size": "XL" } } } } }

Example: Show me all t-shirts in our catalog that are available only in size XL{ ..., "hits" : { "total" : 2, "max_score" : 1.0, "hits" : [ { "_index" : "catalog", "_type" : "tshirt", "_id" : "1", "_score" : 1.0, "_source":{ "sku": "12345-8", "size": [ "S", "M", "XL" ] } }, { "_index" : "catalog", "_type" : "tshirt", "_id" : "2", "_score" : 1.0, "_source":{ "sku": "999-2", "size": [ "XL" ] } } ] } }

www.elastic.co10

Searching multivalue fields for exact equality

Example: Show me all t-shirts in our catalog that are available only in size XL

{ "sku": "12345-8", "size": [ "S", "M", "XL" ] }

{ "sku": "999-2", "size": [ "XL" ] }

Term DocumentsS 1M 1XL 1 , 2

size

www.elastic.co11

Searching multivalue fields for exact equality

Example: Show me all t-shirts in our catalog that are available only in size XL

{ "sku": "12345-8", "size": [ "S", "M", "XL" ], "num_sizes": 3 }

{ "sku": "999-2", "size": [ "XL" ], "num_sizes": 1 }

www.elastic.co12

Searching multivalue fields for exact equality

Example: Show me all t-shirts in our catalog that are available only in size XLGET /catalog/_search { "query": { "filtered": { "filter": { "and": [ { "term": { "size": "XL" } }, { "term": { "num_sizes": 1 } } ] } } } }

{ ..., "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "catalog", "_type" : "tshirt", "_id" : "2", "_score" : 1.0, "_source":{ "sku": "999-2", "size": [ "XL" ], "num_sizes": 1 } } ] } }

www.elastic.co13

Making moving-window filters efficient

Example: Show me all flights that departed in the last hourGET /flights/_search { "query": { "filtered": { "filter": { "range": { "departed_on": { "gt": "now-1h" } } } } }

Not cached!

www.elastic.co14

Making moving-window filters efficient

Example: Show me all flights that departed in the last hourGET /flights/_search { "query": { "filtered": { "filter": { "and": [

{ "departed_on": { "gt": "now-1h/d" } { "range": { "departed_on": { "gt": "now-1h" } ] } } } }

www.elastic.co15

Making moving-window filters efficient

• Filters using now (without rounding) are not cached • Order of filters matters

When filtering by a moving window, filter down to the corresponding fixed window first

www.elastic.co16

Index aliases

What if you want to change your mapping?{ "name": "Coors Field", "location": [ -104.99, 39.76 ] }

you want this field to be mapped as a geo point…

… but you forgot to

explicitly specify the mapping!

… so dynamic mapping will map this field as an (array of) integers

www.elastic.co

What if you want to change your mapping?

1. Create a new index, explicitly specifying correct mapping 2. Create an index alias pointing to both indices - old and new 3. Search using alias 4. Stop indexing into old index + start indexing into new index 5. Reindex old index data into new index 6. Update index alias to point only to new index

PUT /travel_v2 { "travel": { "mappings": { "place": { "type": "geo_point" } } } }

17

Index aliases

www.elastic.co

What if you want to change your mapping?

1. Create a new index, explicitly specifying correct mapping 2. Create an index alias pointing to both indices - old and new 3. Search using alias 4. Stop indexing into old index + start indexing into new index 5. Reindex old index data into new index 6. Update index alias to point only to new index

18

Index aliases

POST /_aliases { "actions": [ { "add": { "index": "travel", "alias": "travel_a" } }, { "add": { "index": "travel_v2", "alias": "travel_a" } } ] }

www.elastic.co

What if you want to change your mapping?

1. Create a new index, explicitly specifying correct mapping 2. Create an index alias pointing to both indices - old and new 3. Search using alias instead of index 4. Stop indexing into old index + start indexing into new index 5. Reindex old index data into new index 6. Update index alias to point only to new index

19

Index aliases

GET /travel_a/_search { ... }

www.elastic.co

What if you want to change your mapping?

1. Create a new index, explicitly specifying correct mapping 2. Create an index alias pointing to both indices - old and new 3. Search using alias instead of index 4. Start indexing new data only into new index 5. Reindex old index data into new index 6. Update index alias to point only to new index

20

Index aliases

PUT /travel_v2/place { "name": "Red Rocks Amphitheater", "location": [ -105.21, 39.66 ]

}

www.elastic.co

What if you want to change your mapping?

1. Create a new index, explicitly specifying correct mapping 2. Create an index alias pointing to both indices - old and new 3. Search using alias instead of index 4. Start indexing new data only into new index 5. Batch reindex old index data into new index 6. Update index alias to point only to new index

21

Index aliases

POST /travel_v2/_bulk { ... }

www.elastic.co

What if you want to change your mapping?

1. Create a new index, explicitly specifying correct mapping 2. Create an index alias pointing to both indices - old and new 3. Search using alias instead of index 4. Start indexing new data only into new index 5. Batch reindex old index data into new index 6. Update index alias to point only to new index

22

Index aliases

POST /_aliases { "actions": [ { "remove": { "index": "travel", "alias": "travel_a" } } ] }

www.elastic.co23

Index aliases

• Can be useful when reindexing data (e.g. due to changed mapping) • Can be used to querying across multiple indices (e.g. last 4 weeks’ data) • Can be used to create a filtered view of an index (like views in SQL)

Index aliases are convenient & cheap - use them early and often!

In future versions of Elasticsearch, you will be able to use index aliases

to restrict the fields returned from an index as well

www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

24

Summary

• Filters and queries • Searching multivalue fields for exact equality • Making moving-window filters efficient • Index aliases

www.elastic.co25

Questions?

https://discuss.elastic.co

Thank you

www.elastic.co26

Extra Slides Follow

Extra Slides

www.elastic.co27

Index aliases

What if you wanted to query a moving-window of data spread across multiple indices?sales-20150503 sales-20150504 sales-20150505 sales-20150506 ... sales-20150517 sales-20150518

POST /_aliases { "actions": [ { "add": { "index": “sales-20150518", "alias": “sales-2w" }}, { "remove": { "index": “sales-20150504", "alias": “sales-2w" }} ] }

POST /_aliases { "actions": [ { "add": { "index": “sales-20150519", "alias": “sales-2w" }}, { "remove": { "index": “sales-20150505", "alias": “sales-2w" }} ] }

sales-20150503 sales-20150504 sales-20150505 sales-20150506 ... sales-20150517 sales-20150518 sales-20150519

Last 2 weeks

Tomorrow

Last 2 weeks

Today

www.elastic.co28

Index aliases

What if you wanted to query a moving-window of data spread across multiple indices?

GET /sales-2w/_search { ... }

www.elastic.co29

Index aliases

What if you wanted one index per size of t-shirt?{ "sku": "12345-8", "size": [ "S", "M", "XL" ] }

{ "sku": "999-2", "size": [ "XL" ] }

www.elastic.co30

Index aliases

What if you wanted one index per size of t-shirt?POST /_aliases { "actions": [ { "add": { "index": “catalog”, "alias": “catalog-small”, "filter": { "term": { "size": “s" } } } }, { "add": { "index": “catalog”, "alias": “catalog-medium” “filter”: { "term": { "size": “m" } } } }, { "add": { "index": “catalog”, "alias": “catalog-xlarge” “filter: { “term”: { “size”: “xl” } } } } ] }

www.elastic.co31

Index aliases

What if you wanted one index per size of t-shirt?GET /catalog-medium/_search { ... }