Elasticsearch Query DSL - Not just for wizards...
256
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. Clinton Gormley @clintongormley Elasticsearch Query DSL … not just for wizards
-
Upload
clintongormley -
Category
Technology
-
view
3.946 -
download
0
description
The Elasticsearch Query DSL is a rich, flexible, powerful query language for full text and structured search, but with power comes complexity. Which of the 40 available queries should you use? What's a filter and when should you use it? How do you combine multiple filters, or multiple queries or queries with filters? To most users, "relevance", and how it is affected by different queries, is a black box. Multi-field queries in particular can be difficult to get right if you don't understand how they work. In this talk, I will explain the Query DSL from the ground up: how filters and queries use the inverted index to find matching documents, how the relevance score is calculated, how to combine the filter/query building blocks into complex statements. And finally, I will talk about the pitfalls of multi-field queries and how to avoid them.
Transcript of Elasticsearch Query DSL - Not just for wizards...
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Clinton Gormley @clintongormley
Elasticsearch Query DSL … not just for wizards
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch.org/guide
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch• real-time
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch• real-time • distributed
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch• real-time • distributed • search
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch• real-time • distributed • search • analytics
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
mapping
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
mapping
analysis
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
mapping
analysis query dsl
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
mapping
analysis query dsl
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
query dsl
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
query dslflexible, powerful query language
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
queries
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
queries filters
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
queries• relevance
filters• boolean yes/no
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
queries• relevance• full text
filters• boolean yes/no• exact values
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
queries• relevance• full text• not cached
filters• boolean yes/no• exact values• cached
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
queries• relevance• full text• not cached• slower
filters• boolean yes/no• exact values• cached• faster
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
queries• relevance• full text• not cached• slower
filters• boolean yes/no• exact values• cached• faster
Filter first, then query remaining docs
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": {...} }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { "match": { "title": "search" }} }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { match_all: {}} }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { match_all: {}} }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { "filtered": { "query": {...}, "filter": {...} } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { "filtered": { "query": { "match": { "title": "search" }}, "filter": {...} } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { "filtered": { "query": { "match": { "title": "search" }}, "filter": { "term": { "status": "active" }} } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { "filtered": { "query": { "match_all": {}}, "filter": { "term": { "status": "active" }} } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { "filtered": { "query": { "match_all": {}}, "filter": { "term": { "status": "active" }} } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how data is indexed
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" } !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" } !{ "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" } !{ "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis" }
where content like “%brown%fox%”
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" } !{ "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis" }
slow & inflexible
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" } !{ "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis" }
“analysis”
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" } !{ "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "title": [quick,brown,rabbits], "content": [brown,rabbits,are,commonly,seen] } !{ "title": [keeping,pets,healthy], "content": [my,quick,brown,fox,eats,rabbits,on,a, regular,basis] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2aarebasisbrowncommonlyeatsfoxmyonquickrabbitsregularseen
field: content
sorted list of unique terms
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2aarebasisbrowncommonlyeatsfoxmyonquickrabbitsregularseen
field: content
where they occur
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2aarebasisbrowncommonlyeatsfoxmyonquickrabbitsregularseen
field: content
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2aarebasisbrowncommonlyeatsfoxmyonquickrabbitsregularseen
field: content
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted indexnot just for text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted indexnumbers, dates, bools, enums
geopoints, geoshapes, etc
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
WHERE field = "value"
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
WHERE field CONTAINS "value"
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
WHERE field CONTAINS "value"
term filter
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"term": { "title": "brown" }
WHERE field CONTAINS "value"
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { !!!! } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { "filtered": { !! } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { "filtered": { "query": { ... }, "filter": { ... } } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { "filtered": { "query": { "match_all": {} }, "filter": { ... } } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { "filtered": { "query": { "match_all": {} }, "filter": { ... } } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { "filtered": { "query": { "match_all": {} }, "filter": { "term": { "title": "brown" }} } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2brownhealthykeepingpetsquickrabbits
field: title
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2brownhealthykeepingpetsquickrabbits
field: title
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"term": { "title": "brown" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"term": { "title": "brown" }
➔ result: bitset[ 1, 0 ]
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"term": { "title": "brown" }
➔ result: bitset[ 1, 0 ]
➔ cache as: "title:brown"
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
WHERE field IN ["val",…]
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
WHERE field IN ["val",…]
terms filter
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"terms": { "title": ["quick", "pets"] }
WHERE field IN ["val",…]
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2brownhealthykeepingpetsquickrabbits
field: title
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"terms": { "title": ["quick","pets"] }
➔ result: bitset[ 1, 1 ]
➔ cache as: "title:quick title:pets"
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
WHERE field >= "val1" AND field < "val2"
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
WHERE field >= "val1" AND field < "val2"
range filter
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"range": { "content":{ "gte": "a", "lt": "m" } }
WHERE field >= "val1" AND field < "val2"
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2aarebasisbrowncommonlyeatsfoxmyonquickrabbitsregularseen
field: content
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
➔ result: bitset[ 1, 1 ]
➔ cache as: "content:[a TO m}"
"range": { "content":{ "gte": "a", "lt": "m" } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"range": { "date":{ "gte": "2014-‐01-‐01", "lt": "2041-‐02-‐01" } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"range": { "date":{ "gte": "2014-‐01-‐01", "lt": "2041-‐02-‐01" } }
numeric/date fields optimised
for range filters
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"range": { "date":{ "gte": "now -‐ 1h" } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"range": { "date":{ "gte": "now -‐ 1h" } }
not cached
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"range": { "date":{ "gte": "now -‐ 1h / h" } }
cached
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
WHERE field IS NOT NULL
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
WHERE field has any term
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
WHERE field has any term
exists filter
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"exists": { "field": "title" }
WHERE field has any term
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
missing filter
WHERE field has no term
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"missing": { "field": "title" }
WHERE field has no term
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
AND … OR … NOT
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
AND … OR … NOT
bool filter
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": [ <filters> ], "should": [ <filters> ], "must_not": [ <filters> ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": [ <filters> ], # AND "should": [ <filters> ], "must_not": [ <filters> ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": [ <filters> ], "should": [ <filters> ], # OR "must_not": [ <filters> ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": [ <filters> ], "should": [ <filters> ], "must_not": [ <filters> ] # NOT }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": { "term": { "title": "rabbits" }}, !!!!!}
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": { "term": { "title": "rabbits" }}, "should": [ { "term": { "title": "quick" }}, { "term": { "content": "quick" }} ], !}
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": { "term": { "title": "rabbits" }}, "should": [ { "term": { "title": "quick" }}, { "term": { "content": "quick" }} ], "must_not": { "term": { "content": "fox" }} }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": [ { "bool": { "should": [ { "term": { "title": "rabbits" }}, { "term": { "content": "rabbits" }} ]}}, { "bool": { "should": [ { "term": { "title": "quick" }}, { "term": { "content": "quick" }} ]}} ], "must_not": { "term": { "content": "fox" }} }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": [ { "bool": { "should": [ { "term": { "title": "rabbits" }}, { "term": { "content": "rabbits" }} ]}}, { "bool": { "should": [ { "term": { "title": "quick" }}, { "term": { "content": "quick" }} ]}} ], "must_not": { "term": { "content": "fox" }} }
not cached
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": [ { "bool": { "should": [ { "term": { "title": "rabbits" }}, { "term": { "content": "rabbits" }} ]}}, { "bool": { "should": [ { "term": { "title": "quick" }}, { "term": { "content": "quick" }} ]}} ], "must_not": { "term": { "content": "fox" }} }
cached
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
result bitset = ! (title:rabbits OR content:rabbits) AND (title:quick OR content:quick) AND NOT content:fox
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
filters• boolean yes/no • exact values • cached • faster
Filter first, then query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how relevant is this term?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how relevant is this term?term query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
≈ term filter + relevance
how relevant is this term?term query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"term": { "title": "brown" }
how relevant is this term?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { !!! } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET /_search{ "query": { "term": { "title": "brown" } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "_index": "myindex", "_type": "mytype", "_id": "1", "_score": 0.5, "_source": { "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" } }
how relevant is this doc?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
relevance score
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
relevance scoreHow common is the term in this doc? ➔ more is better
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
relevance scoreHow common is the term in this doc? ➔ more is better How common is the term in ALL docs? ➔ less is better
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
relevance scoreHow common is the term in this doc? ➔ more is better How common is the term in ALL docs? ➔ less is better How long is this doc? ➔ shorter is better
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene similarityHow common is the term in this doc? ➔ more is better How common is the term in ALL docs? ➔ less is better How long is this doc? ➔ shorter is better
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene similarityHow common is the term in this doc? ➔ more is better How common is the term in ALL docs? ➔ less is better How long is this doc? ➔ shorter is better
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term frequency !
How common is the term in ALL docs? ➔ less is better How long is this doc? ➔ shorter is better
lucene similarity
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term frequency !
How common is the term in ALL docs? ➔ less is better How long is this doc? ➔ shorter is better
lucene similarity
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term frequency !
Inverse document frequency !
How long is this doc? ➔ shorter is better
lucene similarity
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term frequency !
Inverse document frequency !
How long is this doc? ➔ shorter is better
lucene similarity
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term frequency !
Inverse document frequency !
Length norm
lucene similarity
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
AND … OR … NOT
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
AND … OR … NOT
bool query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
like bool filter, but different...
AND … OR … NOT
bool query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": [ <queries> ], # AND "should": [ <queries> ], "must_not": [ <queries> ] # NOT }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": [ <queries> ], # AND "should": [ <queries> ], # Hmmm "must_not": [ <queries> ] # NOT }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": [ <queries> ], # AND "should": [ <queries> ], # Hmmm "must_not": [ <queries> ], # NOT "minimum_should_match": ? # Hmmm }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
no "must" queries"bool": { "should": [ { "term": { "title": "quick" }}, { "term": { "title": "brown" }}, { "term": { "title": "rabbits" }} ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
no "must" queries
at least one must match
"bool": { "should": [ { "term": { "title": "quick" }}, { "term": { "title": "brown" }}, { "term": { "title": "rabbits" }} ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
no "must" queries
minimum_should_match = 1
"bool": { "should": [ { "term": { "title": "quick" }}, { "term": { "title": "brown" }}, { "term": { "title": "rabbits" }} ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": { "term": { "title": "quick"}}, "should": [ { "term": { "title": "brown" }}, { "term": { "title": "rabbits" }} ] }
with "must" queries
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": { "term": { "title": "quick"}}, "should": [ { "term": { "title": "brown" }}, { "term": { "title": "rabbits" }} ] }
with "must" queries
all are optional!
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": { "term": { "title": "quick"}}, "should": [ { "term": { "title": "brown" }}, { "term": { "title": "rabbits" }} ] }
with "must" queries
minimum_should_match = 0
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
bool filter ➔ T/F !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
bool filter ➔ T/F !
bool query ➔ _score
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
_score of bool query =
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
_score of bool query = sum( _score of each query)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
_score of bool query = sum( _score of each query) * num of matching queries
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
_score of bool query = sum( _score of each query) * num of matching queries / num of queries
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
more matching should queries
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
more matching should queries ==
better relevance score
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
trim the long tail"bool": { "should": [ { "term": { "title": "quick" }}, { "term": { "title": "brown" }}, { "term": { "title": "rabbits" }} ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
trim the long tail"bool": { "should": [ { "term": { "title": "quick" }}, { "term": { "title": "brown" }}, { "term": { "title": "rabbits" }} ], "minimum_should_match": "75%" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
trim the long tail"bool": { "should": [ { "term": { "title": "quick" }}, { "term": { "title": "brown" }}, { "term": { "title": "rabbits" }} ], "minimum_should_match": "75%" # 2 of 3 }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
match query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
match queryhigh level query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
match queryhigh level query
understands mapping & analysis
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
match query➔ analyze query string ➔ rewrite query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one word query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match": { "title": "QUICK!" }}
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match": { "title": "QUICK!" }}
title:quick
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match": { "title": "QUICK!" }}
title:quick
{ "term": { "title": "quick" }}
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
multi word query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match": { "title": "QUICK FOX!" }}
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match": { "title": "QUICK FOX!" }}
title:quick OR title:fox
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match": { "title": "QUICK FOX!" }}
title:quick OR title:fox
{ "bool": { "should": [ { "term": { "title": "quick" }}, { "term": { "title": "fox" }} ] }}
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
all words must match
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match": { "title": "QUICK FOX!" } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match": { "title": { "query": "QUICK FOX!", ! } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match": { "title": { "query": "QUICK FOX!", "operator": "and" } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "bool": { "should": [ { "term": { "title": "quick" }}, { "term": { "title": "fox" }} ] } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "bool": { "must": [ { "term": { "title": "quick" }}, { "term": { "title": "fox" }} ] } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
trim long tail
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match": { "title": { "query": "QUICK BROWN FOX!", "minimum_should_match": "75%" } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "bool": { "should": [ { "term": { "title": "quick" }}, { "term": { "title": "brown" }}, { "term": { "title": "fox" }} ], "minimum_should_match": 2 } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
fuzzy queries
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
fuzzy querieslevenshtein edit distance
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
bron ➔ brown
insertion
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
bron ➔ brown
foxs ➔ fox
deletion
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
bron ➔ brown
foxs ➔ fox
kiuck ➔ qiuck
substitution
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
bron ➔ brown
foxs ➔ fox
kiuck ➔ qiuck ➔ quick
transposition
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match": { "title": { "query": "KIUCK BRON FOXS!", "fuzziness": "AUTO" } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
phrase / proximity
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match_phrase": { "title":"QUICK BROWN FOX!" } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "match_phrase": { "title": { "query": "BROWN QUICK FOX!", "slop": "10" } } }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
combine queries
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": { <min_should_match> }, "should": [ { <fuzzy> }, { <proximity> } ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": { <min_should_match> }, "should": [ { <fuzzy> }, { <proximity> } ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
! "match": { "title": { "query": "<words>", "minimum_should_match": "75%" } } !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": { <min_should_match> }, "should": [ { <fuzzy> }, { <proximity> } ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
! "match": { "title": { "query": "<words>", "fuzziness": "AUTO" } } !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": { <min_should_match> }, "should": [ { <fuzzy> }, { <proximity> } ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
! "match_phrase": { "title": { "query": "<words>", "slop": "10" } } !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "must": { <min_should_match> }, "should": [ { <fuzzy> }, { <proximity> } ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
multi-field queries
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
easy!
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "should": [ { "match": {"first": "Reginald" }}, { "match": {"middle": "Kenneth" }}, { "match": {"last": "Dwight" }} ], "minimum_should_match": "75%" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
hard!
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "should": [ { "match": { "title": "quick brown fox" }}, { "match": { "content": "quick brown fox" }} ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" }
{ "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" }
{ "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis" }
better match
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" }
{ "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis" }
But 2 matches wins
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
dis_max query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
all docs which match any query
dis_max query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
all docs which match any query
dis_max query
_score = best matching query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"bool": { "should": [ { "match": { "title": "quick brown fox" }}, { "match": { "content": "quick brown fox" }} ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"dis_max": { "queries": [ { "match": { "title": "quick brown fox" }}, { "match": { "content": "quick brown fox" }} ] }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" }
{ "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" }
{ "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"dis_max": { "queries": [ { "match": { "title": "quick brown fox" }}, { "match": { "content": "quick brown fox" }} ], "tie_breaker": 0.2 }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
all docs which match any query
dis_max query
_score = best matching query + tie_breaker * others
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
multi_match query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
match query on multiple fields
multi_match query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"dis_max": { "queries": [ { "match": { "title": "quick brown fox" }}, { "match": { "content": "quick brown fox" }} ], "tie_breaker": 0.2 }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"multi_match": { "query": "quick brown fox", "fields": [ "title", "content" ] "tie_breaker": 0.2 }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"multi_match": { "query": "quick brown fox", "fields": [ "title", "content" ] "tie_breaker": 0.2, # "type": "best_fields" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
best_fields
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
find whole "concept" in one field
best_fields
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"quick brown fox" in title or content
best_fields
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
best_fields
dis_max"quick brown fox" in title or content
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"title": { "type": "string" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"title": { "type": "string", "fields": { !!!!!!!}}
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"title": { "type": "string", "fields": { "stemmed": { "type": "string", "analyzer": "english" } !!!}}
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"title": { "type": "string", "fields": { "stemmed": { "type": "string", "analyzer": "english" }, "autocomplete": { "type": "string", "analyzer": "edge_ngrams" }}}
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
title: [ brown, fox, jumped ] !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
title: [ brown, fox, jumped ] !title.stemmed: [ brown, fox, jump ] !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
title: [ brown, fox, jumped ] !title.stemmed: [ brown, fox, jump ] !title.autocomplete [ b, br, bro, brow, brown, f, fo, fox, j, ju, jum, jump, jumpe, jumped ]
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"multi_match": { "query": "quick brown fox", "fields": [ "title", "title.stemmed", "title.autocomplete" ] "type": "most_fields" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
most_fields
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
most_fieldsmatch same text
analyzed in different ways
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
most_fieldsmore matching fields = better
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
most_fields
boolmore matching fields = better
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
{ "first": "Reginald", "middle": "Kenneth" "last": "Dwight" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"multi_match": { "query": "Reginald Kenneth Dwight", "fields": [ "first", "middle", "last" ] !}
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"multi_match": { "query": "Reginald Kenneth Dwight", "fields": [ "first", "middle", "last" ] "type": "????" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"multi_match": { "query": "Reginald Kenneth Dwight", "fields": [ "first", "middle", "last" ] "type": "most_fields" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"multi_match": { "query": "Reginald Kenneth Dwight", "fields": [ "first", "middle", "last" ] "type": "most_fields" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
problem
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
problemfield centric
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
( first: Reginald OR first: Kenneth OR first: Dwight ) OR ( middle:Reginald OR middle:Kenneth OR middle:Dwight ) OR ( last: Reginald OR last: Kenneth OR last: Dwight )
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
( first: Reginald OR first: Kenneth OR first: Dwight ) OR ( middle:Reginald OR middle:Kenneth OR middle:Dwight ) OR ( last: Reginald OR last: Kenneth OR last: Dwight )
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
problem
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
problemoperator: and
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
( first: Reginald AND first: Kenneth AND first: Dwight ) OR ( middle:Reginald AND middle:Kenneth AND middle:Dwight ) OR ( last: Reginald AND last: Kenneth AND last: Dwight )
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
( first: Reginald AND first: Kenneth AND first: Dwight ) OR ( middle:Reginald AND middle:Kenneth AND middle:Dwight ) OR ( last: Reginald AND last: Kenneth AND last: Dwight )
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
problem
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
problemterm frequencies
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
problemterm frequencies
first:dwight ➔ common
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
problemterm frequencies
first:dwight ➔ commonlast: dwight ➔ uncommon
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
solution
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
index time solution
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
index time solutionsingle "fullname" field
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"first": { "type": "string" }, "middle": { "type": "string" }, "last": { "type": "string" } !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"first": { "type": "string" }, "middle": { "type": "string" }, "last": { "type": "string" }, "full": { "type": "string" } !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"first": { "type": "string", "copy_to": "full" }, "middle": { "type": "string", "copy_to": "full" }, "last": { "type": "string", "copy_to": "full" }, "full": { "type": "string" } !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"match": { "full": "Reginald Kenneth Dwight", "minimum_should_match": "75%" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
query time solution
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
query time solutionterm-centric query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
( first: Reginald AND first: Kenneth AND first: Dwight ) OR ( middle:Reginald AND middle:Kenneth AND middle:Dwight ) OR ( last: Reginald AND last: Kenneth AND last: Dwight )
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
( first: Reginald OR middle:Reginald OR last: Reginald ) AND ( first: Kenneth OR middle:Kenneth OR last: Kenneth ) AND ( first: Dwight OR middle:Dwight OR last: Dwight )
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
blend(first,middle,last):Reginald AND blend(first,middle,last):Kenneth AND blend(first,middle,last):Dwight
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
blend(first,middle,last):Reginald AND blend(first,middle,last):Kenneth AND blend(first,middle,last):Dwight
blends term frequencies
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cross_fields
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cross_fieldsquery multiple fields as if they were one
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"multi_match": { "query": "Reginald Kenneth Dwight", "fields": [ "first", "middle", "last" ] "type": "cross_fields" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
"multi_match": { "query": "Reginald Kenneth Dwight", "fields": [ "first", "middle", "last" ] "type": "cross_fields", "minimum_should_match": "75%" }
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
best_fields: whole concept in single field
most_fields: same text, different analyzers
cross_fields: treat multiple fields as one
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
understand the building blocks
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
the rest is details
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
thank you@clintongormley
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
thank you@clintongormley
elasticsearch.org/downloads
elasticsearch.com/support
elasticsearch.com/jobs
