Mastico, Cantiere Creativo's new gem!
Writing even simple queries with Elasticsearch is often tedious and complex. Chewy makes things better, but our code won't stay very DRY. Here is our solution for simplifying queries with Chewy and Elasticsearch.
When it comes to building a search system for our projects, we almost always face complex scenarios: we may need cross-searches on multiple fields, in multiple languages, and perhaps full-text search too.
In these cases, the solution we use most often is called Elasticsearch.
Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.
ES is a powerful, complex system that can handle large amounts of data. Of course, it's not a plug-and-play solution, so we rarely use it on its own: there are gems that help us bootstrap the features we need.
One of these gems is called Chewy.
What is Chewy?
Chewy is a high-level framework built on the elasticsearch-ruby client.
Chewy greatly simplifies creating and managing an index, that is, a collection of models sharing similar characteristics, which we can then query to get our search results.
Let's take as an example part of the search system we built for the website of the Uffizi:
class UffiziIndex < Chewy::Index
class << self
def index_name(_suggest = nil)
"#{Rails.env}_uffizi_#{I18n.locale}"
end
end
define_type(
Artwork.includes(:museum),
delete_if: -> { translation_for(I18n.locale).nil? }
) do
field :title, value: -> { title }
field :author, value: -> { author }
field :formatted_text, value: -> { formatted_text }
field :abstract_text, value: -> { abstract_text }
field :formatted_renovation, value: -> { formatted_renovation }
field :location, value: -> { location }
field :technique, value: -> { technique }
end
endIf we wanted to find Botticelli's Venus, we could simply search for “venere”:
UffiziIndex::Artwork.all.query(word: {title: "venere"}).load.to_a
=> [#<Artwork:0x007fa66ec00a28
id: 3,
museum_id: 15,
title: "Nascita di Venere",
author: "Sandro Botticelli (Firenze 1445-1510) ",
position: 109>]As we can see, we get a single result, the exact one. But what would happen if several paintings had “venere” in the title and we only wanted Botticelli's?
In this case, we can chain multiple queries:
UffiziIndex::Artwork.all.query(word: {title: "venere"}).query(word: {author: "botticelli"}).load.to_a
=> [#<Artwork:0x007fa66ec00a28
id: 3,
museum_id: 15,
title: "Nascita di Venere",
author: "Sandro Botticelli (Firenze 1445-1510) ",
position: 109>]The result is the same, but we can already see that if we wanted much more specific queries to narrow down the results further, we'd end up writing quite a few lines of code.
Isn't there a smarter way to do it? Yes. ES provides multi_match, which lets you run multiple searches across multiple fields. Let's see how it behaves:
UffiziIndex::Artwork.all.query(multi_match: {fields: [:title, :author], query: "venere botticelli"}).load.to_a
=> [#<Artwork:0x007fa66eada0b8
id: 3,
museum_id: 15,
title: "Nascita di Venere",
author: "Sandro Botticelli (Firenze 1445-1510) ",
position: 109>,
#<Artwork:0x007fa66eada360
id: 30,
museum_id: 6,
title: "Adorazione dei Magi",
author: "Sandro Botticelli (Firenze 1445-1510)",
position: 2>,
#<Artwork:0x007fa66eada1f8
id: 20,
museum_id: 11,
title: "Fortezza",
author: "Sandro Botticelli (Firenze 1445 -1510)",
position: 2015>]Whereas before we wanted to find the Venus, and only Botticelli's Venus, now we get every painting with “venere” in the title and every author with “Botticelli” in the name. The opposite happened: we got an OR instead of an AND.
There are surely dozens of ways to reach our goal with Chewy, which is why we decided to create a helper to make our lives easier every time we use Toptal's gem.
Meet Mastico!
Mastico simplifies the interface for building queries and gives us a basic Chewy configuration, so that once it's installed we can start searching right away!
chewy_query = UffiziIndex::Artwork.all Mastico::Query.new(fields: [:title], query: "Venere").apply(chewy_query).load.to_a => [#<Artwork:0x007fa6668fc820 id: 3, museum_id: 15, title: "Nascita di Venere", author: "Sandro Botticelli (Firenze 1445-1510) ", position: 109>]
Admittedly, this first query is longer than the Chewy one, but our goal is to cross multiple attributes, so let's see what the query that returns only Botticelli's Venus would look like:
Mastico::Query.new(fields: [:title, :author], query: "Venere Botticelli").apply(chewy_query).load.to_a => [#<Artwork:0x007fa66a4281b0 id: 3, museum_id: 15, title: "Nascita di Venere", author: "Sandro Botticelli (Firenze 1445-1510) ", position: 109>]
We got what we wanted, and the query is only slightly longer than the previous one. Even better, all it takes is adding another field to the fields array to search other attributes too.
Let's look in detail at what runs when we launch this command, which is what we would have had to do by hand with ES: https://gist.github.com/mttmanzo/a7ffb82c312a3b3f4d027fb681e49ef6.
Once fields and query are passed, Mastico starts chaining the search with other values, such as the search type (:word, :prefix, :infix and :fuzzy) and the boost, that is, how much we want to emphasize that word.
This is enough to get us started building even complex searches quickly and easily. But what if I misspell the word?
Mastico handles this automatically with the fuzzy type, so if we searched for “Botticello” we'd still find Botticelli's works.
Mastico::Query.new(fields: [:author], query: "Botticello").apply(chewy_query).load.to_a => [#<Artwork:0x007fa66e9a78f8 id: 30, museum_id: 6, title: "Adorazione dei Magi" author: "Sandro Botticelli (Firenze 1445-1510)", position: 2>, #<Artwork:0x007fa66e9a7678 id: 3, museum_id: 15, title: "Nascita di Venere", author: "Sandro Botticelli (Firenze 1445-1510) ", position: 109>, #<Artwork:0x007fa66e9a77b8 id: 20, museum_id: 11, title: "Fortezza", author: "Sandro Botticelli (Firenze 1445 -1510)", position: 2015>]
Nice, but what if I want to filter out “stop words”? There's a solution for that too: just pass the word_weight attribute to the query:
def word_weight(word)
case word
when "botteghe"
0.0
when /\Ab[ao]tte\z/
0.0
else
1.0
end
end
Mastico::Query.new(fields: [:author], query: "Botticello", word_weight: method(:word_weight)).apply(chewy_query).load.to_aThe returned values represent the boost, which can also be used to emphasize the search on other keywords.
All these options may seem complex to combine, but you can actually chain them in a simple hash:
QUERY_FIELDS = {
title: { types: [:fuzzy] },
formatted_text: { types: [:word] },
author: { boost: 3.0, types: [:word] }, # qua definiamo sia il tipo che il boost, solo per questa parola.
abstract_text: { types: [:fuzzy] },
location: { types: [:infix] },
technique: { types: [:prefix] },
}.freeze
def matching_text_scope(text)
Mastico::Query.new(query: text, fields: QUERY_FIELDS).apply(UffiziIndex::Artwork.all)
endThe features we've just seen are a huge help to us every day on many of our projects, but we can't wait to get feedback from external users and, above all, ideas to improve Mastico! Anyone who wants to contribute can of course do so here https://github.com/cantierecreativo/mastico.
