Searching Relational Data with Elasticsearch

Searching Relational Data
with Elasticsearch
Dr. Renaud Delbru
CTO, Siren Solutions

● CTO, SIREn Solutions
– Search, Big Data, Knowledge Graph
● Lucene / Solr Contributor
– E.g., Cross Data Center Replication
– Lucene Revolution 2013, 2014
– Lucene In Action, 2nd Edition
● Author of the SIREn plugin
Introducing myself

● Open source search
systems
– Lucene, Solr, Elasticsearch
● Document-based model
– Flat key-value model
– Originally developed for
searching full-text documents
Background
firstname John
lastname
title
Smith
Mr Dr

Background
● Data is usually more
complex
– Nested objects
● XML, JSON
● E.g., US patents
– Relations
● RDBMS, RDF, Graph, Documents
with links to entities or other
documents
Article
{
"firstName": "John",
"lastName": "Smith",
"age": 25,
"address" : {
"street" : "21 2nd
Street",
"city" : "New York",
"state" : "NY"
},
"phoneNumber" : [
{ "type" : "home", "number" : "212 555-1234" },
{ "type" : "fax", "number" : "646 555-4567" }
]
}
Person
Company

Crunchbase example
Elastic
Series A
Series B
Data
Collective
Benchmark
Index
Venture

name : Elastic
funding_rounds.round_code : A
funding_rounds.founded_year : 2012
funding_rounds.round_code : B
funding_rounds.founded_year : 2013
funding_rounds.investments.name : Benchmark
funding_rounds.investments.name : Data Collective
funding_rounds.investments.name : Index Ventures
● Pros:
– Relatively easy
– Fast
● Cons:
– Loss of precision, false positive
– Index-time data materialisation
– Data duplication (child)
– Not optimal for updates
Common solutions

name : Elastic
f_r.round_code : A
f_r.founded_year : 2012
f_r.inv.name : Benchmarkname : Elastic
f_r.round_code : A
f_r.inv.name : Data Collectivename : Elastic
f_r.round_code : B
f_r.inv.name : Benchmarkname : Elastic
f_r.round_code : B
f_r.inv.name : Index Ventures
● Pros:
– Relatively easy
– No loss of precision
● Cons:
– Combinatorial explosion
– Duplicate results: query-time grouping is necessary
– Data duplication (parent and child)
Common solutions

● Lucene's BlockJoin
– Feature to provide relational search
– “Nested” type in Elasticsearch
● Model
– One (flat) document per record
– Joins computed at index time
– Related documents are indexed in
a same “block”
{
"company": {
"properties" : {
"funding_rounds" : {
"type" : "nested",
"properties" : {
"investments" : {
"type" : "nested"
} } } } } }
Index-time join

Index-time join
● Pros:
– Fast (join precomputed, data locality)
● Cons:
– High memory usage for complex nested model
Document Block
name : Elastic
country_code : A
...
round_code : A
founded_year : 2012
...
Name : Data Collective
Type : Org
Name : Benchmark
Type : Org
round_code : B
founded_year : 2013
...
Name : Index Venture
Type : Org
Name : Benchmark
Type : Org

Index-time join
● SIREn Plugin
– Plugin to Lucene, Solr, Elasticsearch
– Add native index for nested data type
– http://siren.solutions/siren/overview/
● Model
– One document per “tree”
– Joins computed at index time
– Rich data model (JSON)
● Nested objects, nested arrays, multi-valued
attributes, datatypes
{
"company": {
"properties" : {
"_siren_source" : {
"analyzer" : "concise",
"postings_format" : "Siren10AFor",
"store" : "no",
"type" : "string"
} } } }

Index-time join
name : Elastic
country_code : A
...
round_code : A
founded_year : 2012
...
round_code : B
founded_year : 2013
...
Type : Org
Name : Benchmark
Type : Org
Type : Org
Name : Benchmark
Type : Org
● Pros:
– Fast (join precomputed, data locality)
– Low memory usage, even for complex nested model
● Cons:
1
1.1
1.2
1.1.1
1.1.2
1.2.1
1.2.2

Index-time join
More information on our blog post

Query-time join
● Elasticsearch's Parent-Child
– Query-time join for nested data
● Model
– At index time, child documents should
specify their parent ID with the
_parent field
– Joins computed at query time
{
"company": {},
"investment" : {
"_parent" : {
"type" : "company",
}
},
"investor" : {
"_parent" : {
"type" : "investment",
}
}
}

Query-time join
● Pros:
– Update friendly
– Data locality: parent and child on same shard
● Cons:
– Slower than index-time solutions
– Larger memory use than nested
● A child cannot have more than one parent
name : Elastic
country_code : A
...
round_code : A
founded_year : 2012
...
Type : Org
Name : Benchmark
Type : Org
round_code : B
founded_year : 2013
...
Type : Org
Name : Benchmark
Type : Org

Query-time join
● FilterJoin's Plugin
– Query-time join for relational data
● Inspired from #3278
● Model
– At index time, documents should specify the IDs of their related documents in
a given field
– At query time, lookup ID values from a given field to filter documents from
another index

Query-time join
● Pros:
– Update friendly
– No data duplication
– No index-time data materialisation
● Cons:
– Slower than parent-child
– No data locality principle: network transfer
name : Elastic
country_code : A
...
round_code : A
founded_year : 2012
...
Type : Org
round_code : B
founded_year : 2013
...
Type : Org
Name : Benchmark
Type : Org

● Each solution has its own advantages and disadvantages
– Trade-off between performance, scalability and flexibility
BlockJoin SIREn Parent-Child FilterJoin
Performance ++ ++ + -
Scalability + ++ + +
Flexibility - - + ++
Best for ●Simple nested
model
●Fixed data
●Complex nested
model
●Fixed data
●Simple nested
model
●Dynamic data
●Relational model
●Dynamic data
Summary

Pivot Browser
Knowledge Browser
Crunchbase Demo

Contact Info
76 Tudor Lawn, Newcastle
info@siren.solutions
siren.solutions
We're hiring!

Searching Relational Data with Elasticsearch

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (9)

Similar to Searching Relational Data with Elasticsearch

Similar to Searching Relational Data with Elasticsearch (20)

Recently uploaded

Recently uploaded (20)

Searching Relational Data with Elasticsearch

Editor's Notes