How Lucene Powers the LinkedIn Segmentation and Targeting Platform

How Lucene Powers LinkedIn Segmentation & Targeting Platform

Lucene/SOLR Revolution EU, November 2013 Hien Luu, Raj Rangaswamy

About Us

Hien Luu Rajasekaran Rangaswamy

Agenda

§  Little bit about LinkedIn §  Segmentation & Targeting Platform Overview §  How Lucene powers Segmentation & Targeting

Platform §  Q&A

Our Mission Connect the world’s professionals to make them

more productive and successful.

Our Vision Create economic opportunity for every

professional in the world.

Members First!

The world’s largest professional network Over 65% of members are now international

Company Pages

Languages

>90% Fortune 100 Companies use LinkedIn Talent Soln to hire

Professional searches in 2012

Other Company Facts

•  Headquartered in Mountain View, Calif., with offices around the world! •  LinkedIn has ~4200 full-‐Kme employees located around the world

Source : http://press.linkedin.com/about

SegmentaKon & TargeKng

Segmentation & Targeting

Segmentation & Targeting Attribute types

Bhaskar Ghosh

1. Create attributes

§  Name §  Email §  State §  Occupation §  Etc.

2. Attributes Added to Table

Name Email State OccupaEon …

John Smith jsmith@blah.com California Engineer

Jane Smith smithj@mail.com Nevada HR Manager

3. Create Target Segment: California, Engineer

Name Email State OccupaEon

John Smith jsmith@blah.com California Engineer

Jane Doe jdoe@email.com California Engineer

4. Export List & Send Vendor

Jane Doe jdoe@email.com California Engineer

§  Business definition – Business would like to launch new campaign

often – Business would like to specify targeting criteria

using arbitrary set of attributes – Attributes need to be computed to fulfill the

targeting criteria – The attribute data resides on Hadoop or TD – Business is most comfortable with SQL-like

language

Attribute Computation

Engine

Attribute Serving Engine

Attribute Computation

Engine

Self-service

Support various data sources

Attribute consolidation

Attribute availability

Attribute computation

Attribute Serving Engine

Self-service

Attribute predicate expression

Build segments

Build lists

Serving Engine

count filter sum complex

expressions Σ 1234

LinkedIn Member Attribute table

LinkedIn Segmentation & Targeting Platform

Who are north American recruiters that don’t work for a competitor?

Who are the LinkedIn Talent Solution prospects in Europe?

Who are the job seekers?

LinkedIn Segmentation & Targeting Platform

Complex tree-like attribute predicate expressions

Agenda

§  Architecture –  Indexer Architecture –  Serving Architecture

§  Load Balanced Model §  Next Steps - Distributed Model §  DocValues §  Lessons Learnt §  Why not use an existing solution?

Architecture

StorageLayer

AttributeCreationEngine

AttributeMaterialization

EngineAttributeComputationEngine

AttributeMetastore

AttributeIndexingAttribute

ServingEngine

AttributeServingEngine

LuceneOutputFormat RecordWriter LuceneDocumentWrapper

Document

Index Merger

Web Servers

Indexer

shard 1

shard 2

shard n

Avro data in HDFS

mysql attribute

Hadoop Indexer MR

Attribute Definitions

Mapper K=> AvroKey<GenericRecord> V=> AvroValue<NullWritable> Reducer K=> NullWritable V=> LuceneDocumentWrapper

JSON Predicate Expression

JSON Lucene Query Parser

Inverted Index

Segment & List

Serving

Agenda

Serving – Load Balanced Model

Shard 1

Shared Drive

Shard 2 Shard n

Web Server 2 Web Server nWeb Server 1

Load Balancer

HTTP Request

Serving – Load Balanced Model

But Wait…..

•  Is load balancing alone good enough?

•  What about distribution and failover?

Agenda

Next Steps - Distributed Model

•  A generic cluster management framework

•  Used to manage partitioned and replicated resources in

distributed systems

•  Built on top of Zookeeper that hides the complexity of ZK

primitives

•  Provides distributed features such as leader election, two-

phase commit etc. via a model of state machine

http://helix.incubator.apache.org/

Shard 1

Shard 2

Web Server 2 Web Server 3Web Server 1

Load Balancer

HTTP Request

Scatter Gather

active

standby

Shard 2

Shard3

active

standby

Shard 3

Shard1

active

standby

Shard 1

Shard 2

Web Server 2 Web Server 3Web Server 1

Load Balancer

HTTP Request

Scatter Gather

active

standby

Shard 2

Shard3

active

Shard 3

Shard1

failure

Agenda

DocValues – Use Case

•  Once segments are built, users want to forecast, see a

target revenue projection for the campaigns that they want

to run.

•  Campaigns can be run on various Revenue Models

•  This involves adding per member Propensity Scores and

Dollar Amounts

DocValues – Why not Stored Fields?

Why not use Stored Fields?

•  Stored fields have one indirection

per document resulting in two disk

seeks per document

•  Performance cost quickly adds up

when fetching millions of documents

Document ID

.fdx fetch filepointer to field data

.fdt scan by id until field is found

DocValues – Why not Field Cache?

Why not use Field Cache?

•  Is memory resident

•  Works fine when there is enough memory

•  But keeping millions of un-inverted values in memory is impossible

•  Additional cost to parse values (from String and to String)

DocValues

•  Dense column based storage (1 Value per Document and 1 Column

per field and segment)

•  Accepts primitives

•  No conversion from/to String needed

•  Loads 80x-100x faster than building a FieldCache

•  All the work is done during Indexing

•  DocValue fields can be indexed and stored too

Agenda

Lessons Learnt

Indexing •  Reuse index writers, field and document instances •  Create many partitions and Merge them in a different process •  Rebuild (bootstrap) entire index if possible •  Use partial updates with caution •  Analyze the index Serving •  Reuse a single instance of IndexSearcher •  Limit usage of stored fields and term vectors •  Plan for load balancing and failover •  Cache term frequencies •  Use different machines for Serving and indexing

Agenda

Why not use an existing solution?

•  Doesn’t allow dynamic schema •  Difficult to bootstrap indexes built in

hadoop •  Indexing elevates query latency

•  Doesn’t allow dynamic schema •  Difficult to bootstrap indexes built in

hadoop •  Larger memory overhead •  Comparatively slow

Questions? More info: data.linkedin.com

How Lucene Powers the LinkedIn Segmentation and Targeting Platform

Technology

Transcript of How Lucene Powers the LinkedIn Segmentation and Targeting Platform

Segmentation & Targeting

Mm Segmentation and Targeting

Targeting Research: Segmentation

Steps in Market Levels of Segmentation Segmentation ...kisi.deu.edu.tr/ozge.ozgen/Segmentation, Targeting Positioning.pdf · 14.11.2012 1 Chapter 7 Segmentation, Targeting, and Positioning

Segmentation and Targeting

PPT5- Segmentation Targeting & Positioning

NIKE - Segmentation & Targeting

Segmentation of Targeting

Marketing 7 Segmentation & Targeting. 7.1 Segmentation and Targeting - 7 Bases for segmentation Requirements for effective segmentation Segmentation and.

Segmentation, Targeting & Positioning. Why do this? Overview: Segmentation, Targeting & Positioning.

Steps in Market Levels of Segmentation Segmentation ... Targeting... · 14.11.2012 1 Chapter 7 Segmentation, Targeting, and Positioning Definition •Market Segmentation: –Dividing

Segmentation, Targeting and PositioningTargeting Positioning Segmentation, Targeting and PositioningTargeting Positioning.

Bru segmentation targeting positioning

Segmentation and Targeting Based on Social Architecture · Social segmentation and targeting improves brand performance Applying only traditional stakeholder segmentation and targeting

Segmentation Targeting Positioning

Market Segmentation & Targeting

segmentation, targeting

Segmentation Targeting Postioning

Segmentation & Targeting (ppt)

Segmentation, targeting n positioning