Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
-
Upload
lucidimagination -
Category
Documents
-
view
216 -
download
0
Transcript of Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
1/48
Realtime revolution at workREAL-TIME SEARCH AT YAMMER
May 25, 2011
By Boris Aleksandrovsky
http://www.linkedin.com/in/baleksan
Yammer, Inc.
http://www.linkedin.com/in/baleksan
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
2/48
2
Communication is hard, search is harder
What me grammar? Private language
Conversational language
Time compressed
Transient
Poorly organized
Authority is suspect
Social pressures
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
3/48
3
3
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
4/48
4
Information
Facts
Knowledge
Attention
Engagement
Retention
Challenges - From information to knowledge
4
Messages
Metadata
Personalized Search
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
5/48
5
Agenda
Background
Why search?
Indexing
Search
Tools and methodologies
Lessons learned
Future
Q&A
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
6/48
6
: Putting Social Media to Work
Knowledge
Management:Document-oriented
Enterprise
Collaboration:Outcome-focused
Social Media:People-centric
Yammer makes work
Real-time, Social, Mobile
Collaborative, Contextual
More Human!
Similar to:
Facebook
Twitter
Wikis
Groups
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
7/48
7
Yammer: The Enterprise Social Network
Messaging and Feeds
Direct Messaging
User Profiles
Company Directory
Groups (Internal)
Communities (External)
File Sharing
Applications
Integrations
Web, Desktop, Mobile, Tablet
Translations
Network Consultation and
Support
Easy. Shared. Searchable. Real-time. Where your companys knowledge lives.
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
8/48
8
100,000+ companies, including 85% of the Fortune 500 and growing.
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
9/48
9
What do you discuss at work, and with whom?
What do our
employees think of
our401K program? Is
everybody saving?
Whats the latest
with the XYZ
account?
What are ourrecommendations for
financial and
regulatory reform
given the latest news
about?
What will be
discussed at our
Quarterly Sales
Kickoff?
Where can I find out
more about customer
events here at the ABC
conference? Whos free
to meet up?
How can my team
better prepare for
our next product
release?
Who has
any fresh
ideas for
Who do you need to communicate with, across the company?
How often are the same questions asked?
Who has the answers? Who has new ideas? Who can help?
Who will I be
working with on
this new project?
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
10/48
1010
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
11/48
11
Search use case - Transient Awareness
Reverse-chronological
Simple queries
Facet
Date
Sender
Group
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
12/48
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
13/48
1
3
Challenges for Yammers search engine
More knowledge is generated in realtime
Availability latency < 1 sec
Not always well formed
Complicated relevance story
experts and their reputation
popularity
social graph
tagging/topics
engagement signals
timeliness location
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
14/48
1
4
Team
2 engineers 8 man months
Lots of fun
1
4
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
15/48
15
Indexing
DB to replica
15
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
16/48
1
6
1
6
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
17/48
1
7
Replication
Independent near-replicas based on a single distributedsource of truth
Can (will) get out of sync
Automatic monitoring of replication quality
Are replicas out of sync with other replicas?
number of docs
alert > X
Are replicas out of sync with the DB?
statistical sample of docs
1
7
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
18/48
1
8
Indexing
In-replica to index
1
8
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
19/48
1
9
1
9
30s
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
20/48
20
Why is it hard?
No timeliness guarantee
Fragmentation
Out-of-order deliveries
Index dependencies Need to denormalize the information
Need to build for network partition tolerance and redundancy
But Eventual consistency
Eventual delivery
20
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
21/48
21
How do we cope?
Out of order delivery source of (most) evil
?
A) Assure in-order delivery
buffer and wait
degrades performance, availability and timeliness and is onlyvery eventual consistent
B) Minimize probability and ignore
timestamp precision
clock skew
C) Arbitrate timestamp / vector clocks
semantics
need to index lifecycle events
Need to build for network partition tolerance and21
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
22/48
22
Delete-update race
[create Message hello id=5 ts=12:34:39]
[delete Message hello there id=5 ts=12:45:01]
[modify Message hello there id=5 ts=12:45:01]
22
id timestamp tombstone
5 12:34:39 no
5 12:45:01 yes
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
23/48
2
3
Multiple update race
[create Message hello id=5 ts=12:34:39]
[modify Message hello there now id=5 ts=12:45:01]
[modify Message hello there id=5 ts=12:45:01]
2
3
id timestamp text
5 12:34:39 hello
5 12:45:01 hello there now
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
24/48
2
4
Dupes
[create Message hello id=5 ts=12:34:39]
[like Message id=5 userId=3 ts=12:45:01]
[like Message id=5 userId=3 ts=12:45:02]
[unlike Message id=5 userId=3 ts=12:45:04]
2
4
id timestamp numLikes
5 12:34:39 0
5 12:45:01 1
5 12:45:02 1
5 12:45:04 0
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
25/48
25
Thread example
25
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
26/48
2
6
Zoie
Realtime indexing system Open sourced by LinkedIn
Used by LinkedIn in production for about 3 years
Deployed at dozen or so locations
Thanks Xiaoyang Gu, Yasuhiro Matsuda, John Wang and Lei Wang
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
27/48
2
7
Zoie
Push events into buffer and the transaction log Push buffer into Zoie
When Zoie commits, transaction log is truncated.
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
28/48
2
8
Indexing HA
Cluster queue systems
Round-robin of Rabbits introduce further out-of-order
problems.
Transaction log
Between RabbiMQ dequeue and Zoie disk commit
2
8
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
29/48
2
9
Dual indexing
Primary for serving out
Secondary for reindexing
Verify secondary index consistency
foreach replica do
shutdown
mv secondary to primary
restart
Availability should not be affected except for slight
chance of system failure
2
9
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
30/48
3
0
Index consistency problems
Detect
integrity check against the :source of truth:
Reindex
gaps
whole
reindex into secondary, swap with primary
Repair
patch in place
run on restart
3
0
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
31/48
3
1
Search
3
1
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
32/48
3
2
Goal
50/50-500/100 per partition
50M docs
50 msec P75 - 500 msec P99
100 qps
3
2
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
33/48
3
3
REST-full API over HTTP
http://search.yammer.com:8085/api/search/1/1?query=i&start=0&pageSize=5&f=date,05242001
3
3
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
34/48
3
4
Payload
Payload is usually small json object For security reasons only ids and scores are send out
One page (usually 10 items) x 6 index types.
3
4
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
35/48
3
5
Payload
3
5
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
36/48
3
6
Web Server
Jersey over Jetty
http://jetty.codehaus.org/jetty/
Custom configuration
tuned to the required 100 qps
generally impeccable, occasional lock contention
http://jsr311.java.net/
Annotation driven
Much easier to test
3
6
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
37/48
3
7
Search master
More like a router
Knows about partitioning scheme
Performs load normalization
Call all, take the first
Possible to use multicast
Round Robin
switch to for scale
DLB (Least busy)
Maintains primary SLA metrics
3
7
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
38/48
3
8
Partitioning
Simple Jenkins 64bit hash of networkId
2 level hash to split large partitions
Exception list to split large partition
Limitation: Cannot partition inside a single network
Repartitioning story is expensive
Consistent hashing?
3
8
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
39/48
3
9
Testing
Indexing
Idempotent
Out-of-order delivery
Duplicate and incomplete docs tolerance
10K docs delivered in random order with X% of
dupes and Y% incomplete records
Search
Small manual index by recording event
Unit style tests (testng) with Asserts
3
9
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
40/48
4
0
Production
Measure
Hardware is cheap, people are not
People require more maintenance
Have enough redundancy
4
0
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
41/48
4
1
Metrics
JVM, Queue, Logging and Configuration
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
42/48
4
2
Metrics
Gauges
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
43/48
4
3
Metrics
Meters
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
44/48
4
4
Metrics
Timers
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
45/48
4
5
Metrics
https://github.com/codahale/metrics
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
46/48
4
6
Lessons
Do not underestimate your data model Tradeoff between consistency, RT availability and
correctness
Measure
Flexible partitioning scheme Data recovery plan
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
47/48
4
7
Future
Dynamic routing Zookeeper
Partition rebalancing
Multiple sub-partitions with different SLAs
Work on relevancy
Multiple languages
Document parsing
External data
Scala
-
8/6/2019 Aleksandrovskyboris Real Time Search at Yammer 110531210420 Phpapp01
48/48
4
Q&A Session: Whats On Your Mind?