Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail...
-
Upload
charity-daniel -
Category
Documents
-
view
223 -
download
2
Transcript of Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail...
Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution
Malik Magdon-IsmailRensselaer Polytechnic
Institute
2
Our Group
M. Goldberg M-I B. Szymanski A. Wallace
Students: Mykola Hayvanovich Apirak Hoonlor Stephen Kelley Konstantin Mertsalov
3
Motivation
Communications supporting IED planning have patterns and are correlated….
Analysis of the patterns can reveal the groups as well as their internal group structure.
4
CommunicationsTime: January 12, 2005, 09:35
From: [email protected]
Subject: Hello
Message: Where have you been?
16:06:31] <FreeTrade> Republicans were the worst pacifists before ww1 and ww2[16:06:43] <SweetLeaf> France Fries[16:06:50] <FreeTrade> As a generality, of course their were Republican Hawks.[16:07:13] <FreeTrade> Sweet, good pun but bad story![16:07:18] <SweetLeaf> yup[16:07:23] <Lupine> anyways, he's perpetually tormented by presidential actions[16:07:25] <SweetLeaf> it aint good for no one[16:07:47] <SweetLeaf> I think they knew it was commiing[16:07:51] <FreeTrade> Rossevelt met monthly in New York with mostly trusted Republicans to talk about how to get america into the war.[16:08:10] <FreeTrade> and he spent 2 year with Churchill meeting him sometimes secretly in the ocean to discuss the same topic.[16:08:22] <FreeTrade> Exchanging a lot of letters.[16:08:25] <FreeTrade> telegrams[16:08:28] <Lupine> There really is nothing like a shorn scrotum. It's breathtaking, I suggest you try it.[16:08:55] <FreeTrade> Well they didnt literally meet in the ocean, they were on ships.
5
Streaming ExampleTime From To Message10:00 Alice Charlie Golf tomorrow? Tell everyone.10:05 Charlie Felix Alice mentioned golf tomorrow.10:06 Alice Bob Hey, golf tomorrow. Spread the word.10:12 Alice Bob Tee off: 8am at Pinehurst.10:13 Felix Grace Hey guys, golf tomorrow.10:13 Felix Harry Hey guys, golf tomorrow.10:15 Alice Charlie Pinehurst Tee time: 8am.10:20 Bob Elizabeth We’re playing golf tomorrow.10:20 Bob Dave We’re playing golf tomorrow.10:22 Charlie Felix Tee time 8am at Pinehurst10:25 Bob Elizabeth We tee off 8am at Pinehurst.10:25 Bob Dave We tee off 8am at Pinehurst.10:31 Felix Grace Tee time 8am, Pinehurst.10:31 Felix Harry Tee time 8am, Pinehurst.
6
Streaming ExampleTime From To Message10:00 Alice Charlie Golf tomorrow? Tell everyone.10:05 Charlie Felix Alice mentioned golf tomorrow.10:06 Alice Bob Hey, golf tomorrow. Spread the word.10:12 Alice Bob Tee off: 8am at Pinehurst.10:13 Felix Grace Hey guys, golf tomorrow.10:13 Felix Harry Hey guys, golf tomorrow.10:15 Alice Charlie Pinehurst Tee time: 8am.10:20 Bob Elizabeth We’re playing golf tomorrow.10:20 Bob Dave We’re playing golf tomorrow.10:22 Charlie Felix Tee time 8am at Pinehurst10:25 Bob Elizabeth We tee off 8am at Pinehurst.10:25 Bob Dave We tee off 8am at Pinehurst.10:31 Felix Grace Tee time 8am, Pinehurst.10:31 Felix Harry Tee time 8am, Pinehurst.
7
Streaming ExampleTime From To10:00 Alice Charlie10:05 Charlie Felix10:06 Alice Bob 10:12 Alice Bob 10:13 Felix Grace10:13 Felix Harry10:15 Alice Charlie 10:20 Bob Elizabeth10:20 Bob Dave10:22 Charlie Felix10:25 Bob Elizabeth 10:25 Bob Dave10:31 Felix Grace10:31 Felix Harry
8
Overview: SIGHTS & RDM
b u y , t r a d e . . . b u y
2 t r a d e . . . 2 t r a d e
3 , s e l l . . . 3 , h e l l
Pattern id = 2Pattern = “buy,”
Pattern id = 3Pattern = “2trade”bb
Level 0
Level 1
Level 2
Higher ranked leaders
Group leader
Subgroup leaders
Members
9
Communications
Email, Telephone, Newsgroup, Weblog, Chatrooms, …
Time: January 12, 2005, 09:35
From: [email protected]
Subject: Hello
Message:
Where have you been lately?
Time: January 12, 2005, 09:35
From: [email protected]
Subject: Hello
Message:
Where have you been lately?
11
Communication Graph
What are the social groups/coalitions?
12
Social Groups are Clusters
Clusters may overlap.
13
Social Groups are Clusters
Clusters may overlap.A cluster is a locally defined object.
14
Social Groups are Clusters
Clusters may overlap.A cluster is a locally defined object.
Group members are more introverted than extroverted.
YES NO
15
Social Groups are Clusters
Clusters may overlap.A cluster is a locally defined object.
Group members are more introverted than extroverted.
Social groups (clusters) persist
16
SIGHTSStatistical Identification of Groups Hidden in Time and Space
- System for statistical analysis of social coalitions in communication networks
Data SourcesBlogsEmails (Enron)ChatroomSynthetic data
Coalition DiscoveryOverlapping ClusteringStreaming groupsPersistent groups.
Coalition AnalysisLeadersOpposing groupsTopic matching
VisualizationsSize-Density plotsStatic coalitionsDynamic coalitions
Groups matching analyst topic in red
Size vs. Density Plot
Visualization options
Choose time window
Groupmembers
Different analyses on dataset
Leader index
17
Examples
Two clusters: Electric circuit design; Optimization of Neural Networks:
Intersection: “Sensitivity analysis in degenerate quadratic programming”
Citeseer
ENRON
GROUND TRUTH Group A
Dog Vulture Camel Yassir Hussein Bird (6 others)
Group B Ahmet Saleh Sarwuk Shaid Pavlammed Pavlah Osan Domenik
SIGHTS Group A
Dog Vulture Camel Gopher
Group B Ahmet Saleh Sarwuk Shaid Dajik
Ali Baba Data Set (DoD)
18
Build a classifier to identify the relationship between sender and receiver of a message
EXAMPLE:“Do you have time to meet some time this week?”
“Lets meet 2pm today, ok?”
Which is advisor, which is student?
Recursive Data Mining (RDM)
19
Hierarchical Pattern Construction (recursive definition)Captures patterns; patterns of patterns; patterns of patterns of
patterns… (can even capture long-range patterns)
Pattern Definition
Larger patterns
b u y , t r a d e . . . b u y
2 t r a d e . . . 2 t r a d e
3 , s e l l . . . 3 , h e l l
Pattern id = 2Pattern = “buy,”
Pattern id = 3Pattern = “2trade”
Pattern id = 4Pattern = “3,_ell”
Level 0
Level 1
Level 2
20
Ensemble of classifiersClassifier for each level in the hierarchical approachFeatures gathered from the training messagesGlobal features include average length and number of sentencesApproximate matching allows treatment of noise
A Classifier – Joining the Pieces
21
Binary classification: for a given message m,
is m sent by a person with role r?
r є {CEO, Manager, Trader, Vice-President}
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
CEO Manager Trader Vice-pres.Roles
1 -
F_
sco
re
NB RDM_NB SVM RDM_SVM CPAR
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
CEO Manager Trader Vice-pres.
Roles
1 -
F_
Sco
re
NB RDM_NB SVM RDM_SVM CPAR
Multi-classification: for a given message m,
which role r is the most likely for the sender?
r є {CEO, Manager, Trader, Vice-President}
The bars show the error of classification. Universally RDM_SVM outperforms other classifiers
Results on Enron
22
Summing Up
SIGHTS:Structural; non-semantic; language
independentFinds groups, their dynamics and
structure; visual analytic capabilities.RDM
Uses statistical semantics; language independent
Identifies roles within the group
23
Thank You
http://www.cs.rpi.edu/~magdon