Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf ·...
Transcript of Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf ·...
![Page 1: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/1.jpg)
Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion
William Rand
in collaboration withDavid Darmon, Jimpei Harada, Jared Sylvester, PK Kannan,
Arjuna Ariyaratne, and Michelle Girvan
![Page 2: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/2.jpg)
– First research center focused on the application of complex systems to business and management science
–Working toward a better understanding of emergent patterns arising from nonlinear interactions of individual components
– Examines tools such as: machine learning, agent-‐based modeling, social network analysis, geographic information systems, systems dynamic modeling, and big data tools
Complex Systems and Data Science
http://www.rhsmith.umd.edu/ccb/
Supporters
![Page 3: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/3.jpg)
The Beauty of Big Data
• Individuals leave their footprints (and fingerprints) in the digital sand.
• These traces represent digital / social signals of human behavior.
• We can use these signals to explore the complexity of individuals and digital environments.
![Page 4: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/4.jpg)
The Challenge of Big Data
• What do we do with all this data?
• Aggregation of the data eliminates the richness of it
• The real benefit is when the data is used at the individual level
• How to make sense of that individual-level information?
![Page 5: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/5.jpg)
Agent-Based Modeling
• Agent-Based Modeling provides a way to understand individual-level interactions
• Traditionally, agent-based models use simple rules derived from theory
• If we could create ABMs directly from big data we would have an individual-level detailed model derived directly from digital traces
![Page 6: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/6.jpg)
A Solution
• Use Machine Learning to derive individual-level rules for agents automatically.– Causal State Model (CSM)
• Then use these rules to construct an agent-based model.
• Use this model to forecast individual behavior.
![Page 7: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/7.jpg)
Understanding the Predictive Power of Computational Mechanics and Echo State Networks in Social Media
David Darmon, Jared Sylvester, Michelle Girvan, and William Rand
ter.ps/compmech
![Page 8: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/8.jpg)
Predicting Twitter Engagement Using ML
Opportunity: Social Media is a great marketing channel.Challenge: However, there is a lot of noise, and its not
apparent when users are paying attention.Solution: Understand when users are active.Goal: Create a tool that can accurately predict when
different segments will post content.
Motivation
![Page 9: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/9.jpg)
Twitter users embedded in a 15k user follower network.
Statuses of all users collected over 7 weeks.
Select 3k subset of most frequently tweeting users.
The Dataset
Predicting Twitter Engagement Using ML
![Page 10: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/10.jpg)
2013-08-22 12:54:06 Is Your Gmail Social? How to Use Gmail Daily to Build an Engaged Community | Socially Sorted http://t.co/WVprF4bLPa via 2013-08-22 13:11:22 Facebook's Embedded Posts Now Available to Everyone http://t.co/Cc0q3cSTt62013-08-22 13:14:06 The Credible Hulk http://t.co/q17VrcSdBs2013-08-22 13:29:02 25 Things You Didn’t Know About Ninjas http://t.co/CcJz92sRyy2013-08-22 13:32:59 Twitter Users: Revoke and Reestablish Third Party App Access Now http://t.co/qL9LbnGO6e via @ShellyKramer2013-08-22 13:48:46 10 Brilliant Facebook Marketing Tactics to Increase Reach and Engagement http://t.co/Z4YsudlJ35 via @kathikruse2013-08-22 14:17:11 Google Now Adds Cards for NCAA Football Scores, Concert Tickets, Car Rentals and More http://t.co/BG6Yz2zDyU2013-08-22 15:18:03 What is the NSA Really Up To? [COMIC] http://t.co/hakoq1mFNX2013-08-22 15:39:04 6 Things Every Good Business Blog MUST Have http://t.co/KRGzZRCrbm
Tweet TextTimestamp
User: DanielZeevi
The Setup
Predicting Twitter Engagement Using ML
![Page 11: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/11.jpg)
Bin (in time) Twitter data, giving a discrete time series for each user v at time t:
The Setup
0 1 0 1 1 1 0 1 1 1… …
— user v doesn’t tweet— user v tweets
X(v , t) = 0
X(v , t) = 1
Time
Predicting Twitter Engagement Using ML
![Page 12: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/12.jpg)
Creating Epsilon Machines (Shalizi and Crutchfield, 2001; Crutchfield, 1994)
Assume was generated by a conditionally stationary stochastic process.
Explicitly learn the predictive distribution by grouping together pasts x that give equivalent predictions.
{Xi}Ni=1
P (Xi |X i�1i�L = x)
The Computational Mechanics Framework
A B
C
1
0
1
01
Causal State Model (CSM) built for each user usingCausal State Splitting Reconstruction (CSSR)
Begin with one state and divide as necessary.
![Page 13: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/13.jpg)
Call the model learned the
causal state model (CSM)
for each user. CSM has the unique property of being “maximally predictive and minimally complex”
Learn this state-space representation of the process using
Causal State Splitting Reconstruction (CSSR).
The Computational Mechanics Framework
![Page 14: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/14.jpg)
User: OcioNet
![Page 15: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/15.jpg)
User: HadiJayaPutra
![Page 16: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/16.jpg)
●●●●●●●●●●
●●●
●
●●●
●
●
●●
●●●●●
●●
●
●
●
●
●●●●●●●●
●●●●
●●●●●
●●●
●
●
●●
●●
●●●
●
●●●●●●
●●●●●●
●
●●
●●●●●●●●●
●●●●
●
●●●●●●●●●●●●
●
●●●●
●●●
●●●
●
●●●●
●
●
●
●●●
●
●●●
●●●●●
●●
●
●
●●●●
●●●●●
●●●
●
●●
●
●●
●
●●
●●
●
●●●●●●
●
●
●
●
●●●●●●●●
●●
●●●●
●●
●●●●
●●●●●
●
●
●
●
●
●●
●
●●
●●●
●
●
●●
●●
●
●●
●●●●●●
●●
●
●
●
●●●●
●
●
●
●●
●
●
●●●●●●●●
●
●●
●
●
●
●●●
●
●
●
●
●●●●
●●●
●
●
●●
●
●●●●
●
●
●●
●
●
●●
●
●●●●●●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●●
●●●●●●
●●●
●●●●●●●
●
●
●
●
●
●
●
●●●●●
●●
●
●
●
●●●●●●●●●●●
●●●●
●
●
●●●●
●
●●●●●●
●●
●
●
●●●●
●
●●●●●●
●●●●
●
●●●
●●●●●●
●●●●●●●
●
●●●
●●●
●●
●
●●●●
●●●
●
●
●●●●●●
●
●●●
●●
●●●●●●
●●●●●●●●●●●
●●●●
●●●
●●●
●
●
●
●
●●●
●
●●
●
●●
●
●●●●●●●●●●●●●
●
●●●●
●
●●
●
●●
●●●●
●●
●●
●●●●●●●●
●●●●●
●●●●●●
●
●●
●●●●●
●
●●
●●
●●
●●●
●●
●●●●●●●●●
●
●●
●
●●●●
●
●●●●●●●●
●●●●●
●
●●●●●
●
●●●
●
●
●
●●
●●
0 100 200 300 400 500 600
010
0AnCS
M
●●●
●●
●
●●●
●
●
●●●
●●
●●●
●
●
●●
●●●●
●
●●
●●●●●●●●
●●●●●
●●●●
●
●
●
●
●
●
●
●●●●●
●
●●
●●●
●
●
●●●
●
●●●●
●●●
●●
●
●
●●
●
●
●●●●
●●●●●●
●●
●
●●
●●●
●●●●●
●
●
●●
●●●●●●●●
●●
●
●
●●
●
●
●
●
●
●●
●●●
●●●●●●●●●
●●●
●●●●
●
●●●
●
●●
●●
●
●
●●
●
●
●●●●
●●●●●●●●
●
●
●
●
●●
●
●
●
●●●
●●●●●
●
●●
●●●●●●●
●
●
●
●
●
●●●●
●
●●●●
●●●●
●●●●
●●●
●
●
●●
●
●
●
●●●●
●
●
●●●●●●●
●
●
●●●●
●
●●●●●●●
●●●
●●●
●
●●●●
●●●●●
●
●
●
●
●
●
●
●
●●●
●●
●
●●●●
●●●●
●
●●●
●
●●
●
●
●
●
●●●●
●
●
●
●
●
●●●●●
●
●●●●
●●●
●●
●
●●
●●
●●●●
●●●
●
●
●●●●●●●●●●
●●●
●●●●
●
●
●●●●●●●●
●●
●●●●●●●●●●●
●
●●●●
●
●
●●●●
●
●
●
●●●●●
●●●●●●
●●●●●
●●
●
●●
●
●
●
●●●●●●
●
●●●
●
●
●●●●●
●
●●●●●●●●
●●●
●●
●●●●
●
●●
●●●
●●●
●
●●●
●●●●
●●●
●●●●
●
●
●●●●●●●
●●
●●●●●●●●
●●●●●
●
●●●●●●●
●●●●●
●
●●●●
●
●●●●●●●●●
●●
●
●●
●●●●●●●●●
●
●●●●●
●
●●●●●●●●●
●●
●
●●●●●
●●●
●
●
●
●●●
●
●●●●●●●
●●●●●
0 100 200 300 400 500 600
010
0AnCS
M
●●●●●●●
●
●●●
●●
●
●
●●●●
●●●
●
●●●
●●
●●●●
●●●●
●
●●
●●●
●●
●
●
●●●
●●●●●●●●●●●●●●
●
●●
●
●
●●
●●
●
●●●
●
●
●●
●●●●●●
●●
●
●●●
●
●
●●
●
●●
●●●
●
●
●
●
●●●●
●●●●●
●
●●
●
●●●●
●
●●
●●
●●
●●
●
●●●
●
●●●●
●
●●
●
●
●
●●
●
●
●
●●●●
●
●●●
●●●
●●
●
●●
●●
●
●
●●
●
●●●●
●●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●●
●●●
●●
●●
●
●●●●●●
●●●●●
●
●●
●
●●●●●●
●●
●
●
●●
●●
●●
●●
●●●●●
●●●
●●●●
●
●●
●●●●
●
●●●●
●●●●●●●●●●●●
●●
●
●●
●
●●●●●
●●
●●●●
●●
●
●
●●●●
●●●
●●●●
●●●●●
●●
●
●●●
●
●
●
●
●●●
●●
●
●
●●●
●
●
●
●●●●●●●●●
●●●
●
●
●●●
●●●●
●●●●●●
●●●●●●
●●
●●●●●●
●●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●●●
●
●●●●
●
●●
●
●●●●●
●●●●●●●●●●●●●
●
●
●●●●●●
●
●●●●●●●
●●●●●●
●●
●
●
●●
●●●
●
●●●●
●
●●●
●
●●
●
●●●
●
●●●●
●●●
●
●●●●●●●●●●
●●
●●●●
●●
●
●●●●●
●●●
●
●●●●
●●●●
●●●
●●●●●
●●●●●
●
●
●●
●
●●●●●●●●
●●
●●
●●●
●●
●
●●
●●●●●●
●●
●●●●●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●●
●
●●●●●
●
●●
●
●
0 100 200 300 400 500 600
010
0AnCS
M
●
●
●●
●●
●
●
●●●
●
●●●●●●
●
●
●●●
●●●●●●●●●●
●
●
●●●●●●
●●●●●
●
●●●
●
●●●●
●●●●●
●
●
●
●
●
●
●●●●
●●●●
●●
●
●
●●
●
●●
●
●●
●●
●●●●●●●
●●
●●●●●●
●
●●●
●●●
●
●
●
●●●●
●●
●
●
●●●●●●
●●
●
●●
●●●
●
●●
●●●●
●
●●
●
●●●●
●●●●
●●●●●●●●●
●
●
●●●●
●
●
●●●●
●
●
●●●
●●●●●●
●
●●●●●
●
●●●●
●
●●
●●
●
●●●●●
●
●
●●●
●●
●
●
●●●
●
●●
●●
●●
●●
●
●●●●
●
●●
●●●
●●●
●●●●
●
●●●●●
●●●●●
●●●●●●●●
●
●●●●●●
●
●●●
●
●
●●●
●●●
●
●
●
●●●●●
●
●●●
●
●●●●●
●
●●●
●
●●●●●
●
●●●
●
●●●
●
●●●●●●
●
●
●
●
●●
●
●
●●●●
●
●●●
●●
●
●
●
●
●●●●●
●●
●
●
●●
●●
●
●
●●●●
●
●●●
●
●●
●
●●●●●
●●●●●●
●●●
●
●
●●●●●●●
●●●●
●●●●●●●●●
●●●●●●●
●●●●●●●●
●●
●●●●●
●●●●
●●
●●●
●●●●●●●●
●
●●
●
●
●●
●●●●
●
●
●●
●●●●●●●
●●
●●
●●●
●●●
●
●
●●
●●●●●
●●
●
●
●●●
●●●●●●
●●
●
●
●●
●
●
●●●●●●
●
●
●●●●
●●●●●●●●●●●
●●
●
●●●
●●●●
●
●
●
●
●
●●●
●
●●●
●
●●●●●●●●●●
●●●●●
●
●●●●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●●
●●●●
0 100 200 300 400 500 600
010
0
Time (∆ = 600 seconds)
AnCS
M
Xn(u
2)X
n(u
2)X
n(u
2)
n (∆ = 600 seconds)
Xn(u
2)
Xn(u
3)X
n(u
3)X
n(u
3)
n (∆ = 600 seconds)
Xn(u
3)
Xn(u
4)X
n(u
4)X
n(u
4)
n (∆ = 600 seconds)
Xn(u
4)
Xn(u
2685)
Xn(u
2685)
Xn(u
2685)
n (∆ = 600 seconds)
Xn(u
2685)
Xn(u
1)X
n(u
1)X
n(u
1)
n (∆ = 600 seconds)
Xn(u
1)
...
+ + + + +
Xn(u1) Xn(u2) Xn(u3) Xn(u4) Xn(u2685)
An
![Page 17: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/17.jpg)
Testing Procedure
• Build model for each user separately
• Training: 45 days
• Testing: 4 days
• Look back 10 steps
• Predict ahead 1 step
• 0-1 Loss
• Compare to “majority vote” baseline
Predicting Twitter Engagement Using ML
![Page 18: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/18.jpg)
CSM vs. ESN
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●●●●
●
●
●
● ●● ●●
●
●
●●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●● ●●
●●●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●●●
●
●●
●
●●●
●
●●
●
●●
●
●
●●
●
●
●
● ●●
●
●●
●
●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●●●●●●
●
●
● ●●
●
●
●
●
●●
●
●
●
●
●
● ●●●●
●
●●●
●
●
●
●●●●●●●
●
●
●
●
●
●●
●
●●●●●
●
●
●
●●●●●
●
●● ●
●
●
●
●
●●●●
●
●● ●●
●
●●
●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●●●●
●
●
●
●
●●●●●●
●●
●
●
●●●
●
●
●●●●● ●●
●
●
●
●●
●
●●●
●
●
●
●
●●●
●
●
●
●●●●
●
●
●●
●●
●
●
●●●●
●
●
●
●
●
●●●
●
●●
●
● ●●
●
●
●●
●
●●●
●
●●●●●●●●●
●
●●●
●
●●
●
●
●
●●
●
●
●
●●●●●●
●
●
●
●
●
●
●●●
●
●●●●●
●
●●●●●●●
●
●●●●●●●●●
●
●●
●●
●
●●●●
●
●
●
●
●●
●
●
●
●
●●●●●●●●
● ●●●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●●
●
●●●
●
●
●●●●
●
●
●●●●●
●
●●●●●●
●
●●●●●●●●●●
●
●●●●●●●●●●●●●●●
●●●●
●●
●
●●●●
●
●
●
●
●
●
●
●●●
●
●●●●●●
●
●●●
● ●
●
●
●
●
●●●●●●●
●
●
●●●●●●●
●
●● ●●
●
●●●
●
●●●
●●●
●●●●
●
●●
●
●
●
●
●●●●●●
●●●●
●
●●
●
●
●
●
●
●●●●●
●●●●
●●
●
●
●●
●
●
●
●●●
●
●●●
●
●
●●●●●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●●●●●●●
●
●●●●●●
●
●●●●●●
●
●●●●●●●●
●
●●●●●●●●●●
●
●
●
●
●●●●●●●●●
●
●●
●
●●●●●●
●
●●●●●●●●●●●●●
●
●
●
●
●●●
●
●●●●●
●
●●●●●●●●●●
●
● ●
●
●
●●● ●●
●
●
●●
●●●●●●
●●
●
●
●
●
●
●
●
●
●●●
●●●● ●●●●●●●●●
●●
●●●●●●●●●●●●●●●●●●
●
●●●●●●
●
●●●
●
●
●
●●●●●●
●
●
●
●●●
●
●
●●●●
●
●●
●
●
●●
●
●
●●●
●
●
●●●●●
●
●
●
●●●●●●●● ●●●
●
●●●
●
●●●●
●
●
●●
●
●●●
●
●
●
●●●●
●
●●●●
●
● ●●
●●●●●●
●
●●● ●
●
●
●●●●●●●●●●
●
●●●● ●
●
●●
●
●●●●●
●●●●
● ● ●●●● ●●● ●●
●
●●●
●●●●●●
●●●●●●●
●
●●●
●
●●
●
●●●●●●●●●●●
●
●
●●
●
●●●●●●●●
●
●●●●●●●●● ●●●●●
●
●
●
●●●●
●
●●●●●●●●
●
●●●●●●●●●
●
●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●
●
●●●●●●●●●●●●●
●
●●●●●●●●●● ●●●●●●●●
●
●
●●
●
●
●
●●●●●
●
●
●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●● ● ●●●●●●●●●●● ●●●●
●●●
●
●
●
●●●●●●●● ●●●●●●●
●
●●● ●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●● ●●●●●●● ●
●
●● ●●
●
●
●
●
●●
●●●●●●●●●● ●●●● ●●
●
● ●●●●
●
●●●●●●●●● ●●●●●●●●
●●●●●●
●
●●●●●●●●●●
● ●●●●●●●●●●
●
●●
●
●●●
●
●
●●●●●●●
●
●●●●●●●●
●
●●●
●
●●●●●●
●
●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●
●
●●●●●
●
●
●
●●●●●●●●●●● ●●●●
●
●●●●●●●
●
●●●●
●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●
●
●●●●●●●●
●●●●●●●●●● ●●●●●●●●●●●●●●
●
●●
●
●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●
●
●●●●●●●●●●●●●
●
●
●
●●●
●
●
●
●
●●●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●
●
●
●●
●
●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●● ●●●●●●●
●
●
●●●●●●●
●
●●●●●●●●● ●●
●
●●●●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●
●
●●●●●●●●●●●●●●●●● ●●●●●●●●●
●
●●
●
●●●●●●●●●● ●●● ●
●
●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●● ●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●
●
●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●
●
●●●●●●●●●●●●●●●
●
●●●
●
●●●●●●●●●●●●●●●●
●
●●●●●●●●●● ●●●●●
●
●● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●
●
●●●●
●
●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●
●
●●●●●●● ●●●
●
●●●
0.0 0.2 0.4 0.6
0.0
0.2
0.4
0.6
CSM Improvement
ESN
Impr
ovem
ent
![Page 19: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/19.jpg)
CSM vs. ESN
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
● ●●
●●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●●
● ●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
● ●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●
●
●
●●
●●●● ●
●
●●
●
●
●
●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
● ●
●
● ●●●
●
●
●
●●
●
●
●
●
●
●
●●
● ●●
●
●
●
●
●●
●
●
●
●
●
● ●●
●
●●
●
●
●●
●●●
●
●
●
●●●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●
●●●
●
● ●
●●●
● ●
●
● ●
●
●●
●●● ●
●
●
●
●
●
●●● ●●●● ●
●
●
●
●
●
●
● ●●
●●●
●
●
●
●●
●●
●●●● ●
●
●
●●●
●●
●
●
●
●
●
●●
●
●
● ●
●
●●
●
●
●●●
●
●
●
● ●●● ●
●
●●●●
● ●●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ●● ●●
●
●
●●● ●● ●●●●
●
●
●
●
●●
●
●
●● ●
●
●●
●
●●●
●●
●
●
●
●
●●
●
● ●●
●
● ●
●
● ● ●●●
●
●
●
●
●●●
●
●
●
●● ●
●
●
●
●●
●
●●●●
●
●● ●
●
●
●
● ● ●● ●● ● ●
●
●
●
●●●
● ●●●● ● ●●
●
●●
●●● ●● ●
●●●
●
●
●● ●● ●●
●
●●
●●●●●
●● ●● ●
●
● ●●●●
●
●
●
●● ●●
● ●●●● ●
● ●
●
●
●
●●● ●●●
●● ●
●
●●●
●●● ●●●●
● ●●
●
●
● ●
●
●● ●
●
●
●
●
●●●
●
●
●
●●●
●●
●
● ●
●
●●
●
● ● ●●
●
●
●
●●
● ●●
●
● ● ●
●
● ●●
●
●●
●
●● ●●● ●● ●●●● ●●
●
●● ●●●
●
●
●
●
●●
●
●
●
●●● ● ●●
●
●
●
●
●
●
● ●●
●● ● ●
●●
●
● ●● ● ● ●●
●
● ●●●● ●● ●●
●
●● ● ●
●●●● ●
●
●● ● ●●●
●
●
●
● ●●● ●● ●●
●
●●● ●
●
●●
●●
●
●
●● ●●
●
● ●
●
●●●●
●
●● ●
●●
●●●●
●
●●●
●●●
●
● ●● ●●●
●
● ●●● ● ●● ● ●●
●
●● ●● ● ●● ● ●● ●●●● ●
●
● ● ●●
●
●
● ● ●●●
● ●
●
●
●
●
● ●●
●
● ●●●● ●
●
●● ●●
●
●
●●
●
●
●
●●●
●●●●● ●●●
●● ●
●
●●
●
●●● ●●
●
●●
●
●●● ● ●● ●
●
●
●
●
●●
●
● ●● ●●●
●
●● ●● ●●●
●
●
●
● ●●●● ● ●●●●
●
●●
● ●● ●
●
●●● ●
●
●●●
●●
● ●● ●● ●
●
● ●●
●●
●●●
●
●
●
●
●
●● ●● ●●● ●●
●
● ●●● ● ●
●
● ●● ●● ●● ● ●● ● ● ● ●●
●
●● ●● ● ●● ● ●●
●●
●
●
●
● ●●●● ●● ●
●
●●●
● ● ●●●●
●
● ●● ●
●
●●● ●●●●●
●
●
●
● ●●●
●
●● ●●
●
●●● ● ●●● ●● ●●
●
●
●●●●
●●●
●●
●
●●
●● ● ●●●
●●
●
●
●
●
●
●●
●
● ●● ●●●●●
● ●●●●●● ●
●●
●●● ● ●● ● ●● ●●●● ●● ●
●●
●
● ● ● ●●●●●●
●●
●
●
●●● ●● ●●●●
●●●
●
●● ●● ●
●● ●●● ●●
●
●●●●
●
●
●● ●●●
●
●
●
● ●● ●●● ●●
● ●●
●
●●●
● ●●● ●
●
●
●●
●
●● ● ●
●
●●●● ●●● ●●●●
●
●●
● ● ●●● ●● ●●●
●●
●
●●●● ●●● ●●
●
●
●●●●
●● ● ●●
● ●●●●●
●●●●
●
●
●●●
●
●
●
●●
●
●
●
●●
●
● ● ●● ●● ● ●● ●●● ●●●
●
● ●
●
●●
● ●● ● ●● ●●●●
●
●●●
●● ● ●● ● ●●
●●
●● ●● ● ●●
●●● ●●●
●
●
●
●
●
●●
●
●● ● ●●●● ●
●
● ● ● ●●●●●●
●●●●
●
●● ●● ●●
● ●● ●●●● ●● ●●● ●●● ●● ●● ●
●
● ●● ●● ●● ● ●●●● ●●● ●● ●●
● ● ●● ●●
●
●● ●●● ●● ●●● ●●●● ●●
●●● ●● ●●●
●
●●●● ● ●●
●
● ● ●●●●● ● ●●● ● ●●● ● ●● ● ●●●● ●●
●●● ●● ●●
●●
●●
●
●
●
●●● ●●
●
●
●● ● ●● ●● ● ●●● ●●
●●
● ●●● ●●● ●● ●●●● ●● ●●
●● ●● ●● ●●● ●● ●●●● ●● ●●● ● ●●
●●
●●● ● ●●● ●●
●
● ●
●
● ●●●
●
● ● ●●●● ●● ●●
●●●● ●● ● ●● ●●
●●●● ●●● ● ●●●● ●● ● ● ●●
●
●●● ●●● ●● ●
●
● ●●● ●●●
● ●●
●●
●
●●
●
●
●●● ●● ●●● ● ●●
●
●●
●
●●● ●●
●●●
●
●
● ● ●●●● ●●
●●● ●● ●●● ●● ●● ●●●●● ●● ●●●● ● ●●
● ●● ●●●● ●
●
●● ● ●●
●●●
●
●
● ●● ●●●● ●● ●●●●● ●●●
●●●
●
● ●● ●● ●
●
● ● ● ●●● ● ●●● ●● ●●●
●
●● ●●●●● ●● ●● ●● ● ● ●●●●
●
● ●● ● ●● ●● ● ●● ●●
●●●●
●
●● ● ●●●● ●● ●● ● ●●●●●● ● ●●●● ●● ●● ●●● ●●● ●●●● ●●●
● ●
●● ●●
●
●● ●● ●● ●●●● ●● ●●●● ●●
●
● ●●●
●●●● ●● ●●●●● ●
●
● ●
●
● ●●● ●● ●●● ●●●
●●● ●● ● ●●● ●● ● ●● ●● ●●● ●●● ●● ●● ●●● ● ●●● ●●●●●●
●
● ●●● ●● ●● ●● ● ●●●● ● ● ●●● ●●●● ●
●
●
●
●●
● ●●● ●● ●●
● ●
● ●●● ●●● ●●● ●● ●● ●●● ● ●●
●● ●●
●●● ●●● ●● ●● ●●● ● ●● ● ●● ●● ●●
●
● ● ●●●●●●●● ●● ●● ●● ●●●●● ●● ●●● ●●● ●● ●● ●●
●●● ●●
● ●● ●●● ●● ●● ●●● ●● ●●●●●●● ●●
●● ●●● ●●
●
●
●● ●● ●●●●●●● ●●●● ●●
●
●
●
●● ●●
●
●
●
●● ●●● ●● ● ●● ●●● ●● ●●●● ●●●●●● ●● ●● ● ●● ●●● ●● ● ●● ●●●●
●
●● ●●● ●● ●
●
●●● ● ●●● ● ●●
● ● ●●●●●
●● ●●● ●● ●●● ●●●● ●● ●●● ● ● ●●
● ●●
●● ●●● ●● ●●●● ● ●● ●●●●●● ●●● ●●●
●
●●●●● ●● ●● ●●●●● ●●●● ● ●● ●●● ●●●●●● ●●● ● ●● ●● ●●●● ●● ●●●●● ●●● ●●●●●●●● ● ●● ●
●
●● ●●●● ●●● ●●●●●
●
● ●● ●●●● ●●● ●●●●● ●●● ● ●● ● ●●● ● ●●● ●● ●●● ● ●●● ●●●
● ●●●●●●●
● ●● ●●●● ●●●●● ●●● ●●●●●● ●●●● ●●●● ●● ●● ●●●● ●● ●● ●●●● ●●●●●● ●●● ●●● ●●
●● ●● ●●●●●● ●
●●
●
●●
●
●●●● ●
●
●● ●● ●● ●● ●● ●●●●● ●● ●●●● ●● ●●● ●●●● ●●●● ●● ● ●●●●● ●●● ● ●●●● ●●●● ●● ●●●●
●
●●●● ●●●●● ● ●● ● ●●● ●●●●●● ●● ●● ●● ●● ●● ●● ●●● ●● ●●●●● ●●●● ● ●● ●● ●●●●●● ●● ●●●●
●●●●● ●●● ●●●●● ●●● ●●●●● ●●●● ●● ●●● ● ●●● ●●● ●● ●●● ●● ●
●
●●● ●● ● ●● ●●● ●● ●●●●●● ●● ●●●● ●●● ●●●● ●●● ●●●●●●
●● ●●● ●●● ●●●● ●●● ●●● ●● ●● ●●●● ●●● ●● ●●●
●●●● ●●●● ●●●● ●●●●●●●● ● ●●● ●● ●●● ●
●●● ●● ●● ●●●● ● ●●●●● ●●●● ●●●●
●● ●●● ●● ●●
●
● ●●●● ●●
●
●●● ●●●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
Tweet Rate
Impr
ovem
ent
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
● ●●●
●
●
●●
●
●
●●●
●
●
●
●●
●
●●●
●
●
●●
●
●●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●●●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●●
●● ●
●
●●
●
●●●●
●●
●
●
●
● ●●●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
● ●●●●
● ●
●
●
●
● ●●
●
●
●
●
●●
●●
●
●
●●
●●
●●
●●●
●
●●
●
● ●●●
●●●
●●
●
●
●●
●
●
●
● ●●
●
●●
●
●●
●●
●
●
●
●●
●●
●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●
●●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●● ●●
●
●
●●
●
●
●
●
●
●
●
● ●
●
● ● ●●●●●
●
●
● ●●
●
●
●
●● ●
●
●
●
●
●
●●●●●
●
●● ●
●
●
●
●● ●
● ●●●
●
●
●
●●
●●
●
●●●●
●●
●
●
●●
● ●●
●
● ●●●
●●
●
●● ●●●
● ●●●
●
●●
●●●
● ●● ●
●
●●
●
●
●
●
●
●
●●
●●●
●
●●● ●●
●
●
●
●
● ●●●●●
●●
●
●
●●●
●
●
●●●●
●● ●
●
●
●
● ●
●
●●●
●
●
●
●
●●●
●
●
●
●● ●●●
●
●●
●●
●
●
● ● ●●
●
●
●
●
●● ●●
●
● ●
●
● ●●
●
●
●●●●●
●
●
● ●● ●●●● ●●
●
●● ●
●●●
●
●
●
●●
●
●
●
●●● ● ●●
●
●
●●●
●
● ●●
●
● ● ●● ●
●
● ●● ● ● ●●
●
●●●
●● ●● ●●
●
●●●
●
●
●●● ●
●
●
●
●
●●
●
●●
●
● ●●● ●●
●●●●
●● ●●●
●●
●
●
●
●●
●
●
●
● ●
●
●
●●●
●
●● ●
●
●
●●●●
●
●● ●
●●●
●
● ●● ●●●
●
● ●●● ● ●●●
●●
●
●● ●● ● ●● ● ●●●
●●● ●● ● ● ●●●
●● ● ●●
●
●●
●
●
●
●
● ●●
●
● ●●●● ●
●
●●●
● ●
●
●
●
●
● ●●
●● ● ●
●
●
● ●●● ●● ●
●
●● ●●
●
● ●●
●
●●●
●●●●
●● ●
●
● ●
●
●
●
●
● ●● ●●
●●●
● ●
●
●●●
●
●
●
●
●●●●●
●●●●●
●●
●
●●●
●
●●● ●
●
●●●
●
●
● ●● ●● ●
●
● ●
●
●●●
●
●●
●
●
●
●
●● ●● ●●● ● ●
●
● ●●● ●●
●
●●● ●● ●
●
● ●● ● ● ● ●●
●
●●●
●● ●● ● ●
●
●●
●
●
●● ●●●● ●● ●
●
●●●
● ● ●● ●●
●
● ●● ●●●●● ●●●●●
●
●
●
●
●●●
●
●● ●● ●
●
●● ● ●●● ●● ●●
●
●●●●
●● ●●●
●
●
●●
●● ● ●●●●●
●
●
●●
●
●●
●
● ●●
●●●● ●● ●●●●●● ●
●●
●●● ● ●● ●●● ●●●● ●● ●● ●
●
●●
● ●●●●
●●●
●
●
●
●●● ●● ●
●
●●
●●●
●●
●●● ●
●
● ●
●
●●
●
●
●●
●●●
●
●● ●●●
●
●●
● ●● ●●● ●●● ●●
●
●●●●
●●● ●
●
●
●●
●
●● ●●
●
●
●●● ●●
●●
●●●
●●●●
● ●●● ●
●
●●● ●
●
●
● ●●
● ●●●●●
●
●●●●●
●●
● ●
●
● ●●●
●●
●●●●● ●●●● ●●● ●●
●
●●●
●●● ● ●●
●● ● ●● ●●
●●●●
●
● ●
●
● ●● ●● ● ●● ●●●●
●
●●
●
●● ●●
● ● ●●
●
●● ● ●● ● ●● ●●● ●●●
●
●
●
●●●
●
●
●● ● ●●●● ●
●
●● ● ●●●●● ●
●
●●●
●
●● ●● ●●● ●● ●●●●
●●
●●●
●●
● ●● ●● ●
●
●
●● ●● ●● ● ●●●● ●●● ●● ●● ● ● ●●●●
●
●●
●●● ●● ●●● ●●●● ●● ●●● ●● ●●●●
●●●● ● ●●
●
● ● ●●●●● ● ●●● ● ●
●
● ● ●● ● ●●●● ●● ●●● ●● ●●
●
●
●●
●
●
●
●●● ●●
●
●
●● ● ●● ●● ● ●●● ●●
●●
● ●●● ●●● ●● ●●●● ●● ● ● ●● ●● ●● ●●● ●● ●●●● ●● ●●●●
● ● ●● ●●● ● ●●
● ●●●● ●●●
●●●
●
●
●
●●●● ●● ●● ●●●● ●● ●●
● ●●●●●● ●●● ● ●●●●
●● ● ● ●
●●
●●● ●●● ●● ● ●● ●●● ●● ●
●●●●●
●
●
●
●
●●
●● ●● ●●● ● ●● ●●● ● ●●
●
●●●●●
●
●● ● ●●●● ●● ●●●●● ●●
●●● ●● ●●●
●● ●● ●●●
● ●●
●● ●● ●●●● ● ●●
●
● ●●
●●●●
●
● ●● ●●●●●
● ●●●●● ●●●
●●●
●
● ●● ●● ●
●
●●
● ●●● ● ●●● ●● ●●● ● ●● ●●●●● ●● ●● ●●
●
● ●●●●
●
●
●
● ● ●●
●● ● ●● ● ●●●●●●
●● ● ●●●●●
● ●● ●
●
●●
●●● ● ●●●● ●● ●● ●●● ●●● ●●●● ●●●
● ●
●● ●●●
●● ●● ●● ●●
●● ●● ●●●● ●●●● ●● ●●●●● ●● ●●●
●
● ●
●
● ●●
● ●●● ●● ●●● ●● ● ●●● ●●
● ●●● ●● ● ●● ●● ●●● ●●● ●● ●● ●●● ● ●●● ●
●●●●●
●● ●●● ●● ●● ●●
●
●●●● ● ● ●●● ●●●●
●
●
●●
●●
●
●
●
●
●● ●●
●●
● ●●● ●●
●●●● ●● ●● ●●● ● ●● ●● ●
●●●● ●●● ●● ●● ●●● ● ●● ● ●●
●●
●●
●
● ●●●●●●●●● ●● ●● ●● ●●●●● ●● ●
●● ●●● ●● ●● ●●●●● ●●● ●● ●●● ●● ●● ●●● ●● ●●
●
●
●●● ●● ●● ●●● ●●●
●
●● ●● ●●●●●●● ●●●● ●● ●●
●
●● ● ●
●
●
●
●● ●●● ●● ● ●● ●●● ●● ●●●● ●●●●●● ●● ●●
● ●● ●●● ●● ● ●● ●●●●●
●● ●●● ●● ●
●
●●● ● ●●● ● ●●● ● ●●●● ● ●● ●●● ●● ●●●
●●
●
● ●● ●●● ●●
●●● ● ●●
●
●●● ●●
●●●● ● ●● ●●●●●● ●●● ●●●
●
●●●●● ●● ●● ●●●●● ●●●● ● ●● ●●● ●●●●●● ●●● ● ●● ●● ●●●● ●● ●●●●● ●●● ●●●●●●●● ● ●● ●●●● ●●
●● ●●● ●●●●●
●
● ●● ●●●● ●●● ●●●●● ●●● ● ●● ● ●●● ● ●●● ●● ●●●
●
●●● ●●●
● ●●●●●●● ● ●● ●●●● ●●●●● ●●● ●●●●●● ●
●
●● ●●●● ●● ●●●
●●● ●●
●● ●
●
●● ●●●●●● ●●●
●●● ●●
●
● ●● ●●●●●● ● ●●●●●
●
● ●●● ● ●●● ●● ●● ●● ●● ●●●●● ●● ●●●● ●● ●
●
● ●●●● ●●●● ●● ● ●●●●● ●●● ● ●●●● ●●●
● ●● ●●●●
●
●●●●●
●●●●
● ●● ● ●●● ●●●●●● ●● ●●
●● ●● ●● ●● ●●● ●● ●●●●● ●●●● ● ●● ●● ●●●●●● ●● ●●●
●●●●●● ●●● ●●●●● ●●● ●●●●● ●●●● ●● ●●● ● ●●● ●●● ●● ●●● ●● ●●●●● ●● ● ●● ●●● ●● ●●●●●● ●● ●●●● ●●● ●●●● ●●● ●●●●●●
●● ●●● ●●● ●●●
●
●●● ●●● ●● ●● ●●●● ●●● ●● ●
●●● ●●● ●●●● ●●●● ●●●●●●●● ● ●●● ●● ●●● ●●●● ●● ●● ●●●● ● ●●●●● ●●●● ●●●●
●
●
●●● ●● ●●
●
● ●●●● ●●●●●●
●●●
●
●
CSMESN
![Page 20: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/20.jpg)
User: DanielZeevi
Base Rate:
CSM Rate: ESN Rate:
0.45060.9477 0.9419
![Page 21: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/21.jpg)
Forecasting High Tide:Predicting Times of Elevated
Activity in Online Social MediaDavid Darmon, Jimpei Harada, William Rand,
and Michelle Girvanter.ps/hightide
![Page 22: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/22.jpg)
Motivation
▪ Many individuals, firms, and organizations have interest in spreading information via online social networks.
▪ Spreading a message on social media requires:■ Influence: Position in social network.■ Persuasiveness: Success at retransmission.■ Timing: Success at timing transmission.
![Page 23: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/23.jpg)
●●●
●●
●
●
●●●●●
●
●
●
●●●
●
●●
●●●●
●●●●●●●
●●
●
●●
●
●●●
●●
●
●●
●●●
●●
●●●
●●●
●●
●●●
●●
●
●●
●
●●
●●●●●●●●●●●
●
●
●●●
●●●●●●●●●●●●
●●
●
●●●●●●
●
●●●●●●●●●
●●●●●●●●●●●●●●●●●●●
●●●●
●●
●
●●●●
●●
●●●
●
●●
●
●
●●
●●
●●●
●●●●●●●●
●
●
●
●●
●
●●
●
●●●
●●●
●
●
●●●
●
●●●
●●
●
●●●
●●●●●
●●
●●
●●●●
●
●
●●●
●
●●●●
●
●●
●●
●●
●
●
●●
●●●
●
●●●●●●●●●●●●●
●●
●●●●●●
●●●
●●●
●
●●
●●●●●●
●
●●●
●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●●●
●
●●
●●●●●●●
●
●
●
●●●●
●
●●
●●
●
●●●●
●●●●●●
●
●●
●●
●
●●
●
●●●
●●●●
●
●
●●●
●●
●
●
●
●●
●
●
●
●●●
●●●
●●
●
●●●●
●●●
●●●●●●●
●●●●●●●
●●●●
●
●●●
●
●●●●
●
●●●●●
●●
●
●
●●●●●●
●
●
●
●●
●
●●●●
●●
●
●●●
●
●
●
●●
●●●●
●●
●
●●
●●●●
●
●●●●
●●●●●
●
●●●●●●●
●
●●●●●●●●
●●
●●●●●
●
●●●
●●●●
●●●
●●●
●●●
●●●●●●
●●●●●
●●●●
●●
●●
●
●●●●●
●
●●●●
●
●●●●●
●
●●●●●
●●●
●●
●●
●●●●
●●●
●●●●
●●●
●●●●●●●
●●
●●
●●●●●●
●●●
●●●●●●
●
●●●
●●●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●
●●
●
●●●●●
●
●
●
●●
●
●
●●●
●●
●●●●●●●●●●
●●●
●
●●
●●
●●
●●●
●
●
●●●●●●●●●●●●●●●
●
●
●
●●●●●
●●●●●●●●●●●●●
●●●●●
●●
●●
●
●
●●●●●●●
●●
●●
●●
●
●●
●
●●
●
●●
●●●●
●
●
●●●●●●●●
●
●●
●●●●●
●●●●●
●
●●
●
●
●●●●●
●
●●●
●●●●
●●●
●●●●
●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●
●●
●
●●●
●
●●●
●●●●
●●
●●
●●
●●●●●
●
●
●●●●●●●●●●
●●
●●
●
●●●
●
●●
●●●
●●●●
●●●●●
●●●●●
●
●
●●●
●
●●●●●●●
●●●●●●●●●●
●●
●
●●
●●●●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●●●
●●●
●●
●●●●●●●●●●●
●●
●●●●●●●●●●●●
●
●●●
●
●
●●●●
●●●●
●●
0 200 400 600 800 1000
015
0
An
●
●
●
●●●●●
●●
●
●●●●
●●●●●
●●●●●
●●●
●●
●
●●●●●
●●●●
●
●
●●●●●●●●
●
●●
●●●●
●
●
●●●
●
●
●●
●
●
●
●
●●●●
●
●●
●●●
●
●
●
●
●
●●
●●●●
●
●●●
●
●●●●●
●●
●●
●●
●●●●●●●●●●●●●
●●●
●●●●●●●●●●●●●●●
●
●
●●●●●
●●●●●●
●●
●
●
●
●●
●
●●●
●●
●●●●●
●
●
●●
●
●
●●
●●●●●
●
●●
●●
●
●●●●●
●
●
●●
●●
●
●
●
●●●●●●●
●●
●
●
●
●
●
●●●●●●●●●●
●●●
●●●●●
●●●●
●
●●
●
●●●●●●●●
●●
●●●●●
●
●●●●●●●●●●●●
●●●
●●
●●●●●
●
●●●
●
●●
●●●●●
●
●
●●
●
●
●
●●●
●
●●●
●
●●
●●●
●
●
●
●
●
●
●
●
●●●
●●●●●●
●
●
●●●●
●
●●●●●
●
●●
●
●
●●●●●●
●●●
●●●●
●
●
●
●
●
●
●●
●
●●●●●●
●●
●●●●●●
●●●
●●
●
●●●●●●
●●●●
●●●●●●●●
●●●●●●●●
●●
●
●●
●
●●
●●●●
●
●
●
●
●
●
●
●●
●●●●●●
●●●
●
●●●
●●●
●
●
●●●●●●●
●●
●
●
●
●●●●●●●●●
●●●●●●
●●
●
●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●●
●●●
●●
●●●●
●●
●
●
●
●●
●
●
●
●
●●●●●●●
●●
●●●●●●●
●
●●
●
●
●●●●●●
●●●●●●●●
●●
●●
●
●●●●
●
●●
●●
●●●●
●●●
●●●
●
●
●●
●
●●●
●
●
●
●●●●●●
●
●●●
●●
●
●
●●
●●●
●
●
●
●
●●
●●●●
●
●
●
●●●●
●
●●
●●●
●
●
●●●●●●●
●●●●
●●●●●●●●●
●●●
●●●
●●●●
●●
●●●●●●●●●●●●●
●●●●●
●
●
●
●
●●
●●●●●●●●●●
●
●●
●
●●●●●●
●
●
●●●●
●
●●
●●●●
●
●●
●●●
●●●●●●
●●
●●
●●●●●
●
●
●●
●
●●
●
●
●●●●●●●●●●●●●●●●●●●
●●●●●●●
●
●●
●●●●
●
●●
●●
●
●
●
●●●
●●
●●●
●●●●●
●
●●●●●●●●●●●●●●
●
●
●●
●●
●
●●●
●●●●●●●
●●●
●
●
●
●●●
●●●●●●●●●
●●
●
●●●●●●●●
●
●●
●●
●
●
●●
●●●●
●
●
●
●
●●
●
●●●●
●
●●●●●●●●●●●
●●
●
●
●
●●
●
●●●
●●●
●●●●●
●●●●
●●●
●●●●●
●
●
●
●
●●●
●●●
●
●●●●
●
●●●●
●
●●●
●●●●●●●●●
●●●●●
●●
●●●●
●●●●●●●●●
●●●
●
●●●●●
●●●●
●
0 200 400 600 800 1000
015
0
An
●
●
●
●
●●
●
●
●●●●
●
●
●
●●●
●
●
●●●●●●●
●●●
●●
●
●●
●
●
●●●●
●
●
●●●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●●●●●
●
●●●●●
●●
●
●●●
●●●●●●●●●●●●●●●
●●●●●●
●
●
●
●●●●●●
●●●●●
●●●●●●
●
●●●●
●●
●
●
●
●●
●●
●
●
●●
●●
●
●
●●●●
●
●●●●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●●●●●●
●
●
●
●
●
●
●●
●●
●
●●●●
●●●●
●
●●
●
●
●
●
●●●●●
●●●
●●●●●●●●
●●●
●●●●●●
●●●●
●●
●●●●●●●●●●●
●
●
●●
●●
●
●
●●●●
●●
●
●
●●
●●●
●●
●●●●●
●
●●●●●
●
●●●●●
●
●
●
●
●●●●●●●
●●●●●●●
●
●
●●
●●
●
●●
●
●●
●●
●
●●●●●●●●●●
●
●●●
●
●
●●●
●●●●
●
●●●
●●●●●
●
●
●●●
●
●
●
●
●●
●
●●●●●●●●
●●●●●●●●●●
●●●●●●●●
●●
●●●●●
●●●●●
●●●
●●
●●●●●●●●●
●
●●
●
●●●●●
●●●
●●●
●
●●
●
●
●●●●●●●●
●●●
●
●●
●
●●●
●●●●●
●
●
●
●
●
●●●●●●
●
●
●●●●●
●
●
●
●●●●
●●●●●●
●
●●
●
●●●●●●
●●●
●●●●
●
●●●
●●●●●●
●
●●●●●●●●●●●●●●●●●●
●
●
●
●●
●
●●●
●●●
●●●●
●●●
●
●●
●●●
●●
●
●●
●●
●●
●●●
●
●
●
●
●
●
●●●●
●
●●
●●
●
●●●●
●●
●
●
●
●
●●
●●
●●
●
●●●
●
●●
●●
●●●
●●
●●●
●
●●●●●
●●●●●●
●
●●●●●●●
●
●●
●●●●
●
●●●●
●●●
●●●●●●●●●
●●●●
●●
●●●●●●
●●●●●
●●●●
●●●●●●●
●●●●●
●●●●●
●●●●●●●●●●●
●●
●●●
●●
●
●●
●
●
●
●
●●●●●
●●●●
●
●
●●
●
●●●●
●●●●●●●●
●
●●
●
●
●●
●
●
●
●●●●●
●
●
●●●●
●●●
●●●●●●●●●●●
●●●●●●●●●
●●
●
●●●●●
●●●●●●●●●●●●●●●●●●
●●●●●●●
●●●●●
●
●
●
●●●●●●●
●●●
●
●
●
●
●●●
●●
●●
●
●●●●
●
●●●●●●●
●●
●●●●●
●
●●●●
●●
●●
●●●
●●●●
●●
●
●
●
●
●●●●●●
●●
●
●●●
●
●
●●
●
●●●●●●●●
●
●●
●●●
●
●
●●
●
●
●●●●●●●●●●●
●
●●
●●●
●
●●●●●
●
●●●●●●●●●
●●●●●●●●●
●
●
●●
●●●
●●
●●
0 200 400 600 800 1000
015
0
An
●●●
●
●●●●
●
●●●●
●
●
●
●
●
●
●●
●
●●
●●
●●
●
●●●●
●●●●●
●●●
●
●●●●
●●●●●
●
●
●
●
●●
●●
●●●●●
●
●●●
●
●
●
●
●●●
●●
●
●●●●●
●
●
●
●
●●●●●
●
●
●●●●●●●●●●●●●●●●●●●
●●
●●●●●
●
●●
●●●●
●
●
●●●
●●●●●●●
●
●
●
●
●
●●●●●
●
●●
●
●
●●
●
●
●
●●
●
●
●●●
●●●
●
●
●●●●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●●
●●●●●●
●
●●
●●
●●●●●●●●
●
●
●
●●●
●
●
●
●●●●●●●
●●●●●
●
●
●
●
●
●
●
●
●
●●●
●
●●●
●●●●●●●●●●●
●●●●●●●●●●●●
●●●●●
●●
●
●
●●
●
●●
●●
●●
●●●
●
●
●
●●●●●●●●
●
●
●●
●●
●●●
●●
●●●●●●●
●
●●●
●●●●●
●
●●
●
●●●●●
●●●●
●●●
●
●●
●
●
●●●
●
●●●
●●
●●
●
●
●●●●
●
●
●
●
●
●
●●
●●●●●●
●●
●
●●
●
●●●●●●●●●●●●●●●
●●●●
●
●●●
●●●●
●
●
●
●●●●●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●●●
●
●●
●
●●●●●
●
●●
●
●●●
●
●
●
●●●
●●●●●●
●●●
●●
●●
●
●●●●●●
●
●
●
●●●●●●
●
●
●●
●
●●●●●●
●●●●
●●
●●
●
●
●●●
●
●●
●●●●●
●
●
●
●
●●
●●●●●●●
●●●
●
●
●
●●●●●
●
●●●●
●
●
●●
●●
●●●
●
●
●
●
●
●
●●●●●
●●
●
●
●●
●●
●●●●
●
●
●
●
●
●
●
●●●●●
●
●●●●●●
●
●
●
●
●●●
●
●●●●●●●●●
●●●
●●
●●●●●●●●
●
●●●
●●
●
●●
●
●
●●
●●●
●●●●●●
●●
●
●●●●●●●
●
●
●●●●●●●
●
●●
●
●●●●
●
●●●●●
●●●●●
●●●
●●●●●●
●●●●●●●●●●●●●●●●●
●●
●●
●
●●●
●
●
●●
●●●●●
●
●●●
●
●●●●●
●
●
●●
●
●
●●●
●
●●●
●
●
●●●●
●
●●●
●●
●●
●●
●●●●
●
●●●
●
●
●
●●
●●
●●
●●
●●
●
●●●●
●
●●●
●
●
●●●
●●●
●●●
●●●
●
●●
●
●●
●●●●●●●●●
●
●●
●●●●●●
●
●●●●●●●●●●●
●
●●●
●
●●●●
●●
●
●●●●
●●●●●●●●
●●
●
●●●
●●●●
●
●
●●●
●●●
●
●
●
●●●
●
●
●
●●●●●
●●●●●●
●
●●
●●
●
●
●●
●●●●
●
●●●
●
●
●
●●●
●●●
●●●●●●●
●
●●
●
●
●●●●●●●●●●
●●●●●
●●
●●●●●●●●●●●●●●●●●
●●
●●
●
●
●●
●●●
●
●
●
0 200 400 600 800 1000
015
0
Time (∆ = 600 seconds)
An
![Page 24: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/24.jpg)
Motivation
▪ The ‘optimal’ time varies from day-to-day andweek-to-week.
▪ Can we do better than seasonality by incorporating more information about the user activity?
▪ Compare three models:■ Seasonality
■ Prediction based on day-of-week+time-of-day.
■ Aggregate Autoregressive Model■ Prediction based on overall ‘excitement.’
■ Aggregation-of-Individuals■ Prediction based on individual ‘excitement.’
![Page 25: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/25.jpg)
Methodology — Seasonality
▪ We see that the number of users active showsweek-to-week seasonality.
▪ We estimate the seasonality using an average across W weeks:
▪ Take our seasonality prediction for time n as the value of this estimator at time n:
sn =1
W
X
j2{0,1,...,W�1}
An+jT .
ASn = sn.
Model 1 — Seasonality
▪ We see that the number of users active showsweek-to-week seasonality.
▪ We estimate the seasonality using an average across W weeks:
▪ Take our seasonality prediction for time n as the value of this estimator at time n:
sn =1
W
X
j2{0,1,...,W�1}
An+jT .
ASn = sn.
![Page 26: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/26.jpg)
Model 2 — Aggregate Autoregressive
▪ An extension to seasonality allows the residuals about the seasonality to exhibit memory.
▪ Can think of this as ‘excitement’ in the Twitterverse.
▪ Ex. The capture of Osama bin Laden.
●●●
●●
●
●
●●●●●
●
●
●
●●●
●
●●
●●●●
●●●●●●●
●●
●
●●
●
●●●
●●
●
●●
●●●
●●
●●●
●●●
●●
●●●
●●
●
●●
●
●●
●●●●●●●●●●●
●
●
●●●
●●●●●●●●●●●●
●●
●
●●●●●●
●
●●●●●●●●●
●●●●●●●●●●●●●●●●●●●
●●●●
●●
●
●●●●
●●
●●●
●
●●
●
●
●●
●●
●●●
●●●●●●●●
●
●
●
●●
●
●●
●
●●●
●●●
●
●
●●●
●
●●●
●●
●
●●●
●●●●●
●●
●●
●●●●
●
●
●●●
●
●●●●
●
●●
●●
●●
●
●
●●
●●●
●
●●●●●●●●●●●●●
●●
●●●●●●
●●●
●●●
●
●●
●●●●●●
●
●●●
●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●●●
●
●●
●●●●●●●
●
●
●
●●●●
●
●●
●●
●
●●●●
●●●●●●
●
●●
●●
●
●●
●
●●●
●●●●
●
●
●●●
●●
●
●
●
●●
●
●
●
●●●
●●●
●●
●
●●●●
●●●
●●●●●●●
●●●●●●●
●●●●
●
●●●
●
●●●●
●
●●●●●
●●
●
●
●●●●●●
●
●
●
●●
●
●●●●
●●
●
●●●
●
●
●
●●
●●●●
●●
●
●●
●●●●
●
●●●●
●●●●●
●
●●●●●●●
●
●●●●●●●●
●●
●●●●●
●
●●●
●●●●
●●●
●●●
●●●
●●●●●●
●●●●●
●●●●
●●
●●
●
●●●●●
●
●●●●
●
●●●●●
●
●●●●●
●●●
●●
●●
●●●●
●●●
●●●●
●●●
●●●●●●●
●●
●●
●●●●●●
●●●
●●●●●●
●
●●●
●●●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●
●●
●
●●●●●
●
●
●
●●
●
●
●●●
●●
●●●●●●●●●●
●●●
●
●●
●●
●●
●●●
●
●
●●●●●●●●●●●●●●●
●
●
●
●●●●●
●●●●●●●●●●●●●
●●●●●
●●
●●
●
●
●●●●●●●
●●
●●
●●
●
●●
●
●●
●
●●
●●●●
●
●
●●●●●●●●
●
●●
●●●●●
●●●●●
●
●●
●
●
●●●●●
●
●●●
●●●●
●●●
●●●●
●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●
●●
●
●●●
●
●●●
●●●●
●●
●●
●●
●●●●●
●
●
●●●●●●●●●●
●●
●●
●
●●●
●
●●
●●●
●●●●
●●●●●
●●●●●
●
●
●●●
●
●●●●●●●
●●●●●●●●●●
●●
●
●●
●●●●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●●●
●●●
●●
●●●●●●●●●●●
●●
●●●●●●●●●●●●
●
●●●
●
●
●●●●
●●●●
●●
0 200 400 600 800 1000
015
0
An
●
●
●
●●●●●
●●
●
●●●●
●●●●●
●●●●●
●●●
●●
●
●●●●●
●●●●
●
●
●●●●●●●●
●
●●
●●●●
●
●
●●●
●
●
●●
●
●
●
●
●●●●
●
●●
●●●
●
●
●
●
●
●●
●●●●
●
●●●
●
●●●●●
●●
●●
●●
●●●●●●●●●●●●●
●●●
●●●●●●●●●●●●●●●
●
●
●●●●●
●●●●●●
●●
●
●
●
●●
●
●●●
●●
●●●●●
●
●
●●
●
●
●●
●●●●●
●
●●
●●
●
●●●●●
●
●
●●
●●
●
●
●
●●●●●●●
●●
●
●
●
●
●
●●●●●●●●●●
●●●
●●●●●
●●●●
●
●●
●
●●●●●●●●
●●
●●●●●
●
●●●●●●●●●●●●
●●●
●●
●●●●●
●
●●●
●
●●
●●●●●
●
●
●●
●
●
●
●●●
●
●●●
●
●●
●●●
●
●
●
●
●
●
●
●
●●●
●●●●●●
●
●
●●●●
●
●●●●●
●
●●
●
●
●●●●●●
●●●
●●●●
●
●
●
●
●
●
●●
●
●●●●●●
●●
●●●●●●
●●●
●●
●
●●●●●●
●●●●
●●●●●●●●
●●●●●●●●
●●
●
●●
●
●●
●●●●
●
●
●
●
●
●
●
●●
●●●●●●
●●●
●
●●●
●●●
●
●
●●●●●●●
●●
●
●
●
●●●●●●●●●
●●●●●●
●●
●
●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●●
●●●
●●
●●●●
●●
●
●
●
●●
●
●
●
●
●●●●●●●
●●
●●●●●●●
●
●●
●
●
●●●●●●
●●●●●●●●
●●
●●
●
●●●●
●
●●
●●
●●●●
●●●
●●●
●
●
●●
●
●●●
●
●
●
●●●●●●
●
●●●
●●
●
●
●●
●●●
●
●
●
●
●●
●●●●
●
●
●
●●●●
●
●●
●●●
●
●
●●●●●●●
●●●●
●●●●●●●●●
●●●
●●●
●●●●
●●
●●●●●●●●●●●●●
●●●●●
●
●
●
●
●●
●●●●●●●●●●
●
●●
●
●●●●●●
●
●
●●●●
●
●●
●●●●
●
●●
●●●
●●●●●●
●●
●●
●●●●●
●
●
●●
●
●●
●
●
●●●●●●●●●●●●●●●●●●●
●●●●●●●
●
●●
●●●●
●
●●
●●
●
●
●
●●●
●●
●●●
●●●●●
●
●●●●●●●●●●●●●●
●
●
●●
●●
●
●●●
●●●●●●●
●●●
●
●
●
●●●
●●●●●●●●●
●●
●
●●●●●●●●
●
●●
●●
●
●
●●
●●●●
●
●
●
●
●●
●
●●●●
●
●●●●●●●●●●●
●●
●
●
●
●●
●
●●●
●●●
●●●●●
●●●●
●●●
●●●●●
●
●
●
●
●●●
●●●
●
●●●●
●
●●●●
●
●●●
●●●●●●●●●
●●●●●
●●
●●●●
●●●●●●●●●
●●●
●
●●●●●
●●●●
●
0 200 400 600 800 1000
015
0
An
●
●
●
●
●●
●
●
●●●●
●
●
●
●●●
●
●
●●●●●●●
●●●
●●
●
●●
●
●
●●●●
●
●
●●●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●●●●●
●
●●●●●
●●
●
●●●
●●●●●●●●●●●●●●●
●●●●●●
●
●
●
●●●●●●
●●●●●
●●●●●●
●
●●●●
●●
●
●
●
●●
●●
●
●
●●
●●
●
●
●●●●
●
●●●●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●●●●●●
●
●
●
●
●
●
●●
●●
●
●●●●
●●●●
●
●●
●
●
●
●
●●●●●
●●●
●●●●●●●●
●●●
●●●●●●
●●●●
●●
●●●●●●●●●●●
●
●
●●
●●
●
●
●●●●
●●
●
●
●●
●●●
●●
●●●●●
●
●●●●●
●
●●●●●
●
●
●
●
●●●●●●●
●●●●●●●
●
●
●●
●●
●
●●
●
●●
●●
●
●●●●●●●●●●
●
●●●
●
●
●●●
●●●●
●
●●●
●●●●●
●
●
●●●
●
●
●
●
●●
●
●●●●●●●●
●●●●●●●●●●
●●●●●●●●
●●
●●●●●
●●●●●
●●●
●●
●●●●●●●●●
●
●●
●
●●●●●
●●●
●●●
●
●●
●
●
●●●●●●●●
●●●
●
●●
●
●●●
●●●●●
●
●
●
●
●
●●●●●●
●
●
●●●●●
●
●
●
●●●●
●●●●●●
●
●●
●
●●●●●●
●●●
●●●●
●
●●●
●●●●●●
●
●●●●●●●●●●●●●●●●●●
●
●
●
●●
●
●●●
●●●
●●●●
●●●
●
●●
●●●
●●
●
●●
●●
●●
●●●
●
●
●
●
●
●
●●●●
●
●●
●●
●
●●●●
●●
●
●
●
●
●●
●●
●●
●
●●●
●
●●
●●
●●●
●●
●●●
●
●●●●●
●●●●●●
●
●●●●●●●
●
●●
●●●●
●
●●●●
●●●
●●●●●●●●●
●●●●
●●
●●●●●●
●●●●●
●●●●
●●●●●●●
●●●●●
●●●●●
●●●●●●●●●●●
●●
●●●
●●
●
●●
●
●
●
●
●●●●●
●●●●
●
●
●●
●
●●●●
●●●●●●●●
●
●●
●
●
●●
●
●
●
●●●●●
●
●
●●●●
●●●
●●●●●●●●●●●
●●●●●●●●●
●●
●
●●●●●
●●●●●●●●●●●●●●●●●●
●●●●●●●
●●●●●
●
●
●
●●●●●●●
●●●
●
●
●
●
●●●
●●
●●
●
●●●●
●
●●●●●●●
●●
●●●●●
●
●●●●
●●
●●
●●●
●●●●
●●
●
●
●
●
●●●●●●
●●
●
●●●
●
●
●●
●
●●●●●●●●
●
●●
●●●
●
●
●●
●
●
●●●●●●●●●●●
●
●●
●●●
●
●●●●●
●
●●●●●●●●●
●●●●●●●●●
●
●
●●
●●●
●●
●●
0 200 400 600 800 1000
015
0
An
●●●
●
●●●●
●
●●●●
●
●
●
●
●
●
●●
●
●●
●●
●●
●
●●●●
●●●●●
●●●
●
●●●●
●●●●●
●
●
●
●
●●
●●
●●●●●
●
●●●
●
●
●
●
●●●
●●
●
●●●●●
●
●
●
●
●●●●●
●
●
●●●●●●●●●●●●●●●●●●●
●●
●●●●●
●
●●
●●●●
●
●
●●●
●●●●●●●
●
●
●
●
●
●●●●●
●
●●
●
●
●●
●
●
●
●●
●
●
●●●
●●●
●
●
●●●●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●●
●●●●●●
●
●●
●●
●●●●●●●●
●
●
●
●●●
●
●
●
●●●●●●●
●●●●●
●
●
●
●
●
●
●
●
●
●●●
●
●●●
●●●●●●●●●●●
●●●●●●●●●●●●
●●●●●
●●
●
●
●●
●
●●
●●
●●
●●●
●
●
●
●●●●●●●●
●
●
●●
●●
●●●
●●
●●●●●●●
●
●●●
●●●●●
●
●●
●
●●●●●
●●●●
●●●
●
●●
●
●
●●●
●
●●●
●●
●●
●
●
●●●●
●
●
●
●
●
●
●●
●●●●●●
●●
●
●●
●
●●●●●●●●●●●●●●●
●●●●
●
●●●
●●●●
●
●
●
●●●●●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●●●
●
●●
●
●●●●●
●
●●
●
●●●
●
●
●
●●●
●●●●●●
●●●
●●
●●
●
●●●●●●
●
●
●
●●●●●●
●
●
●●
●
●●●●●●
●●●●
●●
●●
●
●
●●●
●
●●
●●●●●
●
●
●
●
●●
●●●●●●●
●●●
●
●
●
●●●●●
●
●●●●
●
●
●●
●●
●●●
●
●
●
●
●
●
●●●●●
●●
●
●
●●
●●
●●●●
●
●
●
●
●
●
●
●●●●●
●
●●●●●●
●
●
●
●
●●●
●
●●●●●●●●●
●●●
●●
●●●●●●●●
●
●●●
●●
●
●●
●
●
●●
●●●
●●●●●●
●●
●
●●●●●●●
●
●
●●●●●●●
●
●●
●
●●●●
●
●●●●●
●●●●●
●●●
●●●●●●
●●●●●●●●●●●●●●●●●
●●
●●
●
●●●
●
●
●●
●●●●●
●
●●●
●
●●●●●
●
●
●●
●
●
●●●
●
●●●
●
●
●●●●
●
●●●
●●
●●
●●
●●●●
●
●●●
●
●
●
●●
●●
●●
●●
●●
●
●●●●
●
●●●
●
●
●●●
●●●
●●●
●●●
●
●●
●
●●
●●●●●●●●●
●
●●
●●●●●●
●
●●●●●●●●●●●
●
●●●
●
●●●●
●●
●
●●●●
●●●●●●●●
●●
●
●●●
●●●●
●
●
●●●
●●●
●
●
●
●●●
●
●
●
●●●●●
●●●●●●
●
●●
●●
●
●
●●
●●●●
●
●●●
●
●
●
●●●
●●●
●●●●●●●
●
●●
●
●
●●●●●●●●●●
●●●●●
●●
●●●●●●●●●●●●●●●●●
●●
●●
●
●
●●
●●●
●
●
●
0 200 400 600 800 1000
015
0
Time (∆ = 600 seconds)
An
![Page 27: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/27.jpg)
Model 3 — Aggregation of Individuals
▪ The number of individuals active is the aggregation of individual activities.
▪ Previous work showed that many users exhibit stereotypical / predictable behavior.
▪ Predict each user u separately and then aggregate these predictions to get the overall prediction for AnAn.
▪ Use a hidden state representation, called an !-machine, to model the user’s behavior.
![Page 28: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/28.jpg)
●●●●●●●●●●
●●●
●
●●●
●
●
●●
●●●●●
●●
●
●
●
●
●●●●●●●●
●●●●
●●●●●
●●●
●
●
●●
●●
●●●
●
●●●●●●
●●●●●●
●
●●
●●●●●●●●●
●●●●
●
●●●●●●●●●●●●
●
●●●●
●●●
●●●
●
●●●●
●
●
●
●●●
●
●●●
●●●●●
●●
●
●
●●●●
●●●●●
●●●
●
●●
●
●●
●
●●
●●
●
●●●●●●
●
●
●
●
●●●●●●●●
●●
●●●●
●●
●●●●
●●●●●
●
●
●
●
●
●●
●
●●
●●●
●
●
●●
●●
●
●●
●●●●●●
●●
●
●
●
●●●●
●
●
●
●●
●
●
●●●●●●●●
●
●●
●
●
●
●●●
●
●
●
●
●●●●
●●●
●
●
●●
●
●●●●
●
●
●●
●
●
●●
●
●●●●●●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●●
●●●●●●
●●●
●●●●●●●
●
●
●
●
●
●
●
●●●●●
●●
●
●
●
●●●●●●●●●●●
●●●●
●
●
●●●●
●
●●●●●●
●●
●
●
●●●●
●
●●●●●●
●●●●
●
●●●
●●●●●●
●●●●●●●
●
●●●
●●●
●●
●
●●●●
●●●
●
●
●●●●●●
●
●●●
●●
●●●●●●
●●●●●●●●●●●
●●●●
●●●
●●●
●
●
●
●
●●●
●
●●
●
●●
●
●●●●●●●●●●●●●
●
●●●●
●
●●
●
●●
●●●●
●●
●●
●●●●●●●●
●●●●●
●●●●●●
●
●●
●●●●●
●
●●
●●
●●
●●●
●●
●●●●●●●●●
●
●●
●
●●●●
●
●●●●●●●●
●●●●●
●
●●●●●
●
●●●
●
●
●
●●
●●
0 100 200 300 400 500 6000
100
AnCS
M
●●●
●●
●
●●●
●
●
●●●
●●
●●●
●
●
●●
●●●●
●
●●
●●●●●●●●
●●●●●
●●●●
●
●
●
●
●
●
●
●●●●●
●
●●
●●●
●
●
●●●
●
●●●●
●●●
●●
●
●
●●
●
●
●●●●
●●●●●●
●●
●
●●
●●●
●●●●●
●
●
●●
●●●●●●●●
●●
●
●
●●
●
●
●
●
●
●●
●●●
●●●●●●●●●
●●●
●●●●
●
●●●
●
●●
●●
●
●
●●
●
●
●●●●
●●●●●●●●
●
●
●
●
●●
●
●
●
●●●
●●●●●
●
●●
●●●●●●●
●
●
●
●
●
●●●●
●
●●●●
●●●●
●●●●
●●●
●
●
●●
●
●
●
●●●●
●
●
●●●●●●●
●
●
●●●●
●
●●●●●●●
●●●
●●●
●
●●●●
●●●●●
●
●
●
●
●
●
●
●
●●●
●●
●
●●●●
●●●●
●
●●●
●
●●
●
●
●
●
●●●●
●
●
●
●
●
●●●●●
●
●●●●
●●●
●●
●
●●
●●
●●●●
●●●
●
●
●●●●●●●●●●
●●●
●●●●
●
●
●●●●●●●●
●●
●●●●●●●●●●●
●
●●●●
●
●
●●●●
●
●
●
●●●●●
●●●●●●
●●●●●
●●
●
●●
●
●
●
●●●●●●
●
●●●
●
●
●●●●●
●
●●●●●●●●
●●●
●●
●●●●
●
●●
●●●
●●●
●
●●●
●●●●
●●●
●●●●
●
●
●●●●●●●
●●
●●●●●●●●
●●●●●
●
●●●●●●●
●●●●●
●
●●●●
●
●●●●●●●●●
●●
●
●●
●●●●●●●●●
●
●●●●●
●
●●●●●●●●●
●●
●
●●●●●
●●●
●
●
●
●●●
●
●●●●●●●
●●●●●
0 100 200 300 400 500 600
010
0AnCS
M
●●●●●●●
●
●●●
●●
●
●
●●●●
●●●
●
●●●
●●
●●●●
●●●●
●
●●
●●●
●●
●
●
●●●
●●●●●●●●●●●●●●
●
●●
●
●
●●
●●
●
●●●
●
●
●●
●●●●●●
●●
●
●●●
●
●
●●
●
●●
●●●
●
●
●
●
●●●●
●●●●●
●
●●
●
●●●●
●
●●
●●
●●
●●
●
●●●
●
●●●●
●
●●
●
●
●
●●
●
●
●
●●●●
●
●●●
●●●
●●
●
●●
●●
●
●
●●
●
●●●●
●●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●●
●●●
●●
●●
●
●●●●●●
●●●●●
●
●●
●
●●●●●●
●●
●
●
●●
●●
●●
●●
●●●●●
●●●
●●●●
●
●●
●●●●
●
●●●●
●●●●●●●●●●●●
●●
●
●●
●
●●●●●
●●
●●●●
●●
●
●
●●●●
●●●
●●●●
●●●●●
●●
●
●●●
●
●
●
●
●●●
●●
●
●
●●●
●
●
●
●●●●●●●●●
●●●
●
●
●●●
●●●●
●●●●●●
●●●●●●
●●
●●●●●●
●●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●●●
●
●●●●
●
●●
●
●●●●●
●●●●●●●●●●●●●
●
●
●●●●●●
●
●●●●●●●
●●●●●●
●●
●
●
●●
●●●
●
●●●●
●
●●●
●
●●
●
●●●
●
●●●●
●●●
●
●●●●●●●●●●
●●
●●●●
●●
●
●●●●●
●●●
●
●●●●
●●●●
●●●
●●●●●
●●●●●
●
●
●●
●
●●●●●●●●
●●
●●
●●●
●●
●
●●
●●●●●●
●●
●●●●●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●●
●
●●●●●
●
●●
●
●
0 100 200 300 400 500 600
010
0AnCS
M
●
●
●●
●●
●
●
●●●
●
●●●●●●
●
●
●●●
●●●●●●●●●●
●
●
●●●●●●
●●●●●
●
●●●
●
●●●●
●●●●●
●
●
●
●
●
●
●●●●
●●●●
●●
●
●
●●
●
●●
●
●●
●●
●●●●●●●
●●
●●●●●●
●
●●●
●●●
●
●
●
●●●●
●●
●
●
●●●●●●
●●
●
●●
●●●
●
●●
●●●●
●
●●
●
●●●●
●●●●
●●●●●●●●●
●
●
●●●●
●
●
●●●●
●
●
●●●
●●●●●●
●
●●●●●
●
●●●●
●
●●
●●
●
●●●●●
●
●
●●●
●●
●
●
●●●
●
●●
●●
●●
●●
●
●●●●
●
●●
●●●
●●●
●●●●
●
●●●●●
●●●●●
●●●●●●●●
●
●●●●●●
●
●●●
●
●
●●●
●●●
●
●
●
●●●●●
●
●●●
●
●●●●●
●
●●●
●
●●●●●
●
●●●
●
●●●
●
●●●●●●
●
●
●
●
●●
●
●
●●●●
●
●●●
●●
●
●
●
●
●●●●●
●●
●
●
●●
●●
●
●
●●●●
●
●●●
●
●●
●
●●●●●
●●●●●●
●●●
●
●
●●●●●●●
●●●●
●●●●●●●●●
●●●●●●●
●●●●●●●●
●●
●●●●●
●●●●
●●
●●●
●●●●●●●●
●
●●
●
●
●●
●●●●
●
●
●●
●●●●●●●
●●
●●
●●●
●●●
●
●
●●
●●●●●
●●
●
●
●●●
●●●●●●
●●
●
●
●●
●
●
●●●●●●
●
●
●●●●
●●●●●●●●●●●
●●
●
●●●
●●●●
●
●
●
●
●
●●●
●
●●●
●
●●●●●●●●●●
●●●●●
●
●●●●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●●
●●●●
0 100 200 300 400 500 600
010
0
Time (∆ = 600 seconds)
AnCS
M
Xn(u
2)X
n(u
2)X
n(u
2)
n (∆ = 600 seconds)
Xn(u
2)
Xn(u
3)X
n(u
3)X
n(u
3)
n (∆ = 600 seconds)
Xn(u
3)
Xn(u
4)X
n(u
4)X
n(u
4)
n (∆ = 600 seconds)
Xn(u
4)
Xn(u
2685)
Xn(u
2685)
Xn(u
2685)
n (∆ = 600 seconds)
Xn(u
2685)
Xn(u
1)X
n(u
1)X
n(u
1)
n (∆ = 600 seconds)
Xn(u
1)
...
+ + + + +
Xn(u1) Xn(u2) Xn(u3) Xn(u4) Xn(u2685)
An
![Page 29: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/29.jpg)
Forecasting High Tide
▪ Question: How well can we predict that the activity level at a future time n will be at or abovequantile p* of activity for day d?
●●●
●●
●
●
●●●●●
●
●
●
●●●
●
●●
●●●●
●●●●●●●
●●
●
●●
●
●●●
●●
●
●●
●●●
●●
●●●
●●●
●●
●●●
●●
●
●●
●
●●
●●●●●●●●●●●
●
●
●●●
●●●●●●●●●●●●
●●
●
●●●●●●
●
●●●●●●●●●
●●●●●●●●●●●●●●●●●●●
●●●●
●●
●
●●●●
●●
●●●
●
●●
●
●
●●
●●
●●●
●●●●●●●●
●
●
●
●●
●
●●
●
●●●
●●●
●
●
●●●
●
●●●
●●
●
●●●
●●●●●
●●
●●
●●●●
●
●
●●●
●
●●●●
●
●●
●●
●●
●
●
●●
●●●
●
●●●●●●●●●●●●●
●●
●●●●●●
●●●
●●●
●
●●
●●●●●●
●
●●●
●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●●●
●
●●
●●●●●●●
●
●
●
●●●●
●
●●
●●
●
●●●●
●●●●●●
●
●●
●●
●
●●
●
●●●
●●●●
●
●
●●●
●●
●
●
●
●●
●
●
●
●●●
●●●
●●
●
●●●●
●●●
●●●●●●●
●●●●●●●
●●●●
●
●●●
●
●●●●
●
●●●●●
●●
●
●
●●●●●●
●
●
●
●●
●
●●●●
●●
●
●●●
●
●
●
●●
●●●●
●●
●
●●
●●●●
●
●●●●
●●●●●
●
●●●●●●●
●
●●●●●●●●
●●
●●●●●
●
●●●
●●●●
●●●
●●●
●●●
●●●●●●
●●●●●
●●●●
●●
●●
●
●●●●●
●
●●●●
●
●●●●●
●
●●●●●
●●●
●●
●●
●●●●
●●●
●●●●
●●●
●●●●●●●
●●
●●
●●●●●●
●●●
●●●●●●
●
●●●
●●●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●
●●
●
●●●●●
●
●
●
●●
●
●
●●●
●●
●●●●●●●●●●
●●●
●
●●
●●
●●
●●●
●
●
●●●●●●●●●●●●●●●
●
●
●
●●●●●
●●●●●●●●●●●●●
●●●●●
●●
●●
●
●
●●●●●●●
●●
●●
●●
●
●●
●
●●
●
●●
●●●●
●
●
●●●●●●●●
●
●●
●●●●●
●●●●●
●
●●
●
●
●●●●●
●
●●●
●●●●
●●●
●●●●
●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●
●●
●
●●●
●
●●●
●●●●
●●
●●
●●
●●●●●
●
●
●●●●●●●●●●
●●
●●
●
●●●
●
●●
●●●
●●●●
●●●●●
●●●●●
●
●
●●●
●
●●●●●●●
●●●●●●●●●●
●●
●
●●
●●●●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●●●
●●●
●●
●●●●●●●●●●●
●●
●●●●●●●●●●●●
●
●●●
●
●
●●●●
●●●●
●●
0 200 400 600 800 1000
015
0
An
●
●
●
●●●●●
●●
●
●●●●
●●●●●
●●●●●
●●●
●●
●
●●●●●
●●●●
●
●
●●●●●●●●
●
●●
●●●●
●
●
●●●
●
●
●●
●
●
●
●
●●●●
●
●●
●●●
●
●
●
●
●
●●
●●●●
●
●●●
●
●●●●●
●●
●●
●●
●●●●●●●●●●●●●
●●●
●●●●●●●●●●●●●●●
●
●
●●●●●
●●●●●●
●●
●
●
●
●●
●
●●●
●●
●●●●●
●
●
●●
●
●
●●
●●●●●
●
●●
●●
●
●●●●●
●
●
●●
●●
●
●
●
●●●●●●●
●●
●
●
●
●
●
●●●●●●●●●●
●●●
●●●●●
●●●●
●
●●
●
●●●●●●●●
●●
●●●●●
●
●●●●●●●●●●●●
●●●
●●
●●●●●
●
●●●
●
●●
●●●●●
●
●
●●
●
●
●
●●●
●
●●●
●
●●
●●●
●
●
●
●
●
●
●
●
●●●
●●●●●●
●
●
●●●●
●
●●●●●
●
●●
●
●
●●●●●●
●●●
●●●●
●
●
●
●
●
●
●●
●
●●●●●●
●●
●●●●●●
●●●
●●
●
●●●●●●
●●●●
●●●●●●●●
●●●●●●●●
●●
●
●●
●
●●
●●●●
●
●
●
●
●
●
●
●●
●●●●●●
●●●
●
●●●
●●●
●
●
●●●●●●●
●●
●
●
●
●●●●●●●●●
●●●●●●
●●
●
●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●●
●●●
●●
●●●●
●●
●
●
●
●●
●
●
●
●
●●●●●●●
●●
●●●●●●●
●
●●
●
●
●●●●●●
●●●●●●●●
●●
●●
●
●●●●
●
●●
●●
●●●●
●●●
●●●
●
●
●●
●
●●●
●
●
●
●●●●●●
●
●●●
●●
●
●
●●
●●●
●
●
●
●
●●
●●●●
●
●
●
●●●●
●
●●
●●●
●
●
●●●●●●●
●●●●
●●●●●●●●●
●●●
●●●
●●●●
●●
●●●●●●●●●●●●●
●●●●●
●
●
●
●
●●
●●●●●●●●●●
●
●●
●
●●●●●●
●
●
●●●●
●
●●
●●●●
●
●●
●●●
●●●●●●
●●
●●
●●●●●
●
●
●●
●
●●
●
●
●●●●●●●●●●●●●●●●●●●
●●●●●●●
●
●●
●●●●
●
●●
●●
●
●
●
●●●
●●
●●●
●●●●●
●
●●●●●●●●●●●●●●
●
●
●●
●●
●
●●●
●●●●●●●
●●●
●
●
●
●●●
●●●●●●●●●
●●
●
●●●●●●●●
●
●●
●●
●
●
●●
●●●●
●
●
●
●
●●
●
●●●●
●
●●●●●●●●●●●
●●
●
●
●
●●
●
●●●
●●●
●●●●●
●●●●
●●●
●●●●●
●
●
●
●
●●●
●●●
●
●●●●
●
●●●●
●
●●●
●●●●●●●●●
●●●●●
●●
●●●●
●●●●●●●●●
●●●
●
●●●●●
●●●●
●
0 200 400 600 800 1000
015
0
An
●
●
●
●
●●
●
●
●●●●
●
●
●
●●●
●
●
●●●●●●●
●●●
●●
●
●●
●
●
●●●●
●
●
●●●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●●●●●
●
●●●●●
●●
●
●●●
●●●●●●●●●●●●●●●
●●●●●●
●
●
●
●●●●●●
●●●●●
●●●●●●
●
●●●●
●●
●
●
●
●●
●●
●
●
●●
●●
●
●
●●●●
●
●●●●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●●●●●●
●
●
●
●
●
●
●●
●●
●
●●●●
●●●●
●
●●
●
●
●
●
●●●●●
●●●
●●●●●●●●
●●●
●●●●●●
●●●●
●●
●●●●●●●●●●●
●
●
●●
●●
●
●
●●●●
●●
●
●
●●
●●●
●●
●●●●●
●
●●●●●
●
●●●●●
●
●
●
●
●●●●●●●
●●●●●●●
●
●
●●
●●
●
●●
●
●●
●●
●
●●●●●●●●●●
●
●●●
●
●
●●●
●●●●
●
●●●
●●●●●
●
●
●●●
●
●
●
●
●●
●
●●●●●●●●
●●●●●●●●●●
●●●●●●●●
●●
●●●●●
●●●●●
●●●
●●
●●●●●●●●●
●
●●
●
●●●●●
●●●
●●●
●
●●
●
●
●●●●●●●●
●●●
●
●●
●
●●●
●●●●●
●
●
●
●
●
●●●●●●
●
●
●●●●●
●
●
●
●●●●
●●●●●●
●
●●
●
●●●●●●
●●●
●●●●
●
●●●
●●●●●●
●
●●●●●●●●●●●●●●●●●●
●
●
●
●●
●
●●●
●●●
●●●●
●●●
●
●●
●●●
●●
●
●●
●●
●●
●●●
●
●
●
●
●
●
●●●●
●
●●
●●
●
●●●●
●●
●
●
●
●
●●
●●
●●
●
●●●
●
●●
●●
●●●
●●
●●●
●
●●●●●
●●●●●●
●
●●●●●●●
●
●●
●●●●
●
●●●●
●●●
●●●●●●●●●
●●●●
●●
●●●●●●
●●●●●
●●●●
●●●●●●●
●●●●●
●●●●●
●●●●●●●●●●●
●●
●●●
●●
●
●●
●
●
●
●
●●●●●
●●●●
●
●
●●
●
●●●●
●●●●●●●●
●
●●
●
●
●●
●
●
●
●●●●●
●
●
●●●●
●●●
●●●●●●●●●●●
●●●●●●●●●
●●
●
●●●●●
●●●●●●●●●●●●●●●●●●
●●●●●●●
●●●●●
●
●
●
●●●●●●●
●●●
●
●
●
●
●●●
●●
●●
●
●●●●
●
●●●●●●●
●●
●●●●●
●
●●●●
●●
●●
●●●
●●●●
●●
●
●
●
●
●●●●●●
●●
●
●●●
●
●
●●
●
●●●●●●●●
●
●●
●●●
●
●
●●
●
●
●●●●●●●●●●●
●
●●
●●●
●
●●●●●
●
●●●●●●●●●
●●●●●●●●●
●
●
●●
●●●
●●
●●
0 200 400 600 800 1000
015
0
An
●●●
●
●●●●
●
●●●●
●
●
●
●
●
●
●●
●
●●
●●
●●
●
●●●●
●●●●●
●●●
●
●●●●
●●●●●
●
●
●
●
●●
●●
●●●●●
●
●●●
●
●
●
●
●●●
●●
●
●●●●●
●
●
●
●
●●●●●
●
●
●●●●●●●●●●●●●●●●●●●
●●
●●●●●
●
●●
●●●●
●
●
●●●
●●●●●●●
●
●
●
●
●
●●●●●
●
●●
●
●
●●
●
●
●
●●
●
●
●●●
●●●
●
●
●●●●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●●
●●●●●●
●
●●
●●
●●●●●●●●
●
●
●
●●●
●
●
●
●●●●●●●
●●●●●
●
●
●
●
●
●
●
●
●
●●●
●
●●●
●●●●●●●●●●●
●●●●●●●●●●●●
●●●●●
●●
●
●
●●
●
●●
●●
●●
●●●
●
●
●
●●●●●●●●
●
●
●●
●●
●●●
●●
●●●●●●●
●
●●●
●●●●●
●
●●
●
●●●●●
●●●●
●●●
●
●●
●
●
●●●
●
●●●
●●
●●
●
●
●●●●
●
●
●
●
●
●
●●
●●●●●●
●●
●
●●
●
●●●●●●●●●●●●●●●
●●●●
●
●●●
●●●●
●
●
●
●●●●●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●●●
●
●●
●
●●●●●
●
●●
●
●●●
●
●
●
●●●
●●●●●●
●●●
●●
●●
●
●●●●●●
●
●
●
●●●●●●
●
●
●●
●
●●●●●●
●●●●
●●
●●
●
●
●●●
●
●●
●●●●●
●
●
●
●
●●
●●●●●●●
●●●
●
●
●
●●●●●
●
●●●●
●
●
●●
●●
●●●
●
●
●
●
●
●
●●●●●
●●
●
●
●●
●●
●●●●
●
●
●
●
●
●
●
●●●●●
●
●●●●●●
●
●
●
●
●●●
●
●●●●●●●●●
●●●
●●
●●●●●●●●
●
●●●
●●
●
●●
●
●
●●
●●●
●●●●●●
●●
●
●●●●●●●
●
●
●●●●●●●
●
●●
●
●●●●
●
●●●●●
●●●●●
●●●
●●●●●●
●●●●●●●●●●●●●●●●●
●●
●●
●
●●●
●
●
●●
●●●●●
●
●●●
●
●●●●●
●
●
●●
●
●
●●●
●
●●●
●
●
●●●●
●
●●●
●●
●●
●●
●●●●
●
●●●
●
●
●
●●
●●
●●
●●
●●
●
●●●●
●
●●●
●
●
●●●
●●●
●●●
●●●
●
●●
●
●●
●●●●●●●●●
●
●●
●●●●●●
●
●●●●●●●●●●●
●
●●●
●
●●●●
●●
●
●●●●
●●●●●●●●
●●
●
●●●
●●●●
●
●
●●●
●●●
●
●
●
●●●
●
●
●
●●●●●
●●●●●●
●
●●
●●
●
●
●●
●●●●
●
●●●
●
●
●
●●●
●●●
●●●●●●●
●
●●
●
●
●●●●●●●●●●
●●●●●
●●
●●●●●●●●●●●●●●●●●
●●
●●
●
●
●●
●●●
●
●
●
0 200 400 600 800 1000
015
0
Time (∆ = 600 seconds)
An
Num
ber o
f Use
rs R
etw
eetin
gA n
a : FTrued (a) = p⇤
In =
⇢1 : FTrue
d(n) (An) > p⇤
0 : otherwise
![Page 30: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/30.jpg)
Forecasting High Tide
▪ For prediction, use a historical distribution based on the seasonality for day d:
▪ For a predicted activity level , one of predict whether the activity is above the quantile p* using:
FHistd (a) =
1
N�
N�dX
n=N�(d�1)+1
⇥AS
n a⇤.
ˆIn(p) =
⇢1 : FHist
d(n)(ˆAn) > p
0 : otherwise
An {ASn, A
ARn , AAI
n },
![Page 31: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/31.jpg)
Results
SeasonalityAutoregressiveAgg. of Ind.
Year Seasonality Autoregressive Agg. of Individuals
2011 0.778 0.773 0.8252012 0.720 0.773 0.771
Area Under the ROC Curve (AUC):
![Page 32: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/32.jpg)
Attribution Modeling with Machine LearningDavid Darmon, Arjuna Ariyaratne,
William Rand, PK Kannan, and Michelle Girvan
Click on Email Promotion Link
Click on TripAdvisor Link
No conversion No conversion Converts
Day 1 Day 8
Click on Paid Search Link
Day 1
Purchase Funnel
Firm.com Firm.com Firm.com
![Page 33: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/33.jpg)
Conclusions
• Machine Learning plus Agent-Based Modeling provides a robust and powerful solution to harnessing the power of big data
• This method can provide predictive power that exceeds traditional aggregate methods
• Limitation: Requires high-resolution time series data
![Page 34: Using Big Data and Agent-Based Modeling to Understand ...files.meetup.com/2215331/dsdc2015.pdf · Using Big Data and Agent-Based Modeling to Understand Social Media Diffusion William](https://reader034.fdocuments.net/reader034/viewer/2022042204/5ea5de6eab29c20ce906c831/html5/thumbnails/34.jpg)
Future Work
• Construction of a software platform that would automatically generate agent-based models in a uniform way
• Implementation on GPGPU architectures
• Tests of the long-range predictive power of these models