Structure and function of real-world graphs and networks€¦ · – Random people in Nebraska were...
Transcript of Structure and function of real-world graphs and networks€¦ · – Random people in Nebraska were...
Structure and models of
real-world graphs and networks
Jure Leskovec
Machine Learning Department
Carnegie Mellon University
http://www.cs.cmu.edu/~jure/
Jure Leskovec
Networks (graphs)
Jure Leskovec
Examples of networks
• Internet (a)
• citation network (b)
• World Wide Web (c)
(b) (c) (a)
(d)
(e)
• sexual network (d)
• food web (e)
Jure Leskovec
Networks of the real-world (1) • Information networks:
– World Wide Web: hyperlinks
– Citation networks
– Blog networks
• Social networks: people + interactios – Organizational networks
– Communication networks
– Collaboration networks
– Sexual networks
– Collaboration networks
• Technological networks: – Power grid
– Airline, road, river networks
– Telephone networks
– Internet
– Autonomous systems
Florence families Karate club network
Collaboration network Friendship network
Jure Leskovec
Networks of the real-world (2)
• Biological networks
– metabolic networks
– food web
– neural networks
– gene regulatory
networks
• Language networks
– Semantic networks
• Software networks
• …
Yeast protein
interactions
Semantic network
Language network XFree86 network
Jure Leskovec
Types of networks
• Directed/undirected
• Multi graphs (multiple edges between nodes)
• Hyper graphs (edges connecting multiple nodes)
• Bipartite graphs (e.g., papers to authors)
• Weighted networks
• Different type nodes and edges
• Evolving networks:
– Nodes and edges only added
– Nodes, edges added and removed
Jure Leskovec
Traditional approach
• Sociologists were first to study networks:
– Study of patterns of connections between people to understand functioning of the society
– People are nodes, interactions are edges
– Questionares are used to collect link data (hard to obtain, inaccurate, subjective)
– Typical questions: Centrality and connectivity
• Limited to small graphs (~10 nodes) and properties of individual nodes and edges
Jure Leskovec
New approach (1)
• Large networks (e.g., web, internet, on-line
social networks) with millions of nodes
• Many traditional questions not useful anymore:
– Traditional: What happens if a node U is removed?
– Now: What percentage of nodes needs to be
removed to affect network connectivity?
• Focus moves from a single node to study of
statistical properties of the network as a whole
• Can not draw (plot) the network and examine it
Jure Leskovec
New approach (2)
• How the network “looks like” even if I can’t look
at it?
• Need statistical methods and tools to quantify
large networks
• 3 parts/goals:
– Statistical properties of large networks
– Models that help understand these properties
– Predict behavior of networked systems based on
measured structural properties and local rules
governing individual nodes
Jure Leskovec
Statistical properties of networks
• Features that are common to networks of
different types:
– Properties of static networks:
• Small-world effect
• Transitivity or clustering
• Degree distributions (scale free networks)
• Network resilience
• Community structure
• Subgraphs or motifs
– Temporal properties:
• Densification
• Shrinking diameter
Jure Leskovec
Small-world effect (1)
• Six degrees of separation (Milgram 60s) – Random people in Nebraska were asked to send letters to
stockbrokes in Boston
– Letters can only be passed to first-name acquantices
– Only 25% letters reached the goal
– But they reached it in about 6 steps
• Measuring path lengths: – Diameter (longest shortest path): max dij
– Effective diameter: distance at which 90% of all connected pairs of nodes can be reached
– Mean geodesic (shortest) distance l
or
Jure Leskovec
Small-world effect (2)
• Distribution of
shortest path lengths
• Microsoft Messenger
network
– 180 million people
– 1.3 billion edges
– Edge if two people
exchanged at least
one message in one
month period
0 5 10 15 20 25 3010
0
101
102
103
104
105
106
107
108
Distance (Hops)
Num
ber
of
nodes
Pick a random
node, count
how many
nodes are at
distance
1,2,3... hops
7
Jure Leskovec
Small-world effect (3)
• Fact: – If number of vertices within distance r grows exponentially with r, then
mean shortest path length l increases as log n
• Implications: – Information (viruses) spread quickly
– Erdos numbers are small
– Peer to peer networks (for navigation purposes)
• Shortest paths exists
• Humans are able to find the paths: – People only know their friends
– People do not have the global knowledge of the network
• This suggests something special about the structure of the network – On a random graph short paths exists but no one would be able to find
them
Jure Leskovec
Transitivity or Clustering • “friend of a friend is a friend”
• If a connects to b, and b to c,
then with high probability
a connects to c.
• Clustering coefficient C:
C = 3*number of triangles / number of connected triples
• Alternative definition:
Ci = triangles connected to vertex i / number triples
centered on vertex i
– Clustering coefficient:
Ci=1, 1, 1/6, 0, 0
Jure Leskovec
• Clustering coefficient scales as
• It is considerably higher than in a random graph
• It is speculated that in real networks:
C=O(1) as n→∞
In Erdos-Renyi random graph: C=O(n-1)
Transitivity or Clustering (2)
Synonyms network
World Wide Web
Jure Leskovec
Degree distributions (1)
• Let pk denote a fraction of nodes with degree k
• We can plot a histogram of pk vs. k
• In a Erdos-Renyi random graph degree distribution follows Poisson distribution
• Degrees in real networks are heavily skewed to the right
• Distribution has a long tail of values that are far above the mean
• Heavy (long) tail: – Amazon sales
– word length distribution, …
Jure Leskovec
Detour: how long is the long tail?
This is not directly related to
graphs, but it nicely explains
the “long tail” effect. It shows
that there is big market for
niche products.
Jure Leskovec
100
101
102
103
10-6
10-5
10-4
10-3
10-2
0 200 400 600 800 100010
-6
10-5
10-4
10-3
10-2
0 200 400 600 800 10000
0.5
1
1.5
2
2.5
3
3.5x 10
-3
Degree distributions (2)
• Many real world networks contain hubs: highly connected nodes
• We can easily distinguish between exponential and power-law tail by plotting on log-lin and log-log axis
• In scale-free networks maximum degree scales as n1/(α-1)
Degree distribution in
a blog network
lin-lin log-lin
log-log
pk
pk
k
k k
Jure Leskovec
Poisson vs. Scale-free network
Poisson network Scale-free (power-law) network
Function is
scale free if:
f(ax) = b f(x)
(Erdos-Renyi random graph)
Degree distribution is Poisson
Degree
distribution is
Power-law
Jure Leskovec
Degree distribution
number of people a
person talks to on a
Microsoft Messenger
Node degree
Count
X
Highest
degree
Jure Leskovec
Network resilience (1)
• We observe how the connectivity (length of the paths) of the network changes as the vertices get removed
• Vertices can be removed: – Uniformly at random
– In order of decreasing degree
• It is important for epidemiology – Removal of vertices
corresponds to vaccination
Jure Leskovec
Network resilience (2) • Real-world networks are resilient to random attacks
– One has to remove all web-pages of degree > 5 to disconnect the web
– But this is a very small percentage of web pages
• Random network has better resilience to targeted attacks
Fraction of removed nodes
Me
an p
ath
length
Random network
Fraction of removed nodes
Internet (Autonomous systems)
Random
removal
Preferential
removal
Jure Leskovec
Community structure
• Most social networks show community structure – groups have higher density of edges
within than accross groups
– People naturally divide into groups based on interests, age, occupation, …
• How to find communities: – Spectral clustering (embedding into a
low-dim space)
– Hierarchical clustering based on connection strength
– Combinatorial algorithms
– Block models
– Diffusion methods Friendship network of
children in a school
Jure Leskovec
MSN Messenger
Distribution of
Connected components
in MSN Messenger
network
X
Largest
component
Size (number of nodes)
Count
Growth of largest
component over
time in a citation
network
• Graphs have a “giant component ”
• Distribution of connected
components follows a power law
Jure Leskovec
Network motifs (1)
• What are the building blocks (motifs) of
networks?
• Do motifs have specific roles in networks?
• Network motifs detection process:
– Count how many times each subgraph appears
– Compute statistical significance for each subgraph –
probability of appearing in random as much as in real
network
3 node
motifs
Jure Leskovec
Network motifs (2)
• Biological networks
– Feed-forward loop
– Bi-fan motif
• Web graph:
– Feedback with two
mutual diads
– Mutual diad
– Fully connected triad
Jure Leskovec
Network motifs (3)
Transcription
networks
Signal transduction
networks
WWW and
friendship
networks
Word adjacency
networks
Jure Leskovec
Networks over time: Densification • A very basic question: What is the
relation between the number of nodes and the number of edges in a network?
• Networks are becoming denser over time
• The number of edges grows faster than the number of nodes – average degree is increasing
a … densification exponent: 1 ≤ a ≤ 2: – a=1: linear growth – constant out-
degree (assumed in the literature so far)
– a=2: quadratic growth – clique
Internet
Citations
N(t)
N(t)
E(t)
E(t) a=1.2
a=1.7
Jure Leskovec
Densification & degree distribution
• How does densification affect
degree distribution?
• Given densification exponent a, the
degree exponent is:
– (a) For γ=const over time, we obtain
densification only for 1<γ<2, then γ=a/2
– (b) For γ<2 degree distribution has to
evolve according to:
• Power-law: y=b xγ, for γ<2 E[y] = ∞
Degree exponent
over time
γ(t)
pk=kγ
γ(t)
a=1.1
a=1.6
(a)
(b)
Jure Leskovec
Shrinking diameters
• Intuition says that
distances between the
nodes slowly grow as the
network grows (like log n)
• But as the network grows
the distances between
nodes slowly decrease
Internet
Citations
Models of network
generation and evolution
Jure Leskovec
Recap (1)
• Last time we saw: – Large networks (web, on-line social networks) are
here
– Many traditional questions not useful anymore
– We can not plot the network so we need statistical methods and tools to quantify large networks
– 3 parts/goals: • Statistical properties of large networks
• Models that help understand these properties
• Predict behavior of networked systems based on measured structural properties and local rules governing individual nodes
Jure Leskovec
Recap (2)
• We also so features that are common to networks of various types:
• Properties of static networks: – Small-world effect
– Transitivity or clustering
– Degree distributions (scale free networks)
– Network resilience
– Community structure
– Subgraphs or motifs
• Temporal properties: – Densification
– Shrinking diameter
Jure Leskovec
Outline for today
• We will see the network generative models for modeling networks’ features: – Erdos-Renyi random graph
– Exponential random graphs (p*) model
– Small world model
– Preferential attachment
– Community guided attachment
– Forest fire model
• Fitting models to real data – How to generate a synthetic realistic looking network?
Jure Leskovec
(Erodos-Renyi) Random graphs
• Also known as Poisson random graphs or Bernoulli graphs – Given n vertices connect each pair i.i.d. with
probability p
• Two variants: – Gn,p: graph with m edges appears with probability
pm(1-p)M-m, where M=0.5n(n-1) is the max number of edges
– Gn,m: graphs with n nodes, m edges
• Very rich mathematical theory: many properties are exactly solvable
Jure Leskovec
Properties of random graphs
• Degree distribution is Poisson since the
presence and absence of edges is
independent
• Giant component: average degree k=2m/n:
– k=1-ε: all components are of size log n
– k=1+ε: there is 1 component of size n
• All others are of size log n
• They are a tree plus an edge, i.e., cycles
• Diameter: log n / log k
!)1(
k
ezpp
k
np
zkknk
k
Jure Leskovec
Evolution of a random graph fo
r non-G
CC
vert
ices
k
Jure Leskovec
Subgraphs in random graphs
Expected number of
subgraphs H(v,e) in Gn,p is
a
pnp
a
v
v
nXE
eve
!)(
a... # of isomorphic graphs
Jure Leskovec
Random graphs: conclusion
• Pros: – Simple and tractable model
– Phase transitions
– Giant component
• Cons: – Degree distribution
– No community structure
– No degree correlations
• Extensions: – Configuration model
• Random graphs with arbitrary degree sequence
• Excess degree: Degree of a vertex of the end of random edge: qk = k pk
Configuration model
Jure Leskovec
Exponential random graphs
(p* models)
• Comes from social sciences
• Let εi set of measurable properties
of a graph (number of edges,
number of nodes of a given
degree, number of triangles, …)
• Exponential random graph model
defines a probability distribution
over graphs:
Examples of εi
Jure Leskovec
Exponential random graphs
• Includes Erdos-Renyi as a special case
• Assume parameters βi are specified – No analytical solutions for the model
– But can use simulation to sample the graphs:
• Define local moves on a graph: – Addition/removal of edges
– Movement of edges
– Edge swaps
• Parameter estimation: – maximum likelihood
• Problem: – Can’t solve for transitivity (produces
cliques)
– Used to analyze small networks
Example of parameter estimates:
Jure Leskovec
Small-world model • Used for modeling network transitivity
• Many networks assume some kind of geographical proximity
• Small-world model: – Start with a low-dimensional regular lattice
– Rewire: • Add/remove edges to create shortcuts to join remote parts of the
lattice
• For each edge with prob p move the other end to a random vertex
• Rewiring allows to interpolate between regular lattice and random graph
Jure Leskovec
Small-world model
• Regular lattice (p=0): – Clustering coefficient
C=(3k-3)/(4k-2)=3/4
– Mean distance L/4k
• Almost random graph (p=1): – Clustering coefficient C=2k/L
– Mean distance log L / log k
• No power-law degree distribution
Rewiring probability p
Degree distribution
Jure Leskovec
Models of evolution
• Models of network evolution:
– Preferential attachment
– Edge copying model
– Community Guided Attachment
– Forest Fire model
• Models for realistic network generation:
– Kronecker graphs
Jure Leskovec
Preferential attachment
• Models the growth of the network
• Preferential attachment (Price 1965, Albert & Barabasi 1999): – Add a new node, create m out-links
– Probability of linking a node ki is proportional to its degree
• Based on Herbert Simon’s result – Power-laws arise from “Rich get richer” (cumulative advantage)
• Examples (Price 1965 for modeling citations): – Citations: new citations of a paper are proportional to the number
it already has
Jure Leskovec
Preferential attachment
• Leads to power-law degree
distributions
• But:
– all nodes have equal (constant) out-
degree
– one needs a complete knowledge of the
network
• There are many generalizations and
variants, but the preferential
selection is the key ingredient that
leads to power-laws
3 kpk
Jure Leskovec
Edge copying model
• Copying model:
– Add a node and choose k the number of edges to add
– With prob β select k random vertices and link to them
– With prob 1-β edges are copied from a randomly
chosen node
• Generates power-law degree distributions with
exponent 1/(1-β)
• Generates communities
• Related Random-surfer model
Jure Leskovec
Community guided attachment
• Want to model/explain
densification in
networks
• Assume community
structure
• One expects many
within-group
friendships and fewer
cross-group ones
Self-similar university
community structure
CS Math Drama Music
Science Arts
University
Jure Leskovec
Community guided attachment
• Assuming cross-community linking probability
• The Community Guided Attachment leads to Densification Power Law with exponent
– a … densification exponent
– b … community tree branching factor
– c … difficulty constant, 1 ≤ c ≤ b
• If c = 1: easy to cross communities – Then: a=2, quadratic growth of edges – near clique
• If c = b: hard to cross communities – Then: a=1, linear growth of edges – constant out-degree
Jure Leskovec
Forest Fire Model
• Want to model graphs that density and have shrinking diameters
• Intuition:
– How do we meet friends at a party?
– How do we identify references when writing papers?
Jure Leskovec
Forest Fire Model for
directed graphs
• The model has 2 parameters:
– p … forward burning probability
– r … backward burning probability
• The model:
– Each turn a new node v arrives
– Uniformly at random chooses an “ambassador” w
– Flip two geometric coins to determine the number in-
and out-links of w to follow (burn)
– Fire spreads recursively until it dies
– Node v links to all burned nodes
Jure Leskovec
Forest Fire Model
• Simulation experiments
• Forest Fire generates graphs that densify and have shrinking diameter
densification diameter
1.32
N(t)
E(t)
N(t)
dia
me
ter
Jure Leskovec
Forest Fire Model
• Forest Fire also generates graphs with
heavy-tailed degree distribution
in-degree out-degree
count vs. in-degree count vs. out-degree
Jure Leskovec
Forest Fire: Parameter Space
• Fix backward probability r and vary forward burning probability p
• We observe a sharp transition between sparse and clique-like graphs
• Sweet spot is very narrow
Sparse
graph
Clique-like
graph Increasing
diameter
Decreasing
diameter
Constant
diameter
Jure Leskovec
Kronecker graphs
• Want to have a model that can generate a realistic graph: – Static Patterns
• Power Law Degree Distribution
• Small Diameter
• Power Law Eigenvalue and Eigenvector Distribution
– Temporal Patterns • Densification Power Law
• Shrinking/Constant Diameter
• For Kronecker graphs all these properties can actually be proven
Jure Leskovec Adjacency matrix
Kronecker Product – a Graph
Intermediate stage
Adjacency matrix
Jure Leskovec
Kronecker Product – Definition
• The Kronecker product of matrices A and B is given by
• We define a Kronecker product of two graphs as a Kronecker product of their adjacency matrices
N x M K x L
N*K x M*L
Jure Leskovec
Stochastic Kronecker Graphs
• Create N1 N1 probability matrix P1
• Compute the kth Kronecker power Pk
• For each entry puv of Pk include an
edge (u,v) with probability puv
0.5 0.2
0.1 0.3
P1
Instance
Matrix G2
0.25 0.10 0.10 0.04
0.05 0.15 0.02 0.06
0.05 0.02 0.15 0.06
0.01 0.03 0.03 0.09
P2
flip biased
coins
Kronecker
multiplication
Jure Leskovec
Fitting Kronecker to Real Data
• Given a graph G and Kronecker matrix P1 we
can calculate probability that P1 generated G:
P(G|P1): 0.25 0.10 0.10 0.04
0.05 0.15 0.02 0.06
0.05 0.02 0.15 0.06
0.01 0.03 0.03 0.09
0.5 0.2
0.1 0.3
P1 Pk
1 1 0 0
1 1 1 0
0 1 1 1
0 0 1 1 G
σ… node labeling
P(G|P1)
Jure Leskovec
Fitting Kronecker: 2 challenges
• Invariance to node labeling σ (there are N! labelings)
• Calculating P(G|P1) takes O(N2) (since one needs to consider every cell of adjacency matrix)
0.25 0.10 0.10 0.04
0.05 0.15 0.02 0.06
0.05 0.02 0.15 0.06
0.01 0.03 0.03 0.09
0.5 0.2
0.1 0.3
P1 Pk
1 1 0 0
1 1 1 0
0 1 1 1
0 0 1 1 G
P(G|P1)
==
1
2
3
4
2
1
4
3
)(),|()|( 11
PPGPPGP
Jure Leskovec
Fitting Kronecker: Solutions
• Node Labeling: can use MCMC sampling to average over
(all) node labelings
• P(G|P1) takes O(N2): Real graphs are sparse, so calculate
P(Gempty) and then “add” the edges. This takes O(E).
0.25 0.10 0.10 0.04
0.05 0.15 0.02 0.06
0.05 0.02 0.15 0.06
0.01 0.03 0.03 0.09
P=
1 1 0 0
1 1 1 0
0 1 1 1
0 0 1 1
G=
σ… node labeling
Jure Leskovec
Experiments on real AS graph Degree distribution Hop plot
Network value Adjacency matrix eigen values
Jure Leskovec
Why fitting generative models?
• Parameters tell us about the structure of a graph
• Extrapolation: given a graph today, how will it
look in a year?
• Sampling: can I get a smaller graph with similar
properties?
• Anonymization: instead of releasing real graph
(e.g., email network), we can release a synthetic
version of it
Processes taking place
on networks
Jure Leskovec
Epidemiological processes
• The simplest way to spread a virus over the network – S: Susceptible
– I: Infected
– R: Recovered (removed)
• SIS model: 2 parameters – β … virus birth rate
– δ … virus death (recovery) rate
• SIR model: as one gets cured, he or she can not get infected again
SIS model
ζit depends on β and topology
Jure Leskovec
Epidemic threshold for SIS model
• How infectious the virus needs to be to survive in the network?
• First results on power-law networks suggested that any virus will prevail
• New result that works for any topology:
• For s>1 virus prevails
• For s<1 virus dies
λ1… largest eigen value
of graph adjacency matrix
Jure Leskovec
Navigation in small-world networks
• Milgram’s experiment showed:
– (a) short paths exist in networks
– (b) humans are able to find them
• Assume the following setting:
– Nodes of a graph are scattered on a plane
– Given starting node u and we want to reach target
node v
– A small world navigation algorithm navigates the
network by always navigating to a neighbor that is
closest (in Manhattan distance) to target node v
u
v
Jure Leskovec
Navigation in small-world networks
• Start with random lattice: – Each node connects with their 4
immediate neighbors
– Long range links are added with probability proportional to the distance between the points (p(u,v) ~ dα)
• Can be show that only for α=2 delivery time is poly-log in number of nodes n
Deliver time T < nβ
Network creation
Jure Leskovec
Navigation in a real-world network
• Take a social network of
500k bloggers where for
each blogger we know
their geographical location
• Pick two nodes at random
and geographically greedy
navigate the network
• Results:
– 13% success rate (vs. 18%
for Milgram)
Distribution of path lengths
Friendships vs. distance
Jure Leskovec
Navigation in real-world network
• Geographical distance may not be the right kind of distance
• Since population is non-uniform let’s use rank based friendship distance:
i.e., we measure the distance d(u,v) by the number of people living closer to v than u does
• Then
And the proof still works
Jure Leskovec
• Some references used to prepare this talk: – The Structure and Function of Complex Networks, by Mark Newman
– Statistical mechanics of complex networks, by Reka Albert and Albert-Laszlo Barabasi
– Graph Mining: Laws, Generators and Algorithms, by Deepay Chakrabarti and Christos Faloutsos
– An Introduction to Exponential Random Graph (p*) Models for Social Networks by Garry Robins, Pip Pattison, Yuval Kalish and Dean Lusher
– Graph Evolution: Densification and Shrinking Diameters, by Jure Leskovec, Jon Kleinberg and Christos Faloutsos
– Realistic, Mathematically Tractable Graph Generation and Evolution, Using Kronecker Multiplication, by Jure Leskovec, Deepayan Chakrabarti, Jon Kleinberg and Christos Faloutsos
– Navigation in a Small World, by Jon Kleinberg
– Geographic routing in social networks, by David Liben-Nowell, Jasmine Novak, Ravi Kumar, Prabhakar Raghavan, and Andrew Tomkins
– Some plots and slides borrowed from Lada Adamic, Mark Newman, Mark Joseph, Albert Barabasi, Jon Kleinberg, David Lieben-Nowell, Sergi Valverde and Ricard Sole
Jure Leskovec
Rough random material
that did not make it into the
presentation
Jure Leskovec
Bow-tie structure of the web
SCC 56 M
OUT 44 M
IN 44 M
Broder & al. WWW 2000, Dill & al. VLDB 2001
DISC 17 M
TENDRILS 44M
Jure Leskovec
Study of 3 websites • study over three universities’ publicly indexable Web
sites
Jure Leskovec
Australia
In- and out-degree distributions
Jure Leskovec
In- and out-degree distributions
New Zealand
Jure Leskovec
United Kingdom
In- and out-degree distributions
Jure Leskovec
We assume this node would like to connect to a centrally located node; a node whose distances to other nodes is minimized.
dij is the Euclidean distance hj is some measure of the “centrality” of node j α is a parameter – a function of the final number n of points, gauging the relative importance of the two objectives
Jure Leskovec
Fabrikant et al. define 3 possible measures of “centrality” 1. The average number of hops from other nodes 2. The maximum number of hops from another node 3. The number of hops from a fixed center of the tree
Jure Leskovec
α is the crux of the theorem!
Why? Here are some examples:
Fabrikant&al
Jure Leskovec
If α is too low, then the Euclidian distances become unimportant, and the network resembles a star:
Fabrikant&al
Jure Leskovec
But if α grows at least as fast as √n, where n is the final number of points, then distance becomes too important, and minimum spanning trees with high degree occur, but with exponentially vanishing probability – thus not a power law. if α is anywhere in between, we have a power law
Through a rather complex and elaborate proof, Fabrikant&al prove this initial assumption will produce a power law distribution – I’ll save you the math!