Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert.
-
Upload
ashlyn-stevens -
Category
Documents
-
view
218 -
download
2
Transcript of Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert.
Bob-the-Builder vs. Fix-it-Felix
Maintaining Overlays in Dynamic Graphs
Seth Gilbert
What is an overlay?Overlay Networks
Given a collection of (changing) servers:
• Choose a good subset of the edges.
What is an overlay?Overlay Networks
Given a collection of (changing) servers:
• Choose a good subset of the edges.
• Subgraph has low degree.
• Subgraph has low diameter.
What is an overlay?Overlay Networks
Given a collection of (changing) servers:
• Choose a good subset of the edges.
• Subgraph has low degree.
• Subgraph has low diameter.
• Maintain edges as servers come and go.
Can we build it?
Yes we can!
We can fix it!
Bitcoin is broken(So why are you wasting your time here when you could be busy hacking bitcoins?)
What goes wrong?
Bitcoin is a peer-to-peer overlay network• Overlay is used for all communication.• Overlay is assumed to be reliable.
What goes wrong?
Bitcoin is a peer-to-peer overlay network• Overlay is used for all communication.• Overlay is assumed to be reliable.• Nodes arrive and leave all the time.• Malicious/greedy users can partition the
network.
What goes wrong?
Basic overlay maintenance idea:• Connect to a few arbitrary neighbors.• Maintain a fixed number of neighbors.• Accept incoming connection requests.
Details: Eclipse Attacks on Bitcoin’s Peer-to-Peer Network by Heilman, Kendler, Zohar, Goldberg
Bitcoin is a peer-to-peer overlay network• Overlay is used for all communication.• Overlay is assumed to be reliable.• Nodes arrive and leave all the time.• Malicious/greedy users can partition the
network.
Rapid and significant churn• Users constantly arriving and departing.• Network changes continuously.• No stable state.
Malicious / greedy users• Some users may not follow the protocol.• Or all users may be greedy!
Long-lived system
BitcoinWhy is this a hard problem?
I am not going to solve the Bitcoin problem today.
A few existing overlay networks…
ChordKademlia
Pastry Tapestry
Skip+
HSkip+
Patricia Tries
DexXHeal
Forgiving Tree
PGrid
Skipnet
RN Protocol
dHamiltonianCycles
Avatar
Ca-Re-Chord
HyperCubes
HyperRing
Chameleon
Re-Chord
Tiara
Corona
What do we know?Discussion of existing approaches.Is the problem solved?Are the existing solutions practical?
Ongoing workSome ideas that we are thinking about.
Challenge for us to work onTo work on after the workshop is over.
Today’s GoalsA Workshop Talk
Collaborators: Gopal Pandurangan, Peter Robinson, and Amitabh Trahan
Overlay Networks
Overlay NetworksGround Rules
Underlying network:• Collection of nodes.
• Nodes arrive (join). Joining node is connected to someone.
• Nodes leave (fail).
• Precise model TBA.
Overlay NetworksGround Rules
Communication:• Every node has an address.
• Every node has an address book.
• A node can send a message to every node in the address book.
• A node can send an address to another node.
Joe
Sue
JoeSue
< Sue >
Overlay NetworksGoals
Overlay network:• Low degree constant or logarithmic.
• Low diameter logarithmic or polylogarithmic
Note: every existing solution guarantees these properties.
Overlay NetworksGoals
Routable:• There exist short paths…
• … and we can find short paths.
Note: random graphs may not be good!
Overlay NetworksGoals
Other properties:• Good expansion
• Good conductance
Random walks converge quickly
Diameter is small
Overlay NetworksIssues
Churn:• Nodes arrive.
• Nodes leave.
Note: Not considering Byzantine/malicious failures today!
Overlay NetworksIssues
Synchrony vs. Asynchrony:• Synchronous: computation proceeds in rounds.
• Asynchronous: arbitrary message/computation delays
Today: assume computation proceeds in rounds.
Overlay NetworksIssues
Oblivious vs. Adaptive scheduler/adversary:• Oblivious: schedule/arrivals/failures fixed in advance.
• Adaptive: schedule/arrivals/failures depends on execution.
Questions: • Do actions of the algorithm correlate with crashes?
• Is there an attacker using knowledge of the system?
Attacker can crash critical nodes?
Sending many messages can overload a link?
Overlay NetworksMetrics
Rate of churn:• How fast can nodes join and leave?
• What is the maximum rate that an algorithm can tolerate?
Overlay NetworksMetrics
Rate of recovery:• What happens when something goes wrong?
• How fast does the algorithm reconstruct a good overlay?
Overlay NetworksMetrics
Costs:• Message complexity how many messages per round?
• Communication complexity how many bits of communication?
• Quiescent complexity what happens if changes stop?
• Adaptive complexity how do costs relate to the changes?
Overlay NetworksOverlay Networks
Given a collection of (changing) servers:
• Choose a good subset of the edges.
• Subgraph has low degree.
• Subgraph has low diameter.
• Maintain edges as servers come and go.
OverlaysA Play in Three Acts
Act I : Fix-it-Felix and the Half-Life-Hobgoblin
Wherein we meet our hero Felix, as he races to keep up with the Half-Life-Hobgoblin, fixing the overlay as fast as it is being destoryed.
Act II : Bob-the-Builder and the Destabilizing-Demon
Wherein we meet our hero Bob, as he counters the Destabilizing-Demon, rebuilding the overlay no matter how badly damaged.
Act III : Here Be Dragons
Wherein we explore the dangerous and unknown path forward and away from the hobgoblins and demons of our despair.
Basic Statistics• Home: Niceland
• Role: Good Guy
• Antagonist: Ralph
• Tool: magic hammer
• Powers: fixing anything
Introductions
Fix-it-FelixI can fix it!
Fix it:• On insert: fix it!
• On departure: fix it!
• On any change: fix it!
Basic philosophy:• Fix changes as soon as
they occur.
• Fix it right then and there.
• Fixes are local and immediate.
Fix-it-Felix Approach
How to cope with changes?I can fix it!
Approach:• Start with good
overlay.
• Bound rate of churn.
• Fix the overlay faster than the churn destroys it.
• Maintain good properties throughout.
Fix-it-Felix Approach
How to cope with changes?I can fix it!
Measuring ChurnMetric: per-period-churn
In every round:• At most k nodes join every r
rounds.
• At most k nodes leave every r rounds.Ex: k=1, r=log n At most 1 node joins / leaves every log(n) rounds.
2520151050
arrive arrivearrive
departdepart depart
departdepart
Measuring ChurnHalf-life of a System
Minimum time in which either:
• Number of nodes doubles.
• Number of nodes halves.If half life is H and there are n nodes at time t, then between [t, t+H], there are at least (n/2) and at most 2n nodes in the system.
2520151050
arrive arrive arrive arrive departarrive
departdepartarrive
H = 5n = 6
Measuring ChurnHalf-life of a System
Ideal Theorem:
If the half-life of the system is at least θ(log n), then the overlay is always good.
If half life is H and there are n nodes at time t, then between [t, t+H], there are at least (n/2) and at most 2n nodes in the system.
Minimum time in which either:• Number of nodes doubles.
• Number of nodes halves.
Tolerating ChurnChord
Advantages:• Tolerates high rate of churn.
• Overlay has many good properties.
• Simple to implement.
Disadvantages: • Oblivious adversary
• Fragile
• Limitations on where nodes can join?
• Only supports one topology.
Once the ring is disrupted, all is lost!
Easy to attack, at risk of correlated failures.
[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
Basic overlay:• Ring topology
• Nodes distributed randomly on ring.
• Edges:- successor- predecssor- “fingers”
• Approximates a hypercube.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
Maintenance:
• Ignore “fingers.”
- Only matters for routing/lookup.
- Easy to rebuild.
• Maintain successor/predecessor.
• Ensures that ring remains a ring.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
Assume adversary is oblivious:
• Adversary does not know where nodes live on the ring.
• New nodes are inserted “randomly.”
• Nodes are deleted “randomly.”
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
Think about crashes:
• Assume at most n/2 nodes crash.
• Ring is disconnected?
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
Think about crashes:
• Assume at most n/2 nodes crash.
• Maintain connection to log(n) successors.
• With high probability, at least 1 successor survives.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
Think about joining:
• Assume at most n/2 nodes join.
• Joining node is attached “randomly.”
• Creates appendages hanging off ring.
• At most O(log n) joining nodes in one place.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
Repair mechanism:
• Query successors for their successors.
• Query successors for their predecessor.
• Update local state.
• Notify successors of changes.
Note: I’m being a bit imprecise.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
Repair mechanism:
• Query successors for their successors.
• Query successors for their predecessor.
• Update local state.
• Notify successors of changes.
Note: I’m being a bit imprecise.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
Beware: what if this node is deleted?
Time analysis:
• Every O(1) steps (in expectation), one insert/delete is resolved.
• Within O(log n) steps, with high probability, all inserts/deletes from the last half-life are resolved.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
Analysis ideas:
• Define an almost good state.
• Show that in the almost good state, the overlay has good properties.
• Show that if the half-life length Ω(log n), then the overlay is always almost good.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
Tolerating ChurnChord Summary
Advantages:• Tolerates high rate of churn.
• Overlay has many good properties.
• Simple to implement.
Disadvantages: • Oblivious adversary
• Fragile
• Limitations on where nodes can join.
• Only supports one topology.
Once the ring is disrupted, all is lost!
Easy to attack, at risk of correlated failures.
*Re-Chord (KKS11) yields O(n log n) stabilization.
Think about crashes:
• Within O(nc) steps, log(n) consecutive nodes crash.
• If you wait long enough, bad events happen.
• Then the ring is disconnected.
• It will take a long time to fix!
Tolerating Churn
Chord is fragile?
Recent work by Pamela Zave has found bugs/imprecision in original Chord descriptions:
Using lightweight modeling to understand Chord
Why the Chord Ring-Maintenance Protocol is not
Correct How to make Chord correct
Conclusion: Chord works, but only if you really get it right.
Tolerating ChurnChord details are tricky
Approach:• Start with good
overlay.
• Bound rate of churn.
• Fix the overlay faster than the churn destroys it.
• Maintain good properties throughout.
Fix-it-Felix Approach
Chord tolerates churn.I can fix it!
Tolerating ChurnAlternate Fix-it-Felix approach
Load-balanced hypercube:• At most O(log n) arrivals/departures per round.
• Basic idea: map O(log n) nodes to each vertex of a hypercube.
• Load-balance among hypercube vertices to cope with churn.
• Grow/shrink hypercube as needed.
[Kuhn, Schmid, Wattenhofer, IPTPS’05]
Tolerating ChurnAlternate Fix-it-Felix approach
Load-balanced hypercube:• At most O(log n) arrivals/departures per round.
• Basic idea: map O(log n) nodes to each vertex of a hypercube.
• Load-balance among hypercube vertices to cope with churn.
• Grow/shrink hypercube as needed.
[Kuhn, Schmid, Wattenhofer, IPTPS’05]
Interesting ideas: • Map real nodes to a virtual topology.
• Use load balancing to keep real nodes well distributed.
• Tolerates adaptive (and omniscient) adversary.
Note: (unavoidable) weaker per-round adversarial limit.
Tolerating ChurnAlternate Fix-it-Felix approach
Random (dynamic) graph:• Maintain a random constant-degree graph.
• Up to θ(n/log n) churn per round.
• Based on random walks.
[Augustine, Pandurangan, Robinson, Roche, Upfal, FOCS’15]
Disadvantages: • Not routable.
• Oblivious adversary
• Network size remains fixed.
• Limitation on where new nodes can be attached.
Tolerating ChurnAlternate Fix-it-Felix approach
XHeal:• Fixes any graph that has deletions (and insertions).
• Carefully replaces missing nodes with small expanders.
• Preserves good properties of original graph:- Bounded stretch: log(n)- Bounded degree-growth
[Pandurangan, Trehan, PODC’11]
Tolerating ChurnAlternate Fix-it-Felix approach
XHeal:• Fixes any graph that has deletions (and insertions).
• Carefully replaces missing nodes with small expanders.
• Preserves good properties of original graph:- Bounded stretch: log(n)- Bounded degree-growth
[Pandurangan, Trehan, PODC’11]
Interesting aspects: • One change at a time.
• Local repair.
• Works for any topology/graph.
Fix-it-Felix ApproachMoral of the story:
General ideas:• Local repair.• Simple updates.• Maintain approximate structure.
Several existing solutions:• Oblivious adversary.• Varying rates of churn.• Subtle dependence on the joining model.
Challenges:• Adaptive adversary • Tricky analysis, since failures are ongoing.• Hard to keep costs proportional to changes.
I can fix it!
OverlaysA Play in Three Acts
Act I : Fix-it-Felix and the Half-Life-Hobgoblin
Wherein we meet our hero Felix, as he races to keep up with the Half-Life-Hobgoblin, fixing the overlay as fast as it is being destoryed.
Act II : Bob-the-Builder and the Destabilizing-Demon
Wherein we meet our hero Bob, as he counters the Destabilizing-Demon, rebuilding the overlay no matter how badly damaged.
Act III : Here Be Dragons
Wherein we explore the dangerous and unknown path forward and away from the hobgoblins and demons of our despair.
Basic Statistics• Home: Sunflower
Valley
• Role: Building Contractor
• Antagonist: None.
• Tool: Distributed team of building machinery.
• Powers: Teamwork.
Introductions
Bob-the-Builder Can we build it? Yes we can!
Rebuild good overlay:• On insert: rebuild!
• On departure: rebuild!
• On any change: rebuild!
Basic philosophy:• Assume network is in
some arbitrary topology.
• Build a good overlay.
• Build it.. fast!
Bob–the-Builder Approach
How to cope with changes? Can we build it? Yes we can!
From any initial state:• Build a good overlay.
• Fast construction.
• No ongoing churn.
Self-stabilization:• Arbitrary initial
state corruption.
• Converges to a good state.
Bob–the-Builder Approach
How to build an overlay? Can we build it? Yes we can!
Dynamic Graph ModelAssume an arbitrary initial state:
Initially:• Graph is given in an arbitrary connected topology.
• State of the nodes may be corrupted.
Dynamic Graph ModelConstruct a good overlay:
In every round:• Exchange messages with neighbors.
• Adjust edges.
• Improve the overlay.
No further joins/leaves allowed (until overlay is constructed).
Dynamic Graph ModelStabilization:
Eventually:• Overlay is constructed.
• Good properties are guaranteed.
• In good state, joins and leaves may be supported.
From any initial state:• Build a good overlay.
• Fast construction.
• No ongoing churn.
Self-stabilization:• Arbitrary initial
state corruption.
• Converges to a good state.
Bob–the-Builder Approach
How to build an overlay? Can we build it? Yes we can!
Self-Stabilizing OverlaysSkip+ Overlay Graphs:
Advantages:• Stabilizes quickly.
• Supports efficient joins and leaves.
• Simple rules.
Disadvantages: • Large message / communication complexity (in the worst
case).
• Large degree during construction (in the worst case).
• Oblivious adversary (for departures).
• Churn??
• Only supports one topology.
[Jacob, Richa, Scheideler, Schmid, Taubig, PODC’09]
Skip+ OverlaysClassical Skip List
011010 001000 010000 010001 001111 011
Skip+ OverlaysClassical Skip List
011010 001000 010000 010001 001111 011
Advantages:•Fast search / insert in log(n) time.•Fault-tolerant
Skip+ OverlaysClassical Skip List
011010 001000 010000 010001 001111 011
Disadvantage: congestion•Only one root.•Load is not balanced over nodes.
Skip+ OverlaysSkip Graph
011010 001000 010000 010001 001111 011
Skip+ OverlaysSkip+ Graph
011010 001000 010000 010001 001111 011
Skip+ OverlaysHow to build it?
Overlay guarantees:• Fast searches / routing.
• Fast joins / leaves.
• Small diameter.
• Small degree.
Overlay construction issues: • Initially, diameter of the graph may be large.
• How do nodes find their neighbors efficiently?
• How do nodes sort themselves properly into linked lists?
• Leverage parallelism? Not one insertion at a time!
Skip+ OverlaysSimple trick: pointer doubling
In every round:• Send your entire address book to all your neighbors.
• In every round, your “knowledge diameter” doubles.
• Note: each rounds squares the adjacency matrix.
Within log(n) rounds, graph is a clique.
42 8 12
Skip+ OverlaysSimple trick: pointer doubling
Once you have a clique:• Delete edges not in the overlay.
• Final graph matches good topology.
Within log(n)+1 rounds, graph is Skip+.
[Example: Skip List]
Skip+ OverlaysSimple trick: pointer doubling
Bad news:• Degree grows very large: n-1.
• Messages grow very large: θ(n).
• Number of messages per round is large: θ(n2).
Conclusion: fast, but inefficient, overlay construction.
42 8 12
Skip+ OverlayMore efficient construction:
Basic idea:• Minimize expensive doubling steps.• Route nodes directly using existing overlay, if
possible.
Local view:• Each node uses its local view to determine who its
neighbors should be in the final overlay.
Stable edges:• An edge (u,v) is stable if it appears to be in the final
Skip+ overlay according to both the views of u and v.• Otherwise, an edge is temporary.
Skip+ OverlaysSimple rules:
Rule 1: Introduce Friends • Notify stable neighbors of neighbors they should
know about (including yourself).
Rule 2: Forward Temporary Edges (Routing)• Forward a temporary neighbor to a stable neighbor
with the largest shared prefix.
Rule 3a: Introduce All (Pointer Doubling)• If your set of stable neighbors changes (e.g., due to a
change in views), then introduce all your neighbors to each other.
Rule 3b: Linearize• Help organize your neighbors at the same “level” into
a linked list.
• d
Skip+ OverlayHow does it work?
In general:• Guarantees fast O(log n) overlay construction.• Best effort to restrain message size / degree during
construction.
Locally checkable:• Nodes can locally determine whether the overlay is
correct.• If every node thinks it is correct, then it is!
Bad news:• Still can lead to large degree, large messages, and
many messages per round, in the worst case.
Self-Stabilizing OverlaysSkip+ Overlay Graphs:
Advantages:• Stabilizes quickly.
• Supports efficient joins and leaves.
• Simple rules.
Disadvantages: • Large message / communication complexity (in the worst
case).
• Large degree during construction (in the worst case).
• Oblivious adversary (for departures).
• Churn??
• Only supports one topology.
Self-Stabilizing OverlaysSkip+ Overlay Graphs:
Many improvements, similar ideas:• HSkip+ : Heterogeneous bandwidth
• Corona : Deterministic
• TCF-Skip+: Generalization, local detection analysis, etc.
011010 001000 010000010001 001111 011
Self-Stabilizing OverlaysAlternate Bob-the-Builder approach
Patricia-Tree-like Overlay:• Forms a tree from a weakly connected graph.• Asynchronous.• Communication-efficient:
• Small messages, O(1) per round per nod.• Low contention
[Angluin, Aspnes, Chen, Wu, Yin, SPAA’05]
Disadvantages: • Initially: every node has low degree.
• Builds a tree.
• Later version (SSS’07) builds better overlay.
Bob-the-Builder ApproachMoral of the story:
General ideas:• Deterministic, exact final structure.• Converge to final structure.• Graph square or component merging
to converge fast.
Several existing solutions:• Oblivious adversary.• No joins/leaves during stabilization.
Challenges:• How to keep messages small?• How to avoid graph squaring?• How to tolerate ongoing changes?
Can we build it? Yes we can!
OverlaysA Play in Three Acts
Act I : Fix-it-Felix and the Half-Life-Hobgoblin
Wherein we meet our hero Felix, as he races to keep up with the Half-Life-Hobgoblin, fixing the overlay as fast as it is being destoryed.
Act II : Bob-the-Builder and the Destabilizing-Demon
Wherein we meet our hero Bob, as he counters the Destabilizing-Demon, rebuilding the overlay no matter how badly damaged.
Act III : Here Be Dragons
Wherein we explore the dangerous and unknown path forward and away from the hobgoblins and demons of our despair.
Good overlay:• Small diameter / small degree: O(log n).• Routable / good expansion.
Fast and robust construction:• Rapidly formation from any initial state.• Self-stabilizing.
Churn tolerant:• Tolerates large fractions of nodes joining and leaving
every round.• Oblivious adversary (?).
Efficient construction/maintenance:• Small messages (e.g., size O(log n)).• Message-efficient (e.g., 1 message per node per round).
Big PictureGoals:
Collaborators: Gopal Pandurangan, Peter Robinson, and Amitabh Trahan
OverviewPlan:
Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.
Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.
Construction algorithm:• Divide-and-conquer.• Merge and collapse components.
Robustness:• Self-stabilization.• Churn.
Virtual OverlaysWhat is a good topology?
Sequence of graphs G1, G2, …, Gn
• Graph Gj has [j, 2j] nodes.
• Graph Gj has O(log j) degree and diameter.
Expandable:• There exists a mapping from each node in Gj to 1 or 2
nodes in G2j.
Random sampling:• Supports a mechanism for randomly sampling nodes.• E.g., fast random walks.
Permutation routing:• Supports efficiently routing permutations.
Note: many (deterministic) expanders satisfy these requirements.
Virtual OverlaysMap virtual topology to real nodes
For each real node:• Assign O(1) virtual nodes to it.
• Connect two real nodes if any of their virtual nodes are connected.
Inspiration: The Forgiving Tree [HST’12], DeX [PRT’14] and [KSW’05]
Virtual Real
OverviewPlan:
Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.
Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.
Construction algorithm:• Divide-and-conquer.• Merge and collapse components.
Robustness:• Self-stabilization.• Churn.
MergingCombining two overlay networks
Assume two overlay networks:• Each overlay is a properly mapped virtual topology.
• The two overlays are connected by one edge.
Key challenge: merge the two overlay networks.
MergingCombining two overlay networks
Step 1: Grow the virtual topology• Double one overlay (using topology mapping Gn G2n).
• Creates excess virtual nodes.
MergingCombining two overlay networks
Step 1: Grow the virtual topology• Double one overlay (using topology mapping Gn G2n).
• Creates excess virtual nodes.
Step 2: Create new edges• Use permutation routing.
MergingCombining two overlay networks
Step 3: Distribute excess virtual nodes • Send new virtual nodes on a random walk of old
topology.• (Use random sampling of target topology.)
MergingCombining two overlay networks
Step 3: Distribute excess virtual nodes • Send new virtual nodes on a random walk of old
topology.• (Use random sampling of target topology.)
Problem: only one bridge.
MergingCombining two overlay networks
Step 3: Distribute excess virtual nodes • Send new virtual nodes on a random walk of old
topology.• (Use random sampling of target topology.)• Every success creates a new bridge.• Number of bridges doubles (i.e., exponential growth).
MergingCombining two overlay networks
Step 3: Distribute excess virtual nodes • Send new virtual nodes on a random walk of old
topology.• (Use random sampling of target topology.)
Problem: eventually, hard to find an empty node.
MergingCombining two overlay networks
Step 3: Distribute excess virtual nodes • Full nodes send new virtual nodes on random walk of old
topology.
If many nodes are empty, they are easy to find.• Empty nodes send requests on random walk of the old
topology.
If most nodes are full, then there are lots of bridges.
Step 4: Rebalancing and clean up• Balance/reduce the virtual nodes (maybe).• Drop the old topology. (Or keep it as a backup.)
End result: a new instantiation of the virtual topology.
MergingCombining two overlay networks
OverviewPlan:
Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.
Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.
Construction algorithm:• Divide-and-conquer.• Merge and collapse components.
Robustness:• Self-stabilization.• Churn.
Overlay ConstructionDivide-and-Conquer Algorithm
1. Collections: maintain a collection of overlays.• Initially, each node is its own collection.• Collections are connected by edges between (real) nodes.
2. Matching: find a matching in the graph of collections.
• Pair up collections.
3. Merge: combine pairs of collections.• Merge components.• Combine two components into a single new collection.
4. Repeat: while there is > 1 collection, go to Step 2.
Inspired by [Angluin, Aspnes, Chen, Wu, Yin, SPAA’05]
Overlay ConstructionDivide-and-Conquer Algorithm
1. Collections: maintain a collection of overlays.• Initially, each node is its own collection.• Collections are connected by edges between (real) nodes.
2. Matching: find a matching in the graph of collections.
• Pair up collections.
3. Merge: combine pairs of collections.• Merge components.• Combine two components into a single new collection.
4. Repeat: while there is > 1 collection, go to Step 2. Problem: matching may be
small!
Overlay ConstructionDivide-and-Conquer Algorithm
1. Collections: maintain a collection of overlays.• Initially, each node is its own collection.• Collections are connected by edges between (real) nodes.
2. Sparsify: reduce degree of graph of collections.
• Create constant degree graph.• Each node organizes its “children” into a line.
3. Matching: find a matching in the graph of collections.
• Pair up collections.
4. Merge: combine pairs of collections.• Merge components.• Combine two components into a single new collection.
5. Repeat: while there is > 1 collection, go to Step 2.
Overlay ConstructionDivide-and-Conquer Algorithm
Initially: connected graph
Overlay ConstructionDivide-and-Conquer Algorithm
Sparsify: reduce the degree
Overlay ConstructionDivide-and-Conquer Algorithm
Repeat: match and merge
Overlay ConstructionDivide-and-Conquer Algorithm
Repeat: match and merge
Overlay ConstructionDivide-and-Conquer Algorithm
Repeat: match and merge
Overlay ConstructionDivide-and-Conquer Algorithm
Repeat: match and merge
Overlay ConstructionDivide-and-Conquer Algorithm
Repeat: match and merge
Divide-and-Conquer:• Collection has small diameter O(log n) cost to coordinate.• Collection merging: O(polylog n) cost per merge step.• Number of iterations: O(log n) matchings• Overall: O(polylog n) time to form overlay.
Overlay ConstructionEfficiency Analysis
Merging:• Random walks: O(log n) cost• Bridge doubling: O(log n) iterations• Overall: O(polylog n) time to merge topologies.
OverviewPlan:
Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.
Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.
Construction algorithm:• Divide-and-conquer.• Merge and collapse components.
Robustness:• Self-stabilization.• Churn.
Self-StabilizationDetecting Trouble
Self-StabilizationDetecting Trouble
Local detection:• If the topology is bad, some node can detect it.• Virtual topology is deterministic.• Can verify assignment of virtual nodes to real nodes, since
each real node is assigned 1 or 2 virtual nodes.
Detection diameter:• Assume there is a topology problem.• Detection diameter = max distance of any node from a
node that can detect the problem.• In any expander, detection diameter is O(log n), even if
real graph diameter is much larger!
Self-StabilizationDetecting Trouble
Local detection:• If the topology is bad, some node can detect it.• Virtual topology is deterministic.• Can verify assignment of virtual nodes to real nodes, since
each real node is assigned 1 or 2 virtual nodes.
[Defined by [Berns, Ghosh, Pemmaraju: SSS’11]
Detection diameter:• Assume there is a topology problem.• Detection diameter = max distance of any node from a
node that can detect the problem.• In any expander, detection diameter is O(log n), even if
real graph diameter is much larger!
Self-StabilizationDetecting Trouble
Local detection:• If the topology is bad, some node can detect it.• Virtual topology is deterministic.• Can verify assignment of virtual nodes to real nodes, since
each real node is assigned 1 or 2 virtual nodes.Conclusion:
Within O(log n) time, every node learns of a topology error.
Self-stabilizatinoBasic idea:
If a node detects an error:• Flood notification to everyone.• Time: O(log n).
On notification of error:• Restart overlay construction from scratch.• Within O(polylog n) time, new overlay.
Self-stabilizatinoBasic idea:
If a node detects an error:• Flood notification to everyone.• Time: O(log n).
On notification of error:• Restart overlay construction from scratch.• Within O(polylog n) time, new overlay.
Remaining issue: synchronization• Different nodes begin rebuilds at different times.• Components synchronize as they merge.
OverviewPlan:
Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.
Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.
Construction algorithm:• Divide-and-conquer.• Merge and collapse components.
Robustness:• Self-stabilization.• Churn.
ChurnDepartures and Failures
ChurnDepartures and Failures
What types of failures?• Graceful: execute an exit protocol.• Oblivious: adversary decides failures in advance. • Random: adversary fails nodes at random.• Adaptive: adversary chooses failure on-line.• Connected: adversary never disconnects graph.
Overlay tolerates failures• Expanders are highly fault-tolerant.• For “good” target topologies, overlay maintains good
properties, even if nodes fail. • As long as not too many nodes fail, all is good.
ChurnDepartures and Failures
What types of failures?• Graceful: execute an exit protocol.• Oblivious: adversary decides failures in advance. • Random: adversary fails nodes at random.• Adaptive: adversary chooses failure on-line.• Connected: adversary never disconnects graph.
ChurnRepairing failures
Option 1. Local repair • If a node leaves gracefully, hand-off virtual nodes.• If a node crashes, neighbors regenerate virtual nodes.• Rebalance (via random walks).
Conjecture: tolerates small changes.
[as in Kuhn, Schmid, Wattenhofer, IPTPS’05]
Option 2. Periodic Rebuild• Every so often, initiate a complete rebuild.• Assume failures oblivious or random (in some sense).• As long as:
“half-life” > “rebuild time”
then everything continues to work.
ChurnRepairing failures
Option 1. Local repair • If a node leaves gracefully, hand-off virtual nodes.• If a node crashes, neighbors regenerate virtual nodes.• Rebalance (via random walks).
Conjecture: tolerates small changes.
[as in Kuhn, Schmid, Wattenhofer, IPTPS’05]
ChurnKey Question:
Does the construction procedure work when there are ongoing failures?
1. Merging fails if links connecting collections fail:
• Random/oblivious failures not too many links fail / iteration.
2. Collection coordination is more difficult• Can no longer rely on a leader or an aggregation tree in
the collection.• Matching is still doable.• Sparsification is harder.
Joining is easy• Find an extra virtual node.• Route virtual node to new entrant.• Or wait until the next rebuild.
ChurnJoining nodes
How do nodes join?• Random: introduced to a random node.• Oblivious: introduced to a node chosen by the
adversary in advance.• Limited adversarial: introduced to a node, but
adversary cannot introduce more than one at each location.
OverviewPlan:
Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.
Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.
Construction algorithm:• Divide-and-conquer.• Merge and collapse components.
Robustness:• Self-stabilization.• Churn.
OverlaysA Play in Three Acts
Act I : Fix-it-Felix and the Half-Life-Hobgoblin
Wherein we meet our hero Felix, as he races to keep up with the Half-Life-Hobgoblin, fixing the overlay as fast as it is being destoryed.
Act II : Bob-the-Builder and the Destabilizing-Demon
Wherein we meet our hero Bob, as he counters the Destabilizing-Demon, rebuilding the overlay no matter how badly damaged.
Act III : Here Be Dragons
Wherein we explore the dangerous and unknown path forward and away from the hobgoblins and demons of our despair.
Wrap-upOverlay Networks
Many approaches:• One zillion
different base topologies.
• Many different repair techniques.
• Real-world implementations.
• Algorithms and theory.
A few existing overlay networks…
ChordKademlia
Pastry Tapestry
Skip+
HSkip+
Patricia Tries
DexXHeal
Forgiving Tree
PGrid
Skipnet
RN Protocol
dHamiltonianCycles
Avatar
Ca-Re-Chord
HyperCubes
HyperRing
Chameleon
Re-Chord
Tiara
Corona
Wrap-upA few things I have learned
Churn analysis is hard:
• Relatively few papers really allow failures to happen at any time.
• Analysis is handwavy, and sometimes wrong (see, e.g., Chord).
• Can we do a better job of understanding / proving results regarding churn?
We can fix it!
Wrap-upA few things I have learned
Self-stabilization is also hard:
• System can start in any state! Easy to ignore corner cases.
• Still, easier because there are no ongoing failures.
• Tends to be expensive, due to lots of local checking and rebuilding.
Can we build it?
Yes we can!
Wrap-upA few things I have learned
Integrating both churn and self-stabilization:
• Hard to stabilize if failures are ongoing!
• Competing demands: Self-stabilization
treats every failures as a disaster!
Churn-tolerance tries to ignore/repair small failures.
Wrap-upA few things I have learned
Communication costs:
• Many existing solutions rely on “graph squaring” to handle worst-case.
• Churn-tolerance is relatively cheap (when it works).
• Self-stabilization tends to be much more (communication) expensive.
Wrap-upA few things I have learned
No one has any idea how to deal with Byzantine/malicious or greedy participants in a dynamic overlay network!
Exception: see “Brahms: Byzantine resilient random membership sampling” by Bortnikov, Gurevitch, Keidar, Kliot, Shraer
Exception: see “Self-stabilizing and Byzantine-Tolerant Overlay Network” by Dolev, Hoch, and van Renesse
Wrap-upA few things I have learned
Many interesting ideas.
Many interesting techniques.
No one “right” solution yet.
Can we build it?
Yes we can!
We can fix it!