ATAM Architecture Tradeoff Analysis Method Arnon Rotem-Gal-Oz.
The Role of Communication Complexity in Distributed Computing Rotem Oshman.
-
Upload
journey-worsley -
Category
Documents
-
view
221 -
download
3
Transcript of The Role of Communication Complexity in Distributed Computing Rotem Oshman.
![Page 1: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/1.jpg)
The Role of Communication Complexity in Distributed Computing
Rotem Oshman
![Page 2: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/2.jpg)
Background: Distributed Computing
![Page 3: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/3.jpg)
Distributed Computing
• Typical model:– Local computation for free– Charge for “communication”
![Page 4: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/4.jpg)
Distributed Lower Bounds
• “All or nothing”:– If have not communicated, they know nothing
about each other– If communicated, they know everything about
each other• Recently: interest in quantifying the amount of
communication• Natural to use communication complexity
![Page 5: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/5.jpg)
Shared Memory
• Processes communicate by accessing objects in shared memory– Read/write registers– Read-modify-write: CAS, T&S, …
• Typically asynchronous:– Schedule = sequence of process IDs– Adversarially chosen– Sometimes processes may crash
![Page 6: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/6.jpg)
Shared Memory Lower Bounds
• “Covering arguments”• Examples:– Deterministic consensus impossible if one process
may fail [FLP’79]– Mutual exclusion requires shared registers [BL’90]
• Can we introduce a cell-probe-like model for shared memory lower bounds?
![Page 7: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/7.jpg)
Message Passing
• Processes communicate by sending messages– Over some network graph, often complete
• Fully synchronous, fully asynchronous, or anywhere in between
• Processes can crash, recover, cheat, lie,…
• Many successful applications of CC
![Page 8: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/8.jpg)
Some differences…
Complexity measure
Problems
Input
Distributed Computing Comm. Complexity#rounds (limited bandwidth) total #bits
Search Decision
Number-In-Hand Number-on- Forehead (usually)
![Page 9: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/9.jpg)
Message-Passing Models
MESSAGE-PASSING
LOCAL
CONGEST
SHARED BLACKBOARD#rounds total CC
![Page 10: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/10.jpg)
Talk Overview
I. Lower bound techniquesa. CONGEST (#rounds): reductions from 2-party
communication complexityb. Total CC with private channels
II. Shared blackboarda. Number-in-handb. “Not-quite-number-in-hand”
![Page 11: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/11.jpg)
The CONGEST Model
• nodes communicate over a network graph (small diameter)
• Computation proceeds in synchronous rounds• Each message = bits• Several variants:– Communication by unicast vs. local broadcast– Arbitrary graph vs. complete graph
![Page 12: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/12.jpg)
The CONGEST Model
• Arbitrary graph topology: strong lower bounds– [Distributed Verification and Hardness of Distributed
Approximation, by Das Sarma, Holzer, Kor, Korman, Nanongkai, Pandurangan, Peleg, Wattenhofer]• Nearly-tight lower bound on MST approximation• Non-tight lower bounds on all-pairs shortest path,
minimum cut, shortest s-t path, …• Almost all lower bounds
– Very few bounds known, no super-linear bounds
![Page 13: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/13.jpg)
The CONGEST Model
• Complete graph topology, unicast:– Known to be extremely powerful, e.g.,• Sort values in rounds• MST construction in rounds
– No explicit super-constant lower bound known– [Drucker, Kuhn, O. ‘14]: even slightly super-
constant lower bound new ACC lower bound• Complete graph topology, broadcast = shared
blackboard
![Page 14: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/14.jpg)
CONGEST Lower Bounds for Arbitrary Graphs
… by reduction from 2-party disjointness
![Page 15: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/15.jpg)
2-Party Reductions
• Textbook reduction [Kushilevitz Nisan]:Given algorithm for solving task …
Solution for answer for, e.g., Disjointness
bits
𝑌𝑋
Based on
Based on
Simulate
![Page 16: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/16.jpg)
2-Party Reductions
• More generally:Given algorithm for solving task …
Solution for answer for, e.g., Disjointness
𝑌𝑋
Based on
Based on
![Page 17: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/17.jpg)
Multi-Player NIH Communication with Private Channels
![Page 18: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/18.jpg)
The Message-Passing Model
• players• Private channels• Private -bit inputs • Private randomness
• Goal: compute • Cost: total communication
![Page 19: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/19.jpg)
The Coordinator Model
• players, one coordinator• The coordinator has no input
![Page 20: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/20.jpg)
Message-Passing vs. Coordinator
≈
Secure multi-party computation!
![Page 21: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/21.jpg)
Message-Passing Lower Bounds
For players with -bit inputs…• Woodruff, Zhang ‘12: estimation• Phillips, Verbin, Zhang ’12:– for bitwise problems (AND/OR, MAJ, …)
• Woodruff, Zhang ‘13:– for threshold and graph problems
• Braverman, Ellen, O., Pitassi, Vaikuntanathan ‘13: for disjointness
![Page 22: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/22.jpg)
Symmetrization [Phillips, Verbin, Zhang ’12]
Lower bound for -player problem :• Choose hard 2-player problem • Fix hard distribution for , let
Alice:
Bob:
“Smear” across players
![Page 23: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/23.jpg)
Symmetrization [Phillips, Verbin, Zhang ’12]
Lower bound for -player problem :
– -player distribution must be symmetric– Answer to answer to
cost for = cost for
Alice:
Bob:
“Smear” across players
![Page 24: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/24.jpg)
Symmetrization Example: Bitwise-XOR
• 2-party problem:– Input: uniform independent – Goal: Alice outputs Bob’s input,
• Reduction:– Sample uniform , Alice sets – Bob chooses uniformly s.t. – From , Alice can reconstruct
Alice:
Bob:
![Page 25: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/25.jpg)
Set Disjointness
Disj𝑛 ,𝑘 =¿ 𝑖=1¿𝑛¿ 𝑗=1¿𝑘 𝑋 𝑖𝑗¿
?
𝑋 1
𝑋 2
𝑋 3
𝑋 4𝑋 5
![Page 26: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/26.jpg)
Symmetrization vs. Disjointness
• Consider any symmetric distribution…
• How many 0s in coord. , given ?– More than one ⇒ Bob probably sees a zero• Ignore this coordinate
– Only one Pr[ Alice got it ] – 2-party CC
Alice:
Bob:
![Page 27: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/27.jpg)
[BEOPV’13] Lower Bound Outline
1. Direct sum: 2. One-bit lower bound:
Disj𝑛 ,𝑘 =¿ 𝑖=1¿𝑛¿ 𝑗=1¿𝑘 𝑋 𝑖𝑗¿
![Page 28: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/28.jpg)
Reduction from DISJ tograph connectivity [Based on WZ’13]
1
2
3
4
5
6
𝑝1
𝑝2
𝑝𝑘
(Players)
𝑋 𝑖
[𝑛 ]∖⋃𝑋 𝑖
(Elements)
input graph connected
![Page 29: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/29.jpg)
Number-In-Hand Shared Blackboard
![Page 30: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/30.jpg)
Why Should We Care?
• Some fundamental question still open• Natural model for distributed computing– Single-hop wireless network– More generally, abstracts away network topology– Related to MapReduce, etc. [Hegeman and
Pemmaraju’14]
![Page 31: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/31.jpg)
Example: NIH Multi-Party Disjointness
• Trivial upper bound: – Also easy to get
• Simultaneous CC: [Braverman,Ellen,O.,Pitassi,Vaikuntanathan’13]
– In rounds?– Looks like for [current work with Ran Raz]
• Unbounded rounds: [with Mark Braverman]
![Page 32: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/32.jpg)
“Not-Quite Number in Hand”
• In undirected networks, each edge is known to both endpoints
• Distributed graph property testing:– Players , input graph – Input to player : its neighbors in – Goal: test if satisfies property
• Example: subgraph detection [Drucker, Kuhn, O. ‘14]
![Page 33: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/33.jpg)
Example: Lower Bound for
• Claim: -round algorithm for detection solve DISJ in bits
• Reduction outline:– Alice and Bob get inputs – Construct input graph on nodes, such that
contains – Simulate the run of -detection algorithm on
![Page 34: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/34.jpg)
12
34
12
34
12
34
12
34
Construction of from
![Page 35: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/35.jpg)
Construction of from 1
2
34
12
34
12
34
12
34
Top-bottom: Bottom-top:
![Page 36: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/36.jpg)
12
34
12
34
12
34
12
34
Construction of from
(𝟐 ,𝟒 )∈𝑿∩𝒀
![Page 37: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/37.jpg)
Simulating the Algorithm
• Alice simulates nodes, Bob simulates • To simulate one round, each player:– Locally computes message broadcast by each
node it simulates– Sends all messages to the other player– Cost: per round
• Total cost:
![Page 38: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/38.jpg)
What About ?
• More complicated….Can contain regardless of
Solution: use extremal -free
graph Elements of DISJ
= edges of
![Page 39: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/39.jpg)
Upper Bound on Subgraph Detection
• Turán number: max # edges in -free graph on vertices
• Upper bound: solve -subgraph detection in rounds– Example: , nearly tight– Open problem: is this tight for all subgraphs?
![Page 40: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/40.jpg)
Detecting Triangles
• Trivial upper bound: rounds• Lower bound?– 2-party (black box) reduction cannot prove it– For each triangle, one player knows all 3 edges
![Page 41: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/41.jpg)
Triangles to 3-Party NOF Disjointness
• [Ruzsa and Szemerédi ’76]: there is a tripartite graph where
– contains triangles– Each edge in belongs to exactly one triangle
• Reduce from 3-party NOF Disjointness on elements, each representing one triangle in
![Page 42: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/42.jpg)
Triangles to 3-Party NOF Disjointness
• Input: sets of triangles • Let be the unique triangle edge belongs to• Construct , including:– iff – iff – iff
• Note: endpoints of each edge agree on its inclusion!
𝑍
𝑋
𝑌
𝐴
𝐶
𝐵
![Page 43: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/43.jpg)
Triangles to 3-Party NOF Disjointness
• Input: sets of triangles • Triangle appears in • Cost of simulation:– bits per round⇒ Round complexity of triangle detection
for 3-party NOF
𝑍
𝑋
𝑌
𝐴
𝐶
𝐵
![Page 44: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/44.jpg)
3-Party NOF Disjointness
• Randomized CC:– Sherstov’13: – We get nothing:
• Deterministic CC:– Rao and Yehudayoff’14: ⇒ for deterministic triangle detection
![Page 45: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/45.jpg)
Conclusion
MESSAGE-PASSING
LOCAL
CONGEST
SHARED BLACKBOARD#rounds total CC
![Page 46: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/46.jpg)
Directions for Future Research
• Exploiting asynchrony and faults to get stronger communication lower bounds
![Page 47: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/47.jpg)
Example 1: Dynamic Networks
• Abstract model for dynamic networks:– In each round we get a different graph
• [Kuhn, Lynch, O. ‘10]:– Assume each is connected– Any function can be deterministically computed in
rounds using -bit messages– Lower bounds?
![Page 48: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/48.jpg)
Example 1: Dynamic Networks
• For exchanging all inputs:– Determistic, “routing-based” algorithms: [Haeupler,
Kuhn’12]
– Randomized??– Non-routing based??
![Page 49: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/49.jpg)
Example 2: Byzantine Consensus
• processes, synchronous message-passing• Each process receives a bit• Goal:– Everyone outputs the same bit– If everyone received , the output is
• Byzantine faults: up to processes may behave arbitrarily
![Page 50: The Role of Communication Complexity in Distributed Computing Rotem Oshman.](https://reader030.fdocuments.net/reader030/viewer/2022032517/56649c9a5503460f949570b9/html5/thumbnails/50.jpg)
Example 2: Byzantine Consensus
• [King, Saia ‘10]: can solve with total bits• No general lower bound better than