Distributed Training Across the World

Ligeng Zhu Yao Lu Yujun Lin Song HanMassachusetts Institute of Technology, Google

{ligeng, yujunlin, songhan}@mit.edu, luyao11175@gmail.com

Abstract

Traditional synchronous distributed training is performed inside a cluster since itrequires high bandwidth and low latency network (e.g. 25Gb Ethernet or Infini-band). However, in many application scenarios, training data are often distributedacross many geographic locations, where physical distance is long and latency ishigh. Traditional synchronous distributed training cannot scale well under suchlimited network conditions. In this work, we aim to scale distributed learning un-der high-latency network. To achieve this, we propose Delayed and TemporallySparse (DTS) update that enables synchronous training to tolerate extreme networkconditions without compromising accuracy. We benchmark our algorithms onservers deployed across three continents in the world: London (Europe), Tokyo(Asia), Oregon (North America) and Ohio (North America). Under such challeng-ing settings, DTS achieves 90× speedup over traditional methods without loss ofaccuracy on ImageNet.

1 Introduction

Deep neural networks have demonstrated much success in solving large-scale machine learningproblems [22, 35, 16]. However, training deep neural networks may take days or even weeks toconverge. In order to enable training in a reasonable time, distributed training is an importanttechnique and gains increasing attention [23, 11, 31, 15, 17]. To maintain a good scalability, a lowlatency and high bandwidth network is essential for most modern distributed systems. Existingframeworks [8, 43, 29, 1, 3, 30, 34] all require high-end network infrastructure such as 25GbpsEthernet or Infiniband where bandwidth is as large as 10 to 100 Gbps and latency is as small as 1 us.

Bandwidth is easy to increase (e.g. stacking hardware) but latency is hard to improve (physical limits).For example, if we have two servers located at Shanghai and Boston respectively, even at the speed oflight and direct air distance, it still takes 78ms∗ to send and receive a packet. In real world scenario,the latency can be only worse (around 700ms) because indirect routing between internet serviceproviders (ISP) and queuing delay in switches. Such high latency cause severe scalability† issue fordistributed training. Traditional distributed training algorithm scales poorly under such large latency.

In many scenarios, the training data involves privacy-sensitive information, such as personal medicalhistory [20, 19] and keyboard input history [28, 21, 6], thus cannot be centralized to a data centerdue to security and legacy concerns [13]. When datasets are distributed across many locations acrosspublic clouds, private clusters and even edge devices, it is impossible to setup a low latency networkunder long-distance connections and thereby hurts the scalability of training.

In this work, we enable scalable distributed training across different geographical locations withlong distance and high latency network connection. We propose Delayed Update to tolerate latencyby putting off the synchronization barrier to a later iteration; we also propose Temporally Sparse

∗11, 725km× 2/(3× 108m/s) = 78.16ms. Information collected from Google Maps.†If a system can achieve M times speedup on N machines, the scalability is defined as M/N .

33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.

London

TokyoCaliforniaOhio

277m

63Mbps17Mbp183ms

140ms 23Mbp35ms

13Mbp

Table 1-1-1

Latency Bandwidth Scalability

Inside 3 25600 819.2

Across 158750 29 8.192

1

1,000

1,000,000

Latency (us) Bandwidth (Mbps) Scalability (1024 GPUs)

Across the world Inside a cluster

London

TokyoOregonOhio

102ms

210ms 97ms

70ms

Table 1-1

Latency Scalability

Across the world 3 2274

Inside a cluster 479000 23.32

Latency Speed

23

479,000

2,274

3

Inside a cluster Across the world

159,666x longer 98x worse

µs

µs

img/s

img/s

Figure 1: Network conditions across different continents (Left) and the comparison between connec-tions inside a cluster (Right). Different from training inside a data center, long-distance distributedtraining suffers from high latency, which proposes a severe challenge to scale across the world.

Update to amortize the latency and alleviate congestion by reducing the synchronization frequency.To ensure no loss of accuracy, we design a novel error compensation to overcome the staleness forboth vanilla and momentum SGD. We focus on the widely adopted synchronous update using dataparallelism. DTS can maintain high scalability even when latency is as high as 1000ms. This result isalso better than existing state of art technologies such as ECD-PSGD [38]

We benchmark our DTS under a challenging settings: training a deep neural network on four AWSP3 instances located in different continents of the world (Fig. 1). The measured latency is as largeas 277ms (compared to internal latency 2us). In this case, the naive distributed synchronous SGD(SSGD) can only achieve a poor scalability of 0.008. It means in this setting, distributed training with100 servers is even slower than single machine (0.8 v.s. 1.0). Meanwhile DTS achieves scalability of0.72 without compromising the accuracy. In conclusion, our contributions are listed below:

• We propose delayed update to tolerate the latency and temporally sparse update to amortizethe latency. While preserving the accuracy, delayed update tolerates up to 6 seconds latencyand temporally sparse update reduced the traffic congestion by 20×.

• We theoretically justify the convergence rate of our proposed algorithms. We show that bothalgorithms can be as fast original SGD while scaling well under high latency.

• With servers and data distributed in four different countries across the world, we can trainResNet-50 on ImageNet with scalability. To our best knowledge, DTS is the first work thatcan achieve scalable synchronous distributed training under such high latency.

2 Related Work

Distributed Learning becomes ever more important as the sizes of both datasets and models increase.Many studies have been made to explore the efficiency, both at the algorithm level [23, 11, 31, 15, 17]and at the framework level [8, 3, 1, 30, 34]. In most of distributed algorithms, each node performscomputation and exchange updates through network. A key component to improve scalability is toreduce the communication-to-computation ratio. The communication cost is determined by latencyand bandwidth. Conventional studies focus on reducing bandwidth requirements as the latency ofinternal traffic is usually low.

Gradient quantization / compression has been proposed to reduce the data to be transferred. One-bit gradient gradient [33] achieves 10× speedup using 20 GPUs on text-to-speech task. QSGD[5] and Terngrad [42] further improves the trade-off between accuracy and gradient precision.But the theoretical limit of the quantization cannot go beyond 32. To overcome the limitation,gradient sparsification using predefined static threshold [36, 2] and dynamic compression ratio [7]demonstrate that 99% gradients can be pruned with negligible degradation on model performance.[32] combines both quantization and compression to push the ratio to a new level. DGC [26] andDoubleSequeeze [40] explore how to compensate the error to preserve the accuracy better. Theconvergence of quantization and compression is also discussed in [40, 18, 4]. However, latencyremains an issue, especially when servers are not in the same physical location.

Learning on decentralized data has become increasingly popular recently as the growing awarenessof data privacy [13]. For example, Federated Learning [14] aim to jointly train a model without

2

Environment Agent Channel Pruning

Policy gradientDeep-Q Network

Actor Critic

Actions:sparsity for each layer

Observations / Reward: accuracy, FLOPs…

Hardware

Latency

Xxx xxx

11

22

33

44

1 2 3 4

T1

T1

T3

T31

T2

T22 3

T4

T44

T1

T1

T3

T3

T2

T21 2

T4

T43

T1

T1

T2

T2

T3

T3

12

T4

T4

34

:Computation :Synchronization Barrier :Locally update :Send gradients to other

(a) Synchronous distributed learning.

Environment Agent Channel Pruning

Policy gradientDeep-Q Network

Actor Critic

Actions:sparsity for each layer

Observations / Reward: accuracy, FLOPs…

Hardware

Latency

Xxx xxx

11

22

33

44

1 2 3 4

T1

T1

T3

T31

T2

T22 3

T4

T44

T1

T1

T3

T3

T2

T21 2

T4

T43

T1

T1

T2

T2

T3

T3

12

T4

T4

34

:Computation :Synchronization Barrier :Locally update :Send gradients to other

(b) Synchronous distributed learning with delayed update.

Environment Agent Channel Pruning

Policy gradientDeep-Q Network

Actor Critic

Actions:sparsity for each layer

Observations / Reward: accuracy, FLOPs…

Hardware

Latency

Xxx xxx

11

22

33

44

1 2 3 4

T1

T1

T3

T31

T2

T22 3

T4

T44

T1

T1

T3

T3

T2

T21 2

T4

T43

T1

T1

T2

T2

T3

T3

12

T4

T4

34

:Computation :Synchronization Barrier :Locally update :Send gradients to other

(c) Synchronous distributed learning with temporally sparse update.

Figure 2: The visualization of our proposed techniques (Left) and the execution pipeline (Right).Yellow lines and blocks indicate communication, while green ones indicate computation. Delayedupdate covers communication by computation and requires the same number of synchronization.Temporally sparse reduces the synchronization frequency and improve the amortized latency.

centralizing the data and have been used to train models for medical treatments across multiplehospitals [20], analyze patient survival situations from various countries [19] and build predictivekeyboards to improve typing experience [28, 14, 6]. However, existing federated learning worksdo not scale well under high latency network. An orthogonal exploration is Decentralized Trainingwhere only partial synchronization is performed in each update, such as AD-PSGD [25] and D2 [39].None of them has been evaluated on large learning tasks yet.

Asynchronous SGD (ASGD) is derived from [41] and has advantages in unstable latency and faulttolerance. Different from synchronous SGD (SSGD), ASGD relaxes synchronization by allowingtraining on inconsistent models and powers many successful applications such as HOGWILD! [31],BUCKWILD! [10] and dist-belief [11]. However, most of them are implemented through parameterserver [11, 9, 23] and leads to problems like communication congestion and resource congestionwhen scaling up. Moreover, training on inconsistent models leads to different behaviors comparedto training on a single device. Though there are studies discussing the convergences [31, 10] andshowing that ASGD converges as good as SSGD [24], synchronous SGD (SSGD) is usually preferredin practice [15, 37] because it has consistent behaviors when increasing the number of machines andtherefore is easy to develop and deploy.

3 Approach

To motivate this section, we first review the mechanism of synchronous distributed training: at eachstep, each node will first compute gradients locally, then they wait for the collective operation totransmit gradients to each other to calculate the average. They use the averaged gradient to update theweight and continue to the next step. As shown in right of Fig. 2a, when the communication increases, the whole training process would be drastically slowed down.

To address the problem, our proposed algorithm contains two parts: delayed update (Fig. 2b) andtemporally sparse update (Fig. 2c). With the delayed update, instead of waiting for average gradientsto come back, it puts off the synchronization barrier to steps later to tolerate the latency. With thetemporally sparse update, the synchronization frequency is reduced to amortize latency.

3.1 Delayed UpdateIn modern implementations of vanilla SSGD, the transmission is partially pipelined with back-propogration: Synchronizing gradients of nth layer can be performed simultaneously with back-propagating (n − 1)th layer. However, it is not enough to cover communication since latency on

3

long-distance connection is usually larger ( ≥ 100ms) than propagating a layer ( ≈ 11ms). To relaxthe limitation, we design delayed update to put off the the synchronization barrier by n steps as shownin Fig. 2b. To ensure minimal loss on accuracy, we modify the gradient update formula with errorcompensations.

To understand, we start with vanilla SGD: Consider a worker with index j, the weights at timestampn during distributed training are derived via

w(n,j) = w(n−1,j) − γ∇w(n−1) = w(0,j) − γn−1∑i=0

∇w(i) (1)

where ∇w(n−1) is the averaged gradients calculated by gathering all local gradients ∇w(n−1,j). InSSGD, the synchronization has to finish before (n+ 1)th forwarding. In delayed update, we delaythe barrier to (n+ t)th iteration and because of the delay, ∇w(n−1) is not received at timestamp n.We use local gradients to update instead.

w(n,j) = w(n−1,j) − γ∇w(n−1,j) (2)

Meanwhile, stale gradients ∇w(n−t) arrive. At timestamp n − t, we performed the gradient stepusing local gradients∇w(n−t,j), which should be the global averaged∇w(n−t) in SSGD. We correctthe mismatch caused by replacing ∇w(n−t,j) with ∇w(n−t). From Eq. 1, it adds a compensationterm, which is the difference of two gradients.

w′(n,j) = w(n−1,j) − γ∇w(n−1,j)︸ ︷︷ ︸local gradients

−γ(∇w(n−t) −∇w(n−t,j)︸ ︷︷ ︸error compensation

)

= w(n,j) − γ(∇w(n−t) −∇w(n−t,j))

(3)

In original synchronous stochastic gradient descent (SSGD), every worker receives averaged gradientsfrom n− 1 at timestamp n. This requires collective operation (e.g. Allreduce) to execute quickly;otherwise, it will slow down the training process. In delayed update, the barrier of the collectiveoperation is moved from timestamp n to n+ t− 1. It means as long as the gradients are transmittedwithin t× Tcompute, the training is not blocked. To put some real numbers, assume we are trainingResNet-50 on Nvidia Tesla V100, each step takes around 300ms. If we delay the update by 20 steps,even with latency as large as 6 seconds, the communication still can be fully covered by computations.

3.2 Temporally Sparse Update

Delayed update provides a good solution to tolerate latency. However, there still remains two issuesand network congestion is the first one. In the original SSGD, only one transmission is performedduring one iteration. But when using delayed update, there can be up to t transmissions at the sametime. Without a special hacking to low-level network driver, gradients sent earlier may arrive later,which is not what we want. The second one is latency variance. In real world network, especially forthose long-distance connections, the latency varies in a large range. Even though the average latencyis small, some accidentally high latency can slow the system.

To address, we adapt temporally sparse update to reduce the communication frequency and designcorresponding error compensation to guarantee the accuracy. As demonstrated in Fig. 2c, temporallysparse update only synchronizes for every p steps. For those steps where synchronization is notperformed, each worker updates its model with local gradients. For those steps where synchronizationis performed, each worker calculates error compensations based averaged information. As a result,the networking congestion is alleviated and the networking latency is amortized across multiple localupdate steps.

For vanilla SGD, like the delayed update, we can derive the compensation term for temporally sparseupdate from Eq. 1, which is the difference between accumulated gradients. Note the compensation in

4

D, T, D + TTraining curve

| Scalability v.s. latency (same machine)

| Scalability v.s. machine (same latency)

16x4 = 64 cards, batch size 32

latency(ms)

speed Original delay=4 delay=8 delay=12 delay=16 delay=20

0 151 0.79 0.7981 0.7935 0.7935 0.7978 0.7935

1 141 0.73 0.7917 0.7907 0.7907 0.7907 0.7907

5 130 0.68 0.7878 0.7892 0.7892 0.7892 0.7878

10 119 0.62 0.7815 0.7816 0.7816 0.7816 0.7816

50 62.2 0.32 0.7793 0.7866 0.7866 0.7866 0.7866

100 34.2 0.18 0.7726 0.7853 0.7853 0.7853 0.7853

500 7.46 0.04 0.73 0.7812 0.7812 0.7812 0.7812

1000 3.73 0.02 0.53 0.7321 0.7790 0.7789 0.7721

5000 1 0.01 0.03 0.45 0.4510 0.5355 0.7321

Scal

abili

ty

0.0

0.2

0.5

0.7

0.9

Network latency (ms)0 1 5 10 50 100 500 1000 5000

Original delay=8delay=12 delay=20

16x4 = 64 cards, batch size 32-1

latency(ms) original Temporal Sparsity=4 Temporal=8 Temporal=12 Temporal=16 Temporal=20

0 0.7994 0.8053 0.8399 0.8319 0.8639

1 0.7829 0.8054 0.8277 0.8310 0.8523

5 0.7677 0.8090 0.8216 0.8194 0.8436

10 0.7607 0.7994 0.8085 0.8183 0.8328

15 0.7573 0.7895 0.8120 0.8168 0.8286

20 0.7529 0.7876 0.8112 0.8172 0.8263

25 0.7533 0.7826 0.8108 0.8063 0.8261

50 0.7441 0.7820 0.8081 0.8085 0.8189

100 0.7370 0.7806 0.7841 0.8072 0.8127

150 0.6883 0.7561 0.7750 0.8026 0.8129

200 0.6729 0.7330 0.7739 0.7966 0.8043

250 0.6371 0.7261 0.7769 0.7588 0.7924

500 0.5179 0.6460 0.7135 0.7333 0.7529

1000 0.3764 0.5132 0.5959 0.6364 0.6581

1500 0.2905 0.4330 0.5049 0.5711 0.6157

2000 0.2390 0.3696 0.4607 0.5115 0.5548

2500 0.2011 0.3234 0.4080 0.4610 0.5098

5000 0.111 0.2123 0.2656 0.3143 0.3604

Scal

abili

ty

0

0.225

0.45

0.675

0.9

Network Latency (ms)0 1000 2000 3000 4000 5000 6000

Temporal=8 Temporal=12 Temporal Sparsity=4Temporal=16 Temporal=20

0

0.2

0.4

0.6

0.8

latency=1000ms-1

Nodes ideal L=0 d=4, t=4 delay=8, temporal=

delay=12, t=8

delay=20, t=12

4 4 7 8 8 8 8

8 8 14 15 15 16 16

16 16 27 30 30 31 31

32 32 55 58 59 61 63

64 64 110 91 116 121 123

latency=1000ms-1-1

Nodes Nodes D=4, T=4 (200ms) D=20, T=12 (200ms) SSGD (0ms) SSGD (200ms) Ideal

0 0 0 0 0

4 1 0.8 0.8 0.8 1 1

8 2 1.5 1.5 1.6 0.016 2

16 4 3 3 3.1 0.030 4

32 8 5.8 5.9 6.1 0.066 8

64 16 9.1 11.6 12.1 0.128 16

Scal

abili

ty

0

4

8

12

16

Number of servers

0 4 8 12 16

D=4, T=4 (200ms) D=20, T=12 (200ms)SSGD (0ms) SSGD (200ms)Ideal

16x4 = 64 cards, batch size 32-1-1

latency(ms) Original Temporal Sparsity=4

Temporal Sparsity=8

Temporal Sparsity=12

Temporal Sparsity=16

Temporal Sparsity=20

0 0.79 0.7994 0.8053 0.8399 0.8319 0.8639

1 0.76 0.7829 0.8054 0.8277 0.8310 0.8523

5 0.68 0.7677 0.8090 0.8216 0.8194 0.8436

10 0.62 0.7607 0.7994 0.8085 0.8183 0.8328

50 0.32 0.7441 0.7820 0.8081 0.8085 0.8189

100 0.18 0.7370 0.7806 0.7841 0.8072 0.8127

500 0.04 0.5179 0.6460 0.7135 0.7333 0.7529

1000 0.02 0.3764 0.5132 0.5959 0.6364 0.6581

5000 0.01 0.111 0.2123 0.2656 0.3143 0.3604

Table 2

with load balance balanced without load balance

unbalanced

0.9 6.24667472793229 0.9 5.53935234827364

0.85 5.23536863440588 0.85 4.56400742115028

0.8 4.41746119970927 0.8 3.73158046807281

0.7 3.15048025613661 0.7 2.71773154115264

0.6 2.42040902382458 0.6 2.15947329919532

0.5 1.95871011772735 0.5 1.79981186635543

0.4 1.64360026725208 0.4 1.54608167357505

0.3 1.41571093846344 0.3 1.36450079239303

0.2 1.28799022663243 0.2 1.28212446485078

0 1.00005807538185 0 1.00005807538185

Table 2-1

L2 regularization w/o retrain

Accuracy Loss w/o Retrain [L2 reg.]

L1 regularization w/o retrain

L1 regularization w/ retrain

L2 regularization w/ retrain

L2 regularization w/ iterative prune and retrain

0 0 0 0 0 0 0 0 0 0.875 0.0006

0.46 0.3351 -0.0002 0.3351 0 0.3351 0.000139 0.3351 0 0.888888888888889 0.0003

0.72 0.4991 -0.0011 0.4991 -0.0008 0.4991 0.0000389999999999002 0.4991 0.0004 0.9 -0.0008

1.05 0.6657 -0.0092 0.6657 -0.0027 0.6657 -0.00126100000000007 0.6657 0.0018 0.916666666666667 -0.0033

1.27 0.7494 -0.0216 0.7494 -0.0064 0.7494 -0.00216100000000008 0.7494 0.0015 0.928571428571429 -0.006

1.44 0.8 -0.0407 0.8 -0.0125 0.8 -0.0035 0.8 0.0003 0.9375 -0.0104

1.58 0.8335 -0.0739 0.8335 -0.021 0.8335 -0.0057 0.8335 -0.0017 0.94705 -0.0199

1.7 0.8569 -0.119 0.8569 -0.0298 0.8569 -0.00806099999999998 0.8569 -0.00458100000000006 0.95736 -0.042

1.81 0.8749 -0.1875 0.8749 -0.0443 0.8749 -0.010661 0.8749 -0.0081810000000001 0.963 -0.06

2.0 0.899 -0.3617 0.899 -0.0896 0.899 -0.018 0.899 -0.0158

2.27 0.9299 -0.6365 0.9299 -0.3292 0.9299 -0.0413 0.9299 -0.0388

2.78 0.95 -0.7916 0.95 -0.7552 0.95 -0.115561 0.95 -0.088381

Acc

urac

y Lo

ss

-4.5%-4.0%-3.5%-3.0%-2.5%-2.0%-1.5%-1.0%-0.5%0.0%0.5%

Parametes Pruned Away

40% 50% 60% 70% 80% 90% 100%

L2 regularization w/o retrain L1 regularization w/o retrain L1 regularization w/ retrain L2 regularization w/ retrain L2 regularization w/ iterative prune and retrain

Scal

abili

ty

0.0

0.2

0.5

0.7

0.9

Network latency (ms)0 1 5 10 50 100 500 1000 5000

Original Temporal Sparsity=4 Temporal Sparsity=8Temporal Sparsity=12 Temporal Sparsity=16 Temporal Sparsity=20

Delay=4delay=16

(a) The learning curve with different delay steps and scalability under various latency.

D, T, D + TTraining curve

| Scalability v.s. latency (same machine)

| Scalability v.s. machine (same latency)

16x4 = 64 cards, batch size 32

latency(ms)

speed Original delay=4 delay=8 delay=12 delay=16 delay=20

0 151 0.79 0.7981 0.7935 0.7935 0.7978 0.7935

1 141 0.73 0.7917 0.7907 0.7907 0.7907 0.7907

5 130 0.68 0.7878 0.7892 0.7892 0.7892 0.7878

10 119 0.62 0.7815 0.7816 0.7816 0.7816 0.7816

50 62.2 0.32 0.7793 0.7866 0.7866 0.7866 0.7866

100 34.2 0.18 0.7726 0.7853 0.7853 0.7853 0.7853

500 7.46 0.04 0.73 0.7812 0.7812 0.7812 0.7812

1000 3.73 0.02 0.53 0.7321 0.7790 0.7789 0.7721

5000 1 0.01 0.03 0.45 0.4510 0.5355 0.7321

Scal

abili

ty

0.0

0.2

0.5

0.7

0.9

Network latency (ms)0 1 5 10 50 100 500 1000 5000

Original delay=4 delay=8delay=12 delay=16 delay=20

16x4 = 64 cards, batch size 32-1

latency(ms) original Temporal Sparsity=4 Temporal=8 Temporal=12 Temporal=16 Temporal=20

0 0.7994 0.8053 0.8399 0.8319 0.8639

1 0.7829 0.8054 0.8277 0.8310 0.8523

5 0.7677 0.8090 0.8216 0.8194 0.8436

10 0.7607 0.7994 0.8085 0.8183 0.8328

15 0.7573 0.7895 0.8120 0.8168 0.8286

20 0.7529 0.7876 0.8112 0.8172 0.8263

25 0.7533 0.7826 0.8108 0.8063 0.8261

50 0.7441 0.7820 0.8081 0.8085 0.8189

100 0.7370 0.7806 0.7841 0.8072 0.8127

150 0.6883 0.7561 0.7750 0.8026 0.8129

200 0.6729 0.7330 0.7739 0.7966 0.8043

250 0.6371 0.7261 0.7769 0.7588 0.7924

500 0.5179 0.6460 0.7135 0.7333 0.7529

1000 0.3764 0.5132 0.5959 0.6364 0.6581

1500 0.2905 0.4330 0.5049 0.5711 0.6157

2000 0.2390 0.3696 0.4607 0.5115 0.5548

2500 0.2011 0.3234 0.4080 0.4610 0.5098

5000 0.111 0.2123 0.2656 0.3143 0.3604

Scal

abili

ty

0

0.225

0.45

0.675

0.9

Network Latency (ms)0 1000 2000 3000 4000 5000 6000

Temporal=8 Temporal=12 Temporal Sparsity=4Temporal=16 Temporal=20

0

0.2

0.4

0.6

0.8

latency=1000ms-1

Nodes ideal L=0 d=4, t=4 delay=8, temporal=

delay=12, t=8

delay=20, t=12

4 4 7 8 8 8 8

8 8 14 15 15 16 16

16 16 27 30 30 31 31

32 32 55 58 59 61 63

64 64 110 91 116 121 123

latency=1000ms-1-1

Nodes Nodes D=4, T=4 (200ms) D=20, T=12 (200ms) SSGD (0ms) SSGD (200ms) Ideal

0 0 0 0 0

4 1 0.8 0.8 0.8 1 1

8 2 1.5 1.5 1.6 0.016 2

16 4 3 3 3.1 0.030 4

32 8 5.8 5.9 6.1 0.066 8

64 16 9.1 11.6 12.1 0.128 16

Scal

abili

ty

0

4

8

12

16

Number of servers

0 4 8 12 16

D=4, T=4 (200ms) D=20, T=12 (200ms)SSGD (0ms) SSGD (200ms)Ideal

16x4 = 64 cards, batch size 32-1-1

latency(ms) Original Temporal Sparsity=4

Temporal Sparsity=8

Temporal Sparsity=12

Temporal Sparsity=16

Temporal Sparsity=20

0 0.79 0.7994 0.8053 0.8399 0.8319 0.8639

1 0.76 0.7829 0.8054 0.8277 0.8310 0.8523

5 0.68 0.7677 0.8090 0.8216 0.8194 0.8436

10 0.62 0.7607 0.7994 0.8085 0.8183 0.8328

50 0.32 0.7441 0.7820 0.8081 0.8085 0.8189

100 0.18 0.7370 0.7806 0.7841 0.8072 0.8127

500 0.04 0.5179 0.6460 0.7135 0.7333 0.7529

1000 0.02 0.3764 0.5132 0.5959 0.6364 0.6581

5000 0.01 0.111 0.2123 0.2656 0.3143 0.3604

Table 2

with load balance balanced without load balance

unbalanced

0.9 6.24667472793229 0.9 5.53935234827364

0.85 5.23536863440588 0.85 4.56400742115028

0.8 4.41746119970927 0.8 3.73158046807281

0.7 3.15048025613661 0.7 2.71773154115264

0.6 2.42040902382458 0.6 2.15947329919532

0.5 1.95871011772735 0.5 1.79981186635543

0.4 1.64360026725208 0.4 1.54608167357505

0.3 1.41571093846344 0.3 1.36450079239303

0.2 1.28799022663243 0.2 1.28212446485078

0 1.00005807538185 0 1.00005807538185

Table 2-1

L2 regularization w/o retrain

Accuracy Loss w/o Retrain [L2 reg.]

L1 regularization w/o retrain

L1 regularization w/ retrain

L2 regularization w/ retrain

L2 regularization w/ iterative prune and retrain

0 0 0 0 0 0 0 0 0 0.875 0.0006

0.46 0.3351 -0.0002 0.3351 0 0.3351 0.000139 0.3351 0 0.888888888888889 0.0003

0.72 0.4991 -0.0011 0.4991 -0.0008 0.4991 0.0000389999999999002 0.4991 0.0004 0.9 -0.0008

1.05 0.6657 -0.0092 0.6657 -0.0027 0.6657 -0.00126100000000007 0.6657 0.0018 0.916666666666667 -0.0033

1.27 0.7494 -0.0216 0.7494 -0.0064 0.7494 -0.00216100000000008 0.7494 0.0015 0.928571428571429 -0.006

1.44 0.8 -0.0407 0.8 -0.0125 0.8 -0.0035 0.8 0.0003 0.9375 -0.0104

1.58 0.8335 -0.0739 0.8335 -0.021 0.8335 -0.0057 0.8335 -0.0017 0.94705 -0.0199

1.7 0.8569 -0.119 0.8569 -0.0298 0.8569 -0.00806099999999998 0.8569 -0.00458100000000006 0.95736 -0.042

1.81 0.8749 -0.1875 0.8749 -0.0443 0.8749 -0.010661 0.8749 -0.0081810000000001 0.963 -0.06

2.0 0.899 -0.3617 0.899 -0.0896 0.899 -0.018 0.899 -0.0158

2.27 0.9299 -0.6365 0.9299 -0.3292 0.9299 -0.0413 0.9299 -0.0388

2.78 0.95 -0.7916 0.95 -0.7552 0.95 -0.115561 0.95 -0.088381

Acc

urac

y Lo

ss

-4.5%-4.0%-3.5%-3.0%-2.5%-2.0%-1.5%-1.0%-0.5%0.0%0.5%

Parametes Pruned Away

40% 50% 60% 70% 80% 90% 100%

L2 regularization w/o retrain L1 regularization w/o retrain L1 regularization w/ retrain L2 regularization w/ retrain L2 regularization w/ iterative prune and retrain

Scal

abili

ty

0.0

0.2

0.5

0.7

0.9

Network latency (ms)0 1 5 10 50 100 500 1000 5000

temporal sparsity=4 temporal sparsity=16

temporal sparsity=8 temporal sparsity=20

Originaltemporal Sparsity=12

(b) The learning curve with different temporal sparsity and scalability under various latency.Figure 3: Compare the top-1 accuracy and scalability on ImageNet training.

temporally sparse update only applies when n (mod p) ≡ 0.

w′(n,j) = w(n,j) − γ(n−1∑

i=n−p∇w(i) −

n−1∑i=n−p

∇w(i,j)) (4)

Note temporally sparse update is different from gradient accumulation (forward and backwardN times, accumulate the gradients and optimizer steps only 1 time). Gradient accumulation isequivalent to increasing the batch size and requires to adjust learning rate and the parameters of batchnormalization to ensure smooth convergence. Temporally sparsity updates model every iteration andcompensate error every N steps. It does not require any changes to the hyper-parameters.

4 Experiment

We evaluate our DTS on image classification task. We studies ResNet-50 [16] on ImageNet [12].ImageNet contains 1.2 million training images and 50,000 validation images in 1000 classes. Wetrain ImageNet model with momentum SGD using warmup strategy [15] and cosine anneal learningrate decay [27] with 120 epochs. Standard data augmentations random crop and flip are adapted.We set the initial learning rate 0.0125 for which grows linearly w.r.t number of GPUs. For a faircomparison, the training settings are set the same among all experiments.

When performing ablation studies under different latency, we synthesis the network condition usingnetem. When training across the world, the limited network is benchmarked using qperf. Weimplement DST in PyTorch framework [30] and adapt Horovod [34] as the distributed trainingbackend. To make results reproducible, we conduct our experiment on 16 standard Amazon p.largeEC2 instances with eight Nvidia Tesla V100 on each and will release the codebase when lessanonymous.4.1 Results of Delayed UpdateFig. 3a shows the performance and scalability for delayed update with different latency. We startfrom delay steps 4 and gradually increase it. The learning curve shows that with our proposed

5

Top-1%Original SGD 76.63

D=4, T=4, C=1% 76.15D=8, T=8, C=1% 76.32D=12, T=8, C=1% 76.18

D=20, T=12, C=1% 75.81

(a) Delayed updates preserves the accuracy while tol-erate a large latency. D, T, C indicate delayed update,temporal sparsity and gradient compression ratio respec-tively. The results are evaluated on ImageNet dataset.

D, T, D + TTraining curve

| Scalability v.s. latency (same machine)

| Scalability v.s. machine (same latency)

16x4 = 64 cards, batch size 32

latency(ms)

speed Original delay=4 delay=8 delay=12 delay=16 delay=20

0 151 0.79 0.7981 0.7935 0.7935 0.7978 0.7935

1 141 0.73 0.7917 0.7907 0.7907 0.7907 0.7907

5 130 0.68 0.7878 0.7892 0.7892 0.7892 0.7878

10 119 0.62 0.7815 0.7816 0.7816 0.7816 0.7816

50 62.2 0.32 0.7793 0.7866 0.7866 0.7866 0.7866

100 34.2 0.18 0.7726 0.7853 0.7853 0.7853 0.7853

500 7.46 0.04 0.73 0.7812 0.7812 0.7812 0.7812

1000 3.73 0.02 0.53 0.7321 0.7790 0.7789 0.7721

5000 1 0.01 0.03 0.45 0.4510 0.5355 0.7321

Scal

abili

ty

0.0

0.2

0.5

0.7

0.9

Network latency (ms)0 1 5 10 50 100 500 1000 5000

Original delay=4 delay=8delay=12 delay=16 delay=20

16x4 = 64 cards, batch size 32-1

latency(ms) original Temporal Sparsity=4 Temporal=8 Temporal=12 Temporal=16 Temporal=20

0 0.7994 0.8053 0.8399 0.8319 0.8639

1 0.7829 0.8054 0.8277 0.8310 0.8523

5 0.7677 0.8090 0.8216 0.8194 0.8436

10 0.7607 0.7994 0.8085 0.8183 0.8328

15 0.7573 0.7895 0.8120 0.8168 0.8286

20 0.7529 0.7876 0.8112 0.8172 0.8263

25 0.7533 0.7826 0.8108 0.8063 0.8261

50 0.7441 0.7820 0.8081 0.8085 0.8189

100 0.7370 0.7806 0.7841 0.8072 0.8127

150 0.6883 0.7561 0.7750 0.8026 0.8129

200 0.6729 0.7330 0.7739 0.7966 0.8043

250 0.6371 0.7261 0.7769 0.7588 0.7924

500 0.5179 0.6460 0.7135 0.7333 0.7529

1000 0.3764 0.5132 0.5959 0.6364 0.6581

1500 0.2905 0.4330 0.5049 0.5711 0.6157

2000 0.2390 0.3696 0.4607 0.5115 0.5548

2500 0.2011 0.3234 0.4080 0.4610 0.5098

5000 0.111 0.2123 0.2656 0.3143 0.3604

Scal

abili

ty

0

0.225

0.45

0.675

0.9

Network Latency (ms)0 1000 2000 3000 4000 5000 6000

Temporal=8 Temporal=12 Temporal Sparsity=4Temporal=16 Temporal=20

0

0.2

0.4

0.6

0.8

latency=1000ms-1

Nodes ideal L=0 d=4, t=4 delay=8, temporal=

delay=12, t=8

delay=20, t=12

4 4 7 8 8 8 8

8 8 14 15 15 16 16

16 16 27 30 30 31 31

32 32 55 58 59 61 63

64 64 110 91 116 121 123

latency=1000ms-1-1

Nodes Nodes D=4, T=4 (200ms) D=20, T=12 (200ms) SSGD (0ms) SSGD (200ms) Ideal

0 0 0 0 0

4 1 0.8 0.8 0.8 1 1

8 2 1.5 1.5 1.6 0.016 2

16 4 3 3 3.1 0.030 4

32 8 5.8 5.9 6.1 0.066 8

64 16 9.1 11.6 12.1 0.128 16

0

4

8

12

16

Number of servers

0 4 8 12 16

D=4, T=4 (200ms) D=20, T=12 (200ms)SSGD (0ms) SSGD (200ms)Ideal

16x4 = 64 cards, batch size 32-1-1

latency(ms) Original Temporal Sparsity=4

Temporal Sparsity=8

Temporal Sparsity=12

Temporal Sparsity=16

Temporal Sparsity=20

0 0.79 0.7994 0.8053 0.8399 0.8319 0.8639

1 0.76 0.7829 0.8054 0.8277 0.8310 0.8523

5 0.68 0.7677 0.8090 0.8216 0.8194 0.8436

10 0.62 0.7607 0.7994 0.8085 0.8183 0.8328

50 0.32 0.7441 0.7820 0.8081 0.8085 0.8189

100 0.18 0.7370 0.7806 0.7841 0.8072 0.8127

500 0.04 0.5179 0.6460 0.7135 0.7333 0.7529

1000 0.02 0.3764 0.5132 0.5959 0.6364 0.6581

5000 0.01 0.111 0.2123 0.2656 0.3143 0.3604

Table 2

with load balance balanced without load balance

unbalanced

0.9 6.24667472793229 0.9 5.53935234827364

0.85 5.23536863440588 0.85 4.56400742115028

0.8 4.41746119970927 0.8 3.73158046807281

0.7 3.15048025613661 0.7 2.71773154115264

0.6 2.42040902382458 0.6 2.15947329919532

0.5 1.95871011772735 0.5 1.79981186635543

0.4 1.64360026725208 0.4 1.54608167357505

0.3 1.41571093846344 0.3 1.36450079239303

0.2 1.28799022663243 0.2 1.28212446485078

0 1.00005807538185 0 1.00005807538185

Table 2-1

L2 regularization w/o retrain

Accuracy Loss w/o Retrain [L2 reg.]

L1 regularization w/o retrain

L1 regularization w/ retrain

L2 regularization w/ retrain

L2 regularization w/ iterative prune and retrain

0 0 0 0 0 0 0 0 0 0.875 0.0006

0.46 0.3351 -0.0002 0.3351 0 0.3351 0.000139 0.3351 0 0.888888888888889 0.0003

0.72 0.4991 -0.0011 0.4991 -0.0008 0.4991 0.0000389999999999002 0.4991 0.0004 0.9 -0.0008

1.05 0.6657 -0.0092 0.6657 -0.0027 0.6657 -0.00126100000000007 0.6657 0.0018 0.916666666666667 -0.0033

1.27 0.7494 -0.0216 0.7494 -0.0064 0.7494 -0.00216100000000008 0.7494 0.0015 0.928571428571429 -0.006

1.44 0.8 -0.0407 0.8 -0.0125 0.8 -0.0035 0.8 0.0003 0.9375 -0.0104

1.58 0.8335 -0.0739 0.8335 -0.021 0.8335 -0.0057 0.8335 -0.0017 0.94705 -0.0199

1.7 0.8569 -0.119 0.8569 -0.0298 0.8569 -0.00806099999999998 0.8569 -0.00458100000000006 0.95736 -0.042

1.81 0.8749 -0.1875 0.8749 -0.0443 0.8749 -0.010661 0.8749 -0.0081810000000001 0.963 -0.06

2.0 0.899 -0.3617 0.899 -0.0896 0.899 -0.018 0.899 -0.0158

2.27 0.9299 -0.6365 0.9299 -0.3292 0.9299 -0.0413 0.9299 -0.0388

2.78 0.95 -0.7916 0.95 -0.7552 0.95 -0.115561 0.95 -0.088381

Acc

urac

y Lo

ss

-4.5%-4.0%-3.5%-3.0%-2.5%-2.0%-1.5%-1.0%-0.5%0.0%0.5%

Parametes Pruned Away

40% 50% 60% 70% 80% 90% 100%

L2 regularization w/o retrain L1 regularization w/o retrain L1 regularization w/ retrain L2 regularization w/ retrain L2 regularization w/ iterative prune and retrain

Scal

abili

ty

0.0

0.2

0.5

0.7

0.9

Network latency (ms)0 1 5 10 50 100 500 1000 5000

Original Temporal Sparsity=4 Temporal Sparsity=8Temporal Sparsity=12 Temporal Sparsity=16 Temporal Sparsity=20

Spee

dup

(b) Better speedup and scalability with DTS.Measured on servers across the world.

Table 1: DTS demonstrates good scalability and accuracy on world-wide high latency training.

error compensation delayed update demonstrates robust convergence and comparable performancecompared vanilla SSGD, even when delay interval is as large as 20.

In original SSGD and modern deep learning frameworks, though the transmission is partially pipelinedwith backpropogration: Synchronizing gradients of nth layer can be performed simultaneously withback-propagating (n− 1)th layer, it is not enough to fully cover communication since latency on longdistance connection is usually larger ( ≥ 100ms) than propagating a layer (11ms). By delaying thesynchronization, delayed update provides much more spare time for transmission and thus scalesbetter. As shown in Fig. 3a, the larger the delayed interval is, the better tolerance towards latency.The scalability keeps stable until maximum tolerance is exceeded.4.2 Results of Temporally Sparse UpdateThe connection of real-world network has many bothering issues such as unstable latency and conges-tion on routers. Temporally sparse is introduced to amortize these via reducing the synchronizationfrequency. The results are shown in Fig. 3b(Left): we start with temporal sparsity 4 and increase to20. With proposed compensation, the temporally sparse update has a negligible loss on accuracy.Fig. 3(Right) exhibits how temporally sparse training scales under different network latency. Thescalability degrades slower than because temporally sparse update amortizes the latency.

4.3 Distributed Training across the World

With two powerful techniques, we are close to our original target of distributed training across world.To evaluate DTS, we deploy located at four different locations across the world: Tokyo(Japan),London (U.K), Oregon (U.S.A) and Ohio(U.S.A). The latency information is shown in Fig.1. Thereal latency across the ring allreduce is about 479ms through Ethernet. On V100 GPU, ResNet-50takes around 300ms to forward and backward. Therefore delayed update with steps more than 4 isalready enough to tolerate such a network. In terms of latency variance and network congestion, weintegrate temporally sparse update to alleviate which drastically reduces the bandwidth requirementby t times. However, there is still a bottleneck on bandwidth for cross-continent connections (e.g.Tokyo to London). Though theoretically bandwidth is achievable by switching to better internetservice providers, AWS does not provide such an option. As an economical alternative, we adapt deepgradient compression [26] to work through. As shown in Tab. 1a, DTS has little effect on trainingaccuracy. Meanwhile, DTS demonstrates strong scalability (0.72) under high latency (200ms betweenworkers) and this performance is close to what conventional algorithms achieved inside a data center(speedup ratio 0.78 and 1us latency).

5 ConclusionIn this paper, we propose two novel delayed update and temporally sparse update methods to scalesynchronous distributed training on servers located at different continents across the world. Wedemonstrate that delayed synchronization and temporally sparse update are both accuracy preservingwhile tolerate poor network conditions. We believe that our work will open future avenues for a widerange of decentralized learning applications.

6

References

[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado,Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, GeoffreyIrving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg,Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens,Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, FernandaViégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available fromtensorflow.org.

[2] Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. arXivpreprint arXiv:1704.05021, 2017.

[3] Takuya Akiba, Keisuke Fukuda, and Shuji Suzuki. ChainerMN: Scalable Distributed Deep LearningFramework. In Proceedings of Workshop on ML Systems in The Thirty-first Annual Conference on NeuralInformation Processing Systems (NIPS), 2017.

[4] Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cédric Renggli.The convergence of sparsified gradient methods. In Advances in Neural Information Processing Systems,pages 5973–5983, 2018.

[5] Dan Alistarh, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Randomized quantization forcommunication-optimal stochastic gradient descent. arXiv preprint arXiv:1610.02132, 2016.

[6] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov,Chloe Kiddon, Jakub Konecny, Stefano Mazzocchi, H Brendan McMahan, et al. Towards federatedlearning at scale: System design. arXiv preprint arXiv:1902.01046, 2019.

[7] Chia-Yu Chen, Jungwook Choi, Daniel Brand, Ankur Agrawal, Wei Zhang, and Kailash Gopalakrishnan.Adacomp: Adaptive residual gradient compression for data-parallel distributed training. In Thirty-SecondAAAI Conference on Artificial Intelligence, 2018.

[8] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, ChiyuanZhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneousdistributed systems. arXiv preprint arXiv:1512.01274, 2015.

[9] Trishul M Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. Project adam: Buildingan efficient and scalable deep learning training system. In OSDI, volume 14, pages 571–582, 2014.

[10] Christopher M De Sa, Ce Zhang, Kunle Olukotun, and Christopher Ré. Taming the wild: A unified analysisof hogwild-style algorithms. In Advances in neural information processing systems, pages 2674–2682,2015.

[11] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, PaulTucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in neuralinformation processing systems, pages 1223–1231, 2012.

[12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical ImageDatabase. In CVPR09, 2009.

[13] GDPR. The eu general data protection regulation (gdpr). 2016.

[14] Google. Towards federated learning at scale: System design. 2017.

[15] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, AndrewTulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXivpreprint arXiv:1706.02677, 2017.

[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.

[17] Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, ZhenyuGuo, Yuanzhou Yang, Liwei Yu, et al. Highly scalable deep learning training system with mixed-precision:Training imagenet in four minutes. arXiv preprint arXiv:1807.11205, 2018.

[18] Peng Jiang and Gagan Agrawal. A linear speedup analysis of distributed deep learning with sparse andquantized communication. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, andR. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 2525–2536. CurranAssociates, Inc., 2018.

[19] Arthur Jochems, Timo M Deist, Issam El Naqa, Marc Kessler, Chuck Mayo, Jackson Reeves, ShrutiJolly, Martha Matuszak, Randall Ten Haken, Johan van Soest, et al. Developing and validating a survivalprediction model for nsclc patients through distributed learning across 3 countries. International Journalof Radiation Oncology* Biology* Physics, 99(2):344–352, 2017.

7

[20] Arthur Jochems, Timo M Deist, Johan Van Soest, Michael Eble, Paul Bulens, Philippe Coucke, Wim Dries,Philippe Lambin, and Andre Dekker. Distributed learning: developing a predictive model based on datafrom multiple hospitals without data leaving the hospital–a real life proof of concept. Radiotherapy andOncology, 121(3):459–467, 2016.

[21] Jakub Konecný, H. Brendan McMahan, Felix X. Yu, Peter Richtarik, Ananda Theertha Suresh, and DaveBacon. Federated learning: Strategies for improving communication efficiency. In NIPS Workshop onPrivate Multi-Party Machine Learning, 2016.

[22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutionalneural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.

[23] Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, JamesLong, Eugene J Shekita, and Bor-Yiing Su. Scaling distributed machine learning with the parameter server.In 11th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 14), pages583–598, 2014.

[24] Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu. Asynchronous parallel stochastic gradient fornonconvex optimization. In Advances in Neural Information Processing Systems, pages 2737–2745, 2015.

[25] Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradientdescent. arXiv preprint arXiv:1710.06952, 2017.

[26] Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. Deep gradient compression: Reducingthe communication bandwidth for distributed training. arXiv preprint arXiv:1712.01887, 2017.

[27] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprintarXiv:1608.03983, 2016.

[28] H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication-efficientlearning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629, 2016.

[29] Philipp Moritz, Robert Nishihara, Ion Stoica, and Michael I Jordan. Sparknet: Training deep networks inspark. arXiv preprint arXiv:1511.06051, 2015.

[30] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, ZemingLin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W,2017.

[31] Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach toparallelizing stochastic gradient descent. In Advances in neural information processing systems, pages693–701, 2011.

[32] Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojciech Samek. Sparse binary compression:Towards distributed deep learning with minimal communication. CoRR, abs/1805.08768, 2018.

[33] Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and itsapplication to data-parallel distributed training of speech dnns. In Fifteenth Annual Conference of theInternational Speech Communication Association, 2014.

[34] Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensorflow.arXiv preprint arXiv:1802.05799, 2018.

[35] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-tion. arXiv preprint arXiv:1409.1556, 2014.

[36] Nikko Strom. Scalable distributed dnn training using commodity gpu cloud computing. In SixteenthAnnual Conference of the International Speech Communication Association, 2015.

[37] Peng Sun, Wansen Feng, Ruobing Han, Shengen Yan, and Yonggang Wen. Optimizing network perfor-mance for distributed dnn training on gpu clusters: Imagenet/alexnet training in 1.5 minutes. arXiv preprintarXiv:1902.06855, 2019.

[38] Hanlin Tang, Shaoduo Gan, Ce Zhang, Tong Zhang, and Ji Liu. Communication compression fordecentralized training. In Advances in Neural Information Processing Systems, pages 7652–7662, 2018.

[39] Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, and Ji Liu. D2: Decentralized training over decentralizeddata. CoRR, abs/1803.07068, 2018.

[40] Hanlin Tang, Xiangru Lian, Tong Zhang, and Ji Liu. Doublesqueeze: Parallel stochastic gradient descentwith double-pass error-compensated compression. In International Conference on Machine Learning,2019.

[41] John Tsitsiklis, Dimitri Bertsekas, and Michael Athans. Distributed asynchronous deterministic andstochastic gradient optimization algorithms. IEEE transactions on automatic control, 31(9):803–812, 1986.

[42] Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terngrad: Ternarygradients to reduce communication in distributed deep learning. In Advances in neural informationprocessing systems, pages 1509–1519, 2017.

8

[43] Eric P Xing, Qirong Ho, Wei Dai, Jin Kyu Kim, Jinliang Wei, Seunghak Lee, Xun Zheng, Pengtao Xie,Abhimanu Kumar, and Yaoliang Yu. Petuum: A new platform for distributed machine learning on big data.IEEE Transactions on Big Data, 1(2):49–67, 2015.

9