Load Balancing for Interdependent IoT Microservices

Ruozhou Yu, Vishnu Teja Kilari, Guoliang XueArizona State University

Dejun YangColorado School of Mines

Outlines

2

Background and Motivation

System Modeling

Algorithm Design and Analysis

Performance Evaluation

Discussions, Future Work and Conclusions

IoT: The Future Internet

3

q IoT is the future Internet that connects every aspect of ourwork and life.

Environment

Agriculture

Shopping

Manufacturing

Transportation

Home

Healthcare

Travel

Security

Example IoT Applications

4

TrafficAnalysis

ImageProcessing

ObjectDetection

TemporalAnalysis

TrafficPrediction

Monolithic Applications

5

TrafficAnalysis

ImageProcessing

ObjectDetection

TemporalAnalysis

TrafficPrediction

Crowd

Detection

SignalProcessing

DeviceMatching

TemporalAnalysis

PopulationEstimation

Criminal

Searching

ImageProcessing

ObjectDetection

FaceRecognition

FaceComparison

TemporalAnalysis

FaceComparison

TrafficPrediction

PopulationEstimation

DeviceMatching

SignalProcessing

Microservices

6

TrafficAnalysis

Crowd

Detection

Criminal

Searching

ImageProcessing

ObjectDetection

FaceRecognition

Microservices vs. Edge Computing

7

High delayHigh bw overheadSingle point of failureSecurity concerns

ResponsiveLow bw overheadDistributedSecure

Cloud-based

Edge-based

The Microservice Load Balancing Problem

q Edge-based microservices can be easily saturated.

8

ImageProcessing

ImageProcessing

1

ImageProcessing

The Microservice Load Balancing Problem

q Edge-based microservices can be easily saturated.

9

ImageProcessing

2

The Microservice Load Balancing Problem

q Edge-based microservices can be easily saturated.

q Challenge: interdependent microservices.

10

ImageProcessing

1

ImageProcessing

2

TemporalAnalysis

1Traffic

Prediction

ObjectDetection

1

ObjectDetection

2

ObjectDetection

3

TemporalAnalysis

2

Our Approach: Overview

11

Problem Modeling1) DAG-based interdependency graph (App-Graph).2) Compactly modeled infrastructure (Inf-Graph).3) Flexible application instantiation (Real-Graph).4) Joint instantiation finding & load allocation.5) Application QoS requirements.

Algorithmic Results1) Optimal algorithm for QoS-agnostic problem.2) NP-hardness for QoS-aware problem.3) FPTAS for QoS-aware problem.

Next Steps (Future Work)1) Network-aware load balancing.2) Reliability and security.3) Economics-aware microservice composition.

Outlines

12

Background and Motivation

System Modeling

Algorithm Design and Analysis

Performance Evaluation

Discussions, Future Work and Conclusions

Application with Interdependent Microservices

q General DAG-based application graph (App-Graph).v Captures complex interdependencies, unlike existing line graph-based models.

13

Cloud StorageCloud

Storage

Video Proc.Video Proc.

Motion Detect.Motion Detect.

Anomaly Detect.

Anomaly Detect.

Smart Home

Control

Smart Home

ControlData Proc.Data Proc.

Cmd. CenterCmd.

Center

Security Cameras

Motion Sensors

Ambient Sensors

Data InputData Input API CallAPI CallMicroserviceMicroserviceMicroserviceData Input API CallMicroservice

Interdependency edge• Data distribution ratio: output / input.

Microservice

IoT Infrastructure in the Application’sView

q Inf-Graph: deployed microservice instances & their interactions.

14

AA

BB

CC

DD EE

0.5

0.5

1.0

1.0

1.0

1.0

App-Graph

A1A1

B1B1

C1C1

D1D1 E1E1

D2D2 E2E2δ=2.0

(2, 1)

(1, 5)

(1, 4)

(2, 3)

(2, 6)

(2, 4)

(2, 6)

A1

B1

C1

D1 E1

D2 E2δ=2.0

(2, 1)

(1, 5)

(1, 4)

(2, 3)

(2, 6)

(2, 4)

(2, 6)

Inf-Graph

Microservice Instance• Capacity: input data volume.• Processing delay.

Inter-Instance Communication Channel• Transmission delay (=0 in example).

External Demand

Application Instantiation

q Real-Graph: instantiating the App-Graph in the Inf-Graph.

15

A1A1

B1B1

C1C1

D1D1 E1E1

D2D2 E2E2δ=2.0

(2, 1)

(1, 5)

(1, 4)

(2, 3)

(2, 6)

(2, 4)

(2, 6)

A1

B1

C1

D1 E1

D2 E2δ=2.0

(2, 1)

(1, 5)

(1, 4)

(2, 3)

(2, 6)

(2, 4)

(2, 6)

A1A1

B1B1

C1C1

D1D1 E1E1

D2D2 E2E2δ=2.0

(2, 1)

(1, 5)

(1, 4)

(2, 3)

(2, 6)

(2, 4)

(2, 6)

A1

B1

C1

D1 E1

D2 E2δ=2.0

(2, 1)

(1, 5)

(1, 4)

(2, 3)

(2, 6)

(2, 4)

(2, 6)

Inf-Graph

A1

B1

C1 D1 E1

D2 E2

(1, 1)

(0.5, 6)

(0.5, 5)

(1, 9)

(1, 13)

(1, 7) (1, 13)φ=2.0

A1

B1

C1 D1 E1

D2 E2

(1, 1)

(0.5, 6)

(0.5, 5)

(1, 9)

(1, 13)

(1, 7) (1, 13)φ=2.0

Real-Graph

Selected Instance• Impact ratio: how much data this instance

gets if 1 unit is allocated to the source.• Max cumulative delay from source.• Both can be computed based on this graph.

Source Demand Allocation

Source of Demand• A real-graph has a single source of demand.

(There could be multiple sources in Inf-Graph.)

Problem Statement: Overview

q Inputs:v App-Graph: microservices, interdependencies, data distribution ratiosv Inf-Graph: instances, communication channels, capacities, and delays

q Outputs:v A set of Real-Graphs.v External demand allocation for each Real-Graph.

q Constraints:v Load balancing: total load <= instance capacity * 𝛹.v QoS awareness: maximum delay <= D.

q Objective (optimization version):v Minimize maximum delay of all Real-Graphs.

16

Outlines

17

Background and Motivation

System Modeling

Algorithm Design and Analysis

Performance Evaluation

Discussions, Future Work and Conclusions

BLB: Basic (QoS-agnostic) Load Balancing

q Without delay constraint, the problem can be formulated as LP.

18

Variables: Per-link demand allocation function.

Demand per node = external + flow-in.

find f : L 7! R⇤

s.t. �n = �extn +X

l2Lin(n)

f(l), 8n 2 N ;

�n · cn, 8n 2 N ;

r(vn,w)�n =X

l2Lout(n,w)

f(l), 8n,w 2 Vout(vn).

<latexit sha1_base64="m9PQWmRwqT5p5Kb9x2EeaIVayvI=">AAADQnicjVJLb9QwEHbCqyyPbuHIZcQKtAsoyiIkXkKq4MKhQgti26L1NnIcp7Xq2MF2Cqso/40Lf4Abf4ALB0BcOeA8JNrtHhjJ8qeZ7/OMZybOBTc2DL96/pmz585fWLvYu3T5ytX1/sa1baMKTdmUKqH0bkwME1yyqeVWsN1cM5LFgu3Ehy/q+M4R04Yr+dYucjbPyL7kKafEOle04b3Dln20ZcplUgHcxu8LkkD6BLYAZyQ3VtW3PYjj8k21dwcA416rMIEN/ilwwoQlkYRnHdwrW5o7VeX8dwGbIotKAZhL2Iq6MJdVNZSjCtKhGN2D7rFUaSIEyIb76mmTdTkRFgzwxHDANFEWaCT/R66jcnhUUz+4nMeLXlWcKmxdXUteWaALNZLtkxKXYRRE/UEYhI3BaTDuwAB1Non6X3CiaJExaakgxszGYW7nJdGWU8GqHi4Mywk9JPts5qAkGTPzstmBCm45jxuc0u5I147ae1xRksyYRRY7Zj1Psxyrnatis8Kmj+ZuTnlhmaRtorQQ4DajXihIuGbUioUDhGruagV6QDSh1q1dzzVhvPzl02B6P3gchK8fDDafd91YQzfQTTREY/QQbaKXaIKmiHqfvG/eD++n/9n/7v/yf7dU3+s019EJ8//8BUIOB0A=</latexit><latexit sha1_base64="m9PQWmRwqT5p5Kb9x2EeaIVayvI=">AAADQnicjVJLb9QwEHbCqyyPbuHIZcQKtAsoyiIkXkKq4MKhQgti26L1NnIcp7Xq2MF2Cqso/40Lf4Abf4ALB0BcOeA8JNrtHhjJ8qeZ7/OMZybOBTc2DL96/pmz585fWLvYu3T5ytX1/sa1baMKTdmUKqH0bkwME1yyqeVWsN1cM5LFgu3Ehy/q+M4R04Yr+dYucjbPyL7kKafEOle04b3Dln20ZcplUgHcxu8LkkD6BLYAZyQ3VtW3PYjj8k21dwcA416rMIEN/ilwwoQlkYRnHdwrW5o7VeX8dwGbIotKAZhL2Iq6MJdVNZSjCtKhGN2D7rFUaSIEyIb76mmTdTkRFgzwxHDANFEWaCT/R66jcnhUUz+4nMeLXlWcKmxdXUteWaALNZLtkxKXYRRE/UEYhI3BaTDuwAB1Non6X3CiaJExaakgxszGYW7nJdGWU8GqHi4Mywk9JPts5qAkGTPzstmBCm45jxuc0u5I147ae1xRksyYRRY7Zj1Psxyrnatis8Kmj+ZuTnlhmaRtorQQ4DajXihIuGbUioUDhGruagV6QDSh1q1dzzVhvPzl02B6P3gchK8fDDafd91YQzfQTTREY/QQbaKXaIKmiHqfvG/eD++n/9n/7v/yf7dU3+s019EJ8//8BUIOB0A=</latexit><latexit sha1_base64="m9PQWmRwqT5p5Kb9x2EeaIVayvI=">AAADQnicjVJLb9QwEHbCqyyPbuHIZcQKtAsoyiIkXkKq4MKhQgti26L1NnIcp7Xq2MF2Cqso/40Lf4Abf4ALB0BcOeA8JNrtHhjJ8qeZ7/OMZybOBTc2DL96/pmz585fWLvYu3T5ytX1/sa1baMKTdmUKqH0bkwME1yyqeVWsN1cM5LFgu3Ehy/q+M4R04Yr+dYucjbPyL7kKafEOle04b3Dln20ZcplUgHcxu8LkkD6BLYAZyQ3VtW3PYjj8k21dwcA416rMIEN/ilwwoQlkYRnHdwrW5o7VeX8dwGbIotKAZhL2Iq6MJdVNZSjCtKhGN2D7rFUaSIEyIb76mmTdTkRFgzwxHDANFEWaCT/R66jcnhUUz+4nMeLXlWcKmxdXUteWaALNZLtkxKXYRRE/UEYhI3BaTDuwAB1Non6X3CiaJExaakgxszGYW7nJdGWU8GqHi4Mywk9JPts5qAkGTPzstmBCm45jxuc0u5I147ae1xRksyYRRY7Zj1Psxyrnatis8Kmj+ZuTnlhmaRtorQQ4DajXihIuGbUioUDhGruagV6QDSh1q1dzzVhvPzl02B6P3gchK8fDDafd91YQzfQTTREY/QQbaKXaIKmiHqfvG/eD++n/9n/7v/yf7dU3+s019EJ8//8BUIOB0A=</latexit>

Capacity (load balancing) per node.Flow conservation: sum flow towards all

instances of a downstream microservice w= input data * data distribution ratio of w.

BLB: Basic (QoS-agnostic) Load Balancing

q Without delay constraint, the problem can be formulated as LP.

19

Variables: Per-link demand allocation function.

Demand per node = external + flow-in.

find f : L 7! R⇤

s.t. �n = �extn +X

l2Lin(n)

f(l), 8n 2 N ;

�n · cn, 8n 2 N ;

r(vn,w)�n =X

l2Lout(n,w)

f(l), 8n,w 2 Vout(vn).

<latexit sha1_base64="m9PQWmRwqT5p5Kb9x2EeaIVayvI=">AAADQnicjVJLb9QwEHbCqyyPbuHIZcQKtAsoyiIkXkKq4MKhQgti26L1NnIcp7Xq2MF2Cqso/40Lf4Abf4ALB0BcOeA8JNrtHhjJ8qeZ7/OMZybOBTc2DL96/pmz585fWLvYu3T5ytX1/sa1baMKTdmUKqH0bkwME1yyqeVWsN1cM5LFgu3Ehy/q+M4R04Yr+dYucjbPyL7kKafEOle04b3Dln20ZcplUgHcxu8LkkD6BLYAZyQ3VtW3PYjj8k21dwcA416rMIEN/ilwwoQlkYRnHdwrW5o7VeX8dwGbIotKAZhL2Iq6MJdVNZSjCtKhGN2D7rFUaSIEyIb76mmTdTkRFgzwxHDANFEWaCT/R66jcnhUUz+4nMeLXlWcKmxdXUteWaALNZLtkxKXYRRE/UEYhI3BaTDuwAB1Non6X3CiaJExaakgxszGYW7nJdGWU8GqHi4Mywk9JPts5qAkGTPzstmBCm45jxuc0u5I147ae1xRksyYRRY7Zj1Psxyrnatis8Kmj+ZuTnlhmaRtorQQ4DajXihIuGbUioUDhGruagV6QDSh1q1dzzVhvPzl02B6P3gchK8fDDafd91YQzfQTTREY/QQbaKXaIKmiHqfvG/eD++n/9n/7v/yf7dU3+s019EJ8//8BUIOB0A=</latexit><latexit sha1_base64="m9PQWmRwqT5p5Kb9x2EeaIVayvI=">AAADQnicjVJLb9QwEHbCqyyPbuHIZcQKtAsoyiIkXkKq4MKhQgti26L1NnIcp7Xq2MF2Cqso/40Lf4Abf4ALB0BcOeA8JNrtHhjJ8qeZ7/OMZybOBTc2DL96/pmz585fWLvYu3T5ytX1/sa1baMKTdmUKqH0bkwME1yyqeVWsN1cM5LFgu3Ehy/q+M4R04Yr+dYucjbPyL7kKafEOle04b3Dln20ZcplUgHcxu8LkkD6BLYAZyQ3VtW3PYjj8k21dwcA416rMIEN/ilwwoQlkYRnHdwrW5o7VeX8dwGbIotKAZhL2Iq6MJdVNZSjCtKhGN2D7rFUaSIEyIb76mmTdTkRFgzwxHDANFEWaCT/R66jcnhUUz+4nMeLXlWcKmxdXUteWaALNZLtkxKXYRRE/UEYhI3BaTDuwAB1Non6X3CiaJExaakgxszGYW7nJdGWU8GqHi4Mywk9JPts5qAkGTPzstmBCm45jxuc0u5I147ae1xRksyYRRY7Zj1Psxyrnatis8Kmj+ZuTnlhmaRtorQQ4DajXihIuGbUioUDhGruagV6QDSh1q1dzzVhvPzl02B6P3gchK8fDDafd91YQzfQTTREY/QQbaKXaIKmiHqfvG/eD++n/9n/7v/yf7dU3+s019EJ8//8BUIOB0A=</latexit><latexit sha1_base64="m9PQWmRwqT5p5Kb9x2EeaIVayvI=">AAADQnicjVJLb9QwEHbCqyyPbuHIZcQKtAsoyiIkXkKq4MKhQgti26L1NnIcp7Xq2MF2Cqso/40Lf4Abf4ALB0BcOeA8JNrtHhjJ8qeZ7/OMZybOBTc2DL96/pmz585fWLvYu3T5ytX1/sa1baMKTdmUKqH0bkwME1yyqeVWsN1cM5LFgu3Ehy/q+M4R04Yr+dYucjbPyL7kKafEOle04b3Dln20ZcplUgHcxu8LkkD6BLYAZyQ3VtW3PYjj8k21dwcA416rMIEN/ilwwoQlkYRnHdwrW5o7VeX8dwGbIotKAZhL2Iq6MJdVNZSjCtKhGN2D7rFUaSIEyIb76mmTdTkRFgzwxHDANFEWaCT/R66jcnhUUz+4nMeLXlWcKmxdXUteWaALNZLtkxKXYRRE/UEYhI3BaTDuwAB1Non6X3CiaJExaakgxszGYW7nJdGWU8GqHi4Mywk9JPts5qAkGTPzstmBCm45jxuc0u5I147ae1xRksyYRRY7Zj1Psxyrnatis8Kmj+ZuTnlhmaRtorQQ4DajXihIuGbUioUDhGruagV6QDSh1q1dzzVhvPzl02B6P3gchK8fDDafd91YQzfQTTREY/QQbaKXaIKmiHqfvG/eD++n/9n/7v/yf7dU3+s019EJ8//8BUIOB0A=</latexit>

Capacity (load balancing) per node.Flow conservation: sum flow towards all

instances of a downstream microservice w= input data * data distribution ratio of w.

Theorem 1:BLB is optimally solvable in polynomial time.

Real-Graph DecompositionTheorem

q Why do we need such a theorem?

1. Transform any solution of BLB into a set of implementable real-graphs.

2. Define QoS of a load balancing plan (max delay of all real-graphs).

20

Theorem 2:Every demand allocation function f so defined can be decomposed intoat most |N| + |L| real-graphs with positive source demands.

QLB: QoS-aware Load Balancing

q Fully Polynomial-Time Approximation Scheme (FPTAS)can achieve the best trade-off between time and accuracyv Approximation ratio: (1+𝜖) – For maximization problemv Time complexity: O(poly(1/𝜖) × poly(input))v In practice, one can arbitrarily tune 𝜖 to get best accuracy within time limit.

21

Theorem 3:QLB (optimization version) is NP-hard.

Theorems 4&5:QLB (optimization version) admits an FPTAS.

A Brief Overview of Our FPTAS

q Idea: v Pseudo-polynomial time algorithm:

Ø Expand Inf-Graph into a delay-layered graph.Ø Run BLB LP on the expanded graph.

v Discretization via approximate testing:Ø Find delay lower & upper bounds (UB, LB) s.t. UB <= poly(input) * LB.Ø Discretize delay values based on (UB, LB).Ø Run pseudo-polynomial time algorithm.Ø Refine (UB, LB) based on output.

v Efficiency enhancement:Ø Approximate testing to shrink initial bound s.t. UB <= constant * LB.

22

Outlines

23

Background and Motivation

System Modeling

Algorithm Design and Analysis

Performance Evaluation

Discussions, Future Work and Conclusions

Simulation Settings

q Simulated network scenarios:v App-Graph:

Ø 5-layered applications, layer-1 being the source layerØ 10-70 microservices: 10% in layer-1, uniformly distributed in other layersØ 4 in-going edges per microservice in layers 2-5Ø Data distribution ratio: uniformly generated

v Inf-Graph:Ø 1 instance per microservice in source layer, 5-15 in othersØ Linking probability (between interdependent instances): 0.3Ø Source demands: 100-900 unitsØ Node capacities: 10-90 unitsØ Node/Link delays: 0-500/1000 ms

v Load balancing goal: optimal load under BLB, or 2 x optimal load under BLBv Approximation parameter: 𝜖=0.5

q Comparisons:v QLBv BLBv QHU: QoS-aware heuristic, solving BLB minimizing demand-weighted delay

24

Comparison Results

25

Running timePolynomially increased

Max delay (QoS)QLB has improved delayover the other solutions.

Outlines

26

Background and Motivation

System Modeling

Algorithm Design and Analysis

Performance Evaluation

Discussions, Future Work and Conclusions

Other Perspectives and Beyond

q So far, we’ve talked aboutv Basic model: DAG-based apps, Real-Graphsv Processing capacities and delaysv Network delays

q What we didn’t consider in this workv Network topologyv Network capacities & congestionv Routingv Reliability: microservice instance failuresv Incentives, pricingv Payment methods

q A unified approach is still in need for high-performance IoT.

27

Computing Perspective

Networking Perspective

Security Perspective

Economics Perspective

Our Conclusions

q Load Balancing for Interdependent IoT Microservicesv DAG model for applications:App-Graph and Inf-Graphv Application realization with Real-Graph abstractionv System-wide load balancing with QoS (delay) constraints

q Algorithmic solutionsv Optimal solution for QoS-agnostic problemv FPTAS for (NP-hard) QoS-aware problem

q Future directionsv Unified framework for IoT performance optimization

28

Thank you very much!Q&A?

29

NP-Hardness Proof

v0v0 u1u1 v1v11.0 1.0 u2u2 v2v2 uκuκ vκvκ

1.0 1.0 1.0vκ-1vκ-11.0v0 u1 v1

1.0 1.0 u2 v2 uκ vκ1.0 1.0 1.0vκ-1

1.0

n0n0

m1m1

m1m1

n1n1

m2m2

m2m2

n2n2

mκmκ

mκmκ

nκnκ

0

1

0 0

1 1

δ=2.0

nκ-1nκ-1n0

m1

m1

n1

m2

m2

n2

mκ

mκ

nκ

0

1

0 0

1 1

δ=2.0

nκ-1

d=axκd=axκd=ax2d=ax2d=ax1d=ax1

App-graph:

Inf-graph:

30