Applying Control Theory to Stream Processing Systems Wei Xu ([email protected])[email protected]...
-
date post
22-Dec-2015 -
Category
Documents
-
view
219 -
download
1
Transcript of Applying Control Theory to Stream Processing Systems Wei Xu ([email protected])[email protected]...
Applying Control Theory to Stream Processing Systems
Wei Xu ([email protected])Bill Kramer ([email protected])
Joe Hellerstein ( [email protected] )
Description of the system
TCQComplex internal structure
Input BufferData Source
TCQ drops tuples silently if result queue is full
Why do we need control?• Data source does not provide accurate data rate
0 1 2 3 4 5 6
x 105
0
500
1000
1500
2000
2500
3000
3500
4000
4500
time (ms)
num
ber
of t
uple
s pe
r se
c
desired load
actual load
Why do we need control?• TCQ node drops tuples when result queue fill up
0 1 2 3 4 5 6 7 8 9
x 105
0
1000
2000
3000
time (ms)
tup
les p
er
se
c
source data rateend-to-end drop rate
0 1 2 3 4 5 6 7 8 9
x 105
0
1000
2000
3000
time (ms)
tup
les p
er
se
c
output rate of buffer
0 1 2 3 4 5 6 7 8 9
x 105
0
2
4
6x 10
5
time (ms)
fre
e s
pa
ce
(K
B)
free space
Buffer
Source
TCQ
Result Q
Control Problems
• Providing an accurate data source– Get the actual data rate
• Regulate queue length on TCQ node– Prevent dropping tuples – Maximize throughput (and adapts when distur
bance happens)
The Control Architecture
P Controller
PI Controller
Result – An accurate data source
P Controller with Pre-compensation PI Controller
0 2 4 6 8 10
x 105
0
500
1000
1500
2000
2500
3000
time (ms)
tup
les
pe
r se
c
desired loadactual load
0 2 4 6 8 10
x 105
0
500
1000
1500
2000
2500
3000
time (ms)
tup
les
pe
r se
c
desired loadactual load
Result – regulating queue length
Buffer
Source
TCQ
Result Q
0 1 2 3 4 5 6 7 8 9 10
x 105
0
1000
2000
3000
time (ms)
tup
les
pe
r se
csource data rateend-to-end drop rate
0 1 2 3 4 5 6 7 8 9 10
x 105
0
1000
2000
3000
time (ms)
tup
les
pe
r se
c
output rate of buffer
0 1 2 3 4 5 6 7 8 9 10
x 105
2
4
6x 10
5
time (ms)
fre
e s
pa
ce (
KB
)
free space
Result – Under CPU Contention
Buffer
Source
TCQ
Result Q
0 1 2 3 4 5 6 7 8 9
x 105
0
1000
2000
3000
time (ms)
tup
les
pe
r se
csource data rateend-to-end drop rate
0 1 2 3 4 5 6 7 8 9
x 105
0
1000
2000
3000
time (ms)
tup
les
pe
r se
c
output rate of buffer
0 1 2 3 4 5 6 7 8 9
x 105
0
2
4
6x 10
5
time (ms)
fre
e s
pa
ce (
KB
)
free space
Why theory is useful?• One of my implementations .. What happened?
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
x 105
0
2000
4000
time (ms)
tup
les p
er
se
c
source data rateend-to-end drop rate
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
x 105
0
5000
10000
time (ms)
tup
les p
er
se
c
output rate of buffer
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
x 105
0
2
4
6x 10
5
time (ms)
fre
e s
pa
ce
(K
B)
free space
Buffer
Source
TCQ
Result Q
What is going on?
Queue LengthController
Desired Queue length
Actual Queue Length
Data Rate to TCQControlled
Output Thread
(Code Reuse)
Tricky part of parameter estimation
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5
x 105
-3
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5x 10
5
u: number of tuples per sec
y: f
ree
spac
e on
que
ue
Model evaluation – Making the system operate in desired range
Non-Linear range
Easy for data source, but queue length ..
0 1 2 3 4 5 6 7 8 9
x 105
0
1
2
3
4
5
6x 10
5
time (ms)
free
spac
e (K
B)
free space
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
x 105
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
5.5x 10
5
time (ms)
fre
e sp
ace
(K
B)
free space
-1 0 1 2 3 4 5
x 105
-1
0
1
2
3
4
5x 10
5
u: number of tuples per sec
y: fr
ee
spa
ce o
n qu
eue
Free Space
Data rate vs free space
Settling Time and Overshoot matters
P Controller
A lot of small disturbance in a Java programIncremental garbage collection
5 5.1 5.2 5.3 5.4 5.5
x 105
1400
1420
1440
1460
1480
1500
1520
1540
1560
1580
time (ms)
num
ber
of t
uple
s pe
r se
c
desired load
actual load
5.3 5.4 5.5 5.6 5.7 5.8 5.9
x 105
1280
1300
1320
1340
1360
1380
1400
1420
1440
time (ms)
num
ber
of t
uple
s pe
r se
c
desired load
actual load
PI Controller
Conclusion
• Advantages of feedback control– Make system more robust under disturbance– Treat complex systems as black boxes
• Cope with the system characteristics instead of having to change it
– Encourage reporting system statistics– Implementation is easy and has theoretical
guarantees
Future Work
• Load balancer
• Smaller sample time to reduce disturbance caused by Java GC?
• Controller on scheduling of system shared by multiple streams
Description of the System
Revised
DataSource
Input Buffer
TCQ Node
Queue length
RoutingLogic
Load SplitterTCQ Node
Tuples
Tuples
Tuple Blocks
Operation of Load Splitter1. Arriving blocks wait in Input Buffer2. Tuples are routed to balance TCQ queue lengths3. Stop routing if queue length is too large to avoid tuple discards
Compare to Open Loop Control
We know
Y(k) , and we know what we want y(k+1) to be.. Use transfer function to solve for u(k)…
(Expected result – accuracy and disturbance ) -- do be done
Estimation of the transfer function
-3000 -2000 -1000 0 1000 2000 3000-3000
-2000
-1000
0
1000
2000
3000
u: desired tuples per sec
y: a
ctua
l tup
les
per
sec
y(k+1)=ay(k)+bu(k)
Regression
Tricky part of parameter estimation
Model evaluation – A data rate that make it operate in linear range
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
x 105
0
1000
2000
3000
time (ms)
nu
mb
er
of
tup
les
pe
r s
ec
desired load
actual loadend drop
tcq drop
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
x 105
0
1000
2000
3000
time (ms)
blo
ck
re
sp
on
se
tim
e
block response time
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
x 105
0
2
4
6x 10
5
time (ms)
fre
e s
pa
ce
on
re
su
lt q
ue
ue