Stream Data• Operator Ordering Query Optimiza-
tion
• Query Index
Query Optimization• Operator Ordering Problem• Assumption
– A query consists of a set of commutative filters– Filter
• Drop or Select• Overall processing costs can vary widely across dif -
ferent filter order
• Ex– Filter O1 drops 1, 3, 5– Filter O2 drops 2, 4, 6– Let an input stream be 2, 4, 6.– The cost of Operator Order O2,O1is cheaper than that of O1, O2
Operator ordering• Operator Ordering– Choose efficient order– The optimal order is changed over time.
• Eddy[4]– Tuple routing Technique– An operator dropping many tuples has high priority
Operator ordering• A-Greedy[9]
– Query Cost C
– d(i|j) denotes the conditional probability that i th oper-ator Of(i) will drop a tuple e, given that e was not dropped by any of operators Of(1), Of(2),..., Of(j).
– ti represents the expect time for Of(i) to process one tu-ple
• Goal Minimized C– Greedy heuristic rule which rearrange the operator or-
der satisfying the following formula
Operator ordering
• A-Greedy– Profiler
• To obtain conditional selectivity d(i|j), profiling is used.• In profiling, a tuple e which is dropped during processing is se-
lected with probability p• Then, profiler artificially applies e to all operators and generate
a profile tuple whose attribute bi is 1 if Oi drops e– Reoptimizer
• Keeps the operator order• Maintains a matrix view• Ex) first row: O4 drops most tuples, second row : reports the numbers of tuples which are not
dropped by O4 droped by O1,O3, and O2.
Profile matrix view
Operator ordering• Problem of A-Greedy– Profiling overhead– A normal tuple may be dropped by an
operator, but a tuple for profiling is ap-plied to all operators.
– In other words, when 10% data of input are profiled, the increment of system overheads is greater than 10%.
• Push-based data source• High and unpredictable data rates• Problem– Load > Capacity
– Load Shedding: eliminate excess load by dropping data
Load Shedding[8]
AuroraApp QoS...
...
App QoS...
App QoS
...
...
s
s
m
s
È
m
s
Slide
Tumble
m
s
ss
ss
ss
mmss
ssss
ÈÈ
ss
mm ÈÈÈÈ
ssss mm App
TumbleTumble App
QoS: Aurora
• QoS• Specifies “Utility” Of Imperfect Query Results
Delay-Based (specify utility of late results)Delivery-Based, Value-Based (specify utility of partial results)
• QoS Influences… Scheduling, Storage Management, Load Shedding
QoS
Output value
1
0
QoS
% messages delivered
1
0100 0
B
QoS
delay
1
0
goodzoneA C
Load Shedding: Aurora• Two Load Shedding Techniques:
• Random Tuple DropsAdd DROP box to network (DROP a special case of FILTER)Position to affect queries w/ tolerant delivery-based QoS reqts
• Semantic Load SheddingFILTER values with low utility (acc to value-based QoS)
• Load Coefficient
Load Shedding: Aurora
• Best location of Drop operator–Maximize cycle gain, minimize utility
loss– Cycle gain: processor cycles gained fro
each percentage of tuples dropped• G(x) = R*(x*L-D) R: input rate, L is load coefficient
– Loss/Gain ratio the smaller, the better
Load Shedding: Aurora
Drop x%
R L
D cycles/tuple
Loss-tolerant graph
• Load Sheddingwhere, when, how much.– Where ->[8], How much [26[
• Particularly, in multi-Query Environments• Ex) Two Query, Q1 and Q2Data size = 24, Processing cost per tuple = cOverall cost = 24*2*c = 48cSystem capability = 30cGoal : Min G = ((1-rp)/rp )*fp
where rp is the fraction to be considered for a query Qp
fp is actual frequency of tuples to be result.Assume fa =1, fb =4Plan 1) Uniform ra = rb =15 G = 3Plan 2) Proportional fb/fa = 4 6:24 ra= 6/24, rb = 24/24 G=3Plan 3) Optimal ra = 10/24, rb 20/24 G = 2.2
Load Shedding[26]
• Estimate fp– Let bi = 1 if a tuple ti is a query result. Otherwise bi =0– fp = bi
– Each tuple ti is processed with a probability rq and dis-card with a probability 1-rq
– Let Xi = bi/rq with a probability rq and Xi = 0 with a probability 1-rq
– Estimate fp = Xi• E(fp) = E(Xi) = bi = fp
– Var(fp) =((1-rq)/rq) *fp• Variance means average error ep
Load Shedding[26]
– Let S is a set of query, |S|= N– Error vector E = [e1,…, eN]– Importance of queries V = [v1,…,vN]– Resource Cost C = [c1,…cN]– Processing ratio r = [r1,…, rN]– Total resource limitation = L– Data Size = W
• Goal : Constraint rC = ri*ci <= L/W• Minimize G = EV= ei*vi
– Apply eq=((1-rq)/rq) *fp
– G= - fi*vi + G1 where G1 = (fj*vj)/rj
– To minimize G, it suffices to minimize G1
– non-linear programming(separable and convex resource allocation)– Sorting O(NlogN)– In the paper, suggest O(N) algorithm
Load Shedding[26]
Query Index• Invoke all query whenever data arrives
– Query Index
• Property of Stream Data– Locality– ex. the temperature in near future will be similar to the
current temperature– Some or all queries will be reused in near future
Query Index– The number of registered queries is huge– Overhead to find out the proper queries which
can evaluate the input stream item.
– IBS(Interval Binary Search Tree)
– R-Tree• Multi-Dimensional data access method• Range conditions of Queries are overlaped.• Many nodes should be traversed due to a large
amount of overlap of query conditions
Query Index• IBS[10]
– Use balanced binary search tree for query indexes– When a data item arrives, balanced binary search
trees and hash table are probed with the value of tu-ples
– Not appropriate to general range queries which have two bounded conditions• Each condition is indexed in individual binary tree. un-
necessary partial result
Query Conditions
q1: R.a 1 and R.a < 10q2: R.a > 5q3: R.a > 7q4: R.a = 4q5: R.a = 6
5
1 7
q1 q2 q3
10
q1
1=q14=q46=q5
Group Filter for R.a
<
> =
!=
• Query Processing Based on Spatial Join[26]– Query- represented as a region– Data – represented as a point
• Batch mode• Accumulate arriving data elements and process continuous
queries Set of data represented as a region– Uses Spatial Indexes for data set and queries
Query Index
• A set of data region• Query region– compute overlap relationships
• In [26], Use Corner Transformation – n-dim object 2n-dim point
Query Index
Query Index– BMQ-Index [11]• DMR List is a list of DNi
– DNi = <DRi,+DQSet, -DQSet>– DRi is a matching Region (bi-1, bi)– +DQSet is a set of queries whose lower bound lk
= bi-1
– -DQSet is a set of queries whose upper bound uk = bi-1
• A stream table keeps the recently accessed DNi
Query Conditions
q1: R.a 1 and R.a < 10q2: R.a > 5q3: R.a > 7q4: R.a = 4q5: R.a = 6
1 4 5 6 7 10 inf
q1
q2
q3
DN1 DN2 DN3 DN4 DN5 DN6 DN6
{+q1} {+q2} {+q3} {-q1} {-q2,-q3}
stream table
Query Index• QSet(t) is a set of queries for data vt• Let vt be in DNj and vt+1 be in DNh,
– e.g., bj-1 <= vt < bj and bh-1<= vt+1 < bh• Then QSet(t+1) is obtained as follows
• For example• vt = 4.5, QSet(t) = {q1}• if vt+1 = 12,
– U+DQSet = {q2,q3}– U-DQSet = {q1}– Thus QSet(t+1) = {q2,q3}
1 4 5 6 7 10 inf
q1
q2
q3
DN1 DN2 DN3 DN4 DN5 DN6 DN6
{+q1} {+q2} {+q3} {-q1} {-q2,-q3}
stream table
Query Index• Problem of BMQ-Index
– If the forthcoming data is quite different from the current data, many DRM nodes should be retrieved like a linear search
– Support only (l, u) style condition. • q4 and q5 is not registered
– does not work correctly on the boundary condition.
1 4 5 6 7 10 inf
q1
q2
q3
DN1 DN2 DN3 DN4 DN5 DN6 DN6
{+q1} {+q2} {+q3} {-q1} {-q2,-q3}
stream table
Let vt = 5.5 and QSet(t) = {q1,q2}If vt+1 = 5, Then QSet(t+1) is also {q1,q2}But, actual query set of vt+1 is {q1}.
• [1] D. Carney, U. Cetintemel, M. Cherniack, C. Convey, S. Lee,G. Seidman, M. Stonebraker, N. Tatbul, and S. Zdonik. Monitoring streams–a new class of data management applications. In Proc. 28th Intl. Conf. on Very Large Data Bases, Aug. 2002.
• [2] A. Arasu, B. Babcock, S. Babu, M. Datar, K. Ito, R. Motwani, I. Nishizawa, U. Srivastava, D. Thomas, R. Varma, J. Widom, J., “Stream: The stanford stream data manager”, IEEE Data Engineering Bulletin, Vol 26, No 1, pp. 19-26, 2003.
• [3]J. M. Hellerstein, M. J. Franklin, S. Chandrasekaran, A. Deshpande, K. Hildrum, S. Madden, V. Raman, V., M. A. Shah, “Adaptive query processing: Technology in evolution”, IEEE Data Engineering Bulletin, Vol 23, No 2, pp. 7-18, 2000.
• [4] R. Avnur, J. M. Hellerstein, “Eddies: Continuously adaptive query processing”, In Proceedings of ACM SIGMOD Conference, pp. 261-272, 2000.
• [5] Brain Babcock et.al, “Chain: Operator scheduling for Memory minimization in Data Stream Systems,” ACM SIGMOD 2003.
• [6] Don Carney et.al, “Operator Scheduling in a Data Stream Manager”, VLDB 2003• [7] B. Pielech, “Adaptive scheduling algorithm selection in a streaming query system,” Master thesis ,
Worcester polytechnic institute, 2004.• [8] N Tatbul, U Çetintemel, S Zdonik, M Cherniack, M Stonebraker, “Load shedding in a data stream man-
ager”, VLDB 2003.• [9]. Babu, S., Motwani, R., Munagala, K., Nishizawa, I., Widom, J.: Adaptive ordering of pipelined stream
filters. In: Proceedings of ACM SIGMOD Conference. (2004) 407–418• [10] S. Madden, M.A. Shah, J.M. Hellerstein, V. Raman, “Continuously adaptive continuous queries over
streams”, In Proceedings of ACM SIGMOD Conference, 2002.• [11] Jinwon Lee, Seungwoo Kang, Youngki Lee, SangJeong Lee, and Junehwa Song, "
BMQ-Processor: A High-Performance Border Crossing Event Detection Framework for Large-scale Monitoring Applications", IEEE Transactions on Knowledge and Data Engineering (TKDE), Vol. 21, No. 2, pp 234-252, February 2009
Reference
• [12] S. Madden et.al., “TAG: Aggregation Service for Ad-Hoc Sensor Networks”, OSDI, 2002• [13] N. Shrivastava et.al., “Medians and Beyond: New Aggregation Techniques for Sensor Networks,”
ACM Sensys 2004• [14] N. Trigoni et.al., “Multi-Query Optimization for Sensor Networks” DCOSS 2005• [15]N. Trigoni, et.al., "Routing and Processing Multiple Aggregate Queries in Sensor Networks,“ ACM
SenSys, 2006.• [16] A. Deshpande et.al., "Model-Driven Data Acquisition in Sensor Networks,“ VLDB, 2004.• [17] D. Chu et.al., "Approximate Data Collection in Sensor Networks using Probabilistic Models,“
ICDE, 2006• [18] D. Tulone et. al., “PAQ: Time Series Forecasting For Approximate Query Answering In Sensor
Networks,” European Conf. Wireless Sensor Networks, 2006• [19] A. Deligiannakis et.al., “Compressing Historical Information in Sensor Networks,” ACM SIGMOD
2004• [20] A. Jain et.al., “Adaptive Stream Resource Management Using Kalman Filters,” ACM SIGMOD
2004• [21] X. Yang et.al., “In-Network Execution of Monitoring Queries in Sensor Networks,” ACM SIGMOD
2007.• [22]M. Stern et.al., “Towards Efficient Processing of General-Purpose Joins in Sensor Networks,” ICDE
2009.• [23]A. Pandit et.al, “ Communication-Efficient Implementation of Range-Joins in Sensor Networks,”
International Conference on Database Systems for Advanced Applications (DASFAA), 2006• [24] H. Yu et.al, “In-Network Join Processing for Sensor Networks,” APWeb 2006.• [25] A. Coman et.al, “On Join Location in Sensor Networks,” MDM 2007.• [26] H.S. Lin, J.G. Lee, M.J. Lee, K.Y. Whang, I.Y. Song ,” Continuous Query Processing in Data
Streams Using Duality of Data and Queries,” ACM SIGMOD 2006.• [27] B. Mozafari, C. Zaniolo, “Optimal Load Shedding with Aggregates and Mining Queries,” ICDE
2010.
Reference
Top Related