Big Events
14
MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Big Events Hans-Arno Jacobsen Middleware Systems Research Group MSRG.org
description
Big Events. Hans-Arno Jacobsen Middleware Systems Research Group MSRG.org. Big Event Data. Traditional Big Data Domain vs. Rest of Universe. There are other emerging domains with needs similar to Big Data Smart grids Smart cities …. - PowerPoint PPT Presentation
Transcript of Big Events
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORG
Big Events
Hans-Arno JacobsenMiddleware Systems Research GroupMSRG.org
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORG
Big Event Data
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORG
H.-A. Jacobsen
Traditional Big Data Domain vs. Rest of Universe
• There are other emerging domains with needs similar to Big Data– Smart grids– Smart cities …
My first message: There are other relevant Big
Data domains – beware!
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORG
H.-A. Jacobsen
Smart Grids for Taming The Energy Problem
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORG
H.-A. Jacobsen
Relevance of Smart Grids
• Increasing penetration of variable renewable energy sources like wind and solar et al.
• Paradigm shift from demand-following supply to supply-following demand
• Need for new large-scale information system infrastructure to control demand
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORG
H.-A. Jacobsen
Distributed Generation, Flexible Loads and Energy Storage
• Come in big numbers• Show unique behavior (users, weather, equipment, …)• Have to be monitored and controlled
Big event data challenge
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORG
H.-A. Jacobsen
Solar Photovoltaic Power Generation
• High frequency measurements required
• Several metrics of interest, many spatially distributed measurement points
1 127 253 379 505 631 757 883 1009113512611387-200
0
200
400
600
800
1000
1200direct normal solar irradiance
minutes
watt
s/m
^2
1 121 241 361 481 601 721 841 961 10811201132105
10152025303540
air temperature
minutes
degr
ees c
elsiu
s
1 124 247 370 493 616 739 862 985 1108123113540
50
100
150
200
250
diffuse solar irradiance
minutes
watt
s/m
^2
Source: National Oceanic & Atmospheric Administration (U.S.)
~2.3 TB per year and 1k panels
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORG
H.-A. Jacobsen
Use of PEVs as Grid Resource
-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6
-0.2
-0.15
-0.1
-0.05
0
0.05
GPS coordinates
Trip 1 Trip 2 Trip 3 Trip 4 Trip 5 Trip 6
delta longitude
dlet
a la
titud
e
• High frequency measurements required
• Important for SG applications: Continuous update of trip destination and energy level at destination
Source: Auto21 Project, University of Winnipeg
~ 0.5 TB per year and 1k vehicles
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORG
H.-A. Jacobsen
Electric Power Consumption
0:00:00 3:39:00 7:18:00 10:57:00 14:36:00 18:15:00 21:54:00012345678
real power
time
kwatt
s
0:00:00 3:30:00 7:00:00 10:30:0014:00:0017:30:0021:00:000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
reactive power
time
kVAR
0:00:00 3:27:00 6:54:00 10:21:0013:48:0017:15:0020:42:00215220225230235240245250255
voltage
Time
Volts
• Very high frequency measurements required (e.g., for inferring device on/off events, grid stability, etc.)
• Several metrics of interest (household electricity meters, single devices, etc.)
Source: UCI Machine Learning Repository
~ 27.5 PB per year and 1k homes
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORG
H.-A. Jacobsen
Traditional Big Data Domain vs. Rest of Universe
My second message: Detecting events in real-
time in the sea of Big Data is just as
important.
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORGTowards Big Events
• Many non-traditional scenarios that require filtering of Big Events at large scales
• … scenarios that require filtering & storage of events at large scales
• Filtering & storage of “event streams”
• Filtering & storage of “event showers”
H.-A. Jacobsen
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORGEvent Showers vs. Event Streams
Event Showers• Partially ordered sets of events• No single event schema • Events vary in shape and size
from one to the next• Processing of many event
expressions• Tends to require support for
aggregation• Broader model & paradigm
(dissemination, matching, coordination)
Event Stream Processing• Linearly ordered event
sequences• Schema-based, single schema
per stream• Stream tuples follow schema• More single-expression
processing-based• Aggregation is a key requirement• Focused on processing
queries/expressions over event streams
H.-A. Jacobsen
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORG
H.-A. Jacobsen
Conclusions
• Big Events are Big Data in motion
• Processing Big Data in real-time to detect events of interest is important as well
• There are other emerging application domains; let us watch out for themMy final message: Big
Data Benchmarking efforts should take this
into account.
MIDDLEWARE SYSTEMSRESEARCH GROUP
MSRG.ORG
H.-A. Jacobsen
Acknowledgements
• C. Goebel for help with smart grid slides
