Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Event Processing Course
Filtering and transformation(Relates to Chapter 8)
2Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Lecture outline
Filtering
Transformations
Derivations
FFT examples
Some code examples
3Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Filter on input terminal: by event type
Event Producer
Event Processing Agent
Filter expression:“allow only triangle and square events”
A filter expression (assertion) takes the form of a predicate that is evaluated against an event. The event passes the filter if the predicate evaluates to TRUE
and fails the filter if the predicate evaluates to FALSE[ .
4Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Filter EPA
Filtering
Filter EPA
Filtered Out
Filtered In
Non-Filterable
Input terminal filter expression
Principal filter expression
5Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Two consecutive filters
Filter 1 Out
In
Non-Filterable
Membership level = Gold Filter 2
Out
In
Non-Filterable
Membership level = SIlver
Gold
Silver
Bronze
Silver
BronzeSilver
Bronze
Gold
6Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Various filtering
Filtering
Filter EPA
Filtered Out
Filtered In
Non-Filterable
Input terminal filter expression
Principal filter expression
Partitionselection
Context expression
7Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Filter features
Feature Interpretation
Attribute inclusion Checks whether an attribute with that name exists in the event instance
Attribute has value Checks that the value of the attribute is not null
Data type assertion Checks that the attribute's value is valid with respect to its data type, for example
that it is valid as a number, string or timestamp
Binary relation Tests the value of an attribute against a constant or against the value of another
attribute, using a relation such as <, >, = , , , or
Conjunction Expression is the conjunction ("AND") of two or more expressions
Disjunction Expression is the disjunction ("OR") of two or more expressions
Negation Expression is the negation of another expression
8Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
XPATH filters
Expression returns Success condition Example
Node-set Node-set contains at least one node /occurrenceTime
Boolean Value is TRUE /ranking > 5
Number Value is non-zero 25
String String has non-zero length “OK”
9Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Stateful filters
First m—This passes the first m event instances in the context partition window.
Last m—This passes the last (most recent) m event instances in the window.
Random m—This passes a random set of m instances.
10Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Transportation types
Translate Compose
Aggregate Enrich
Split
Project
11Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Transportation logic
Derivation
Output
Failure
Input terminal filter expression
Derivation expression
Partitionselection
Context expression
12Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Project EPA
Projection List{Ticker, Vwap}
Project EPATicker=“xyz”
MinPrice = 46.7MaxPrice = 49.5
Vwap = 48.3
Ticker=“xyz”Vwap = 48.3
Input event Output event
13Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Translate EPA
Derivation Expression:OrderType = “Low”Price = Price/1.15Total = NumItems * Price
Translate EPA
NumItems =10Price = 17
OrderType =“Low”Price = 14.78Total = 170
Input event Output event
14Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Enrich EPA
Derivation
Enrich EPA
Output
Failure
Input terminal filter expression
Derivation expression
Partitionselection
Context expression
event
global state
Query expression
Query Parameters
15Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Multiple results policy for ENRICH
The multiple results policy defines the behavior of an enrich event processing agent when its query returns more than one result. The possible policy values are: first, last, every, and combine.
The four values for this policy are defined as follows:
First—Use only the first row that is returned.
Last—Use only the last row that is returned.
Every—A separate output event is generated, one for each row that is returned.
Combine—A single output event is returned, but the derivation rules have access to all the rows when preparing the output
16Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Split EPA
Derivation
Output 1
Failure
Input terminal filter expression
Derivation expressions
Partitionselection
Context expression
Output 2
Output…
Split EPA
17Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Aggregate EPA
Derivation
Output
Failure
Input terminal filter expression
Derivation expression
Partitionselection
Context expression
Aggregate EPA
State accumulation
18Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Aggregate EPA
Derivation expression:Driver = any(Driver);AssignmentCount = count();
RequestId = R429243Driver = “Arthur Boyd”RequiredPickUpTime = 15:30RequiredDeliveryTime = 16:30Store = “Exotic Flowers”
Driver = “Arthur Boyd”Assignment Count = 3
RequestId = R427763Driver = “Arthur Boyd”RequiredPickUpTime = 15:30RequiredDeliveryTime = 16:30Store = “Exotic Flowers”
Aggregate EPA
RequestId = R429531Driver = “Arthur Boyd”RequiredPickUpTime = 15:30RequiredDeliveryTime = 16:30Store = “Exotic Flowers”
Segmentation context: DriverEvent Interval Context:
Initator = Assignment
19Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Aggregation operators
Name Argument Returns
first Attribute Name Attribute value taken from the first event that arrived in the
context partition
last Attribute Name Attribute value taken from the last event that arrived in the
context partition
any Attribute Name Attribute value taken from an arbitrary event in the context
partition
min Attribute Name Smallest of all the values of the attribute
max Attribute Name Largest of all the values of the attribute
sum Attribute Name Sum of all the values of the attribute
avg Attribute Name Arithmetic mean of all the values of the attribute
stdev Attribute Name Standard deviation of all the values of the attribute
distinct Attribute Name Number of distinct values of the attribute
concat Attribute Name A list that contains all the attribute values
dConcat Attribute Name A list that contains all the distinct attribute values
count - Number of event instances in the current context partition
partitionCount - Number of active partitions for the current context
globalCount - Number of event instances in all active partitions of the current
context
20Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Compose EPA
Derivation
Output
Failure
Input terminal filter expression
Derivation expression
Partitionselection
Context expression
Compose EPA
Accumulation and matching
Input terminal filter expression
Left
Right
Buffer specification
Match condition
21Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Compose EPA example
Derivation expression:Registration= Left/Registration Entry = Left/OccurrenceTimeExit = Right/OccurrenceTime
Registration = ABC123Arrival = 15:14:12Departure = 15:16:23
Compose EPA
Match expression:Left/Registration = Right/Registration
Buffer SpecificationLeft =10 minutesRight = 0
OccurenceTime = 15:14:12Registration = ABC123
OccurenceTime = 15:16.23Registration = ABC123
Registration = ABC123Entry = 15:14:12Exit = 15:16:23
OccurrenceTime=15:14:12Registration = ABC123
OccurrenceTime=15:16.23Registration = ABC123
22Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Compose policies
Left buffer specification—This controls how many event instances from the Left input terminal should be retained. It can be specified either as a count of instances or as a time interval.
Right buffer specification—This controls how many event instances from the Right input terminal should be retained. It can be specified either as a count of instances or as a time interval.
Unmatched Left Policy—This states what should happen when an event is evicted from the left buffer if that event hasn’t been matched with anything prior to eviction.
Unmatched Right Policy—This states what should happen when an event is evicted from the right buffer if that event hasn’t been matched with anything prior to eviction.
Match condition—This is the condition used to judge whether an event from the left stream matches one from the right stream. It can be a simple equality test, such as Left/A = Left/B, or a more complex expression involving both events, such as the XPath expression count(Left/A) = count(Right/B) + 7.
23Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Derivation expression
A derivation expression is an expression that assigns values to the attributes of the derived event. A derivation expression can refer to values of the input event attributes.
24Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Header derivation
Attribute Name Default output value Settable by derivation step?
eventIdentity New system-generated identity No
event Type Specified by output terminal definition Yes
detectionTime See earlier discussion No
occurrenceTime As detectionTime Yes1
eventAnnotation Not present Yes
eventCertainty Not present Yes
eventSource Identity of the event processing agent Yes
1 See section 11.1.2 for further discussion
25Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
FFD – Bid Request Creator EPA
Driver Status Table
Derivation expression:Copy attributes from input;driversList = Driver;
Enrich EPA
Query expression:SELECT Driver FROM DriverStatusWHERE Ranking > = ? AND Location = ?
North East5Mark Smith
North East7Arthur Boyd
………
South West5Joan Winter
LocationRankingDriver
Query parameters:minimumRankingneighborhood
requestId = R429531addresseeLocation = 5 Main StrequiredPickUpTime = 15:30requiredDeliveryTime = 16:30store = Exotic FlowersminimumRanking = 5driversList = Arthur Boyd, Mark Smith
requestId = R429531addresseeLocation = 5 Main StrequiredPickUpTime = 15:30requiredDeliveryTime = 16:30store = Exotic FlowersminimumRanking = 5neighborhood = North East
Multiple results policy:Combine
26Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
FFD – Bid Enrichment
Store Reference
Enrich EPA
Query Expression:SELECT Auto FROM StoreRef
WHERE Store = ?
requestId = R429531driver = Arthur BoydcommittedPickUpTime = 15:30store = Exotic Flowers
requestId = R429531driver = Arthur BoydcommittedPickUpTime = 15:30store = Exotic Flowersauto = No
Query Parameter = store
Filter EPAOut
In
Non-Filterable
Auto = Yes
To manual
To auto
Bid enrichment
Bid routing
…
North East
South East
Region
…
No
Yes
Auto
17 East Ave3Blooms R Us
………
33 High St5Exotic Flowers
LocationMinRankStore
27Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
FFD - Assignment manager EPA
Derivation expression:Copy attributes of right event
Compose EPA
Match expression:Right/committedPickupTime < Left/requiredPickupTime + 5 minutes
Unmatched left policy: Fail
Buffer Specification:Left = 1 eventRight = 0 events
Bid Request
Delivery BidDelivery Bid
Left
Right
Segmentation context: requestIdEvent Interval Context:
Initiator = Bid RequestExpiration = +2 minutes
Delivery BidDelivery Bid
Output
Failure
No Bidders alert
Failure derivation expression:Copy requestId
28Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Daily Statistics Creator
Aggregate EPA
Fixed interval context: Daily
driver = Arthur Boydday = 2009-02-14assignmentCount = 3
day = 2009-02-14assignmentMean = 5.2assignmentSTDV= 1.2
Derivation expression:day = any(day); assignmentMean=avg(assignmentCount);assignmentSTDV=stdev(assignmentCount);
driver = Arthur Boydday = 2009-02-14assignmentCount = 3
29Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Streambase example
30Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Rulecore example <Rule name="CreateAutomaticAssignments" limit="?" evalMode="once" level="2"> <Description>This is rule CreateAutomaticAssignments</Description> <Initialize> <Assert> <Event> <base:XPath>sim:EventDef[@eventType="BidRequest"]</base:XPath> </Event> <Expression> <Property name=”Store”> <InList name=”AutomaticAssignmentStore”/> </Property> </Expression> </Assert> </Initialize> <Views> <ViewRef name="CreateAutomaticAssignments"> <base:XPath>sim:ViewDef[@name=”CreateAutomaticAssignments”]</base:XPath> </ViewRef> </Views> <Situations> <SituationRef name="CreateAutomaticAssignments"> <base:XPath>sim:SituationDef[@name="CreateAutomaticAssignments"]</base:XPat h> </SituationRef> </Situations> <Actions> <SituationDetected situationName="CreateAutomaticAssignments"> <ActionRef name="CreateAutomaticAssignments" eventVisibility="external"> <base:XPath>sim:ActionDef[@name="CreateAutomaticAssignments"]</base:XPath > </ActionRef> </SituationDetected> </Actions> </Rule>
31Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Apama example
32Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Esper example
insert into BidRequest(requestId, store, location, pickupTime, deliveryTime, storeManual) select d.requestId, d.store, d.location, d.pickupTime, d.deliveryTime, s.manual from DeliveryRequest d unidirectional, GPSLocationW g //,sql:DomainDB['select ranking from Driver where driver = ${g.driver} and ranking > ${d.minimumRanking}'] ,method:Domain.driverRankLookup(g.driver) r ,method:Domain.isStoreManualLookup(d.store) s where Geo.distanceKM(g.location, d.location) < 10 and r.ranking >= d.minimumRanking;
33Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Lecture summary
In this lecture we have looked into:
Filtering
Transformation types
Aggregation derivations
Some code examples
Top Related