Event Processing Course

Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion

Event Processing Course

Filtering and transformation(Relates to Chapter 8)

2Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion

Lecture outline

Filtering

Transformations

Derivations

FFT examples

Some code examples


Filter on input terminal: by event type

Event Producer

Event Processing Agent

Filter expression:“allow only triangle and square events”

A filter expression (assertion) takes the form of a predicate that is evaluated against an event. The event passes the filter if the predicate evaluates to TRUE

and fails the filter if the predicate evaluates to FALSE[ .


Filter EPA

Filtering

Filter EPA

Filtered Out

Filtered In

Non-Filterable

Input terminal filter expression

Principal filter expression


Two consecutive filters

Filter 1 Out

In

Non-Filterable

Membership level = Gold Filter 2

Out

In

Non-Filterable

Membership level = SIlver

Gold

Silver

Bronze

Silver

BronzeSilver

Bronze

Gold


Various filtering

Filtering

Filter EPA

Filtered Out

Filtered In

Non-Filterable


Principal filter expression

Partitionselection

Context expression


Filter features

Feature Interpretation

Attribute inclusion Checks whether an attribute with that name exists in the event instance

Attribute has value Checks that the value of the attribute is not null

Data type assertion Checks that the attribute's value is valid with respect to its data type, for example

that it is valid as a number, string or timestamp

Binary relation Tests the value of an attribute against a constant or against the value of another

attribute, using a relation such as <, >, = , , , or

Conjunction Expression is the conjunction ("AND") of two or more expressions

Disjunction Expression is the disjunction ("OR") of two or more expressions

Negation Expression is the negation of another expression


XPATH filters

Expression returns Success condition Example

Node-set Node-set contains at least one node /occurrenceTime

Boolean Value is TRUE /ranking > 5

Number Value is non-zero 25

String String has non-zero length “OK”


Stateful filters

First m—This passes the first m event instances in the context partition window.

Last m—This passes the last (most recent) m event instances in the window.

Random m—This passes a random set of m instances.


Transportation types

Translate Compose

Aggregate Enrich

Split

Project


Transportation logic

Derivation

Output

Failure


Derivation expression

Partitionselection

Context expression


Project EPA

Projection List{Ticker, Vwap}

Project EPATicker=“xyz”

MinPrice = 46.7MaxPrice = 49.5

Vwap = 48.3

Ticker=“xyz”Vwap = 48.3

Input event Output event


Translate EPA

Derivation Expression:OrderType = “Low”Price = Price/1.15Total = NumItems * Price

Translate EPA

NumItems =10Price = 17

OrderType =“Low”Price = 14.78Total = 170

Input event Output event


Enrich EPA

Derivation

Enrich EPA

Output

Failure



Partitionselection

Context expression

event

global state

Query expression

Query Parameters


Multiple results policy for ENRICH

The multiple results policy defines the behavior of an enrich event processing agent when its query returns more than one result. The possible policy values are: first, last, every, and combine.

The four values for this policy are defined as follows:

First—Use only the first row that is returned.

Last—Use only the last row that is returned.

Every—A separate output event is generated, one for each row that is returned.

Combine—A single output event is returned, but the derivation rules have access to all the rows when preparing the output


Split EPA

Derivation

Output 1

Failure


Derivation expressions

Partitionselection

Context expression

Output 2

Output…

Split EPA


Aggregate EPA

Derivation

Output

Failure



Partitionselection

Context expression

Aggregate EPA

State accumulation


Aggregate EPA

Derivation expression:Driver = any(Driver);AssignmentCount = count();

RequestId = R429243Driver = “Arthur Boyd”RequiredPickUpTime = 15:30RequiredDeliveryTime = 16:30Store = “Exotic Flowers”

Driver = “Arthur Boyd”Assignment Count = 3


Aggregate EPA


Segmentation context: DriverEvent Interval Context:

Initator = Assignment


Aggregation operators

Name Argument Returns

first Attribute Name Attribute value taken from the first event that arrived in the

context partition

last Attribute Name Attribute value taken from the last event that arrived in the

context partition

any Attribute Name Attribute value taken from an arbitrary event in the context

partition

min Attribute Name Smallest of all the values of the attribute

max Attribute Name Largest of all the values of the attribute

sum Attribute Name Sum of all the values of the attribute

avg Attribute Name Arithmetic mean of all the values of the attribute

stdev Attribute Name Standard deviation of all the values of the attribute

distinct Attribute Name Number of distinct values of the attribute

concat Attribute Name A list that contains all the attribute values

dConcat Attribute Name A list that contains all the distinct attribute values

count - Number of event instances in the current context partition

partitionCount - Number of active partitions for the current context

globalCount - Number of event instances in all active partitions of the current

context


Compose EPA

Derivation

Output

Failure



Partitionselection

Context expression

Compose EPA

Accumulation and matching


Left

Right

Buffer specification

Match condition


Compose EPA example

Derivation expression:Registration= Left/Registration Entry = Left/OccurrenceTimeExit = Right/OccurrenceTime

Registration = ABC123Arrival = 15:14:12Departure = 15:16:23

Compose EPA

Match expression:Left/Registration = Right/Registration

Buffer SpecificationLeft =10 minutesRight = 0

OccurenceTime = 15:14:12Registration = ABC123

OccurenceTime = 15:16.23Registration = ABC123

Registration = ABC123Entry = 15:14:12Exit = 15:16:23

OccurrenceTime=15:14:12Registration = ABC123

OccurrenceTime=15:16.23Registration = ABC123


Compose policies

Left buffer specification—This controls how many event instances from the Left input terminal should be retained. It can be specified either as a count of instances or as a time interval.

Right buffer specification—This controls how many event instances from the Right input terminal should be retained. It can be specified either as a count of instances or as a time interval.

Unmatched Left Policy—This states what should happen when an event is evicted from the left buffer if that event hasn’t been matched with anything prior to eviction.

Unmatched Right Policy—This states what should happen when an event is evicted from the right buffer if that event hasn’t been matched with anything prior to eviction.

Match condition—This is the condition used to judge whether an event from the left stream matches one from the right stream. It can be a simple equality test, such as Left/A = Left/B, or a more complex expression involving both events, such as the XPath expression count(Left/A) = count(Right/B) + 7.



A derivation expression is an expression that assigns values to the attributes of the derived event. A derivation expression can refer to values of the input event attributes.


Header derivation

Attribute Name Default output value Settable by derivation step?

eventIdentity New system-generated identity No

event Type Specified by output terminal definition Yes

detectionTime See earlier discussion No

occurrenceTime As detectionTime Yes1

eventAnnotation Not present Yes

eventCertainty Not present Yes

eventSource Identity of the event processing agent Yes

1 See section 11.1.2 for further discussion


FFD – Bid Request Creator EPA

Driver Status Table

Derivation expression:Copy attributes from input;driversList = Driver;

Enrich EPA

Query expression:SELECT Driver FROM DriverStatusWHERE Ranking > = ? AND Location = ?

North East5Mark Smith

North East7Arthur Boyd

………

South West5Joan Winter

LocationRankingDriver

Query parameters:minimumRankingneighborhood

requestId = R429531addresseeLocation = 5 Main StrequiredPickUpTime = 15:30requiredDeliveryTime = 16:30store = Exotic FlowersminimumRanking = 5driversList = Arthur Boyd, Mark Smith

requestId = R429531addresseeLocation = 5 Main StrequiredPickUpTime = 15:30requiredDeliveryTime = 16:30store = Exotic FlowersminimumRanking = 5neighborhood = North East

Multiple results policy:Combine


FFD – Bid Enrichment

Store Reference

Enrich EPA

Query Expression:SELECT Auto FROM StoreRef

WHERE Store = ?

requestId = R429531driver = Arthur BoydcommittedPickUpTime = 15:30store = Exotic Flowers

requestId = R429531driver = Arthur BoydcommittedPickUpTime = 15:30store = Exotic Flowersauto = No

Query Parameter = store

Filter EPAOut

In

Non-Filterable

Auto = Yes

To manual

To auto

Bid enrichment

Bid routing

…

North East

South East

Region

…

No

Yes

Auto

17 East Ave3Blooms R Us

………

33 High St5Exotic Flowers

LocationMinRankStore


FFD - Assignment manager EPA

Derivation expression:Copy attributes of right event

Compose EPA

Match expression:Right/committedPickupTime < Left/requiredPickupTime + 5 minutes

Unmatched left policy: Fail

Buffer Specification:Left = 1 eventRight = 0 events

Bid Request

Delivery BidDelivery Bid

Left

Right

Segmentation context: requestIdEvent Interval Context:

Initiator = Bid RequestExpiration = +2 minutes

Delivery BidDelivery Bid

Output

Failure

No Bidders alert

Failure derivation expression:Copy requestId


Daily Statistics Creator

Aggregate EPA

Fixed interval context: Daily

driver = Arthur Boydday = 2009-02-14assignmentCount = 3

day = 2009-02-14assignmentMean = 5.2assignmentSTDV= 1.2

Derivation expression:day = any(day); assignmentMean=avg(assignmentCount);assignmentSTDV=stdev(assignmentCount);

driver = Arthur Boydday = 2009-02-14assignmentCount = 3


Streambase example


Rulecore example <Rule name="CreateAutomaticAssignments" limit="?" evalMode="once" level="2"> <Description>This is rule CreateAutomaticAssignments</Description> <Initialize> <Assert> <Event> <base:XPath>sim:EventDef[@eventType="BidRequest"]</base:XPath> </Event> <Expression> <Property name=”Store”> <InList name=”AutomaticAssignmentStore”/> </Property> </Expression> </Assert> </Initialize> <Views> <ViewRef name="CreateAutomaticAssignments"> <base:XPath>sim:ViewDef[@name=”CreateAutomaticAssignments”]</base:XPath> </ViewRef> </Views> <Situations> <SituationRef name="CreateAutomaticAssignments"> <base:XPath>sim:SituationDef[@name="CreateAutomaticAssignments"]</base:XPat h> </SituationRef> </Situations> <Actions> <SituationDetected situationName="CreateAutomaticAssignments"> <ActionRef name="CreateAutomaticAssignments" eventVisibility="external"> <base:XPath>sim:ActionDef[@name="CreateAutomaticAssignments"]</base:XPath > </ActionRef> </SituationDetected> </Actions> </Rule>


Apama example


Esper example

insert into BidRequest(requestId, store, location, pickupTime, deliveryTime, storeManual) select d.requestId, d.store, d.location, d.pickupTime, d.deliveryTime, s.manual from DeliveryRequest d unidirectional, GPSLocationW g //,sql:DomainDB['select ranking from Driver where driver = ${g.driver} and ranking > ${d.minimumRanking}'] ,method:Domain.driverRankLookup(g.driver) r ,method:Domain.isStoreManualLookup(d.store) s where Geo.distanceKM(g.location, d.location) < 10 and r.ranking >= d.minimumRanking;


Lecture summary

In this lecture we have looked into:

Filtering

Transformation types

Aggregation derivations

Some code examples

Event Processing Course

Documents

Transcript of Event Processing Course