Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data…...

35
Tasty Topics! Novel approaches using Topic Filtering

Transcript of Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data…...

Page 1: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

TastyTopics!Novelapproachesusing

TopicFiltering

Page 2: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

It’sallaboutme

[email protected]

https://www.linkedin.com/in/tomfairbairn

Page 3: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Pub/Subrevision• Distributed• Decoupled• Fanin/Fanout• Persistence• RegisterinterestinTopic

Imagecredit:pubnub.com

Page 4: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Topics

Topic≠ 𝑇𝑎𝑔!

food/apple/slices

*/*/slices=>

food/apple/slicesfood/ham/slices

food_apple_slices

List{food_apple_slices,food_ham_slices}Stringsearch?

Page 5: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

WhoCares?• Simpler• Consistent• Reducesunnecessarydatacopies– E.g.InIoT reducesunnecessarysensorreads

Page 6: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

UseCase1

MigratingYourDataFormat

Page 7: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Case1:MigratingYourDataFormat{“Person” : “Tom”,

“Team” : “Magicians”,

“Mobile” : “07746 244422”,

“EmployeeId” : 6

}

{

“Employee” :

[“Id” : “0000732006”,

“Name” : “Tom Fairbairn”,

“PhoneNum” : “+44(0)7746244422”,

“DirectReports”: [],

”ReportsTo”: “Ben Taieb”

]

}

Page 8: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

CI/CD

“Employee” : [“Id” : “0000732006”, “Name” : “Tom

Fairbairn”,

Tag

Build

Test

Release

Page 9: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

DataFormat– read/writeGson gson = new Gson();empolyeeData = gson.fromJson(data, employee.class);

public class employee {private String Person;private String Team;private String Mobile;public int EmployeeId;

…}

public class employee {private class employeeData {private String Id;private String Name;private String PhoneNum;private String[] DirectReports;private String ReportsTo;

}…}

Tag:v1.0 Tag:v2.0

Page 10: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

DataFormattopicprivate String versionedTopic =

“london/employee/json/$GIT_TAG_NAME/[…]”;

session.subscribe(versionedTopic);

producer.send(message, versionedTopic);

Page 11: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

UseCase2

Monitoring

Page 12: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Monitoring

Page 13: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Aquickdiversion

https://ip/endpoint

Websockets

Page 14: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Aquickdiversion-again

V1

V3

V5

Page 15: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Pub/SubMonitoringoverPub/Sub!

Pub/Sub

Anyapp Monitoringapp

Monitoring/ManagementAPI

MQTT ReST

Page 16: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:
Page 17: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

UseCase3

Replay

Page 18: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

Dealingwithsharedstate– Ployglot persistence?– Replay“stateoftheworld”frommessagestream

Case3:Replay/Event-streaming

Page 19: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Replay– queuesthatcansubscribe

Queue

app/control

app/config

data/app/…

• QueueBrowser • TTL • LVQ

Page 20: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

UseCase4

Authorisation

Page 21: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Case4:Authorisation

Pub/Sub

Application Subscriptionmanager

1. Request:log-in2. Servicecalculatessubscriptions3. Serviceappliessubscriptions

forapp4. ReplywithOK5. Appreceivesmatchingdata

OnBehalfOf

Page 22: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

OnBehalfOf• Clienthasnoawarenessoftopics/services– Nochancetoguessotherservices– Nowork/exposureatclient

• Fullypluggablearchitecture

Page 23: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Authorisation

Pub/Sub

Anyapp Subscriptionapp

1. Requestaccountbalance2. Servicecalculatessubscriptions3. Servicesubscribesforapp4. Appreceivesmatchingdata

Page 24: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

UseCase5

Findthenearest…Geo-locationusingtopics

Page 25: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Case2:FindTheNearest…InRealTime

Source:ICOMP2016,A.L.Lee,RangedFilteringofStreamingNumericData…usingTopic-BasedPub/SubMessaging

Page 26: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Geo-filteringtopicPublishtotopicwithlocation:<app>/<type>/<lat>/<long>/<vehicle>/<id>

geo/sim/51.520150/-00.097330/CAR/00021

WhereisCAR00021?subscribe(“geo/sim/*/*/CAR/00021”);

Page 27: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Geo-filteringlocationsubscribe(“geo/sim/51.52015*/-00.09733*/>”);

Match:lat 51.520150to51.520159long-000.097330to-000.097339

Page 28: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Geo-filteringlocationsubscribe(“geo/sim/51.52*/-00.09*/>”);

Match:lat 51.520to51.529999long-0.090to-0.099999

Page 29: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Geo-filteringlocationsubscribe(“geo/sim/51.52*/-00.09*/>”,

“geo/sim/51.516*/-00.092*/>”,

“geo/sim/51.516*/-00.093*/>”,

“geo/sim/51.516*/-00.094*/>”,

“geo/sim/51.516*/-00.096*/>”,

“geo/sim/51.517*/-00.092*/>”,

“geo/sim/51.517*/-00.093*/>”,

“geo/sim/51.517*/-00.094*/>”,

“geo/sim/51.517*/-00.096*/>”,

// repeat for 51.518 and .519

);

Page 30: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Geo-filteringlocation• Createanypolygon

– Accuracyatmetrelevel

– Circles,arcs…• Subscriptions

generatedonce• Matchesthen

streaminwithnoextracomputation

Page 31: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Geo-filteringlocationalgorithm• Dividespaceintorectanglesalignedtosubscriptions• Throwawayrectangleswithnomatch

Page 32: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Geo-filteringlocationalgorithm• Repeat:divideremainingrectanglesby10• Throwawayrectangleswithnomatch

Page 33: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Geo-filteringlocationalgorithmdeployment

• Library?

Pub/Sub

Geo-filteringapp Subscriptionapp

1. Requestsubscriptionsforshape2. Servicecalculatessubscriptions3. Servicesubscribesforapp4. Appreceivesmatchingdata

Page 34: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Lastcodesnippetif

Topic

then

Page 35: Tasty Topics!€¦ · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish to topic with location:

Monitoring

ComeandseeusatourBooth!


Turbo Chameleon 64 User Manual - icomp

Turbo Chameleon 64 User Manual - icomp

1 Information Filtering Rong Jin. 2 Outline Brief introduction to information filtering Collaborative filtering Adaptive filtering.

1 Information Filtering Rong Jin. 2 Outline Brief introduction to information filtering Collaborative filtering Adaptive filtering.

news filtering topic detection and tracking · 2006-11-15 · • Topic Detection and Tracking – focused on detecting and tracking events in news • novelty detection: does this

news filtering topic detection and tracking · 2006-11-15 · • Topic Detection and Tracking – focused on detecting and tracking events in news • novelty detection: does this

news filtering topic detection and tracking · • TDT and TREC. • Usually the starting point is a few example documents on each topic. • TDT topics are events in news. • TREC

news filtering topic detection and tracking · • TDT and TREC. • Usually the starting point is a few example documents on each topic. • TDT topics are events in news. • TREC

Topic 2 Firewalls2.ist.psu.edu/ist451/451-Topic 2-Firewalls.doc · Web viewFor example, packet filtering works at network layer. It is faster however it does less inspection. Application

Topic 2 Firewalls2.ist.psu.edu/ist451/451-Topic 2-Firewalls.doc · Web viewFor example, packet filtering works at network layer. It is faster however it does less inspection. Application

Tasty Topics! - QCon · 2017. 3. 15. · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish

Tasty Topics! - QCon · 2017. 3. 15. · Source: ICOMP 2016, A.L. Lee, Ranged Filtering of Streaming Numeric Data… using Topic-Based Pub/Sub Messaging Geo-filtering topic Publish

Nurhayati icomp surabaya

Nurhayati icomp surabaya

Institute for Computational Mechanics in Propulsion (ICOMP) · PDF fileinstitute for computational mechanics in propulsion (icomp) first year summary icomp first annual report september

Institute for Computational Mechanics in Propulsion (ICOMP) · PDF fileinstitute for computational mechanics in propulsion (icomp) first year summary icomp first annual report september

Real-Time Top-R Topic Detection on Twitter with Topic Hijack Filtering

Real-Time Top-R Topic Detection on Twitter with Topic Hijack Filtering

Median Filtering and Median Filtering and Morphological Filtering

Median Filtering and Median Filtering and Morphological Filtering

Research overview Murat Demirbas University at Buffalo, SUNY CSE Dept. iComp.

Research overview Murat Demirbas University at Buffalo, SUNY CSE Dept. iComp.

Navigating a Change in Service Providers Prash Shukla Global Shares Brock Benson iComp LLC.

Navigating a Change in Service Providers Prash Shukla Global Shares Brock Benson iComp LLC.

Institute for Computational Mechanics in Propulsion (ICOMP) · The ICOMP research staff for 1996 is shown in Table I. A total of thirty-three researchers were in residence at ICOMP

Institute for Computational Mechanics in Propulsion (ICOMP) · The ICOMP research staff for 1996 is shown in Table I. A total of thirty-three researchers were in residence at ICOMP

Notes for course EE1.1 Circuit Analysis 2004-05 TOPIC 7 ...cas.ee.ic.ac.uk/people/dhaigh/EE1.1 Files/Notes/7 Frequency... · TOPIC 7 – FREQUENCY RESPONSE AND FILTERING ... circuits

Notes for course EE1.1 Circuit Analysis 2004-05 TOPIC 7 ...cas.ee.ic.ac.uk/people/dhaigh/EE1.1 Files/Notes/7 Frequency... · TOPIC 7 – FREQUENCY RESPONSE AND FILTERING ... circuits

en.SafeWatch Filtering - Watch List Filtering

en.SafeWatch Filtering - Watch List Filtering

Proposal Jasa IT Support iComp

Proposal Jasa IT Support iComp

Recommendations via Collaborative Filtering. Recommendations Relevant for movies, restaurants, hotels…. Recommendation Systems is a very hot topic in.

Recommendations via Collaborative Filtering. Recommendations Relevant for movies, restaurants, hotels…. Recommendation Systems is a very hot topic in.

1 Topic 2 – Lesson 4 Packet Filtering Part I. 2 Basic Questions What is packet filtering? What is packet filtering? What elements are inside an IP header?

1 Topic 2 – Lesson 4 Packet Filtering Part I. 2 Basic Questions What is packet filtering? What is packet filtering? What elements are inside an IP header?

Pattern-based Topic Modelling and its Application for ...eprints.qut.edu.au/83982/1/Yang_Gao_Thesis.pdf · Pattern-based Topic Modelling and its Application for Information Filtering

Pattern-based Topic Modelling and its Application for ...eprints.qut.edu.au/83982/1/Yang_Gao_Thesis.pdf · Pattern-based Topic Modelling and its Application for Information Filtering

Improving Collaborative Filtering Based Recommenders Using Topic Modelling

Improving Collaborative Filtering Based Recommenders Using Topic Modelling