Eagle6 Enterprise Situational Awareness
-
Upload
mongodb -
Category
Technology
-
view
190 -
download
2
description
Transcript of Eagle6 Enterprise Situational Awareness
![Page 1: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/1.jpg)
Confidential & Proprietary
Eagle6 Enterprise Situational Awareness
MongoDC 2014
14 Oct 2014
![Page 2: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/2.jpg)
Confidential & Proprietary
Adam Bell, Director, Product Management◦ 15+ years enterprise software solutions experience◦ 10+ years enterprise architecture◦ Healthcare◦ Technology◦ MongoDB user since 2012
Introduction
![Page 3: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/3.jpg)
Confidential & Proprietary
About Rivera Group Service-Disabled Veteran
Owned Small Business Minority-Owned Business
Primary NAICS Codes:◦ 541511◦ 541512
Established: 2002*
Subject Matter Expertise:
Enterprise Software Development
Business Process Reengineering (BPR)
Proprietary Software: Eagle6 Modeling & Simulation
*Over 30% of Rivera Group Employees Are Veterans
![Page 4: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/4.jpg)
Confidential & Proprietary
Eagle6 is a modeling tool that automatically collects system data (application code, database schemas, log files, etc.) and provides an ability to continuously monitor for unwanted system states (bugs) that may result in system degradation and/or system outages.
Eagle6 – Enterprise Situational Awareness
![Page 5: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/5.jpg)
Confidential & Proprietary
Large sets of Multi-Dimensional Data Heavy Read and Write Heavy Audit Requirements Fast Near Real-Time Analytics Analytics are User Driven
Our Use Case
![Page 6: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/6.jpg)
Confidential & Proprietary
About Our Documents◦ 1000(s) of leafs on documents◦ Document Sizes (Bson):
2300 bytes (2.3Kb) - 729699 bytes (729Kb)◦ Need to quickly add new data structures
Multi-Dimensional Data
![Page 7: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/7.jpg)
Confidential & Proprietary
Sample Document (Slice)"network" : {
"host" : [{
"host" : {"network" : null,"ips" : [
"6PNfL9bV7BJO"],"names" : [ ],"mx" : [ ],"txt" : [ ],"srv" : [ ],"ns" : [ ]
}},{
"host" : {"network" : null,"ips" : [
"yqb6q7er3DvWf"],"names" : [
"tQypbmzVrEZHtWG1n"],"mx" : [ ],"txt" : [ ],"srv" : [ ],"ns" : [ ]
}}
],}
![Page 8: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/8.jpg)
Confidential & Proprietary
Do not drop transactions Capturing large volumes of real-time data
(web access logs, database transactions, etc)
Read & Write Heavy
![Page 9: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/9.jpg)
Confidential & Proprietary
Indexing has been a challenge MongoDB only supports 64 indexes To many indexes defeats the purpose of
indexes Scenarios exist where we will not know what
fields the user needs until they request them
Indexing - Challenges
![Page 10: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/10.jpg)
Confidential & Proprietary
Key Value Approach TODO: Need an example of KV indexes
Indexing - Solution
![Page 11: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/11.jpg)
Confidential & Proprietary
Deployment Architecture
![Page 12: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/12.jpg)
Confidential & Proprietary
Shard Key is an object hash Goal is equal distribution of data across
shards
◦ Example {
hash: '00003820efcff8b669b055606813bcd360ace3f43fbf9c129845b3028992eacabcaef8cd13796dc7a96b7a5f38b0efaceaadecfd537c72eaec8a8f9c10a00a1e’,
offset: -1}
Shard Key
![Page 13: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/13.jpg)
Confidential & Proprietary
Needed a flexible way to Aggregate Needed a way to represent Aggregations to
end users with out writing functions Needed a way to cache frequently run
analytics
Real-time User Driven Analytics
![Page 14: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/14.jpg)
Confidential & Proprietary
Provides a Rich set of operations for aggregating data
We have been using the Aggregation Framework since MongoDB 2.2
Aggregation Pipelines have enabled us to do smart caching
JSON versions of the pipeline allow end users flexibility without writing Map/Reduce code
Analytics - Aggregation Framework
![Page 15: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/15.jpg)
Confidential & Proprietary
Example Pipeline { "$match": { ”servers": { "$exists": true } } }, { "$group": { "_id": ”$serversr", "clients": { "$addToSet": ”$client.names } } }, { "$project": { "title": { "$join": "$_id.names" }, "_id": { "id": { "$join": "$_id.ips" } }, "clients": 1, "server": "$_id" } }, { "$group": { "_id": "$_id", "server": { "$addToSet": "$server" }, "clients": { "$addToSet": "$clients" }, "title": { "$addToSet": "$title" } } },
{ "$unwind": "$clients" }, { "$unwind": "$clients" }, { "$project": { "clients": { "$join": "$clients" }, "server": "$server", "title": "$title" } }, { "$group": { "_id": "$_id", "server": { "$addToSet": "$server" }, "clients": { "$addToSet": "$clients" }, "title": { "$addToSet": "$title" } } }, { "$unwind": "$title" }, { "$unwind": "$title" }
![Page 16: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/16.jpg)
Confidential & Proprietary
Ported Aggregation framework to Javascript ◦ Custom built extensions
Accumulators $stdDev
Expressions $regex $slice
Document Sources $projectPrevious $split
Pipelines for Everyone
![Page 17: Eagle6 Enterprise Situational Awareness](https://reader033.fdocuments.net/reader033/viewer/2022061220/5484feffb479590a0d8b4d32/html5/thumbnails/17.jpg)
Confidential & Proprietary
Questions