MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

37
MongoDB at Medtronic Jeff Lemmerman Matt Chimento 1 Medtronic Energy and Component Center aimer: We are not currently managing any Device Patient Data in MongoDB

description

 

Transcript of MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

Page 1: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

MongoDB at Medtronic

Jeff LemmermanMatt Chimento

1

Medtronic Energy and Component Center

*Disclaimer: We are not currently managing any Device Patient Data in MongoDB

Page 2: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

| MDT Confidential2

Agenda

• Intro• Challenge• Approach• Opportunities• Takeaways

Page 3: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

3

Medtronic Energy and Component Center• MECC est. 1976

• Manufacture components for devices

• Census – 1200 Employees

• Plant Size – 190,000 Square Feet40,000 Manufacturing15,000 R&D Labs38,000 Office97,000 Common, Support, Warehouse

Page 4: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

| MDT Confidential4

Every 3 Seconds…

Page 5: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

Where does our data come from?

Page 6: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

6

Where stored?

Big Data: Volume, Variety, Velocity

Page 7: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

| MDT Confidential7

Without MongoDB

Page 8: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

| MDT Confidential8

Data Management Vision

Page 9: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

| MDT Confidential9

Reporting and Analytics VarietyOperations Dashboards SPC Charts

Component Summary Time Series

Page 10: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

| MDT Confidential10

Component Dataset In Excel

Page 11: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

11

How Is Data Retrieved?

Page 12: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

12

Loading Data Into Central Repository

Page 13: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

13

Component Dataset in RDBMS

Page 14: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

14

Ideal Future State

Medtronic Confidential

Page 15: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

15

Loading Components Into MongoDB

Page 16: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

16

Ideal Future State

Medtronic Confidential

Page 17: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

| MDT Confidential17

Page 18: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

| MDT Confidential18

Simple, Fast Queries – Complete History

Page 19: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

19

Medtronic Confidential

Component Time Series Data

Page 20: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

20

Collect and Store Raw Time Series Data

Medtronic Confidential

Page 21: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

Our experience with MongoDB

• Consulting/Training has been excellent• Support agreement has been under-utilized

– Emails for security updates etc. are prompt– Release cycle is frequent

• MongoDB Monitoring Service– Potential concerns storing db stats externally– MMS can now be hosted locally

• MongoDB Certification now available– Udacity course, “Data Wrangling with MongoDB”

Page 22: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

Gaps

• Enterprise acceptance of “new” approach• Integration with off-the-shelf reporting and

analytics tools• User interface for managing MongoDB cluster• Developer familiarity with JSON and MongoDB• 21 CFR Part 11 Compliance – Audit Tracing

Page 23: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

Thank You

Questions?

23

Page 24: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

| MDT Confidential24

Page 25: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

Key Features• Data stored as documents (JSON-like BSON)

– Flexible-schema– In schema design, think about optimizing for read vs. storage

• Full CRUD support (Create, Read, Update, Delete)– Atomic in-place updates– Ad-hoc queries: Equality, RegEx, Ranges, Geospatial

• Secondary indexes• Replication – redundancy, failover• Sharding – partitioning for read/write scalability• Terminology

– Collection = Table– Index = Index– Document = Row– Column = Field– Joining = Embedding & Linking

Page 26: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

26

Creating Components

• .insert() will always try to create new document• .save() if _id already exists will update • If document doesn’t have _id field it is added

Page 27: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

27

Reading Components

Page 28: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

28

Reading ComponentsReturns Null

Page 29: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

29

Updating Components

• $set keyword used for partial updates• Without $set keyword entire document is replaced• {multi : true} to update multiple documents

Page 30: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

30

Deleting ComponentsWorks like .find()

Drops collection

Drops database

Page 31: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

31

Creating Components – Add()

Page 32: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

32

Reading Components

Page 33: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

33

Updating Components – Save()

Save sends entire document back to server

Page 34: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

34

Updating Components – Update()

Update only sends changes

Page 35: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

35

Deleting Components

Needed to add reference to Repo class

Page 36: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

36

Automapping

Page 37: MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

37