Using neo4j for enterprise metadata requirements

We help organisations get more value from their Data

ArchitectureLean Data specialists

Service delivery through:

Systems Integration

Onsite Consulting

Onsite / Offsite Managed Services

Data Strategy consulting

Regulatory, compliance &

Financial Crime

Investment Management, Operations & Research

Risk Management

KYC, SCV

What is metadata & why is it useful?

What is metadata?

Metadata is information about the structures that contain the actual data.

Who’s worked on a metadata centric project?

Data Governance

Enterprise Architecture

Master Data Management

Enterprise Data Modelling

Enterprise Data Definition

Enterprise Data Warehousing

Enterprise Data Quality

Enterprise Data Integration

Legal / Regulatory e.g. GDPR, BCBS 239

Enterprise Knowledge Management

Why are we so bad at Enterprise Metadata

Management?

Most organisations attempts at managing metadata have failed. Why?

Past failures

Scope & definitions

Data people don’t play well with other data people

Business case

Approach & tooling

Where do we typically bury store metadata?

Data & IT Governance tools

Enterprise Architecture

Business Process modelling

Data Modelling

Log storage

CMDB

Policy & Standards documents

How do we typically try to integrate & report on metadata?

The ‘so what’ questions of metadata

Tell me which Data Elements are most

critical

Tell me where this value

originated & where it goes

Help me understand & enforce business &

technical rules

Tell me to which level standards & policies are adhered to and

help me

Provide me with rich & interactive visualisations rather than long policies

that sit on shared drives…

Help me understand the context &

meaning of my data

Tell me which people, processes & IT components are impacted

by an IT event

for metadata

There’s lots of exciting (& scary) stuff happening in the world of metadata right now

Forward Engineering / Metadata OLTP

Reverse Engineering / Metadata OLAP

The era of cheap storage & Data Lakes

Why don’t we just retain

EVERYTHING to be on the safe side?

Why don’t we treat & manage metadata like

‘real’ data!?

The ‘so what’ questions of metadata

Tell me which Data Elements are most

critical

Tell me where this value

originated & where it goes

Help me understand & enforce business &

technical rules

Tell me to which level standards & policies are adhered to and

help me

Provide me with rich & interactive visualisations rather than long policies

that sit on shared drives…

Help me understand the context &

meaning of my data

Tell me which people, processes & IT components are impacted

by an IT event

Which downstream databases & processes are affected by this data event / defect?

MATCH (n:DQTest)-[l*]->(C:Column)-[b]-(t:Table)-[y]-(d:Database)-[x*1..3]-(p)

where p:Database OR p:Process

AND n.name = 'Address Check' return p

DQTest Column Table

Database

Process

Neo4j for metadata OLTP & OLAP requirements

Architecture

Forward Engineering / OLTP

Schemaless Graph model offers flexibility as metadata requirements evolve

Suitable for complex business rules & data structures – hierarchies, taxonomies etc.

Suitable for real-time metadata requirements – alerting, schema validation, real-time MDM / ETL etc.

Highly scalable

Reverse Engineering / OLAP

Flexible data model makes defining constraints simple

Cypher – very simple & intuitive

Can apply empirical techniques to traditionally contentious issues:

E.g. Definitions

Community support & online content is great

A Neo based metadata lake

MetadataScientist

Architects, Modellers &

BA’s

Reports & self-service visualisations

Harvesting Metadata

‘Data’ & ‘IT’ teams

Enrich

Analyse & build apps

Analyse

Ingest

DDL

Enhance

Demo – interesting Open Data demo.

Got an idea? Speak to us

Rapid POCs – often in weeks

Connected Data –

July 12th in Mayfair

Questions

http://connected-data.london/