Unleashing the Power of Apache Atlas with Apache Ranger

33
Unleashing the power of Apache Atlas with Apache Ranger Virtual Data Connector Project NIGEL JONES [email protected] DATAWORKS, MUNICH, APRIL 2017 Apache®, Apache Atlas, Apache Ranger & other Apache project names referenced are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

Transcript of Unleashing the Power of Apache Atlas with Apache Ranger

Page 1: Unleashing the Power of Apache Atlas with Apache Ranger

Unl

eash

ing

the

pow

er o

f Apa

che

Atla

s with

Apa

che

Rang

er

Virtu

al D

ata

Conn

ecto

r Pro

ject

NIG

EL JO

NES

JON

ESN

@U

K.IB

M.C

OM

DAT

AWO

RKS,

MU

NIC

H, A

PRIL

201

7A

pach

e®, A

pach

e A

tlas,

Apa

che

Rang

er &

oth

er A

pach

e pr

ojec

t nam

es re

fere

nced

are

eith

er re

giste

red

trade

mar

ks o

r tra

dem

arks

of t

he A

pach

eSo

ftwar

e Fo

unda

tion

in th

e U

nite

d St

ates

and

/or o

ther

cou

ntrie

s. N

o en

dors

emen

t by

The

Apa

che

Softw

are

Foun

datio

n is

impl

ied

by th

e us

e of

thes

e m

arks

.

Page 2: Unleashing the Power of Apache Atlas with Apache Ranger

Abo

ut M

e –

Nig

el Jo

nes

•ht

tps:/

/ww

w.lin

kedi

n.co

m/in

/nig

elljo

nes/

•jo

nesn

@uk

.ibm

.com

(Any

one

still

use

emai

l?)

•@

plan

etf1

– n

oisy

, f1,

ele

ctric

veh

icle

s, fo

od &

drin

k …

. A sp

lit o

f wor

k/lif

eac

coun

ts di

dn’t

wor

k fo

r me!

•A

nd o

f cou

rse

the

Apa

che

Atla

s & R

ange

r mai

ling

lists

& JI

RA!

•Sc

ienc

e fa

n at

scho

ol u

ni. I

t was

clo

ud c

ham

bers

bac

k th

en…

now

just

the

clou

dJ

•IB

M H

ursle

y, U

K si

nce

1990

•La

st 3

year

s foc

us o

n D

ata

Lake

, Inf

orm

atio

n G

over

nanc

e, O

pen

Met

adat

a

Page 3: Unleashing the Power of Apache Atlas with Apache Ranger

The

Prob

lem

…..

WH

Y A

RE

WE

HER

E…..

Page 4: Unleashing the Power of Apache Atlas with Apache Ranger

Dat

a?

•W

hat d

ata

do I

have

?•

Wha

t doe

s it m

ean?

•W

here

is it

?•

Who

has

acc

ess t

o it?

•W

ho o

wns

it?

•W

hat q

ualit

y is

it?

•H

ow d

oes i

t rel

ate

to o

ther

dat

a?•

How

to I

cont

rol,

audi

t & u

nder

stan

d ac

cess

?

Page 5: Unleashing the Power of Apache Atlas with Apache Ranger

Reg

ulat

ory

need

s

•A

dher

e to

regu

latio

ns li

ke B

CB

S-23

9 an

d G

DPR

•N

eed

to k

now

mea

ning

, val

ue o

f the

dat

a

•D

emon

stra

te p

roce

sses

in p

lace

to g

over

n ac

cess

•A

udit

•Si

gnifi

cant

fine

s if r

ules

bre

ache

d

•W

hils

t ens

urin

g ea

sy, r

eady

acc

ess t

o ap

prop

riate

dat

a fo

r dat

a pr

ofes

sion

als t

o su

ppor

tan

agi

le b

usin

ess

Page 6: Unleashing the Power of Apache Atlas with Apache Ranger

So w

hat d

o w

e ne

ed to

add

ress

this

?

Page 7: Unleashing the Power of Apache Atlas with Apache Ranger

Met

adat

a..

•M

etad

ata

enab

les d

ata

to b

e us

ed o

utsid

e of

the

appl

icat

ion

that

cre

ated

it.

•A

naly

tics a

nd d

ecisi

on m

akin

g

•N

ew b

usin

ess a

pplic

atio

ns

•Re

porti

ng a

nd c

ompl

ianc

e

•M

etad

ata

desc

ribes

the

form

at a

nd c

onte

nt o

f dat

a al

low

ing

peop

le to

judg

e w

hich

data

set t

o us

e fo

r a n

ew p

roje

ct•

Stru

ctur

e

•M

eani

ng

•O

rigin

•Va

lid v

alue

s and

qua

lity

•U

sage

and

ow

ners

hip

•Re

gula

tions

and

cla

ssifi

catio

ns th

at a

pply

Page 8: Unleashing the Power of Apache Atlas with Apache Ranger

Whi

ch c

an su

ppor

t…

•A

n en

terp

rise

data

cat

alog

ue th

at li

sts a

ll da

ta in

clud

ing

whe

re it

is, w

hat i

t is,

who

owns

it, i

t’s m

eani

ng, q

ualit

y, w

here

it c

ame

from

, an

d ca

n fu

lly d

escr

ibe

it’s

busi

ness

con

text

& h

ow th

e da

ta s

houl

d be

gov

erne

d….

•Su

bjec

t Mat

ter e

xper

ts se

arch

ing,

col

labo

ratin

g, fe

edin

g ba

ck a

bout

thei

r dat

ane

eds a

nd u

se•

Aut

omat

ed g

over

nanc

e ac

tions

to p

rote

ct a

nd m

anag

e in

clud

ing

audi

ting,

mon

itorin

g, q

ualit

y co

ntro

l, rig

hts m

anag

emen

t

Page 9: Unleashing the Power of Apache Atlas with Apache Ranger

But e

asily

•O

pen

fram

ewor

ks &

API

s•

Aut

omat

ic c

olle

ctio

n &

disc

over

y of

met

adat

a in

a d

ynam

ic h

eter

ogen

eous

envi

ronm

ent

•U

sing

pred

efine

d sta

ndar

ds fo

r glo

ssar

ies,

sche

mas

, rul

es, r

egul

atio

ns to

redu

ceco

st•

Chea

p to

inte

grat

e ne

w to

ols

•N

o pr

oprie

tary

lock

-in &

ass

umpt

ions

that

all

tool

s are

from

one

suite

or v

endo

r•

Avoi

ding

silo

s•

Dist

ribut

ed a

nd O

pen

Page 10: Unleashing the Power of Apache Atlas with Apache Ranger

The

visi

on

Ope

nand

Unifie

dMetadata

Page 11: Unleashing the Power of Apache Atlas with Apache Ranger

Virt

ualiz

atio

n D

ata

Con

nect

or p

roje

ct

Page 12: Unleashing the Power of Apache Atlas with Apache Ranger

Dat

a vi

rtual

izat

ion

proj

ect

•Co

llabo

ratio

n –

IBM

, sev

eral

ban

ks &

ope

n co

mm

unity

•A

Dat

a La

ke e

nviro

nmen

t•

Not

just

Had

oop,

but

oth

er so

urce

s too

•Bu

sines

s Ter

ms,

Clas

sifica

tions

, Met

adat

a ric

h

•O

ffer v

irtua

lized

vie

ws.

Expo

se re

latio

nal d

ata

with

bus

ines

s ter

ms

•M

anag

e A

cces

s to

reso

urce

s – p

erm

it, d

eny,

log,

filte

r/mas

k …

. TH

ROU

GH

MET

AD

ATA

•O

pen,

plu

ggab

le

•W

orki

ng th

roug

h us

e ca

ses,

desig

n, in

itial

MV

P (th

is ye

ar)

•Cr

itiqu

e, fe

edba

ck is

wel

com

ed. W

e’re

look

ing

for g

uida

nce

and

supp

ort f

rom

the

Atla

s & R

ange

r com

mun

ities

as w

ell a

s con

tribu

te o

ur id

eas

•Pr

opos

ed c

hang

es a

ll go

thro

ugh

mai

ling

list a

nd JI

RA fo

r fee

dbac

k

Page 13: Unleashing the Power of Apache Atlas with Apache Ranger

Apa

che

Atla

s

•“A

tlas i

s a sc

alab

le a

nd e

xten

sible

set o

f cor

e fo

unda

tiona

l gov

erna

nce

serv

ices

–en

ablin

g en

terp

rises

to e

ffect

ivel

y an

d ef

ficie

ntly

mee

t the

ir co

mpl

ianc

ere

quire

men

ts w

ithin

Had

oop

and

allo

ws i

nteg

ratio

n w

ith th

e w

hole

ent

erpr

ise d

ata

ecos

yste

m.”

…. h

ttp://

ww

w.ap

ache

.org

•O

pen

Com

mun

ity --

Apa

che

Incu

bato

r sin

ce M

ay 2

015

•Ty

pe a

gnos

tic m

etad

ata

store

•RE

ST A

PI &

UI

•Su

ppor

ts m

any

Had

oop

com

pone

nts i

nclu

ding

HBa

se, H

ive,

Sqo

op, S

torm

&ot

hers

Page 14: Unleashing the Power of Apache Atlas with Apache Ranger

Apa

che

Ran

ger

•C

entra

lized

secu

rity

adm

inis

tratio

n to

man

age

all s

ecur

ity re

late

d ta

sks i

n a

cent

ral

UI o

r usi

ng R

EST

API

s.•

Fine

gra

ined

aut

horiz

atio

n to

do

a sp

ecifi

c ac

tion

and/

or o

pera

tion

with

Had

oop

com

pone

nt/to

ol a

nd m

anag

ed th

roug

h a

cent

ral a

dmin

istra

tion

tool

•St

anda

rdiz

e au

thor

izat

ion

met

hod

acro

ss a

ll H

adoo

p co

mpo

nent

s.•

Enha

nced

supp

ort f

or d

iffer

ent a

utho

rizat

ion

met

hods

- R

ole

base

d ac

cess

con

trol,

attri

bute

bas

ed a

cces

s con

trol e

tc.

•C

entra

lize

audi

ting

of u

ser a

cces

s and

adm

inis

trativ

e ac

tions

(sec

urity

rela

ted)

with

in a

ll th

e co

mpo

nent

s of H

adoo

p.

… fr

om h

ttp://

rang

er.a

pach

e.or

g

Page 15: Unleashing the Power of Apache Atlas with Apache Ranger

Proj

ect I

nter

actio

ns

Sear

ch/R

epor

t

Gai

anD

B

•Se

arch

for l

ist o

f ass

ets b

y m

etad

ata

•Se

arch

for d

ata

•Re

porti

ng to

ol o

btai

ns d

ata

to d

raw

repo

rt

Und

erly

ing

data

, sql

, hiv

e,H

DFS

, Ora

cle,

Net

ezza

etc

Man

ages

logi

cal v

iew

s

Dep

loys

rule

s, pu

shes

clas

sifica

tions

, sou

rce

for

user

role

s (no

t use

rs)

+ran

ger p

lugi

n to

per

mit/

deny

, mas

k et

c

Pulls

rule

s. cl

assifi

catio

ns

RDBM

SH

adoo

p

Apa

che

Atla

s

Apa

che

Rang

erA

pach

e So

lr

Page 16: Unleashing the Power of Apache Atlas with Apache Ranger

Why

Atla

s and

Ran

ger?

•O

pen

Sour

ce e

ssen

tial t

o fo

rmin

g an

act

ive

ecos

yste

m•

Visio

n, a

ctiv

e co

mm

unity

& e

volv

ing

– ab

ility

to c

ontri

bute

& w

ork

with

oth

ers t

opr

ovid

e th

e be

st so

lutio

n•

Alre

ady

have

goo

d co

re c

apab

ilitie

s•

Atla

s typ

e sy

stem

is v

ery

flexi

ble

•Ra

nger

offe

rs a

rang

e of

pol

icy

type

s and

pro

vide

s a p

lugg

able

fram

ewor

k

•A

lread

y cr

oss p

roje

ct in

tegr

atio

n•

Use

of t

ag b

ased

pol

icie

in R

ange

r sou

rced

from

Atla

s

•Ca

n be

use

d in

depe

nden

tly o

f ful

l Had

oop

stack

Page 17: Unleashing the Power of Apache Atlas with Apache Ranger

Refi

ned

virtu

al c

onne

ctor

scop

e sc

ope

Gai

anD

B

Ranger

Plugin

Titan

(Graph

DB,

Metadata

Repo

sitory)

Ranger

Confi

g

Ran

ger S

erve

r

Atla

sPo

ll Po

licie

s

OMAS

OMRS

IGC

Pre

Post

CreateView

Metadata

Extractp

hysic

almetadata

Manage

Logical

Tables

Virt

ualiz

er

Ret

rieve

met

a da

ta

Ret

rieve

met

a da

taR

etrie

ve m

eta

data

Push

met

a da

ta

Oracle

Netezza

Hive

Tables

Push

and

que

ry m

eta

data

Dat

a La

ke R

epos

itorie

s

Meta

Data

Dat

a La

ke V

irtua

lizat

ion

tag-sync

rule-sync

Confi

g(egPolicies,

AuditloglocaMo

n)

LDAP

AuditLog

Mappe

r

Se

arch

for d

ata/

repo

rting

Push

and

que

rym

etad

ata

Meta

Data

Nav

igat

or

Meta

Data

Dat

amee

r

Page 18: Unleashing the Power of Apache Atlas with Apache Ranger

Gai

anD

B &

Virt

ualiz

er

•G

aian

DB

•O

pen

Sour

ce

•Fe

dera

ted,

self

lear

ning

, dyn

amic

con

figur

atio

n

•Ba

sed

on A

pach

e D

erby

•A

lread

y ha

d “p

olic

y” su

ppor

t – w

e’re

plu

ggin

g in

Ran

ger f

orth

is pr

ojec

t

•Vi

rtual

izer

•Li

stens

to e

vent

not

ifica

tions

on

asse

ts et

c

•Cr

eate

s vie

w d

efini

tions

in G

aian

DB,

and

new

Atla

s API

s to

store

met

adat

a. C

ould

use

diff

eren

t virt

ual e

ngin

e..

•D

esig

ned

to b

e op

en to

oth

er v

irtua

lizat

ion

tech

nolo

gies

.

LT1

LT2

DS2

DS1

DS3

Polic

yPl

ugin

(rang

er)

Virtu

aliz

erA

tlas

Gai

anD

B su

ppor

ts fe

dera

tion

– no

t use

d fo

r MV

P

Page 19: Unleashing the Power of Apache Atlas with Apache Ranger

Atla

s – g

loss

ary

enha

ncem

ents

•G

et A

tlas c

lose

r to

parit

y w

ith c

omm

erci

al o

fferin

gs•

Bus

ines

s Ter

ms –

cat

egor

ies,

cate

gory

hie

rarc

hies

•H

as-a

, is-

a, ty

pe-o

f, sy

nony

m, a

nton

ym, a

rbitr

ary

rela

tions

hips

•A

sset

s map

ped

to B

usin

ess T

erm

s•

Cla

ssifi

catio

ns•

Hie

rarc

hy

•N

avig

able

map

ping

s to

reta

in a

bilit

y to

flat

ten

tags

to ra

nger

•In

stea

d of

hiv

e co

lum

n EM

P_SA

LARY

-> S

PI, n

ow c

an b

e EM

P_SA

LARY

-> S

ALA

RY ->

SPI …

•U

sed

to d

rive

gove

rnan

ce

•AT

LAS-

1410

Page 20: Unleashing the Power of Apache Atlas with Apache Ranger

Atla

s – o

ther

enh

ance

men

ts

•Co

nsum

er C

entri

c A

PIs

•O

pen

Met

adat

a A

cces

s Ser

vice

s (O

MA

S)

•RE

ST &

mor

e K

afka

not

ifica

tions

•A

sset

, Cat

alog

, Con

nect

or, G

loss

ary,

Gov

erna

nce

Act

ion,

Gov

erna

nce

Defi

nitio

ns,

Info

rmat

ion

View

, Rol

es a

nd A

cces

s

•Re

posit

ory

leve

l API

s•

Ope

n M

etad

ata

Repo

sitor

y Se

rvic

es (O

MRS

)

•RE

ST &

mor

e K

afka

not

ifica

tions

•Pl

ugga

bilit

y th

roug

h an

Ope

n Co

nnec

tor F

ram

ewor

k to

oth

er m

etad

ata

repo

sitor

ies –

distr

ibut

ed a

nd O

pen

•St

anda

rd d

ata

mod

el/c

ore

•En

hanc

emen

t to

core

mod

el –

ver

sioni

ng, e

xter

nal l

inka

ge e

tc

•M

ore

stand

ard

type

s ie

for a

ll re

latio

nal d

atab

ases

to e

ase

shar

ing

Page 21: Unleashing the Power of Apache Atlas with Apache Ranger

Ran

ger a

reas

bei

ng lo

oked

at

•B

uild

ing

a pl

ugin

for G

aian

DB

•A

cces

s con

trol,

sim

ple

mas

king

. Mor

e la

ter

•U

ser s

ynch

roni

zatio

n (la

rge

#use

rs, r

ole

of A

tlas)

•C

hang

es to

tag

sync

pro

cess

for N

ew g

loss

ary

prop

osal

•A

s mor

e m

etad

ata

goes

into

Atla

s, it

beco

mes

sour

ce fo

r gen

erat

ion

of so

me

kind

sof

pol

icie

s. W

here

is th

e m

aste

r?•

Gen

erat

ing

rang

er ru

les f

rom

gov

erna

nce

defin

ition

s

•H

ow a

bout

con

trol o

f acc

ess t

o A

tlas i

tsel

f?

•A

side

: Int

erfa

ces u

sed

by e

nfor

cem

ent e

ngin

es (s

uch

as to

get

cla

ssifi

catio

n da

ta)

need

to b

e ef

ficie

nt –

thes

e sh

ould

wor

k fo

r pro

ject

s lik

e A

pach

e Se

ntry

as w

ell a

sA

tlas

Page 22: Unleashing the Power of Apache Atlas with Apache Ranger

Beyo

nd th

e M

VP

•O

pen

Disc

over

y Fr

amew

ork

•Co

nsid

er o

ther

secu

rity

enfo

rcem

ent e

ngin

es –

such

as A

pach

e Se

ntry

& d

rivin

gm

ore

capa

bilit

y ar

ound

rule

s & g

over

nanc

e ac

tions

from

Atla

s met

adat

a•

Wor

k on

stan

dard

mod

els t

o su

ppor

t diff

eren

t dom

ains

•Li

neag

e•

From

hig

h le

vel d

esig

n lin

eage

thro

ugh

to o

pera

tiona

l det

ail.

Logs

vs g

raph

….

•A

PI m

etad

ata

•In

frastr

uctu

re –

Janu

sGra

ph…

•A

bstra

ctio

n ad

ded

by IB

M in

last

few

mon

ths f

or ti

tan

1

Page 23: Unleashing the Power of Apache Atlas with Apache Ranger

The

visi

on•

An

ente

rpris

e da

ta c

atal

og th

at li

sts a

ll of

you

r dat

a, w

here

it is

loca

ted,

its o

rigin

(lin

eage

),ow

ner,

stru

ctur

e, m

eani

ng, c

lass

ifica

tion

and

qual

ity•

Span

ning

syst

ems b

oth

on p

rem

ise

and

clou

d pr

ovid

ers

•H

oste

d lo

cally

to y

our d

ata

plat

form

s but

inte

grat

ed to

pro

vide

the

ente

rpris

e vi

ew

•N

ew d

ata

tool

s (fr

om a

ny v

endo

r) c

onne

ct to

you

r dat

a ca

talo

g ou

t of t

he b

ox•

No

vend

or lo

ck-in

; nor

exp

ensi

ve p

opul

atio

n of

yet

ano

ther

pro

prie

tary

silo

ed m

etad

ata

repo

sito

ry

•M

etad

ata

is a

dded

aut

omat

ical

ly to

the

cata

log

as n

ew d

ata

is c

reat

ed•

Exte

nsib

le d

isco

very

pro

cess

es c

hara

cter

ise

and

clas

sify

the

data

•In

tere

sted

par

ties a

nd p

roce

sses

are

not

ified

•Su

bjec

t mat

ter e

xper

ts c

olla

bora

ting

arou

nd th

e da

ta•

Loca

te th

e da

ta th

ey n

eed,

qui

ckly

and

effi

cien

tly

•Fe

ed b

ack

thei

r kno

wle

dge

abou

t the

dat

a an

d th

e us

es th

ey h

ave

mad

e ab

out i

t to

help

oth

ers a

ndsu

ppor

t eco

nom

ic e

valu

atio

n of

dat

a

•A

utom

ated

gov

erna

nce

proc

esse

s pro

tect

and

man

age

your

dat

a•

Met

adat

a-dr

iven

acc

ess c

ontro

l

Page 24: Unleashing the Power of Apache Atlas with Apache Ranger

Sum

mar

y

•A

tlas c

an h

elp

us h

ave

an in

dust

ry w

ide

com

mon

met

adat

a pl

atfo

rm a

roun

d w

hich

avi

bran

t eco

syst

em c

an e

volv

e•

Not

onl

y in

Had

oop

but m

ore

broa

dly

•M

etad

ata

driv

en g

over

nanc

e ca

n be

scal

able

& e

nabl

e us

to m

anag

e ou

r dat

a be

tter,

and

be c

ompl

iant

with

regu

latio

ns•

The

idea

s pre

sent

ed h

ere

reso

nate

with

man

y pe

ople

we’

ve sp

oken

to•

Get

invo

lved

! I’d

love

to h

ear t

he fe

edba

ck o

n th

is a

ppro

ach!

•C

omm

ent o

n th

e JI

RA

S, a

sk q

uest

ions

, con

tribu

te, d

isag

ree…

;-)

•Lo

ok a

t JIR

A T

ag “

Virt

ualD

ataC

onne

ctor

” or

star

t at A

TLA

S-16

89

•A

tlas w

iki

•“I

nnov

atio

n ha

ppen

s bes

t not

in is

olat

ion

but i

n co

llabo

ratio

n” (k

eyno

te)

•TH

AN

KS!

Page 25: Unleashing the Power of Apache Atlas with Apache Ranger

Que

stion

s

Afte

r thi

s tal

kjo

nesn

@uk

.ibm

.com

17:5

0 Ro

om 4

– S

ecur

ity &

Gov

erna

nce

BOF

z z z z z z z

Que

stion

s?

Page 26: Unleashing the Power of Apache Atlas with Apache Ranger

Back

up c

harts

Page 27: Unleashing the Power of Apache Atlas with Apache Ranger

Atlas

graphD

B“gaiandb

IG C

IGCRE

STAPI

Oracle

Data

HDFS

Data

Netezza

Data

P-JDBC

P-JDBC

P-JDBC

GAF

OMAS

Virtual

Asset

OMAS

Search

Sear

ch/E

xplo

re U

I

Catalog

OMAS

OMR

S

OMR

S

GAF

Pre

GAF

Post

Conn

ectorF

ramew

ork

*

Atlasbo

undarie

sDevelop

edinPOC

MaynotbeinPOCiniNally

*Maybehardcode

datfirst

Conn

ector

Fram

ework

ATLAS

Virt

ualiz

er

Arc

hite

ctur

e

Page 28: Unleashing the Power of Apache Atlas with Apache Ranger

Met

adat

a ar

eas

and

type

s

PolicyM

etadata(Principles,

Regula6ons,Standards,Approaches,

RuleSpecifica6ons,RolesandM

etrics)

Governance

Ac6onsand

Processes

Augmenta6on

Mapping

Implementa6on

ConnectorDirectories

Access

Access

Inform

a6on

Auditor

Integra6on

Developer

Business

Analyst

Data

Scien6st

Inform

a6on

Worker

Inform

a6on

Owner

Inform

a6on

Governor

Inform

a6on

Steward

Data

Quality

Analyst

BusinessObjectsand

Rela6onships,Taxonomiesand

Ontologies

BusinessAMributes

Organiza6on

Inform

a6on

Curator

TeamingM

etadata

(peopleprofiles,communi6es,

projects,

notebooks,…)

ModelsandSchemas

3

2

4 5

PhysicalAssetDescrip6ons

(Datastores,APIs,

modelsandcomponents)

AssetCollec6ons

(Sets,TypedSets,Type

OrganizedSets)

Inform

a6onViews

Rights

Management

ReferenceData

FeedbackM

etadata

(tags,comments,ra6ngs,…)

Classifica6on

Schemes

C l a s s if i c a 6 o n

Strategy

SubjectAreaDefini6on

CampaignsandProjects

Infrastructureandsystems

Rollout

1

Discovery

Metadata(profiledata,technical

classifica6on,dataclassifica6on,

dataqualityassessment,…)

Augmenta6on

Instrument

Associa6on

Inform

a6onProcess

Instrumenta6on(designlineage)

6

7

Page 29: Unleashing the Power of Apache Atlas with Apache Ranger

Use

r & G

roup

/Rol

e sy

nchr

oniz

atio

n

Use

rSyn

c2

LDA

P ho

lds r

ole-

mem

bers

hip

(LD

AP

grou

ps) –

cou

ld a

lso b

eA

ctiv

e D

irect

ory

ATLA

S m

anag

es d

efini

tive

list o

f rol

es <

that

are

use

d fo

rat

las m

anag

ed so

urce

s>

•Co

rpor

ate

LDA

P ha

s a h

uge

num

ber o

f use

rs/g

roup

s•

Rang

er c

urre

ntly

nee

ds to

sync

all

•In

futu

re p

erha

ps w

e es

tabl

ish g

roup

/role

mem

bers

hip

durin

g au

then

ticat

ion

•Ca

pabi

lity

for a

ltern

ativ

e so

urce

cou

ld b

e m

erge

d in

toba

se U

serS

ync

LDA

P lo

okup

->gr

oup:

mem

ber

Gov

erna

nce

Act

ion

OM

AS

- get

Role

s

Apa

che

Rang

er

LDA

P

Apa

che

Atla

s

Page 30: Unleashing the Power of Apache Atlas with Apache Ranger

Atla

s Glo

ssar

y v2

: Tag

Syn

c to

Ran

ger

TagS

ync2

ATLA

S gl

ossa

ry m

anag

es a

soph

istic

ated

ent

erpr

ise g

loss

ary

struc

ture

•A

tlas G

loss

ary

v2 P

ropo

sed

in A

TLA

S-14

10 (D

avid

Rad

ley)

Syn

c Bu

ilds o

n ex

istin

g ta

gsyn

c ap

proa

ch•

New

API

in A

tlas w

ill fl

atte

n cl

assifi

catio

n str

uctu

re•

No

chan

ges t

o ra

nger

– b

ut e

xpos

ing

riche

r cla

ssifi

catio

n co

uld

be a

rea

of fu

ture

wor

k

Gov

erna

nce

Act

ion

OM

AS

Confi

dent

ial

Sala

ry

emp_

renu

m

Busin

ess

Term

Hiv

e Co

lum

n

Busin

ess

Term

Confi

dent

ial

emp_

renu

mH

ive

Colu

mn

Tag

Apa

che

Rang

er

Apa

che

Atla

s

Page 31: Unleashing the Power of Apache Atlas with Apache Ranger

Polic

y (R

ule)

sync

hron

izat

ion

Rule

Sync

•G

ener

ate

polic

ies i

n Ra

nger

bas

ed o

ff en

titie

s in

Atla

s•

Curre

ntly

des

igni

ng h

ow th

is w

orks

•Sc

oped

by

polic

y se

rvic

e so

exi

sting

Ran

ger U

I app

roac

h sti

ll w

orks

Gov

erna

nce

Act

ion

OM

AS

- get

Rule

s

Role

Clas

sifica

tions

Ass

et

Rang

er R

ule

Act

ion

Apa

che

Rang

erA

pach

e A

tlas

Page 32: Unleashing the Power of Apache Atlas with Apache Ranger

Virtu

alD

ataC

onne

ctor

JIRA

S 20

1704

02

•RA

NG

ER-

1488

•RA

NG

ER-

1487

•RA

NG

ER-

1486

•RA

NG

ER-

1485

•RA

NG

ER-

1464

•RA

NG

ER-

1454

•RA

NG

ER-

1234

•RA

NG

ER-

•Cr

eate

Ran

ger p

lugi

n fo

r gai

andb

•ge

nera

te ru

les f

rom

Gov

erna

nce

defin

ition

s in

Atla

s

•N

ew u

sers

ync

alte

rnat

ive

for A

tlas (

vdc)

•Ra

nger

supp

ort f

or V

irtua

l Dat

a Co

nnec

tor P

roje

ct (A

TLA

S)

•Su

ppor

t Atla

s v2

glos

sary

in A

tlas p

lugi

n (fo

r acc

ess c

ontro

l to

term

s etc

)

•Su

ppor

t of A

tlas v

2 gl

ossa

ry A

PI p

ropo

sal f

or ta

g so

urce

•Po

st-ev

alua

tion

phas

e us

er e

xten

sions

•Ra

nger

Sou

rce:

ecl

ipse

•A

dd d

ata

mas

king

for t

ag b

ased

pol

icie

s

•G

over

nanc

e A

ctio

n Fr

amew

ork

OM

AS

•Sa

mpl

e as

sets

to su

ppor

t Virt

ual C

onne

ctor

Pro

ject

•O

MA

S In

terfa

ces f

or A

tlas

•Bu

ild A

TLA

S us

ing

Doc

ker

Page 33: Unleashing the Power of Apache Atlas with Apache Ranger

Ref

eren

ces

•A

pach

e A

tlas -

http

://at

las.a

pach

e.or

g/•

Top

leve

l JIR

A fo

r thi

s act

ivity

http

s://i

ssue

s.apa

che.

org/

jira/

brow

se/A

TLA

S-16

89

•A

pach

e R

ange

r - h

ttp://

rang

er.a

pach

e.or

g/•

Gai

anD

B•

http

s://g

ithub

.com

/gai

andb

/gai

andb

•ht

tps:

//dev

elop

er.ib

m.c

om/o

pen/

open

proj

ects

/gai

an-d

atab

ase/

•Th

e ca

se fo

r ope

n m

etad

ata

– A

.M.C

hess

ell

•ht

tp://

ww

w.ib

mbi

gdat

ahub

.com

/blo

g/ca

se-o

pen-

met

adat

a