Towards a Stream-Centered Enterprise

Kafka Summit 2016

Gabriel Commeau

Stream Data Platform Overview

Data Ingest

Multi-Datacenter Deployment

o Avro versus json, protobuf, xml, thrifto Some guidelines for Avro schema:

• Document it all• Use explicit names• Provide default value• Avoid complex union type• If possible avoid byte arrays

o Use:• Common core schemas• Proper nesting• Enum + String

Schema Management

Avro Schema Evolution

Enum• Reader can add value• Writer can remove value

Don’t• Change namespace• Change name

• Unless default value• Change type

Do• Use default value• Use union type

containing null

Union Type• Change type order ok• Reader can add type• Writer can remove type

CompatibilityOk• Change order• Change, add, remove,

reorder alias