DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT,...

133
DevOps for Dishwashers Bringing grown-up practices to the Internet of Things Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au #YOWConnected Sep 2017 Christopher Biggs, Accelerando Consulting @unixbigot @accelerando_au 1 / 133

Transcript of DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT,...

Page 1: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

DevOps for DishwashersBringing grown-up practices to the Internet of Things

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Christopher Biggs, Accelerando Consulting

@unixbigot @accelerando_au

1 / 133

Page 2: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Who am I?

Christopher Biggs — @unixbigot — @accelerando_au

▷ Brisbane, Australia

▷ Former developer, architect, development manager

▷ Founder, Accelerando Consulting

▷ Full service consultancy - chips to cloud

▷ IoT, DevOps, Big Data

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017 2 / 133

Page 3: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Why Devops?

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Problems

3 / 133

Page 4: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Why Dishwashers?

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Problems

4 / 133

Page 5: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

"Software is eating the world"

-- Mark Andreesen

Wall St Journal, Six years ago last month.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Problems

5 / 133

Page 6: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Interlude: What do I mean by "DevOps"?

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017 6 / 133

Page 7: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

DevOps is NOT THIS

DevOps is not a thing you do,

it's the way you do things.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Problems DevOps?

7 / 133

Page 8: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

DevOps is NOT THIS

Empower everyone

to maximise value.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Problems DevOps?

8 / 133

Page 9: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

"When every Thing is connected, Everything is connected"-- Me

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017 9 / 133

Page 10: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Welcome to the Internet of Things

pop. 10 Trillion

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape

10 / 133

Page 11: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

1936 (Information Pandemic Year Zero)

10-7 device/person

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape

11 / 133

Page 12: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Mainframe era

10-4 device/person

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape

12 / 133

Page 13: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Minicomputer era

10-2 device/person

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape

13 / 133

Page 14: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Desktop era

100 device/person

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape

14 / 133

Page 15: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Mobile era

100.5 devices/person

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape

15 / 133

Page 16: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Cloud era [YOU ARE HERE]

~101 devices/person

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape

16 / 133

Page 17: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Internet of things

103 devices/person

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape

17 / 133

Page 18: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Solve the next problem, not thelast one

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges

18 / 133

Page 19: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Beware of false analogies and straight line trends

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges

19 / 133

Page 20: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Observe, Orient, Decide, Act

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges

20 / 133

Page 21: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

"Bad people will break your stuff"

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape ChallengesLandscape Challenges Risks

Do you want to know more? "The Internet of Scary Things" christopher.biggs.id.au/talk

21 / 133

Page 22: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Everything is awful

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape ChallengesLandscape Challenges Risks

22 / 133

Page 23: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Everything is awful

and the awful is on recurl http://my-dishwasher/../../../../../../../../../../../../etc/shadow

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape ChallengesLandscape Challenges Risks

23 / 133

Page 24: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

It's not rocket science

No really, I mean actual rockets.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges

24 / 133

Page 25: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Desiderata

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Agenda Landscape ChallengesLandscape Challenges Risks Desiderata

Page 26: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Select appropriate tools and platforms

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges

26 / 133

Page 27: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Comprehensive identity management

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges

27 / 133

Page 28: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Automate for developer and user convenience

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges

28 / 133

Page 29: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Testing and testability kept front-of-mind

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges

29 / 133

Page 30: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Train, and Audit, and keep doing both

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges

30 / 133

Page 31: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Monitor and react (automatically)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges

31 / 133

Page 32: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Platforms

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017 32 / 133

Page 33: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

People are more expensive than circuits

(Sorry, robots)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges Solutions Platforms

33 / 133

Page 34: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Hardware is DevOps too

(Robots, I hope this makes it up to you :)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges Solutions Platforms

34 / 133

Page 35: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Open and well supported

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges Solutions Platforms

35 / 133

Page 36: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case study: ARM v7 and Debian Linux

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges Solutions Platforms

36 / 133

Page 37: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Meet the #3 top-sellingcomputer of all time

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges Solutions Platforms

37 / 133

Page 38: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Artisanal free-range small-batch Linux?

No.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges Solutions Platforms

38 / 133

Page 39: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Without the Internet, it's just a Thing.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges Solutions Platforms

39 / 133

Page 40: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

"Is there anybody out there?"-- Pink Floyd

(also, my lighting controller)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges Solutions Platforms

40 / 133

Page 41: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

"Put the robot back in the ocean, kid."-- Oceanographers, everwhere

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges Solutions Platforms

41 / 133

Page 42: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Management is not a dirty word

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Agenda Landscape Challenges Solutions Platforms

42 / 133

Page 43: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Development

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017 43 / 133

Page 44: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Aside: Go Serverless

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev

44 / 133

Page 45: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Nice languages are portable, memory-safe and asynchronous.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev

45 / 133

Page 46: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case study: Javascript

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev

46 / 133

Page 47: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case Study - Go

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Landscape Challenges Solutions Platforms Dev

Page 48: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Naughty languages are like a tightrope over a pit of spikes.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev

48 / 133

Page 49: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

'The wi password is "abc123';cat /etc/passwd#" '

Say no to shell scripts.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev

49 / 133

Page 50: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Use, and reuse, a framework.

Yours, mine, Google, Amazon, Microsoft, Apple, whatever.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev

50 / 133

Page 51: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

AWS IoT

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev

51 / 133

Page 52: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Choose your own framework adventure

▷ Amazon IoT and Greengrass

▷ Google IoT

▷ Azure IoT

▷ Open Connectivity Foundation IoTivity

▷ Resin.io

▷ Mongoose-OS

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev

52 / 133

Page 53: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Containment is complexity management

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev

53 / 133

Page 54: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

You can run Docker on a $6.95 linux computer

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev

54 / 133

Page 55: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Use your CI to produce docker images as artifacts

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev

55 / 133

Page 56: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Testing

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017 56 / 133

Page 57: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Total Infrastructure Awareness

replicate your whole ecosystem

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

57 / 133

Page 58: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

No snow ake servers.

A dev team should have access to disposable instances of everything

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

58 / 133

Page 59: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

DevOps is a disaster, every day

and that's good

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

59 / 133

Page 60: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Painful testing practices beget painfully bad testing

provide easy test data

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

60 / 133

Page 61: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Quick xes are good

and cheap

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

61 / 133

Page 62: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Listen to that annoying hipster tech blogger

At least four eyeballs per line.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

62 / 133

Page 63: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Test-before-merge

Never* "break the build"

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

* well, almost never

63 / 133

Page 64: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Fail fast

Unit tests rst, followed by slower end-to-end tests

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

64 / 133

Page 65: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Every pair of eyeballs costs $$$

(and no, you can't save $-½ by poking one out)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

65 / 133

Page 66: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Do not poke customers with sticks (either)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

(Wherein Christopher does math to "prove" a point)

66 / 133

Page 67: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

A modern embedded system is faster than a Cray 1

But that's still reaaally slow

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

67 / 133

Page 68: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Cross-platform CI pipelines

Option zero: cross-platform languages are win

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

68 / 133

Page 69: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Cross-platform CI pipelines

Option one: Emulate

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

69 / 133

Page 70: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Cross-platform CI pipelines

Option two: Enrol embedded systems in your CI

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

70 / 133

Page 71: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case Study: My Pipelines

Minimal requirement: one x86 and one ARM server (eg Raspberry Pi)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

71 / 133

Page 72: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case Study: My Pipelines

Stage 1: common policy checks

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

72 / 133

Page 73: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case Study: My Pipelines

Stage 2: compile and package

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

73 / 133

Page 74: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case Study: My Pipelines

Stage 3: Testing (on target arch)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

74 / 133

Page 75: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case Study: My Pipelines

Stage 4: Deploy to container registry

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

75 / 133

Page 76: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case Study: My Pipelines

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

76 / 133

Page 77: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case Study: My Pipelines# note the pi build job can be run on x86 because Go is awesomebuild for pi: stage: build script: - make installdeps image contents ARCH=pi artifacts: paths: - GPIOpower - GPIOpower_docker_pi.tar.gz - GPIOpower_contents_pi.tar.gz## Run tests on RasPi#test on pi: stage: test tags: - pi script: - make test

deploy to staging: stage: deploy environment: name: staging only: - masterscript:

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Landscape Challenges Solutions Platforms Dev QA

Page 78: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Package separate lab and eld artifacts

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

78 / 133

Page 79: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Defeat laziness by making it easier to do the right thing

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

79 / 133

Page 80: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Regression tests, longitudinal tests

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

80 / 133

Page 81: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Dashboards as "live tests"

Containerise your BI stack and write dashboards alongsidecode

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

81 / 133

Page 82: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Dashboards learn "green" state and alert on red

Obligatory reference to Machine Learning

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA

Do you want to know more? "Continuous Dashboarding" christopher.biggs.id.au/talk

82 / 133

Page 83: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Deployment

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017 83 / 133

Page 84: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Orchestrate: Never doanything by hand.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

84 / 133

Page 85: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Build a provisioning work ow

Customise a clean OS (via ethernet or emulation)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

85 / 133

Page 86: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Robo-con gure the target system from a provisioning system

Then save a lesystem image

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

86 / 133

Page 87: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

How do you create a provisioning system?

Turtles all the way down!

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

Do you want to know more? github.com/unixbigot/kevin

87 / 133

Page 88: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case study - My orchestration scripts

1. Create a read only recovery partition

2. Install SaltStack orchestration minion (now switch protocols)

3. Set timezone, locale, etc.

4. Change default passwords

5. Configure network

6. Provision message bus clients

7. Install language runtimes (nodejs, java etc.) if needed

8. Configure VPN client

9. Fetch initial application containers

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Landscape Challenges Solutions Platforms Dev QA Deployment

Page 89: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Zoom level 1: Device roles# top.sls

'G@roles:controller': - match: compound - salt.syndic - mqtt.relay - mqtt.client - net.hostapd_bridge - net.gpsd - fleetvalid.station-console 'G@roles:sensor': - match: compound - os.initramfs - os.hostname - net.aws.cli - docker - fleetvalid.docker_auth - fleetvalid.service-advertiser 'G@roles:inroad_sensor' - match: compound - fleetvalid.inroad_sensor

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

89 / 133

Page 90: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Zoom level 2: Device pro le# fleetvalid/inroad_sensor.sls

include: - fleetvalid.aggregator - fleetvalid.rfid - fleetvalid.radar - fleetvalid.vibration - fleetvalid.rfidpower

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

90 / 133

Page 91: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Zoom level 3: Device component# fleetvalid/rfid.sls

{% set image = reg.path + '/roadcurtain/' + rfid + 'rfid-' + pillar.fleetvalid.arch + ':' + pillar.fleetvalid

fleetvalid-rfid-image: docker_image.present: - name: {{image}}

fleetvalid-rfid: docker_container.running: - image: {{image}} - links: - fleetvalid-aggregator:aggregator - fleetvalid-rfidpower:power - binds: - {{rfid_device}}:{{rfid_device}} - environment: - RCAGGREGATOR: aggregator:9091 - POWER_API: power:80 - privileged: True

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

91 / 133

Page 92: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Zoom level 4: System con gurationgpsd: pkg.installed: - pkgs: - gpsd - chrony

gpsd-device-configuration: file.replace: - name: /etc/default/gpsd - pattern: ^DEVICES=.* - repl: DEVICES="{{pillar.gps.device}}" - append_if_not_found: True

gpsd-running: service.running: - name: gpsd - enable: True

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

92 / 133

Page 93: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Hey, that all sounds a bit like PaaS

Yeah, it does.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

93 / 133

Page 94: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Amazon AWS Greengrass

IoT PaaS built on AWS IOT + AWS Lambda

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

94 / 133

Page 95: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Resin.io

IoT PaaS with Linux and Docker

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

Do you want to know more? https://resin.io/

"The Internet of Scary Things" christopher.biggs.id.au/talk

95 / 133

Page 96: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Apache MyNewt

Embedded component-based OS for wireless sensors

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

96 / 133

Page 97: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Mongoose-OS

Multiplatform embedded OS with cloud integration and remote upgrade

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

Do you want to know more? http://mongoose-os.com/

"IoT in two Minutes" christopher.biggs.id.au/talk "Javascript Rules My Life (CampJS 2017)" christopher.biggs.id.au/talk

97 / 133

Page 98: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Automate PKI enrolment

IoT Makes PKI Easy.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

-er

98 / 133

Page 99: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Your "secure key distribution channel" is a cardboard box

1. Build and sign a root certificate

2. Upload root cert to the SaltStack master

3. Create minion certificate

4. Install minion certificate on minion

5. Upload a copy to the master

6. All automatically

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

99 / 133

Page 100: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Containerised version control

Use your docker registry the way it was intended

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

100 / 133

Page 101: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Use Salt "grains" to de ne which container to use(i.e. live, staging, dev or other)

fleetvalid-rfid-image: docker_image.present: - name: ((image)) - force: True

fleetvalid-rfid: docker_container.running: - image: ((image)) - links: - fleetvalid-aggregator:aggregator - fleetvalid-rfidpower:power - binds: - ((rfid_device)):((rfid_device)) - environment: - RCAGGREGATOR: aggregator:9091 - POWER_API: power:80

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment

101 / 133

Page 102: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Maintenance

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017 102 / 133

Page 103: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Self-care

FLASH memory longevity tweaks

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance

103 / 133

Page 104: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Boot to ramdisk

Sanity check, then proceed to target environment

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance

104 / 133

Page 105: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Recovery mode

(ab)use DHCP

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance

105 / 133

Page 106: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Liveness monitoring

If a device goes silent, notify the site custodian

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance

106 / 133

Page 107: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Sickness monitoring

Use an audible or visual attention signal (think smoke-alarms)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance

107 / 133

Page 108: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

While you were sleeping

Intermittent connections

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance

108 / 133

Page 109: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Kill or Cure

Feature/component disable

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance

109 / 133

Page 110: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Kill or Cure

The Cassini Solution

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance

110 / 133

Page 111: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Monitoring (platform data)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017 111 / 133

Page 112: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Heartbeats

SaltStack's presence monitor

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring

112 / 133

Page 113: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Flatliners

Detect missing devices

(i.e. known to saltstack but not connected)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring

113 / 133

Page 114: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Health stats

SaltStack beacons - cpu/memory, often

Full process list, less often

Network stats

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring

114 / 133

Page 115: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Pour it all into a data lake

And pretend to be a Bond Villain

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring

Do you want to know more? "Continuous Dashboarding" christopher.biggs.id.au/talk

115 / 133

Page 116: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case study: Log pooling for a building safety startup

Ram disk on local ARM devices

Streaming to cloud with Filebeat

Processing with Logstash

Set a storage budget and expire to meet the budget

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring

116 / 133

Page 117: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Measurement (application data)

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017 117 / 133

Page 118: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Use orchestration message bus

SaltStack message bus is the fast, lightweight ZeroMQ

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement

118 / 133

Page 119: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Can you use your orchestration bus for application events?

Yes, with care

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement

119 / 133

Page 120: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Extend orchestration system with custom modules

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement

120 / 133

Page 121: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Record as much as you can, digest later

Shove all your client data in ElasticSearch

Purge oldest indexes until CFO stops whinging

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement

121 / 133

Page 122: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case study: Saltstack plus ELK

Bridge orchestration bus to application message bus

Engine module at top level master (or intermediate)

Gateway messages to elasticsearch, via logstash

Want MQTT? You already built a PKI to deploy it in 2 minutes

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement

122 / 133

Page 123: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Case study: MQTT plus ELK

"Rapids Rivers Ponds"

MQTT brokers at each site

Broker in the cloud federates with on-site brokers

Logstash MQTT plugin subscribes to all events

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement

Do you want to know more? "Implementing Microservice Architectures" Fred George, YOW 2014

123 / 133

Page 124: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Visualisation

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017 124 / 133

Page 125: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Real time status

Liveness, resources, environment

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation

125 / 133

Page 126: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Measure your KPIs

Whatever makes you money, count it

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation

126 / 133

Page 127: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Measure your KPIs

Set high and low water marks, alert on them

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation

127 / 133

Page 128: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Measure your KPIs

Pay: Elastic and other vendors have commercial alert engines

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation

128 / 133

Page 129: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Measure your KPIs

Free: Node-RED makes a good FOSS alerting engine

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation

Do you want to know more? "Continuous Dashboarding" christopher.biggs.id.au/talk

129 / 133

Page 130: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Longitudinal comparisons

View long-term trends in KPIs

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation

130 / 133

Page 131: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Longitudinal comparisons

Pay attention to device longevity, wear, etc.

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation

131 / 133

Page 132: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Summary▷ Lots of devices, too many to administer by hand

▷ Swimming in a soup of malware and bad actors

▷ Choose tools that support quality

▷ Pipelines for automated build/test/stage

▷ (Ab)use traditional cloud management tools for IoT Fleet

▷ Message bus all the things

▷ Big data now, play later

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWC d S 2017

Landscape Challenges Solutions Coda Summary

Page 133: DevOps for Dishwashers - YOW! Conferences · + Full service consultancy - chips to cloud + IoT, DevOps, Big Data Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

Resources, QuestionsMy SaltStack rules for IoT - github.com/unixbigot/kevin

Related talks - http://christopher.biggs.id.au/#talks

Me - Christopher Biggs

Twitter: @unixbigot

Email: [email protected]

Slides, and getting my advice: http://christopher.biggs.id.au/

Accelerando Consulting - IoT, DevOps, Big Data - https://accelerando.com.au/

Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au

#YOWConnected Sep 2017

Landscape Challenges Solutions Coda Summary Resources

133 / 133