Skim down AGL Application Framework to bridge …...Skim down AGL Application Framework to bridge...

Skim down AGL Application Framework to bridge AGL with hard realtime subsystems

Dresden AGL/AMM Oct/2018Linux Realtime Technical Lead

[email protected]

Oct-2019Micro Binder Architecture 2

Who are we?


Where ?

LORIENT

vannes

● https://iot.bzh/en/● http://github.com/iotbzh

https://iot.bzh/en/

http://github.com/iotbzh


Bridging AGL Application Framework with hard realtime subsystems

Introduction


How to broaden AGL Connectivity

● Up to the Cloud– Keeping critical services in realtime contexts

● Down to:– (very) smaller, hard realtime systems

You’re at the right place !

Feb-2018 7



Plan

● The RT Quest– What is RealTime in an OS ?

– Soft vs Hard Realtime in Linux

– Which Automotive Apps need RT ?

– Impacts on Linux RT Applications

– Turn “ON” Linux RealTime● Porting to lighter OS

– Binder & bindings

– Impacts on binder

– Defining a boundary to portability

– Inter SOC communication● Focus on the can-low-level binding● To infinity (→smaller) and beyond !● Suggested roadmap


Plan











The RT Quest

● Path to RealTime is not straightforward !


Some urban legends about RealTime

● “I wont’t get RealTime issues, and won’t have to deal with thread priorities, as long as my CPU load keeps low”

Hey buddy, have you ever heard about:

Interrrupts ?

Ping flood ?

Unpredictive (because not tested) time spent in code branches ?

Device latencies with poor driver implementations (eg, SD card) ?

Freezes due to tight loops ?

Feb-2018 12

Some urban legends about RealTime

● “I do not care memory allocations, I have got 4 Go of available RAM !”

Feb-2018 13

What is RealTime in an OS?

● RealTime means “on time” and not “faster”– Realtime is about predictability

● Typically Realtime address following concerns– 80% => Could my execution code be disturbed ?

● If then:– For how much time ?– How often ?– By who ?

– 20% => How big is my latency● Time lost within an external event, and the time my code can

start to handle it ?● Maximum latency fluctuation ?

Feb-2018 14

Linux & Soft/Hard Realtime

● Soft realtime● Periodical tasks/events of several milliseconds● Some acceptable unpredictable delays (10/100 ms)● Often implemented by resource controls● Latency of few ms with exceptionnal unpredictable

fluctuation +-10ms

● Hard realtime● Total Latency magnitude of 10/100 us● Predictable and short delays (< 250ms)

● Current vanilla Linux kernel is Soft RealTime

Feb-2018 15

Which Automotive Apps need RT ?

● Soft Realtime● Data Acquisition● Audio/Video

● Hard Realtime● Cluster● Emergency/Safety signal● Dead reckoning● LIDAR acquisition● Trajectory control● ...

Driv erle ss c ars

Feb-2018 16

RT Options inside AGL

Cluster

Carte handling

Localistion management

POI

CAN GPS

Geopositioning Virtual Signal

Multi ECU & Cloud Aware Architecture

Entertainement

CAN-BUS Virtual Signal

Gyro, AcelerometerCAN-BUS

LIN-BUS

Engine-CAN-BUS

ABS

Transport & ACL

Head Unix

Direction Indication

Cloud

Log Analytics

No-SQL Engine

Statistics & Analytics

Transport & ACL

My Car Portal

Paiement

Subcriptions

Preference

Preferences &

Custumisation

MongoDB Engine

Paiement Service

Cluster Virtual Signal

Transport & ACL

Navigation Service

Maintenance Portal

Know Bugs

Maintenances

Service Packs

Soft RealTimeHard RealTime

Feb-2018 17

Linux RT Application Impact

● Standard Linux: a simple “Ping Flood“ will lag applications.● Linux network IRQs preempt applications too often and for too long, which

significatly increase the latency.

● PREEMPT_RT reduces scheduling latency● Replaces most spinlock by mutexes● Support threaded IRQs● Supports hight resolution timers

Feb-2018 18

Turn “ON” Linux RealTime.

● Objectives● Decrease Application Latencies● Garanty that high priority tasks will not be bothered by

lower priority ones.● Make sure interrupts cannot lag your critical apps.

● Soft Realtime (Standard Kernel)● Container, Ggroups, ...

● Hard Realtime (Kernel must be patched)● PREEMPT-RT● IPIPE+Xenomai

Feb-2018 19

Preempt_RT vs Xenomai

● Xenomai● Xenomai supports to legacy RT non-POSIX applications (eg: VxWorks, PSOS) through

skins● Dual Kernel solution brings more performances when no more than 4 cores run RT

threads● More confidence on the whole RT application (eg: /proc/xenomai statistics)● Integrated debug features● Misses some critical Unix develoment tools (eg: Valgrind, LTTng)

● Preempt_RT● Almost Vanilla Linux (no API/ABI changes)● Continuous testing in OSADL QA farm● No need for extra userspace libraries● Less confidence in app, harder to debug, needs extra code for RT monitoring

Feb-2018 20

Xenomai Dual Kernel Mode

Being replaced by Dovetail

Feb-2018 21

Preempt_RT Latency

Source: http://www.emlid.com/raspberry-pi-real-time-kernel Latency (us)


Xenomai Latency

Feb-2018 23

Xenomai & Prempt_RT convergence

● Xenomai 3.x offer dual kernel and PREEMPT_RT option (with the same high level API and skins)

● Dual kernel latency remains significantly better● Some options are not available in both

solutions (eg: RtNet only runs on dual kernel)

Feb-2018 24

RT Kernel is only a start (1/4)PREEMPT_RT kernel tuning

● Realtime requires more kernel tuning and clean behaviours on the application side.

● Enable CONFIG_PREEMPT_RT_FULL & CONFIG_HIGHRES_TIMERS to get <1ms precision

● Disable CONFIG_CPU_FREQ !● Might conflict with power management● … and disable THERMAL framework

Feb-2018 25

RT Kernel is only a start (2/4)Impact on Applications

● There are strict rules to follow and actions to take in the application:● Stack pre-faulting● Virtual Memory locking● Fine tuning of threads priorities● malloc() and friends chasing, to avoid page faults (can be difficult with

some C++ libraries, eg BOOST)● Forbidden usage of system(); popen(); execve() … in runtime.● Monitoring run-away threads (ie, tight loops in RT contexts) to prevent

system hanging (and to allow debugging).● clock_nanosleep is your friend, for writing periodic tasks● Carefull initialization parameters of pthread_mutex, default ones do not

have PTHREAD_PRIO_INHERIT !● Fancy some LTTng sessions ? (does not work with IPIPE)

Feb-2018 26

RT Kernel is only a start (3/4)Choose the right clock !

● CLOCK_REALTIME is NOT RealTime ! – Dayligt savings

– NTP adjustements (even THE big jump, when enabled)● CLOCK_MONOTONIC (affected in speed by NTP)

or● CLOCK_MONOTONIC_RAW will only vary depending on the quality

of the quartz and voltage and temperature compensation quality

Feb-2018 27

RT Kernel is only a start (4/4)Impact on Applications :

Last but not least

● Not everything can be RT● Providing high priority to some task means

than the other will inherit of low priority● Base you flow on lock (semaphore) and not on

thread priority● Get rid of any spin lock

Feb-2018 28

Impact on Applications : C specific issues (& workarounds)

● malloc()/realloc() do not lead to pagefault always, (through sys_brk() or sys_mmap_pgoff() because of internal memory pool of the glibc

● Thus, a RT ‘leak’ may be hard to reproduce● Using GDB with a breakpoint on malloc() is usually

sufficient● Another technique, less intrusive, is to use Memory

Allocation Hooks of the glibc.● Some companies allow malloc() for initializations, and

always forbid the free() !

Feb-2018 29

C++ specific issues

● In C++, dynamic allocations are not always explicit

● Example: std::vector growing● In some extra libraries (eg, boost), memory

allocations may be completly out of control (in addition of alien-only-friendly backtraces)

Feb-2018 30

Plan










Feb-2018 31


● Cars are made of several systems with different connectivity or responsiveness constraints

● Communication between cluster and subsystems involves several protocol stacks

● Bringing access security to embedded control systems

● Low-level filtering of devices events is leaner than in high level application(s)


What is the AGL binder

● Bound to a systemd service● Dynamic loader of applications (bindings)● Embeds:

– a tiny http server

– websockets based on Unix named sockets● Provides an event loop and timers API

Feb-2018 33

Use case of the binder

cloudAPPLI

BINDERA

BINDER

BAPPLI

BINDERC

BINDING

BINDERD

BINDING

Connected carMaster

ECU

HTTPS+WSS/TCP HTTPS+WSS/TCP

WS/UDS

WSS/TCP

WS: WebSocketWSS: WS SecuredUDS: Unix Domain

SocketECU: Engine Control

Unit

BINDERBINDING


Binder/Binding model

●Binder–Container process–Transport–Security–Standardized Sync/Async API–Expose API through HTTP or WebSocket

● Protected by token●Binding

–One or more API published through binder–Provided as:

● A native library weekly coupled (threading allowed)● A proxy to a remote service

APPLICATION

BINDERafb-daemon

BINDING

BINDING

BINDING

SECURITYCONTEXT

http ws

BINDERafb-daemon

BINDING

SECURITYCONTEXT

A

B


Some existing binders & bindings in AGL

● agl-service-audio-4a– Highlevel API

– Softmixer

– 4a-HALs

– AlsaCore● agl-service-can-low-level● agl-service-iiodevices


Porting the Binder to a micro-OS

● Leave the Unix world– No unix sockets

– Inter-binding messages should be lighter, less footprint, no big json data.

– Binary json could be an option

– Avoid copying data when possible● Break the existing dependency to libsystemd

– But keep the timers and event loop API● Need to have an OS abstraction layer for non POSIX systems● Add new communication transport with other system chips

? or shared memory


Inter SOC communication

● Depends on the subsystem connectivity– Subsystem on same PCB → shared memory or

dedicated hw channel (i2c …)

– Remote subsystem → specific bus, or IP in the best case

● Access token can given by an external authentification server

● Can use the encryption as well


Defining a boundary

● POSIX may be a too strong prerequisite (not available for all the embedded systems, or only partial support

● Assuming Glibc (or µCLibc) seems fair enough● Incomplete libc implies to have more OS

abstraction

or

µClibc

Glibc

Feb-2018 39

Plan









– Inter SOC communication

– Security● Focus on the can-low-level binding● To infinity (→small) and beyond !● Suggested roadmap


Focus on the can-low-levelbinding

41

CAN agent base on AGL framework

● Clear Isolation ● Low level CAN operation only depends on equipment● High level business logic dedicated to applications

● Security Built In● Navigation APP may access GeoLocation but not Telephony● Implement statistic/counter to monitor unexpected behavior

● Leverage AGL framework● API transparency for client applications● Reuse existing technology (faster, cheaper, safer)

42

AGL CAN Mapping

Binder

CAN Low Level Binding(s)

● Decoding / Encoding● Authentication / Crypto / Firewalling● Transaction (set… ack ...)● Stats & Maths● Caching (low freq. Signals, get() call)● Debug

CAN High Level Binding(s)

● Logic● Aggregation (« vehicle.doors.any.open »)● Advanced Ops

CAN BUS

CAN frames - 011010010

Signals - « vehicle.doors.left.open »(Binder Events)

UI

Publish Subscribe

43

Low Level Binding

● Low Level Binding● Binary encoding decoding● Generate Application Friendly Signal Name & Values

● Close to Automatic Code Generation● OpenXC CAN Vector definition in JSON● Other CAN analysers formats, as CANoe XML (TBD)

● Include Basic Filtering & Statistic● GT/GE LT/LE● Timer, Cycle, Timestamps● Counter: last value, average, invalid ID, …

● Can be ship to developer as binary only

44

From OpenXC to AGL CAN Binding

CAN Low Level Binding(Shared Library)

OPENXC Signal

Description(JSON)

Low LevelBinding

Static Code(AGL)

CANDecoding/EncodingC Code(vendor)

OptionalMessageHandlers(vendor)

AGLCode

Generator

C/C++Compiler

OPENXC Signal

Description(JSON)

OPENXC Signal

Description(JSON)

45

Current Binding Capabilities

● Asynchronous signalization● Basic Subscribe/Unsubscribe event model● Transport onto WebSocket/DBus

● AGL App-Framework event signaling API● handle = afb_daemon_make_event (bindif, name)● afb_req_subscribe(request, handle)● afb_req_unsubscribe(request, handle)● afb_event_push(handle, json_object_get(object));

46

Event Subscription

Waiting verb call

[Subscribe/Unsubscribe]

Find corresponding signals name

Create event handler

Make subscription or unsubscription

[No name provided]

[Name provided]

For each signal

Add recurring request to the diagnostic manager

[Diagnostic message]

[Regular CAN message]

Create Diagnostic handle

[No more signals to process]

Take all signals

Vehicle.*.doors.*50<Vehicle.speed<100

Freq=10Hz

47

OpenXC Specification● Messages

● bus - name of initialized CAN buses where this message can be found● name - The name of the CAN message● handler – Function name applied to entire raw CAN message● max_frequency – slow down high level signal frequency (ex: 1/10)● signals - A list of CAN signal objects that are in this message

● Signals● generic_name - The name of the associated generic signal name● bit_position - Staring bit position of this signal within the message● bit_size - The width in bits of the signal● factor - The signal value is multiplied by this if set● offset - This is added to the signal value if set● decoder - type & function name applied to signal bitfield to decode it● encoder - Idem to encode a value in bitfield to send over CAN bus● states - Mapping between descriptive states and values from CAN● max_frequency – slow down high level signal frequency (ex: 1/10)● send_same – if false only send signal when value change● ignore, enabled - Control parameters about activation level for a signal● writable – signal is writable on CAN

48

OpenXC Sample Definition

"0x620": { "bus": "can0",

“name”: “doors”, "signals": { "doors.driver.open": { "generic_name": "doors.driver.open", "bit_position": 78, "bit_size": 1, "factor": 0, "offset": 0, "decoder": "decoder_t::booleanDecoder"}, "doors.passenger.open": { "generic_name": "doors.passenger.open", "bit_position": 79, "bit_size": 1, "factor": 0, "offset": 0, "decoder": "decoder_t::booleanDecoder"},[...]


Plan





– Turn “ON” Linux RealTime● RT Options inside AGL● Porting to lighter OS





– Security● Focus on the can-low-level binding● To infinity (→ small) and beyond !● Suggested Roadmap


And next ?

● Going to very smaller systems● Scalability and OS-agnosticism will be the

keys● Would eventually imply new transport

implementations (serial, i2c, specific usb ...)


Plan





– Turn “ON” Linux RealTime● RT Options inside AGL● Porting to lighter OS





– Security● Focus on the can-low-level binding● To infinity (→ small) and beyond !● Suggested Roadmap


Suggested Roadmap

AGLBinder

POSIXBinder

Zephyr / STM32 POC- map POSIX calls to Zephyr syscalls

PREEMPT_RT POC

- Self-monitoring service- Priorities policies

MapSocketcan

Calls toZephyr

CAN driver

Low-levelCAN

on ZephyrBinder on

ZephyrBinder on

Zephyr

RT Binder

RT Low-level

CAN

PortableRT Binder

Lean Memory

allocations

Memfriendly

Remove theDependencyto systemd

pre-RT

Smartmessaging

Xenomai POC

- Priorities policies

RT Binder

RT Low-level

CAN


Questions ?

Skim down AGL Application Framework to bridge …...Skim down AGL Application Framework to bridge...

Documents

Transcript of Skim down AGL Application Framework to bridge …...Skim down AGL Application Framework to bridge...