Temporal Isolation of Hard Real-Time Applications on...

Temporal Isolation of Hard Real-Time Applications on Many-core Processors Quentin Perret, Pascal Maurère, Eric Noulard, Claire Pagetti, Pascal Sainrat, Benoit Triquet

RTAS - April 2016 Quentin Perret - Temporal Isolation of Hard Real-Time Applications on Many-core Processors

RTAS April 12th, 2016 - Vienna

© AIRBUS Operations S.A.S. All rights reserved. Confidential and proprietary document.

Outline


How to build manycore-based avionics ?

1 / 38


Outline

• Certification constraints • Predictability, WCET estimation • Mixed-criticality, non propagation of faults



1 / 38


Outline


• Industrial constraints • Decoupled development, test and integration of systems • Incremental certification



1 / 38


Outline


• Industrial constraints • Decoupled development, test and integration of systems • Incremental certification

[1] Quentin Perret, Pascal Maurère, Eric Noulard, Claire Pagetti, Pascal Sainrat and Benoit Triquet « Predictable composition of memory accesses on many-core processors » in ERTS’16



1 / 38


Why the Kalray MPPA ?

• Massively parallel computation power • 288 cores

• Low power

• Good temporal properties

• K1 cores are fully timing compositional

• Demonstrated by AbsInt in the context of the CERTAINTY project


2 / 38


Why the Kalray MPPA ?

• Massively parallel computation power • 288 cores

• Low power

• Good temporal properties

• K1 cores are fully timing compositional

• Demonstrated by AbsInt in the context of the CERTAINTY project


Interesting candidate for future avionics computers

2 / 38


Kalray MPPA: overview

• Version : Andey

• 256 (+32) cores

• 16 Compute clusters

• 4 I/O clusters

• 2 Networks on Chip


Fig: Overview of the Kalray MPPA

3 / 38


Accesses to the external memory


Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Compute cluster B

Compute cluster A

Core 2

Core 1

DMA 1

DMA 2

R

R

NoC

DDR Controller

A

I/O cluster

Bank

1

Bank

2

Bank

3

Bank

4

Bank

5

Bank

6

Bank

7

Bank

8

DDRx-SDRAM

4 / 38




Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Compute cluster B

Compute cluster A

Core 2

Core 1

DMA 1

DMA 2

R

R

NoC

DDR Controller

A

I/O cluster

Bank

1

Bank

2

Bank

3

Bank

4

Bank

5

Bank

6

Bank

7

Bank

8

DDRx-SDRAM

Data 1

Destination 4 / 38




Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Compute cluster B

Compute cluster A

Core 2

Core 1

DMA 1

DMA 2

R

R

NoC

DDR Controller

A

I/O cluster

Bank

1

Bank

2

Bank

3

Bank

4

Bank

5

Bank

6

Bank

7

Bank

8

DDRx-SDRAM

Data 1

?

?

?

Destination 4 / 38




Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Compute cluster B

Compute cluster A

Core 2

Core 1

DMA 1

DMA 2

R

R

NoC

DDR Controller

A

I/O cluster

Bank

1

Bank

2

Bank

3

Bank

4

Bank

5

Bank

6

Bank

7

Bank

8

DDRx-SDRAM Data 1

Destination

?

?

?

4 / 38




Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Compute cluster B

Compute cluster A

Core 2

Core 1

DMA 1

DMA 2

R

R

NoC

DDR Controller

A

I/O cluster

Bank

1

Bank

2

Bank

3

Bank

4

Bank

5

Bank

6

Bank

7

Bank

8

DDRx-SDRAM

?

?

?

Destination

Data 1

4 / 38




Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Compute cluster B

Compute cluster A

Core 2

Core 1

DMA 1

DMA 2

R

R

NoC

DDR Controller

A

I/O cluster

Bank

1

Bank

2

Bank

3

Bank

4

Bank

5

Bank

6

Bank

7

Bank

8

DDRx-SDRAM

Dat

a 1

Destination 4 / 38


Workflow


Partitions Partitions Partitions Partitions

5 / 38


Workflow



Partitions Partitions Partitions Resource budget

Partitions Partitions Partitions Budget

5 / 38


Workflow





Architecture

Describe

5 / 38


Workflow





Architecture

Describe

Exec. model

5 / 38


Workflow





Architecture

Describe

Allocate Exec. model

5 / 38


Workflow




Partitions Partitions Partitions HW config.


Architecture

Describe


5 / 38


Workflow






Architecture

Describe


Hypervisor

5 / 38


Workflow






Architecture

Describe


Compile Hypervisor

5 / 38


Workflow


Executable





Architecture

Describe


Compile Hypervisor

5 / 38


Workflow


Executable





Architecture

Describe


Compile Hypervisor

Mastering execution

Budgeting

5 / 38


Agenda

I. Execution model

II. Implementation

III. Experiments

IV. Conclusion


6 / 38


Execution model

RULE 1.

In a compute cluster, each local SRAM bank and each

core can be used by at most one partition.


7 / 38


Rule 1


Bank 4

Bank 3

Bank 2

Bank 1

Core 4

Core 3

Core 2

A

A

A

A

Core 1

8 / 38


Rule 1


Bank 4

Bank 3

Bank 2

Bank 1

Core 4

Core 3

Core 2

A

A

A

A

Core 1

Par

titio

n 1

Par

titio

n 2

8 / 38


Rule 1


Bank 4

Bank 3

Bank 2

Bank 1

Core 4

Core 3

Core 2

A

A

A

A

Core 1

Par

titio

n 1

Par

titio

n 2

DMA

8 / 38


Rule 1


Bank 4

Bank 3

Bank 2

Bank 1

Core 4

Core 3

Core 2

A

A

A

A

Core 1

Par

titio

n 1

Par

titio

n 2

DMA

RM

8 / 38


Execution model

RULE 2.

NoC transactions are performed during strictly periodic

slots defined off-line. Two slots sharing a NoC resource

must not overlap.


9 / 38


Rule 2


R R R R

R R R R

R R R R

R R R R

10 / 38


Rule 2


R R R R

R R R R

R R R R

R R R R

A

B

10 / 38


Rule 2


R R R R

R R R R

R R R R

R R R R

A

B

D

C

10 / 38


Rule 2


R R R R

R R R R

R R R R

R R R R

A

B

D

C

Lx

10 / 38


Rule 2


R R R R

R R R R

R R R R

R R R R

A

B

D

C

A to

B

t

Lx

Using temporal offset :

10 / 38


Rule 2


R R R R

R R R R

R R R R

R R R R

A

B

D

C

A to

B

C to

D

t

t

Lx


10 / 38


Rule 2


R R R R

R R R R

R R R R

R R R R

A

B

D

C

A to

B

C to

D

t

t

Lx

Offset


10 / 38


Rule 2


R R R R

R R R R

R R R R

R R R R

A

B

D

C

A to

B

C to

D

t

t

t

Lx

Lx

Offset


10 / 38


Rule 2


R R R R

R R R R

R R R R

R R R R

A

B

D

C

Lx

Different routes.

10 / 38


Rule 2


R R R R

R R R R

R R R R

R R R R

A

B

D

C

Different routes.

10 / 38


Execution model

RULE 3.

All the buffers sent during the strictly periodic

NoC slots must be defined off-line.


11 / 38


Rule 3

Not required to achieve composability but:

• It simplifies the implementation of the execution model

• It reduces the combinatorial explosion of WCET estimation

through static analysis

• It eases the memory coverage of measurement based

validations


12 / 38


Execution model

RULE 4.

A bank of the DDR-SDRAM can be used by several

partitions if and only if they never access it simultaneously.


13 / 38


Rule 4


R R R R

R R R R

…

DDR Controller

Ban

k 1

Ban

k 2

Ban

k 3

Ban

k 4

Ban

k 5

Ban

k 6

Ban

k 7

Ban

k 8

14 / 38


Rule 4


R R R R

R R R R

…

DDR Controller

Ban

k 1

Ban

k 2

Ban

k 3

Ban

k 4

Ban

k 5

Ban

k 6

Ban

k 7

Ban

k 8

A B C

14 / 38


Rule 4


R R R R

R R R R

…

DDR Controller

Ban

k 1

Ban

k 2

Ban

k 3

Ban

k 4

Ban

k 5

Ban

k 6

Ban

k 7

Ban

k 8

A B C

A B C

14 / 38


Rule 4


R R R R

R R R R

…

DDR Controller

Ban

k 1

Ban

k 2

Ban

k 3

Ban

k 4

Ban

k 5

Ban

k 6

Ban

k 7

Ban

k 8

A B C

A B C A

14 / 38


Rule 4


R R R R

R R R R

…

DDR Controller

Ban

k 1

Ban

k 2

Ban

k 3

Ban

k 4

Ban

k 5

Ban

k 6

Ban

k 7

Ban

k 8

A B C

A B C A

B

14 / 38


Rule 4


R R R R

R R R R

…

DDR Controller

Ban

k 1

Ban

k 2

Ban

k 3

Ban

k 4

Ban

k 5

Ban

k 6

Ban

k 7

Ban

k 8

A B C

A B C A

B

C

14 / 38


Rule 4


R R R R

R R R R

…

DDR Controller

Ban

k 1

Ban

k 2

Ban

k 3

Ban

k 4

Ban

k 5

Ban

k 6

Ban

k 7

Ban

k 8

A B C

A B C A

B

C

A+B

+C

14 / 38


Execution model

The big picture:

1. Spatial partitioning inside compute clusters

2. TDM scheduling of the NoC

3. Buffers sent defined off-line

4. No conflict at DDR-SDRAM bank level


15 / 38


Resource budget

Relies on 3 notions:

1. A Partition Node (or PN): processing ressources • Number of cores • Number of local SRAM banks

2. An I/O Node (or ION): transactions with I/O peripherals • A (part of a) core on an IO cluster

3. A Partition Communication (or PC): communications between PN / ION • Source and destination PN / ION • Period & duration of strictly periodic slot


16 / 38


Agenda

I. Execution model

II. Implementation

III. Experiments

IV. Conclusion


17 / 38


Hypervisor (or HV)

• Executed by RM on each cluster

• Private local SRAM bank (HV code + scheduling tables)

• Applies the hardware configurations

• Manages the DMA

• Enforces respect of the TDM NoC schedule

• Global notion of time is easy to obtain on the MPPA


18 / 38


Systick

• HVs of all cluster periodically activated simultaneously

• Period of hypervisor denoted as Systick

• Lower bound on Systick is the WCET of HV

• Start & end of both Tx and Rx slots simultaneously

• With measures : WCET(HV) < 5µs

• Should be easy & efficient with static analysis


19 / 38


Rule 1 : spatial partioning in compute clusters

• Local SRAM addressing is configured to non-interleaved mode

• PEs set-up a static MMU config. during boot while in privileged mode

• Once booted, PEs run user application in user mode only

• Handlers are fixed by HV

• Syscalls are ignored

• Cores shutdown on traps


20 / 38


Rules 2 & 3 : TDM scheduling of the NoC & static buffers


𝑃𝑃1

𝑃𝑃2

𝐵𝐴

𝐵𝐵

𝐵𝐶

𝐵𝐷

𝐵𝐴 𝐵𝐴 𝐵𝐷 𝐵𝐷 𝐵𝐵 𝐵𝐵 𝐵𝐶

𝑃𝑃1 𝑃𝑃1 𝑃𝑃1 𝑃𝑃1 𝑃𝑃1 𝑃𝑃2 𝑃𝑃2

𝑇𝐻 𝑇𝐻

𝑻𝒊 𝑪𝒊 𝑶𝒊 𝑷𝑪𝟏 6 2 0

𝑷𝑪𝟐 12 3 3

State

DMA

Buffers lists: • 𝑃𝑃1 ∶ { 𝐵𝐴,𝐵𝐵 ,𝐵𝐶 } • 𝑃𝑃2 ∶ { 𝐵𝐷 }

PC parameters: Scheduling table:

State 𝑷𝑪𝟏 Idle 𝑷𝑪𝟐 𝑷𝑪𝟏 Idle

Duration 2 1 3 2 4

21 / 38




𝑃𝑃1

𝑃𝑃2

𝐵𝐴

𝐵𝐵

𝐵𝐶

𝐵𝐷



𝑇𝐻


𝑷𝑪𝟐 12 3 3

State

DMA




Duration 2 1 3 2 4

𝑻𝑯 = lcm 𝑻𝟏,𝑻𝟐

𝑇𝐻

21 / 38




𝑃𝑃1

𝑃𝑃2

𝐵𝐴

𝐵𝐵

𝐵𝐶

𝐵𝐷



𝑇𝐻 𝑇𝐻


𝑷𝑪𝟐 12 3 3

State

DMA




Duration 2 1 3 2 4

21 / 38


Rule 4: no interferences at DDR-SDRAM level

• Avoid interferences by design

• Additional constraint when computing the NoC schedule

• 2 PCs targeting the same bank (on the same DDR controller)

cannot overlap in time.


22 / 38


Agenda

I. Execution model

II. Implementation

III. Experiments

IV. Conclusion


23 / 38


The ROSACE case study [2]


engine

elevator aircraft_dynamics

Vz_filter

az_filter

h_filter

q_filter

Va_filter

altitude_hold Vz_control

Va_control

Environment simulation

Controller

200Hz

200Hz

200Hz

100Hz

100Hz

100Hz

100Hz

100Hz

h_c

Va_c

10Hz

10Hz

50Hz 50Hz

50Hz

[2] Claire Pagetti, David Saussié, Romain Gratia, Eric Noulard and Pierre Siron « The ROSACE case study: From Simulink specification to multi/many-core execution » in RTAS’14

24 / 38


Budget of ROSACE’s partition

• 2 PNs • PN 1 • whole ROSACE application • 5 cores / 3 local SRAM banks

• PN 2 • Set Point Generator (or SPG) • 1 core / 1 local SRAM bank

• 1 ION

• 3 PCs


IO1

C0 C2

SPG ROSACE

DDR-SDRAM

𝑷𝑪𝟏 / 10Hz

𝑷𝑪𝟐 / 200Hz

𝑷𝑪 𝟑

/ 20

0Hz

25 / 38


Concurrent application: ImInv

• Multithreaded image inversion algorithm

• Periodically receives a picture

• Periodically sends it’s inverse

• Configurable

• Memory / NoC utilization depend on picture size

• 512x512, 256x256 or 128x128

• 2, 4, 8 or 16 PEs

• Simple to implement


26 / 38


Experiments

SCENARIO 1.

Reference scenario, ROSACE in isolation.


27 / 38


Scenario 1

• 10,000 executions

• Validation of functional behaviour

• Log of execution times


IO1

C0 C2

SPG ROSACE

DDR-SDRAM

𝑷𝑪𝟏 / 10Hz

𝑷𝑪𝟐 / 200Hz

𝑷𝑪 𝟑

/ 20

0Hz

28 / 38


Experiments

SCENARIO 2.

Interferences on the NoC.


29 / 38


Scenario 2

• 1st partition: same as in scenario 1


IO1

C0 C2

SPG ROSACE

DDR-SDRAM

𝑷𝑪𝟏 / 10Hz

𝑷𝑪𝟐 / 200Hz

𝑷𝑪 𝟑

/200

Hz

30 / 38


Scenario 2


• 2nd partition:

• 2 instances of ImInv (I1 and I2)

• Each instance in one PN (8 cores, 7 banks each)


IO1

C0 C2

SPG ROSACE

DDR-SDRAM

𝑷𝑪𝟏 / 10Hz

𝑷𝑪𝟐 / 200Hz

𝑷𝑪 𝟑

/200

Hz

C3 C4

I1 I2

30 / 38


Scenario 2


• 2nd partition:

• 2 instances of ImInv (I1 and I2)

• Each instance in one PN (8 cores, 7 banks each)

• 2 PCs to connect the PNs (500 Hz)


IO1

C0 C2

SPG ROSACE

DDR-SDRAM

𝑷𝑪𝟏 / 10Hz

𝑷𝑪𝟐 / 200Hz

𝑷𝑪 𝟑

/200

Hz

C3 C4

I1 I2

𝑷𝑪𝟑 / 500Hz

𝑷𝑪𝟒 / 500Hz

30 / 38


Scenario 2

• Almost no variations in execution times

• Observed WCET and BCET are equal

• Differences of ET averages < 1%

• 100% of the NoC packets received within the intended NoC slots

• No delay related to « lost » packets


31 / 38


Experiments

SCENARIO 3.

Interferences at DDR-SDRAM level.


32 / 38


Scenario 3



IO1

C0 C2

SPG ROSACE

DDR-SDRAM

𝑷𝑪𝟏 / 10Hz

𝑷𝑪𝟐 / 200Hz 𝑷𝑪 𝟑

/ 200

Hz

33 / 38


Scenario 3


• 2nd partition:

• 1 instance of ImInv

• 1 PN (same as in scenario 2)


IO1

C0 C2

SPG ROSACE

DDR-SDRAM

𝑷𝑪𝟏 / 10Hz

𝑷𝑪𝟐 / 200Hz 𝑷𝑪 𝟑

/ 200

Hz

C4

I1

33 / 38


Scenario 3


• 2nd partition:



• 1 ION (same bank as partition 1)


IO1

C0 C2

SPG ROSACE

DDR-SDRAM

𝑷𝑪𝟏 / 10Hz

𝑷𝑪𝟐 / 200Hz 𝑷𝑪 𝟑

/ 200

Hz

C4

I1

IO2

33 / 38


Scenario 3


• 2nd partition:



• 1 ION (same bank as partition 1)

• 2 PCs


IO1

C0 C2

SPG ROSACE

DDR-SDRAM

𝑷𝑪𝟏 / 10Hz

𝑷𝑪𝟐 / 200Hz 𝑷𝑪 𝟑

/ 200

Hz

C4

I1

IO2

𝑷𝑪𝟑 / 500H

z 𝑷𝑪 𝟒

/ 50

0Hz

33 / 38


Scenario 3

• Almost no variations in execution times

• Observed WCET and BCET are equal

• Differences of ET averages < 1%

• 100% of the NoC packets received within the intended NoC slots

• No delay related to « lost » packets

• Observation of DDR traces show no interleaved accesses to the

shared bank


34 / 38


Agenda

I. Execution model

II. Implementation

III. Experiments

IV. Conclusion


35 / 38


Conclusion

• Many-core processors such as the Kalray MPPA are good

candidates for future avionics computers.

• The time-composability on such platforms can be achieved thanks

to an appropriate execution model.

• Implementation of the execution model is feasible and the

experiments exhibit promising temporal properties.


36 / 38


Perspectives

• Online management of best effort NoC traffic to fill the partially

unused slots

• Automatic budgeting seems feasible for instrinsically cyclic

applications (e.g., control-command)

• Evaluate the approach with applications of industrial size


37 / 38



© AIRBUS Operations S.A.S. All rights reserved. Confidential and proprietary document. This document and all information contained herein is the sole property of AIRBUS Operations S.A.S. No intellectual property rights are granted by the delivery of this document or the disclosure of its content. This document shall not be reproduced or disclosed to a third party without the express written consent of AIRBUS Operations S.A.S. This document and its content shall not be used for any purpose other than that for which it is supplied. The statements made herein do not constitute an offer. They are based on the mentioned assumptions and are expressed in good faith. Where the supporting grounds for these statements are not shown, AIRBUS Operations S.A.S will be pleased to explain the basis thereof. AIRBUS, its logo, A300, A310, A318, A319, A320, A321, A330, A340, A350, A380, A400M are registered trademarks.

Thank you

Any questions ?

38 / 38


Kalray MPPA: the NoC

• Data-NoC (DNoC)

• 2D torus

• Wormhole switching

• Packets sent by DMAs

• Control-NoC (CNoC)

• Short control messages

• Synchronization …


Fig: NoC topology of the Kalray MPPA


Kalray MPPA: Compute clusters

• 16 Processing Elements

• 1 Resource Manager (RM)

• 2 MiB of SRAM

• 16 banks

• accessible in parallel

• 1 DMA

• 2 NoC interfaces


PE0 PE1 PE4 PE5

PE2 PE3 PE6 PE7

PE8 PE9 PE12 PE13

PE10 PE11 PE14 PE15

Bank 0

Bank 1

Bank 2

Bank 3

Bank 4

Bank 5

Bank 6

Bank 7

Bank 8

Bank 9

Bank 10

Bank 11

Bank 12

Bank 13

Bank 14

Bank 15

Resource Manager DMA

SRAM SRAM Fig: Compute cluster Kalray MPPA


Kalray MPPA: I/O Clusters

• 4 Resource Managers

• 4x2 NoC interfaces

• 4 DMAs

• DDR/ETH controllers

• Other I/Os

• PCIe

• i2c

• GPIOs

• …


RM 0

DMA 0

RM 1 RM 2 RM 3

DMA 1 DMA 2 DMA 3

DDR Controller PCIe

GPIO / I2C /

UART / … …

Fig: I/O cluster Kalray MPPA




Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Compute cluster B

Compute cluster A

Core 2

Core 1

DMA 1

DMA 2

R

R

NoC

DDR Controller

A

I/O Cluster

Bank

1

Bank

2

Bank

3

Bank

4

Bank

5

Bank

6

Bank

7

Bank

8

DDRx-SDRAM

?

?

?

?




Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Compute cluster B

Compute cluster A

Core 2

Core 1

DMA 1

DMA 2

R

R

NoC

DDR Controller

A

I/O Cluster

Bank

1

Bank

2

Bank

3

Bank

4

Bank

5

Bank

6

Bank

7

Bank

8

DDRx-SDRAM

?

?

?

?

?

?

?




Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Bank 4

Bank 3

Bank 2

Bank 1

Core 3

Core 2

Core 1

DMA A

A

A

A

Local SRAM

R

Compute cluster B

Compute cluster A

Core 2

Core 1

DMA 1

DMA 2

R

R

NoC

DDR Controller

A

I/O Cluster

Bank

1

Bank

2

Bank

3

Bank

4

Bank

5

Bank

6

Bank

7

Bank

8

DDRx-SDRAM

?

?

?

?

?

?

?

?

?


Means of observation

• Observation of time-triggered systems is often intrusive, and thus,

challenging

• Should be taken into account in the design

• Example: specific PC for logging

• Other means • Post processing of memory footprints (ex: dump of DDR-SDRAM)

• Non-intrusive on-line observation of DDR-SDRAM accesses with a DDR interposer / analyser

• Asynchronous Log to server using the C-NoC (with care)


Temporal Isolation of Hard Real-Time Applications on...

Documents

Transcript of Temporal Isolation of Hard Real-Time Applications on...