Multicore Expert Series: QorIQ Platform Data Path Acceleration … · 2016. 3. 12. · 10Gbps...

TM

Freescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire,

ColdFire+, C-Ware, the Energy Efficient Solutions logo, Kinetis, mobileGT, PowerQUICC,

Processor Expert, QorIQ, Qorivva, StarCore, Symphony and VortiQa are trademarks of

Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. Airfast, BeeKit, BeeStack,

CoreNet, Flexis, MagniV, MXC, Platform in a Package, QorIQ Qonverge, QUICC Engine,

Ready Play, SafeAssure, the SafeAssure logo, SMARTMOS, TurboLink, Vybrid and

Xtrinsic are trademarks of Freescale Semiconductor, Inc. All other product or service

names are the property of their respective owners. © 2012 Freescale Semiconductor, Inc.

.

Sam Siu

Systems and Applications Engineer

June 2012

2

Freescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, ColdFire+, C-Ware, the Energy Efficient Solutions logo, Kinetis,

mobileGT, PowerQUICC, Processor Expert, QorIQ, Qorivva, StarCore, Symphony and VortiQa are trademarks of Freescale Semiconductor, Inc.,

Reg. U.S. Pat. & Tm. Off. Airfast, BeeKit, BeeStack, CoreNet, Flexis, MagniV, MXC, Platform in a Package, QorIQ Qonverge, QUICC Engine,

Ready Play, SafeAssure, the SafeAssure logo, SMARTMOS, TurboLink, Vybrid and Xtrinsic are trademarks of Freescale Semiconductor, Inc. All

other product or service names are the property of their respective owners. © 2012 Freescale Semiconductor, Inc.

• DPAA

• BMan Enablement

• FMan Enablement

• QMan Enablement

• SEC Enablement

• PME Enablement

• DCE Enablement (T series only)

• RMan Enablement

• DCB Enablement (T series only)

• Conclusion

3






64-bit High

End

Up to 2.2 GHz

4 – 8 Cores

Up to 1.5 GHz

2 – 4 Cores

Up to 1.5 GHz

1 – 2 Cores

Up to 1.2 GHz

1 – 2 Cores

Up to 1 GHz

QorIQ P5 P5020, P5010

QorIQ P3 P3041

QorIQ P4 P4080, P4040

QorIQ P2 P2040, P2020,

P2010

QorIQ P1 P1010, P1011, P1012,

P1013, P1014, P1015,

P1016, P1017, P1020,

P1021, P1022, P1023,

P1024, P1025

Service Provider

Routers

Network Admission

Controls

Storage

Networks

Radio Network

Control

Serving Node

Router

Metro Carrier

Edge Router

IMS

Controller

Unified Threat

Mgmt

Base

Station

VoIP Carrier-Class

Media Gateway

Wireless

Media Gateway

Integrated

Services Router

Network Attached

Storage

Home Media

Hub

Converged Media

Gateway

SSL, IPSec,

Firewall

Access

Gateway

Switching

Enterprise

WAP

QorIQ T4 T4240, T4160

16 – 24 cores

Up to 2 GHz

QorIQ T5

QorIQ T3

QorIQ T2

Aerospace

& Defense

Metro Carrier

Edge Router

Media

Gateway

Service Provider

Routers

Storage

Networks

Network

Attached

Storage

Integrated

Services

Router

4






Hardware Accelerators

FMAN

Frame

Manager

50 Gbps aggregate Parse,

Classify, Distribute

BMAN

Buffer

Manager

64 buffer pools

QMAN

Queue

Manager

Up to 224 queues

RMAN

Rapid IO

Manager

Seamless mapping sRIO

to DPAA

SEC

Security

40Gbps: IPSec, SSL

Public Key 25K/s 1024b

RSA

PME

Pattern

Matching

10Gbps aggregate

DCE

Data

Compression

20Gbps aggregate

Saving CPU Cycles for higher value work

Compress and

Decompress

traffic across the

Internet

Protects against

internal and

external Internet

attacks

Frees CPU from

draining repetitive

RSA, VPN and

HTTPs traffic

Identifies traffic

and targets CPU

or accelerator

New Enhanced

Line rate

50Gbps

Networking

Quality of Service

for FCoE in

converged data

center networking

5






QorIQ Devices

DPAA Feature List

Revision Number

FMan QMan BMan SEC PME RMan DCE RE

P1023 4.0 2.0 2.0 4.2 n/a n/a n/a n/a

P4080 rev2 2.0 1.1 1.0 4.0 2.1 n/a n/a n/a

P2040, P2041

P3041,

P4040,

3.0 1.2 1.0 4.2 2.1 1.0 n/a n/a

P5020, P5010 3.0 1.2 1.0 4.2 2.1 1.0 n/a 1.0

T4240, T4216 6.1 3.1 2.1 5.0 2.1 1.0 1.0 n/a

• QorIQ P class devices has “Datapath Three Speed Ethernet Controller (dTSEC) and 10-Gigabit Ethernet Media Access Controller (10GEC)

• QorIQ T class devices has Ethernet Media Access Controller (EMAC)

6






• DPAA is design to balance the performance of Accelerators with seamless Integrations

− ANY packet to ANY core to ANY accelerator or network interface efficiently WITHOUT locks or semaphores.

• “Infrastructure” components

− Queue Manager (QMan)

− Buffer Manager (BMan)

• “Accelerator” Components

− Cores

− Frame Manager (FMan)

− RapidIO Message Manager (RMan)

− Cryptographic accelerator (SEC)

− Pattern matching engine (PME)

− Decompression/Compression Engine (DCE)

− DCB (Data Center Bridging)

− RAID Engine (RE)

• CoreNet

− Provides the interconnect between the cores and the DPAA infrastructure as well as access to memory.

D$ I$

D$ I$ L2$

e500mc Core

D$ I$

CoreNet™ Coherency Fabric

Buffer

Mgr

Frame Manager

1GE 1GE

1GE 1GE 10GE

D$ I$

D$ I$ L2$

e500mc Core

D$ I$

Queue

Manager

Sec 4.x PME 2

RMan RE

Parse, Classify, Distribute

Buffer

1/10G 1/10G 1G

1G

1G

1G

Frame Manager

1G

1G

D$ I$

D$ I$ L2$

e6500 Core

D$ I$

D$ I$

D$ I$ L2$

e6500

Core

D$ I$

DCE DCB

P Series T Series

… …

… …

7






• Buffer: Unit of contiguous memory, allocated by software

• Frame: Buffer(s) that hold a data element (generally a packet)

− Frames can be single buffers or multiple buffers (scatter/gather lists)

A “simple frame” has one delimited data element

A “multi buffer frame” has two or more data elements

• Frame Descriptor (FD): Proxy structure used to represent frames

• Frame Queue:

− FIFO of related Frames Descriptor.(e.g. TCP session)

− The basic queuing structure supported by QMan

Frame Queue Descriptor (FQD): Structure used to manage Frame Queues

Buffer

Buffer

Ethernet

Frame Pre-

amble

Dest

addr

Src

addr Type Data CRC

Buffer Buffer

FD

FD

FQD FD FD FD FD …

8






Length

Simple Frame Multi-buffer Frame

(Scatter/gather)

D PID

BPID

Address

Offset

Status/Cmd

000

Buffer

Frame Descriptor

D PID

BPID

Address

Offset

Length

Status/Cmd

100

Frame Descriptor

Address

Length

BPID

Offset

00

Address

Length

BPID

Offset (=0)

00

Address

Length

BPID

Offset (=0)

01

…

Data

Data

Data

S/G List

Packet

0 1 2 3 4 5 6 7 8 9 1

0

1

1

1

2

1

3

1

4

1

5

1

6

1

7

1

8

1

9

2

0

2

1

2

2

2

3

2

4

2

5

2

6

2

7

2

8

2

9

3

0

3

1

D

D

LIODN

offset

BPID ELIO

DN

offset

- - - - addr

addr (cont)

Fmt Offset Length

STATUS/CMD

9






01

Address

Length

BPID

Offset

10

Address

Length

BPID

Offset

00

Data

D PID

BPID

Address

CongWght

Status/Cmd

001

Compound

Frame Descriptor

Compound Frame

S/G List

Address

Length

BPID

Offset

00

Address

Length

BPID

Offset (=0)

00

Address

Length

BPID

Offset (=0)

01

Data

Data

Data

Output: SGL (or Buffer)

Input: Buffer (or SGT)

Frame Format Code

Member(Buffer/SG), Final

CongWght = Congestion Weight Data

Stream Configuration Frame

Address

Length

BPID

Offset

01

• Compound frames allows related data to be passed in a single unit to the DPAA Accelerators.

10






FQD Selected Field Description:

• FQD_LINK: Link to the next FQD in a queue of FQDs, used for Work Queues

• ORPRWS: ORP Restoration Window Size

• OA: ORP Auto Advance NESN Window Size

• ODP_SEQ: ODP Sequence Number

• ORP_NESN: ORP Next Expected Sequence Number.

• ORP_EA_HPTR, ORP_EA_TPTR: ORP Early Arrival Head and Tail Pointer

• PFDR_HPTR, PFDR_TPTR : PFDR Head and Tail Pointer

• CONTEXT_A, CONTEXT_B: Frame Queue Context A and B

• STATE: FQ State

• DEST_WQ: Destination Work Queue

• ICS_SURP: Intra-Class Scheduling Surplus or Deficit.

• IS: Intra-Class Scheduling Surplus or Deficit identifier

• ICS_CRED: Intra-Class Scheduling Credit

• CONG_ID: Congestion Group ID

• RA[1-2]_SFDR_PTR: SFDR Pointer for Recently Arrived frame # 1 and 2

• TD_MANT, TD_EXP : Tail Drop threshold Exponent and Mantissa

• C: FQD in external memory or in cache (Qman 1.1)

• X: XON or XOFF for flow control command (Qman1.1)

0 1 2 3 4 5 6 7 8 9 1

0

1

1

1

2

1

3

1

4

1

5

1

6

1

7

1

8

1

9

2

0

2

1

2

2

2

3

2

4

2

5

2

6

2

7

2

8

2

9

3

0

3

1

- - - - - - - - FQD_LINK

OR

PR

WS

OA

ODP_SEQ ORP_NESN

OL

WS

ORP_EA_HSEQ ORP_EA_TSEQ

ORP_EA_HPTR ORP_EA_TPTR

ORP_EA_TPTR PFDR_HPTR

(cont…) PFDR_TPTR

CONTEXT_A

(cont…)

CONTEXT_B

FQ_CTRL

FE

R

ST

A

TE

DEST_WQ

ICS_SURP

IS ICS_CRED

BYTE_CNT

CONG_ID FRM_CNT

NR

A

OA

C

C RA1_SFDR_PTR X

IT - - - RA2_SFDR_PTR

NO

D

OAL OD1_SFDR_PTR OAL OD2_SFDR_PTR

NP

C

- - - OD3_SFDR_PTR - - - TD_MANT TD_EXP

11






• AE: Frame Annotation Stash Exclusive.

• DE: Frame Data Stash Exclusive

• CE: FQ Context Stash Exclusive

− 0: Stash transaction issued as DIRECT0. PAMU translate this to LDEC

− 1: Stash transaction issued as DIRECT1. PAMU translate this to LDECPE/LDECFE.

• AS: Frame Annotation Stashing Size

• DS: Frame Data Stashing Size

• CS: FQ Context Stashing Size

− Number of 64 byte coherency granules (0, 1, 2, or 3) of Frame Annotation to be stashed.

• ADDR: FQ Context Address

− the first 64 byte coherency granule containing the FQ context information to be stashed.

0 1 2 3 4 5 6 7 8 9 1

0

1

1

1

2

1

3

1

4

1

5

1

6

1

7

1

8

1

9

2

0

2

1

2

2

2

3

2

4

2

5

2

6

2

7

2

8

2

9

3

0

3

1

- - - - -

AE

D

E

CE

- - AS DS CS - - - - - - - - ADDR

3

2

3

3

3

4

3

5

3

6

3

7

3

8

3

9

4

0

4

1

4

2

4

3

4

4

4

5

4

6

4

7

4

8

4

9

5

0

5

1

5

2

5

3

5

4

5

5

5

6

5

7

5

8

5

9

6

0

6

1

6

2

6

3

ADDR - - - - - -

12






Power Architecture™

Core

D-Cache I-Cache

L2 Cache


Core

D-Cache I-Cache

L2 Cache


Core

D-Cache I-Cache

L2 Cache

• FMan receives packets

− allocates internal buffers

− retrieves data from MAC

• BMI

− acquires a buffer from BMan

− uses DMA to store data in it

• Parse+classify+keygen select a queue and policer profile

• Policer “colors” and optionally discards frame

• QMan applies active queue management and enqueues frame

• Frame is enqueued to one of a pool of cores

• Available core dequeue FD for processing MAC

BMI

Parser

Classifier

Keygen

Policer

QMI

WRED

Enqueue

Dequeue

To

Memory

10GE GE GE GE GE

Frame Manager

(FMan) DMA

Policer Keygen

(Distribution)

Parser Classifier

QMI

BMI

Memory Buffer

Manager

Queue

Manager

D

WQ0

WQ1

WQ2

WQ3

WQ4

WQ5

WQ6

WQ7


Core

D-Cache I-Cache

L2 Cache

ENQ

FD

DEQ

Return

Buf Ptr

Request

Buffer

DDR

D

PKT

D

PKT

DDR

13






• DPAA

• BMan Enablement

• FMan Enablement

• QMan Enablement

• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

14






• Standardized command interface to SW and HW

− Up to 66 Software portals for software: resolves any Multi Core race scenario

− Up to 6 HW portal per HW block: simplified command for HW Accelerators

− Up to 64 separate pools of free buffers

• BMan keeps a small per-pool stockpile of buffer pointers in internal memory

− stockpile of 64 buffer pointers per pool, Maximum 2G buffer pointers

− Absorbs bursts of acquire/release commands without external memory access

− minimized access to memory for buffer pool management.

• Pools (buffer pointers) overflow into DRAM

• LIFO buffer allocation policy

− A released buffer is immediately used for receiving new data, using cache lines previously allocated

… … SW Portal HW Portal

System Memory

……

Per-Pool

Buffer Stacks

Central Service

Sequencer

On chip Memory

……

Per-Pool

Stockpile

Com

fort Z

one

16-5

5

63 D

eep

Fetch/Flush

Sequencer

Depletion

Threshold

Free Pool

Low Watermark

Power Architecture®

Core FMan, SEC, PME, etc

CoreNet

15






• Software portals have 2 components

− Management commands:

• Command Registers (BCSPi_CR) : acquire 1-8 buffers OR query availability

• Response Registers (BCSPi_RR0 / RR1): buffer address OR Buffer Pool Availability and Depletion state

− Buffer Release:

• Release Command Ring (RCR) Entry (BCSPi_RCRj): Circular FIFO

• Interrupts can be used to signal availability of space (in RCR) and that pools are depleted and require replenishment (RCR Interrupt Threshold Register)

Release

Command

Ring (RCR)

Core

BMan

PI

CI

acquire/

query

BMan

Command

Register

Interrupts

BMan

Response

Registers

Buffer Ptr/

BP Snapshot

Acquire Resp

VERB BPID

BUF0

BUF1

BUF2

BUF3

BUF4

BUF5

BUF6

BUF7

Bit 1-3: 1h

Bit 4-7: # of Buf Acquire Cmd

VERB

BPID

Bit 1-3: 1h(Acq)

Bit 4-7: # of Buf

Bit 2-7: Buf Pool ID

Release Cmd

VERB

BPID0 BUF0

BPID1 BUF1

BPID2 BUF2

BPID3 BUF3

BPID4 BUF4

BPID5 BUF5

BPID6 BUF6

BPID7 BUF7

Bit 1-3:

2h = Rel Buf to the pool

3h = Rel each B to the pool

Bit 4-7: # of Buf

16






0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Verb

X38

BPID

bp1

Buffer Pointer 0 - BPID

bp1

Buffer Pointer 1

- bp1 Buffer Pointer 2 - bp1 Buffer Pointer 3



0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Verb

x18

BPID

Bp1

Buffer Pointer 0 - - Buffer Pointer 1

- - Buffer Pointer 2 - - Buffer Pointer 3



• BMan Command Type

− BMan command registers (BCSPi_CR) are 64B long.

− Command Verb (1B) + Buffer Pool ID (1B)

Bit 1-3: Response Type. Valid encodings are:

• 001 = Acquire buffers (Acquire)

• 010= Release buffers to the pool identified in byte field 1 (Release)

• 011= Release each buffer to the pool identified in byte field immediately preceding its buffer field (Release)

• 100=Query buffer pool state, depletion and availability.

• 110= Invalid command (Response)

• 111= Stockpile ECC Error (Response)

Bit 4-7: Number of buffers associated with command type, maximum 8

• 0h = Zero buffers. 1h = One buffer .... 8h = Eight buffers

Returns up to eight 48bit buffer addresses

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Verb

x18

BPID

bp1

- - - - - - - - - - - - - -

- - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - -

17






BMan functions like paper tray in a photocopier.

Photo Copier BMan

Each tray holds same paper size Buffer Pool holds fixed size buffer pointers

Different trays for different paper type Different BPs for different sizes Buffers

Stock up papers that you need Configure a contiguous memory region

Initially fill up the paper tray Fill the stockpile with buffer pointers

Add paper when the tray runs low Refill/remove BPs when it runs low/high

Does not detect what is in the tray BMan does not check the size of buffer

18






• DPAA

• BMan Enablement

• FMan Enablement

• QMan Enablement

• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

19






• FMan combines the Ethernet network interfaces with packet distribution logic to provide intelligent distribution and queuing decisions for incoming traffic at line rate.

• FMan key new features for QorIQ T4 processors.

− Six 1G/2.5G multirate Ethernet MACs (mEMACs) per Frame Manager

− Two 10G multirate Ethernet MACs (mEMACs) per Frame Manager

− QMan interface: Supports priority based flow control message pass from Ethernet MAC to Qman

− Comply with IEEE 803.3az (Energy efficient ethernet) and IEEE 802.1QBbb, in addition of IEEE Std 802.3®, IEEE 802.3u, IEEE 802.3x, IEEE 802.3z, IEEE 802.3ac, IEEE 802.3ab, and IEEE-1588 v2 (clock synchronization over Ethernet)

− Port Virtualization: Virtual Storage profile (SPID) selection after classification or distribution function evaluation.

− Rx port multicast support.

− Egress Shaping.

− Offline port: able to copy the frame into new buffers and enqueue back to the QMan.

20






MAC Rx and validate

BMI streams and allocate internal buffer

for incoming frame + IC (Internal Context)

Calculate raw L4 checksum for parser

Based upon Layer-2 packet size, BMI requests

“right sized” buffer to BMan

BMI instructs DMA to transfer frame

to external buffer

Parse / Classify / Distribute

Determine queue ID#

Per-group policing(RFC2698/4115)

QMI instructs QMan to enqueues FD

BMI releases internal buffer on completion

QMan active queue mngt

and scheduling (WRED)

Core dequeue & processing

Rx

BMI instructs DMA to write

frame IC and header to ext. Buffer

Multiple

stages

FMan

Parser

Internal Ctx

Shared

Memory

BMI

QMI

1GE 10GE

DMA

Keygen

Classify/ Distrib

Policer

MACs

FPM BMan

QMan

BMI

BMI / BMan

BMI / DMA

PCD

Policer

BMI / DMA

QMI / BMI

QMan / BMan

QMan

Core / SW

MAC

FMC

10GE 1GE 1GE 1GE 1GE 1GE

QorIQ™ T4240

FD

BP

Frame

21






• Performs parsing of common L2/L3/L4 headers, including tunneled protocols

• Can be augmented by the user to parse other standard protocols

• Can also parse proprietary, user-defined headers at any layer:

− Self-describing, using standard fields such as proprietary Ethertype, Protocol ID, Next Header, etc.

− Non-Self-Describing through configuration.

• Parse results, including proprietary fields, can be used by the classifier, and/or software.

• Soft parse can modify any field in parse results

Ethernet

VLAN

VLAN

Tunneled

IPv4

MPLS

s=0

s=1*

s=1*

s=1*

IPSec

IPv6

PPPoE

IPv4

TCP/UDP

Other L3

PPP

Other L4

Tunneled

IPv6

Min

Encap

to

Other

L4

MPLS

s=0

MPLS

GRE

Other L2

22






B 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1x

Logic

al P

ort ID

Sh

imR

L2R L3R L4

R

Cla

ssific

atio

n

Pla

n ID

Nxt

Hdr

Running

Sum

Flags,

Flag

offset

Rout T

ype

RH

P

2x

Sh

im

Offs

et_

1

Shim

Offs

et_

2

IP P

id o

ffset

Eth

Offs

et

LLC

+S

nap

Offs

et

VL

AN

TC

I

offs

et_

1

VLA

NT

CI

offs

et_

n

LastE

TypeO

ffset

PP

PoE

Offs

et

MP

LS

Offs

et_

1

MP

LS

Offs

et_

n

IPO

ffset_

1

IPO

ffset_

n

GR

E

Offs

et

L4 O

ffset

NxtH

dr

offs

et

The parser stores all parsing results in the Parse Array located inside the parser.

Ethernet

VLAN

Incoming frame

Other L3

DA EthType

8100 VLAN ... TOS ... PID ... SIP DIP SPORT DPORT .... SA

EthType

0800

Parse Result Array (simplified)

EthType

<UDF>

UDF

Length

UDF

Content

compare ethtype to UDF

ethtype

read UDF length

write UDF result

write UDF offset

write parse continue offset

IPv4

TCP/UDP

23






• 256 Classification Plans

− Indicates which fields of a parsed packet are of interest for key generation

• 32 Key Generation schemes

− Direct Method

Used for port based or post coarse classification

− Indirect Method

Based on the parsed protocol stack of the frame and the source port

The presence (or absence) of valid headers can direct the scheme used

Build Key

Hash Key

Mask Bits

FQ Base

Addition

Logical ‘OR’

with Special

Fields

Parser Result/Frame

56 byte Key

8 byte Hash Key

24 bit field

24 bit FQID

Classification

Plan/LCV

Exact Key for

Table Lookup

24






• Up to three level tree search at line rate

− Up to 256 bytes per tree level

− Up to 512 bytes of total key data for the three level

• Each table is an ordered list of entries, returning the first match

− Keys for the initial table are generated from parse results/KeyGen

− Keys for subsequent tables are generated from parse and previous lookup results

− Keys can be generated from any fields in the frame, including proprietary UDF

− Each entry is individually maskable.

• Each table entry has Action description

− Queue ID, next table, hash and distribute, or Drop.

• Output of classifier

− Single queue

− Set of queues and a KeyGen scheme if distributing

− Policing Profile

• Policing Profile

• Packet Handling

• Drop

• FQ ID

• FQ generation

Key Action

FQ x

FQ y

Table x

Table y

Key Action

FQ x

FQ y

Table x

Table y

Key Action

FQ x

FQ y

Table q

Table r

KeyGen

Key Action

FQ x

FQ y

Table x

Table y

Key Action

FQ x

FQ y

Table x

Table y

Key Action

FQ x

FQ y

Drop

FQ z

25






Core

Core

Eth

IP Prot = SCTP

Classify on

IP TOS

Control Path

Pool of

cores

Ded

icate

d c

han

nel

Po

ol c

han

nel

IP Prot != SCTP

FQ

Data Path

Hash (n-tuple)

Classify on

(IP/UDP/…propr. Field)

26






Dedicated channel

WQ

0

WQ

1

WQ

2

WQ

3

WQ

4

WQ

5

WQ

6

WQ

7

Pool channel

WQ

0

WQ

1

WQ

2

WQ

3

WQ

4

WQ

5

WQ

6

WQ

7

Eth

L2 L3-IP

Prot TOS UDP

L4

SCTP (Control)

Policer

UDP (Dataplane)

Hash

n-tuple IP-UDP

TOS

portal

CORE

portal

CORE

portal

CORE

Combining exact match + Hash

27






• rootfs/etc/fmc/config/8c-128fq-p.xml

…

<distribution name="ipv4eth0">

<queue count="128" base="0x3800"/>

<key>

<fieldref name="ipv4.src"/>

<fieldref name="ipv4.dst"/>

<fieldref name="ipv4.tos"/>

</key>

</distribution>

<distribution name="ipv4eth1">

<queue count="128" base="0x3880"/>

<key>

<fieldref name="ipv4.src"/>

<fieldref name="ipv4.dst"/>

<fieldref name="ipv4.tos"/>

</key>

</distribution>

…

0 1 2 3 4 5 6 7 8 9 1

0

1

1

1

2

1

3

1

4

1

5

1

6

1

7

1

8

1

9

2

0

2

1

2

2

2

3

2

4

2

5

2

6

2

7

2

8

2

9

3

0

3

1

Version HdrLen TOS Length

Identification Flag

s

Offset

TTL Protocol Checksum

Source Address

Destination Address

Options + Padding

Data

28






• DPAA

• BMan Enablement

• FMan Enablement

• QMan Enablement

− QMan Building Blocks

− QMan Functions

− QMan Scheduling

− Order Restoration, Order Preservation and Atomicity

− Congestion Management and Avoidance

• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

29






• T4xxx has a total of 50 Software Portals (SP), increase from 10 SP found in the P class processors.

• Supports Customer Edge Egress Traffic Management (CEETM) that provides hierarchical class based scheduling and traffic shaping:

− Available as an alternate to FQ/WQ scheduling mode on the egress side of specific direct connect portals

− Enhanced class based scheduling supporting 16 class queues per channel

− Token bucket based dual rate shaping representing Committed Rate (CR) and Excess Rate (ER)

− Congestion avoidance mechanism equivalent to that provided by FQ congestion groups

• A total of 48 algorithmic sequencers are provided, allowing multiple enqueue/dequeue operations to execute simultaneously.

• Support up to 295M enqueue/dequeue operations per second.

30






FD

PID

BPID

ADDR

FORMAT

LENGTH

• Frame Descriptor (FD)

− The basic queue Element that describe a frame

− Usually a single IO packet will use a single frame

− Other scenarios: commands with no buffer

• Frame Queue Descriptor (FQD)

− A linked list of FD’s

− Usually a frame queue is associated with a flow or interface

− Enqueue operation must include the target FQ as a parameter

− Dequeue operation may use FQ as a parameter for operation

− Head of frame queue can be associated to ODP

• Work Queue Structure

− Linked list of FQD

− Hold flows of the same priority and designation

− Dequeue operation may use WQ as a parameter for operation

• Channel

− Set of eight WQ channel served by a single type of entity

− Dequeue from channel can be configured to be:

strict priority

round robin (Simple, Weighted or Deficit)

− Dequeue may use channel as a parameter for operation

FD

PID

BPID

ADDR

FORMAT

LENGTH

Frame

FD

PID

BPID

ADDR

FORMAT

LENGTH

FD

PID

BPID

ADDR

FORMAT

LENGTH

FD

PID

BPID

ADDR

FORMAT

LENGTH

WQ

FQD FQD FQD

FD/Frame FD/Frame FD/Frame

Frame

B

… Frame

B

WQ0

WQ1

WQ2

WQ3

WQ4

WQ5

WQ6

WQ7

Channel

B

FQD

OD1_SFDR_PTR

OD2_SFDR_PTR

OD3_SFDR_PTR

PFDR_HPTR

PFDR_TPTR

RA1_SFDR_PTR

RA2_SFDR_PTR

…

Context A/B

…

192.168.1.1:80 TCP Flow 10.10.10.100:16734

FD

PID

BPID

ADDR

FORMAT

LENGTH

31






• Portals are the interface between QMan and the accelerator which use them

− Direct Connect Portals has direct connect signals to Dedicated Channel

Dedicated Channels are always serviced by a single entity, e.g. FMan, etc.

− Software Portals use CoreNet as the physical interconnect to the processor core.

Each Software Portal serves a Dedicated Channel, and optionally services Pool Channels.

Software and QMan interact by “reading” and “writing” data across CoreNet

• Each channel consists of 8 WQs, and thus there are 8 possible priorities.

• QMan contains a total of 110 channels for T4240

− 16 Dedicated Channels (Sub Portals) per Frame Manager’s Direct Connect Portal

− 1 for SEC (8 SPs), 1 for PME, 1 for DCE, 1 for RMan

− Up to 50 CoreNet dedicated channels for software portals

− 15 for CoreNet pool channels which are shared by all software portals

Channel

WQ WQ WQ WQ WQ WQ WQ WQ

Software Portal

Channel


Channel WQ WQ WQ WQ WQ WQ WQ WQ

Pool

Channel Dedicated

Channel

Dedicated

Channel

Software Portal Direct Connect Portal

Channel


Dedicated

Channel

e6500 e6500 FMan/SEC/PME

WQ

Software Portal

32






• DPAA

• BMan Enablement

• FMan Enablement

• QMan Enablement


− QMan Functions

− QMan Scheduling



• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

33






• QMan provides a way to inter-connect DPAA components

− Cores (including IPC)

− Hardware offload accelerators

− Network interfaces – Frame Manager

• Queue management

− High performance interfaces (“portals”) for enqueue/dequeue

− Internal buffering of queue/frame data to enhance performance

• Congestion avoidance and management

− RED/WRED

− Tail drop for single queues and aggregates of queues

− Congestion notification for “loss-less” flow control

• Load spreading across processing engines (cores, HW accelerators)

− Order restoration

− Order preservation/atomicity

• Delivery to cache/HW accelerators of per queue context information with the data (Frames)

− This is an important offload for software using hardware accelerators

FQD

Cache

Queue Manager

(QMan)

…

FMan

FMan

SEC

PME …

…

…

…

…

FD

Memory

Queuing

Engines

Software Portals

…

CoreNet To Cores

Dire

ct C

on

ne

ct p

orta

ls

DCE

RMan

34






• Software portals have 4 components

− Enqueue: EQCR

− Dequeue: Command registers + DQRR

− Messages: MR (e.g. enqueue rejections)

− Management commands: command/response registers

• Interrupts can be used to signal availability of data or space (in EQCR)

• Rings provide finite size FIFOs

− Up to 16 entries for DQRR, 8 entries for EQCR and MR

• Portal components are implemented inside QMan to reduce access latency

− Unlike traditional BD rings which are in “memory” and “registers”

• QMan can “push” (stash) DQRR entries across CoreNet into the appropriate core’s cache

• PI and CI are the basic mechanisms used with rings but other forms of notification of data availability and data consumption are supported

• When these other mechanisms are used QMan maintains PI/CI

Dequeue

Response Ring

(DQRR)

Enqueue

Command Ring

(EQCR)

Dequeue

Commands

Core

QMan

DECR_PI

DE

CR

_C

I

EQ

CR

_P

I

EQCR_CI

Management

Command

(CR)

Message Ring

(MR)

PI

CI

Dequeue Interface

Interrupts

Enqueue Interface

Response

Registers

(RR0/1)

Response

verb status

SEQ number

FQID

CNTXTB

FD

ENQ CMD

Cmd verb

…

Seq # ORP

FQID + TAG

FD

FQ Init

Cmd Verb

WE_MASK

FQID Count

ORPC CGID

DEST_WQ

……

35






• DCA is a form of “indirect” Consumer Index (CI) update

• DCA is targeted at forwarding applications where most frames which are received are then transmitted

• Each entry in the TX ring (EQCR) can contain a DCA acknowledge for the corresponding entry on the RX (DQRR) ring via the QCSPi_DQRR_DCAP register, or by adding an embedded DCA to enqueue commands issued in the EQCR

Dequeue Response

Ring (DQRR)

… PI

CI

Enqueue Command

Ring (EQCR)

PI

CI

0 1 2 3 4 5 6 7 8 9 1

0

1

1

1

2

1

3

1

4

1

5

1

6

1

7

1

8

1

9

2

0

2

1

2

2

2

3

2

4

2

5

2

6

2

7

2

8

2

9

3

0

3

1

VE

RB

D

CA

SE

QN

UM

- ORP - FQID TAG FD

3

2

3

3

3

4

3

5

3

6

3

7

3

8

3

9

4

0

4

1

4

2

4

3

4

4

4

5

4

6

4

7

4

8

4

9

5

0

5

1

5

2

5

3

5

4

5

5

5

6

5

7

5

8

5

9

6

0

6

1

6

2

6

3

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

0 1 2 3 4 5 6 7 8 9 1

0

1

1

1

2

1

3

1

4

1

5

1

6

1

7

1

8

1

9

2

0

2

1

2

2

2

3

2

4

2

5

2

6

2

7

2

8

2

9

3

0

3

1

CI_VECTOR - - - - - - - S -

PK

- - DCAP_

CI

If E

an

ble

d

QCSPi_DQRR_DCAP Register

36






• In addition to stashing DQRR entries into cache, QMan’s software portals can also “warm” a core’s (L1 or L2) cache with frame and queue related data

− Actual frame data for single buffer frames

− Scatter gather list for multi-buffer frames

− Frame “annotations”

Data between Address and Offset at start of frame

Used to pass additional information about the frame which is not “frame data” e.g. FM parse results

− Data referenced by FQ Context

• Stashing options can be configured on a per FQ basis

• Cache warming are performed at the time that the frame is dequeued (i.e., DQRR entry is created)

37






• DPAA Overview

• BMan Enablement

• FMan Enablement

• QMan Enablement


− QMan Functions

− QMan Scheduling



• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

38






FQ FQ FQ

FQ FQ

• Static flow-based distribution (Dedicated Channel)

− Set of WQs with different FQs directed statically to different cores

− Distribution of frames (selection of FQ) is based on hash keys, ensuring that packets from the same traffic flow will always go to the same cores

− Static not dynamic, doesn't react to core load, assign work to the cores in a static or fixed manner

• Adaptive load balancing (Pool Channel)

− Load spread the packets (or the Frame Queue) to the cores based on actual core availability/readiness

− QMan provides two mechanisms to deal with out of order packets:

Order preservation: ensure that related packets are processed in order (and typically one at a time); can also provide “atomicity” – atomic access to data

Order restoration: allows frames to be processed out of order and then restores their order later on before they are transmitted

Channel


FQ

Software Portal

FQ FQ

Channel


FQ FQ

Channel WQ WQ WQ WQ WQ WQ WQ WQ

FQ FQ FQ

Pool

Channel Dedicated

Channel

Dedicated

Channel

Software Portal

39






0 Out of Service State:

− FQ ready for Initialization to a Park state.

− Software can transform Retired FQ to Out of Service.

1 Retired State:

− This state is a precursor to the Out of Service State.

− Placed in this state upon command at the earliest possible opportunity.

2 Tentatively Scheduled State

− consumer has given scheduling control of the FQ to the QMan.

− the eligibility criteria is not currently met.

3 Truly Scheduled State

− eligibility criteria is currently met

− FQ is queued in a WQ waiting to become the active or held active.

4 Parked State:

− The consumer (rather than the QMan) has scheduling control of the FQ.

− Producers may enqueue to FQs in this state.

5 Active State:

− FQ is temporarily coupled to a single portal.

− Consumers may dequeue and Producers may also enqueue to this FQ.

Held Active State:

− FQ cannot be scheduled or consumed via any other portal.

− Consumers may dequeue and Producers may also enqueue to this FQ.

− Released from the portal after all frames have been consumed.

Held Suspended State:

− A FQ moves to this state when it is no longer eligible for dequeue.

− Consumers cannot dequeue. Producers may still enqueue to it.

Initia

lization

SW

Schedule

FQ

T

ie to P

ort

al

SW Portal Dequeue Dispatcher

Channel S

chedule

r

Active FQ SDQCR

Select Channel

With Priority Word

High 0

High 1

Elevated Low

Medium

Low

RR

RR

RR

RR

RR

Priority

Compare

40






• QMan supports 2 modes on software portals:

− Push Mode:

Qman continues to push entries into DQRR in attempt to keep it “full”

QMan provides 2 command registers

• One register is “static” and QMan repeatedly executes this command

• One register is “volatile” and QMan executes that command a limited number of times

Push mode is “just like” a BD ring

− Pull Mode:

QMan provides a single command register

Software must issue a new command for each dequeue operation

• Push mode is the most common mode

• Pull mode offers more control to applications

41






• Class Scheduler schedules WQ

− A Class scheduler per channel

− Two levels of scheduling:

Use Weighted Interleaved Round Robin to schedule within the medium priority queues (2 to 4) and low priority queues (5 to 7)

Strict priority of all 8 WQs with programmable elevation (CS_ELEV) of the low priority tier over the medium priority tier

− Maintain active FQ states transition

Keeps track of the last RR winner, selection counters, elevation pending counter

• Intra-Class scheduling schedules FQ

− Schedule a frame queue within a work queue

− Use Modified Deficit Round Robin with ICS_CRED + ICS_SURP

− First Dequeue surplus = surplus + credit

− Dequeue 1-3 frames,

− subtract frame length(s) bytes from surplus

− If surplus > 0, dequeue 1-3 frames more

− If surplus <=0, reschedule Frame Queue

Cla

ss S

chedule

rs FD

FD

FQD

FQD

FD

FD

v. h

igh

hig

h

med.

med.

med.

low

low

low

Dedicated Channel 1

FD

FD

v. h

igh

hig

h

med.

m

ed.

m

ed.

lo

w

low

low

Pool Channel 15

FQD

FQD

FD

FD

v. h

igh

hig

h

med.

m

ed.

med.

low

low

low

Pool Channel 1

15 bit Credit 16 bit Surplus

WQ

Channel S

chedule

r

Active FQ SDQCR

Gate with per-channel

DQ Enables

High 0

High 1

Elevated Low

Medium

Low

RR

RR

RR

RR

RR

Priority

Compare

SW Portal Dequeue Dispatcher

Select Channels

With Priority Word

42






Dedicated Channel

Work Queue 4

Work Queue 5

Work Queue 6

FQ 4002 FQ 0003 FQ 4999

Work Queue 7

FD 4001-a

FD 4001-b

FD 4001-c

FD 4001-d

FD 4001-e

FD 5001-a

FD 5001-b

FD 5001-c

FD 5001-d

FD 5001-e

FD 6001-a

FD 6001-b

FD 6001-c

FD 6001-d

FD 6001-e

FQ 5001 FQ 5002 FQ 5003 FQ 5999

FQ 6001 FQ 6002 FQ 6003 FQ 6999

FQ 7001 FQ 7002 FQ 6003 FQ 7999

FD 4002-a

FD 4002-b

FD 4002-c

FD 4002-d

FD 4002-e

FD 5002-a

FD 5002-b

FD 5002-c

FD 5002-d

FD 5002-e

FD 6002-a

FD 6002-b

FD 6002-c

FD 6002-d

FD 6002-e

FD 4003-a

FD 4003-b

FD 4003-c

FD 4003-d

FD 4003-e

FD 5003-a

FD 5003-b

FD 5003-c

FD 5003-d

FD 5003-e

FD 6003-a

FD 6003-b

FD 6003-c

FD 6003-d

FD 6003-e

FD 4999-a

FD 4999-b

FD 4999-c

FD 4999-d

FD 4999-e

FD 5999-a

FD 5999-b

FD 5999-c

FD 5999-d

FD 5999-e

FD 6999-a

FD 6999-b

FD 6999-c

FD 6999-d

FD 6999-e

e500 core X QMan

…

……

……

……

……

SDQCR

- S

S

F

C

D

P

- - DCT

Token

DQ SRC

DQ SRC (cont)

Push Mode

surplus

> 0

Park FQ4001

Get next FQ in line

i.e., FQ4002

Y

N

FQ 4001

43






Dedicated Channel

Work Queue 4

Work Queue 5

Work Queue 6

FQ 4002 FQ 0003 FQ 4999

Work Queue 7

FD 4001-a

FD 4001-b

FD 4001-c

FD 4001-d

FD 4001-e

FD 5001-a

FD 5001-b

FD 5001-c

FD 5001-d

FD 5001-e

FD 6001-a

FD 6001-b

FD 6001-c

FD 6001-d

FD 6001-e

FQ 5001 FQ 5002 FQ 5003 FQ 5999

FQ 6001 FQ 6002 FQ 6003 FQ 6999

FQ 7001 FQ 7002 FQ 6003 FQ 7999

FD 4002-a

FD 4002-b

FD 4002-c

FD 4002-d

FD 4002-e

FD 5002-a

FD 5002-b

FD 5002-c

FD 5002-d

FD 5002-e

FD 6002-a

FD 6002-b

FD 6002-c

FD 6002-d

FD 6002-e

FD 4003-a

FD 4003-b

FD 4003-c

FD 4003-d

FD 4003-e

FD 5003-a

FD 5003-b

FD 5003-c

FD 5003-d

FD 5003-e

FD 6003-a

FD 6003-b

FD 6003-c

FD 6003-d

FD 6003-e

FD 4999-a

FD 4999-b

FD 4999-c

FD 4999-d

FD 4999-e

FD 5999-a

FD 5999-b

FD 5999-c

FD 5999-d

FD 5999-e

FD 6999-a

FD 6999-b

FD 6999-c

FD 6999-d

FD 6999-e

e500 core X QMan

…

……

……

……

……

Push Mode FQ 4001 VDQCR

P E Number of Frames

FQID

……

……

=FQ5003 =FQ6002

44






• QMan 1.2 (i.e. QorIQ T42xx) supports egress traffic management by provides hierarchical class based scheduling and traffic shaping.

• On a specific QMan Direct Connect Portals (DCPs), each sub-portal that supports CEETM can be configured to use either the regular FQ/WQ scheduling mode OR CEETM scheduling mode.

− A given sub-portal can switch between FQ/WQ and CEETM scheduling mode.

− CEETM supports up to 8 logical network interfaces (LNI) that can each be mapped to a DCP sub-portal, whereas a DCP can support up to 16 sub-portals.

− CEETM is supported on only a subset of the DCP portals, not on all DCP portals.

− A single instance of the CEETM is associated with a single DCP and therefore a single egress I/O module.

• CEETM maintains the following functionality equivalent of regular FQ/WQ scheduling mode:

− Congestion management capabilities including WRED

− Dequeued frame context (Context_A and Context_B)

− Priority or traffic class flow control

• FQID xF00000 – xFFFFFF are reserved for CEETM as Logical FQIDs (LFQID).

− xF00000 – xF00FFF (4k) for DCP portal 0 (i.e. FMAN0)

− xF10000 – xF10FFF (4k) for DCP portal 1 (i.e. FMAN1)

45






Shape Aware

Fair Scheduling

StrictPriority StrictPriority

• Logics

− Green denotes logic units and signal paths that relate to the request and fulfillment of Committed Rate (CR) packet transmission opportunities.

− Yellow denotes the same for Excess Rate (ER).

− Black denotes logic units and signal paths that are used for unshaped opportunities or that operate consistently whether used for CR or ER opportunities.

• Scheduler

− Channel Scheduler: channels are selected to send frame from Class Queues.

− Class scheduler: frames are selected from Class Queues . Class 0 has highest priority.

• Algorithm

− Strict Priority (SP)

− Weighted Scheduling

− Shaped Aware Fair Scheduling (SAFS)

− Weighted Bandwidth Fair Scheduling (WBFS)

Strict Priority

Shape Aware

Fair Scheduling

Weighted

Scheduling

StrictPriority StrictPriority StrictPriority

WBFS WBFS WBFS WBFS WBFS

CQ

8

CQ

9

CQ

10

CQ

11

CQ

12

CQ

14

CQ

13

CQ

15

CQ

0

CQ

1

CQ

2

CQ

3

CQ

4

CQ

5

CQ

6

CQ

7

Network IF

Cha

nn

el S

ch

ed

ule

r

for

LN

I #

9

Class Scheduler Ch6

unshaped

8 Indpt, 8 grp Classes

Class Scheduler Ch7

Shaped

3 Indpt, 7 grp

Class Scheduler Ch8

Shaped

2 Indpt, 8 grp

Token Bucket Shaper for Committed Rate

Token Bucket Shaper for Excess Rate

46






• Weighted Bandwidth Fair Scheduling (WBFS) is used to schedule packets from queues within a priority group (A or B group) such that each gets a “fair” amount of bandwidth made available to that priority group.

• The premises for fairness for algorithm is:

− available bandwidth is divided and offered equally to all classes.

− offered bandwidth in excess of a class’s demand is to be re-offered equally to classes with unmet demand.

Initial Distribution First

ReDistribution

Second

Redistribution

Total BW

Attained

BW available 10G 1.5G .2G 0G

Number of classes

with unmet demand 5 3 2

Bandwidth to be

offer to each class 2G .5G .1G

Demand Offered &

Retained

Unmet

Demand

Offered &

Retained

Unmet

Demand

Offered &

Retained

Class 0 .5G .5G 0 .5G

Class 1 2G 2G 0 2G

Class 2 2.3G 2G .3G .3G 0 2.3G

Class 3 3G 2G 1G .5G .5G .1G 2.6G

Class 4 4G 2G 2G .5G 1.5G .1G 2.6G

Total Consumption 11.8G 8.5G 1.3G .2G 10G

47






• DPAA Overview

• BMan Enablement

• FMan Enablement

• QMan Enablement


− QMan Functions

− QMan Scheduling



• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

48






• There are two basic approaches to addressing this requirement:

− Order restoration

Take note of the correct order (or sequence) of packets before processing starts and restore the packets to that order before they are transmitted

− Order preservation

Ensure that related packets are processed in order (and typically one at a time)

Order preservation can also provide “atomicity” – atomic access to data used in processing the frame

• QMan requires that related frames (which must be transmitted in order) be placed on the same frame queue for both of these approaches

This does not mean that only related frames are placed on a given FQ

Many sets of related frames can be placed on an FQ

Frame Manager is responsible for achieving this

F1P1

F0P2

F0P1


Core

D-Cache I-Cache

L2 Cache


Core

D-Cache I-Cache

L2 Cache

F1P1

F0P2

F0P1


Core

D-Cache I-Cache

L2 Cache


Core

D-Cache I-Cache

L2 Cache

Ordered packets

processed in parallel

Ordered packets

queued temporally to

single processing

element

49






Qu

eu

e M

an

age

r


Core

D-Cache I-Cache

L2 Cache

• QMan’s order restoration support has two components:

• Order Definition Point (ODP)

− A point defined relative sequence to each Frames’ pass

− ODP id is associated to FQ-ID

− assigning a monotonically increasing 14 bits sequence number to a series of frames

− QM supports single ODP on head of queue

− ODP can be made anywhere in the system i.e. SW can be an ODP

• Order Restoration Point (ORP)

− A point relative sequence associated to single ODP is restored

− Allows frames insertion into the flow (single sequence number with more/last indication)

• Behavior highlights

− Configurable number of “in-flight” packets per ORP

− resection of ORP is part of enqueue command but Queue tail is not associated to ORP i.e., enqueue to single destination queue can respect many ORP’s

− Note: Frames are not “marked”

WQ0

WQ1

WQ2

WQ3

WQ4

WQ5

WQ6

WQ7


Core

D-Cache I-Cache

L2 Cache

DEQ

WQ0

WQ1

WQ2

WQ3

WQ4

WQ5

WQ6

WQ7

FQD A

…

ODP Seq #

… FD_A

FD_A

FD_A

FQD

…

ORP_NESN

… FD_A2

FD_A1

FD_B2

FQD

…

ORP_NESN

… FD_A3

FD_B3

FD_B1

WQ0

WQ1

WQ2

WQ3

WQ4

WQ5

WQ6

WQ7 FQD B

…

ODP Seq #

… FD_B

FD_B

FD_B

ODP-A

ODP-B

DEQ FD_A3

FD_A2

FD_A1

FD_B3

FD_B2

FD_B1

ORP-B

ORP-A

ENQ

ENQ

FD_A3

FD_A2

FD_A1

FD_B3

FD_B2

FD_B1

ORP-…

AS

OD

P-A

A

S O

DP

-B

50






FQD

…

ORPRWS

OA FD_A2

FD_A1

FD_A3

ORP-A

• Treatment of a FD is determined by:

− ORP Restoration Window Size (ORPRWS)

− ORP Auto Advance NESN Window Toggle (OA)

− ODP Sequence Number (SEQ)

− ORP Next Expected Sequence Number (NESN)

− ORP Acceptable Late Arrival Window Size (OLWS)

− ORP Early Arrival Head Sequence Number (EA_HSEQ)

− ORP Early Arrival Tail Sequence Number (EA_TSEQ)

− ORP Early Arrival Head Pointer (EA_HPTR)

− ORP Early Arrival Tail Pointer (EA_HPTR)

Note: SEQ is 14 bit wide. Create a sliding (8k - 1) sliding windows before and after NESN

FD_A9

FD_A8

FD_A7

ODP_SEQ

ORP_NESN

OLWS

ORP_EA_HSEQ

ORP_EA_TSEQ

ORP_EA_HPTR

ORP_EA_TPTR

…

Sequence

Number

-8191

to 8191

Auto Advance

Window

(EQ RWS)

Late Arrival

Rejection Window

Acceptable Late

Arrival Window

(0 to 8K)

Restoration

Window

(32 to 4K)

Early Arrival

Rejection Window

Late Arrival Early Arrival

NESN (e.g. #4)

…

…

Expecting

FD_A4

FD_A7

FD_A9

51






• DPAA Overview

• BMan Enablement

• FMan Enablement

• QMan Enablement


− QMan Functions

− QMan Scheduling



• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

52






• Both FM and QMan are involved in supporting congestion management and avoidance

• QMan provides the following support

− Congestion management (loss-less flow control, threshold/tail drop)

− Congestion avoidance (RED/WRED)

• Congestion Groups (CG) define granularity

− Every frame queue has a tail drop threshold and configured to a congestion group

− Congestion calculations are done on groups of queues

− Congestion avoidance/management calculations configured in the CG

• Congestion Group Details

− 256 Congestion Groups

− Time aware weighted average queue depth of all queues in the CG

− 3 color configurable WRED curves

− Enqueued packet may be rejected (discarded) due to WRED policy

− Instantaneous CG depth +/- hysteresis value can initiate congestion state messages to enqueue sources

Lossless flow control on interfaces supporting PAUSE semantics

FMan

QMan

CG 1 CG 2

Aggregate

Q Occupancy

Dis

ca

rd

Pro

ba

bili

ty

p

p

Congestion state message

p

QMI

10G 1G 1G 1G 1G

pause

53






• On enqueue:

− The color for the frame being enqueued is used to select a probability curve

− The frame may be selected for random discard

Mgmt Cmd

Cmd Verb

WE_MASK

WR_PARM_G

WR_PARM_Y

WR_PARM_R

WR_EN_G

WR_EN_Y

WR_EN_R

…

Initialize CGR: Verb = 50h.

Modify CGR: Verb = 51h.

Bit 0-4 (msbits): Reserved.

Bit 5: Write enable for WR_PARM_G

Bit 6: Write enable for WR_PARM_Y

Bit 7: Write enable for WR_PARM_R

Bit 8: Write enable for WR_EN_G

Bit 9: Write enable for WR_EN_Y

Bit 10: Write enable for WR_EN_R

Bit 11: Write enable for CSCN_EN

Bit 12: Write enable for CSCN_TARG

Bit 13: Write enable for CSTD_EN

Bit 14: Write enable for CS_THRES

Bit 15: Reserved

CGR

WRED Green Enable

WRED Green Parameters

WRED Yellow Enable

WRED Yellow Parameters

WRED Red Enable

WRED RED Parameters

CSCN Enable

CSCN Target

Congestion State Tail Drop Enable

Congestion State Threshold

Congestion State

Group Instantaneous Byte Count

Group Average Byte Count

TimeStamp

Bits 0-7: MA

Bits 8-12: Mn

Bits 13-19: SA

Bits 20-25: Sn

Bits 26-31: Pn

ENQ and DEQ

upd avg queue len

54






• DPAA

• BMan Enablement

• FMan Enablement

• QMan Enablement

• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

55






Supports protocol processing for the following: • IPSec • 802.1ae (MACSEC) • SSL/TLS/DTLS • 3GPP RLC • LTE PDCP • SRTP • 802.11i (WiFi) • 802.16e (WiMax)

Job Queue

Controller

Descriptor

Controllers

CHAs

DM

A

RT

IC

Queue

Interface

Job Ring I/F

Public Key Hardware Accelerators (PKHA)

RSA and Diffie-Hellman (to 4096b)

Elliptic curve cryptography (1023b)

Data Encryption Standard Accelerators (DESA)

DES, 3DES (2K, 3K)

ECB, CBC, OFB modes

Advanced Encryption Standard Accelerators (AESA)

Key lengths of 128-, 192-, and 256-bit

ECB, CBC, CTR, CCM, GCM, CMAC,

OFB, CFB, and XTS

ARC Four Hardware Accelerators (AFHA)

Compatible with RC4 algorithm

Message Digest Hardware Accelerators (MDHA)

SHA-1, SHA-2 256,384,512-bit digests

MD5 128-bit digest

HMAC with all algorithms

Kasumi/F8 Hardware Accelerators (KFHA)

F8 , F9 as required for 3GPP

A5/3 for GSM and EDGE

GEA-3 for GPRS

Snow 3G Hardware Accelerators (STHA)

Implements Snow 3.0

ZUC Hardware Accelerators (ZHA)

Implements 128-EEA3 & 128-EIA3

CRC Unit

Standard and user defined polynomials

Random Number Generator, random IV generation

56






SRIO

Message

Unit

DMA

PCIe

18-Lane 5GHz SERDES

PCIe SRIO PCIe

CoreNet™

1024-Kbyte

Frontside

L3 Cache

64-bit

DDR-2 / 3

Memory Controller P4080E

32-Kbyte

I-Cache


e500-mc Core

32-Kbyte

I-Cache

128-Kbyte

Backside

L2 Cache

SRIO

Watchpoint Cross Trigger

Perf Monitor

CoreNet Trace

Aurora

Real Time Debug

Security 4.0

Pattern

Match

Engine

2.0

Queue

Mgr.

Buffer

Mgr.

eLBIU

M2SB

Test

Port/

SAP

Frame Manager

1GE 1GE

1GE 1GE 10GE


Buffer

1024-Kbyte

Frontside

L3 Cache

64-bit

DDR-2 / 3

Memory Controller

PAMU

Coherency Fabric PAMU PAMU PAMU PAMU

1GE 1GE

1GE 1GE 10GE


Buffer

Frame Manager

Peripheral

Access

Mgmt Unit

eOpenPIC

Power Mgmt

2x USB 2.0/ULPI

SD/MMC

Clocks/Reset

DUART

2x I 2 C

SPI

GPIO

PreBoot Loader

Security Monitor

Internal BootROM

CCSR

57






• JQ Controller take inputs from:

− JR (Direct Mode)

− QI (DPAA Mode)

− RTIC

• DEscriptor COntroller

− 8x T4240

− 5x P4080

− 3x P3041/P2040

• CHA Control Block

• Crypto Hardware Accelerator (CHA)

− Dedicated CHAs

8x AESA, MDHA, CRCA, KFHA, DESA

− Pool CHAs

RNG, AFHA, PKHA, STHA, ZUCE, ZUCA

• Watch Dog Timer

− Monitors DECOs for prolonged inactivity

Queue Interface Job Prep Logic

Job Queue Controller

DECO Pool

DECO 0

Descriptor

Buffer

DECO 7

R FDs

SP1 0 000

SP2 0 001

SP3 0 101

SP4 0 011

SP5 1 111

FQ FQ FQ FQ FQ

1 E E E D E

2 D E E D E

3 E E E E E

SP Status FQ ID List

Holding

Tank 0

Holding

Tank 7

Holding Tank Pool

Job Queues JR 0

JR 1

JR 2

JR 3

DM

A

Descriptor

Buffer

Watch

Dog

CCB 0 CCB 7

RTIC

Buffer

Mgr

Queue

Manager DDR/CoreNet (Shared Desc, Frame)

. . . . . . .

Arbiter

AFHA

Arbiter RNG

Arbiter Arbiter Arbiter

PKHA STHA f8

STHA f9

MDHA

CRCA

AESA

KFHA DESA

MDHA

CRCA

AESA

KFHA DESA

PKHA

PKHA AFHA

STHA f8

STHA f9

ZUEA

ZUCE ZUEA

ZUCE

. . . . . . .

DM

A

DM

A


e6500 Core

58






Arbiter

AFHA

Arbiter RNG

Arbiter Arbiter Arbiter

PKHA STHA f8

STHA f9

MDHA

CRCA

AESA

KFHA DESA

MDHA

CRCA

AESA

KFHA DESA

PKHA

PKHA AFHA

STHA f8

STHA f9

ZUEA

ZUCE ZUEA

ZUCE

• QI has room for more work, issues dequeue

request.

• Qman selects FQ and provides 1 FD along with

FQ Information

• QI creates [internal] Job Descriptor

• QI transfers completed Job Descriptor into one of

the Holding Tanks

• Job Queue Controller finds an available DECO,

transfers JD1 to it

• DECO initiates DMA of SD and places it in

Descriptor Buffer with JD from Holding Tank

• DECO executes descriptor commands, loading

registers and FIFOs in its CCB

• CCB obtains and controls CHA(s) to process the

data per DECO commands

• DECO commands DMA to store results and any

updated context to system memory

• As input buffers are being emptied, DECO tells QI,

which may release them back to BMan

• Upon completion of all processing through CCB,

DECO resets CCB

• DECO informs QI that JD1 has completed with

status code X, data of length Y has been written to

address Z

• QI creates outbound FD, enqueues to Qman

using FQID from Ctx B field

Queue Interface Job Prep Logic

Job Queue Controller

DECO Pool

DECO 0

Descriptor

Buffer

DECO 7

R FDs

SP1 0 000

SP2 0 001

SP3 0 101

SP4 0 011

SP5 1 111

FQ FQ FQ FQ FQ

1 E E E D E

2 D E E D E

3 E E E E E

SP Status FQ ID List

Holding

Tank 0

Holding

Tank 7

Holding Tank Pool

Job Queues JR 0

JR 1

JR 2

JR 3

DM

A

Descriptor

Buffer

CCB 0 CCB 7

Buffer

Mgr

Queue

Manager FD1

JD1

. . . . . . .

. . . . . . .

DDR/CoreNet (Shared Desc, Frame)

59






• DPAA

• BMan Enablement

• FMan Enablement

• QMan Enablement

• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

60






• Regex support plus significant extensions:

− Patterns can be split into 256 sets each of which can contain 16 subsets

− 32K patterns of up to 128B length

− 9.6 Gbps raw performance

• Combined hash/NFA technology

− No “explosion” in number of patterns due to wildcards

− Low system memory utilization

− Fast pattern database compiles and incremental updates

• Matching across “work units”

− Finds patterns in streamed data

• Pipeline of processing

− PME offers pipeline of filtering, matching, and behavior base engine for complete pattern matching solution

On-Chip

System

Bus

Interface

Pattern

Matcher

Frame

Agent

(PMFA)

Data

Examination

Engine

(DXE)

Stateful

Rule

Engine

(SRE)

Key

Element

Scanning

Engine

(KES)

Hash

Tables

Access to Pattern Descriptors and State

Pattern Matching Engine components

Cache Cache

User Definable Reports

Cor

eNet

B

Ma

n

QM

an

61






• PME Frame Descriptor Commands

− b111 NOP NOP Command

− b101 FCR Flow Context Read Command

− b100 FCW Flow Context Write Command

− b001 PMTCC Table Configuration Command

− b000 SCAN Scan Command

0 1 2 3 4 5 6 7 8 9 1

0

1

1

1

2

1

3

1

4

1

5

1

6

1

7

1

8

1

9

2

0

2

1

2

2

2

3

2

4

2

5

2

6

2

7

2

8

2

9

3

0

3

1

DD LIODN offset BPID ELIODN

offset

- - - - addr

addr (cont)

Format Offset Length

Status/CMD

Scan

b000

SRV

M

F S/

R

E SET Subset

62






I W A N T T O S E A R C H F R E E

• Patterns

− Patt1 /free/ tag=0x0001

− Patt2 /freescale/ tag=0x0002

• KES

− Compare hash value of incoming data(frames) against all patterns

• DXE

− Retrieve the pattern with matched hash value for a final comparison.

• SRE

− Optionally post process match result before sending the report to the CPU

On-Chip

System

Bus

Interface

Pattern

Matcher

Frame

Agent

(PMFA)

Data

Examination

Engine

(DXE)

Stateful

Rule

Engine

(SRE)

Key

Element

Scanning

Engine

(KES)

Hash

Tables

Access to Pattern Descriptors and State

Cache Cache

User Definable Reports

Cor

eNet

B

Ma

n Q

Ma

n

192.168.1.1:80 TCP 10.10.10.100:16734

192.168.1.1:25 TCP 10.10.10.100:17784

192.168.1.1:1863 TCP 10.10.10.100:16855

DDR

Memory

flowA:FD1: 192.168.1.1:80->10.10.10.100:16734 “I want to search free “

flowA:FD2: 192.168.1.1:80->10.10.10.100:16734 “scale FTF 2011 event schedule”

Frame Queue: A

FD1

Patt1 /free/

tag=0x0001

FD2

63






• DPAA

• BMan Enablement

• FMan Enablement

• QMan Enablement

• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

64






• Deflate

− As specified as in RFC1951

• GZIP

− As specified in RFC1952

• Zlib

− As specified in RFC1950

− Interoperable with the zlib 1.2.5 compression library

• Encoding

− supports Base 64 encoding and decoding (RFC4648).

• Operate up to 600Mhz

− 10Gbps Compress

− 10Gbps Decompress

− 20Gbps Aggregate

32KB

History

Frame

Agent

QMan

I/F

BMan

I/F

Bus

I/F

Decompressor

Compressor

QMan

Portal

BMan

Portal

To

Corenet

4KB

History

65






• The Status/Command word in the dequeued FD allows software to modify the processing of individual frames while retaining the performance advantages of enqueuing to a FQ for flow based processing.

• The three most significant bits of the Command /Status field of the Frame Descriptor have the following meaning:

0 1 2 3 4 5 6 7 8 9 1

0

1

1

1

2

1

3

1

4

1

5

1

6

1

7

1

8

1

9

2

0

2

1

2

2

2

3

2

4

2

5

2

6

2

7

2

8

2

9

3

0

3

1

DD LIODN offset BPID ELIODN

offset

- - - - addr

addr (cont)

Format Offset Length

CMD Token: Pass through data that is echoed with the returned Frame.

3 MSB Description

000 Process Command Command Encoding

001 Reserved

010 Reserved

011 Reserved

100 Context Invalidate Command Token

101 Reserved

110 Reserved

111 NOP Command Token

0 1 2 3 4 5 6 7 8 9 1

0

1

1

1

2

1

3

1

4

1

5

1

6

1

7

1

8

1

9

2

0

2

1

2

2

2

3

2

4

2

5

2

6

2

7

2

8

2

9

3

0

3

1

CM

D

OO

Z

Flu

sh

SC

RF

R

I

RB

B

64

CF

-

CE

UH

C

US

PC

U

SD

C

SC

US

Status

(output Frame)

66






• SW enqueues work to DCE via Frame Queues. FQs define the flow for stateful processing.

• FQ initialization creates a location for the DCE to use when storing flow stream context.

• Each work item within the flow is defined by a Frame Descriptor, which includes length, pointer, offsets, and commands.

• DCE has separate channels for compress and decompress.

Command

FQs

Flow

Stream

Context

Context_A

Flow

Stream

Context

Context_A

Direct C

on

ne

ct

Po

rtal

DCE

WQ6

WQ7

ch

an

ne

l WQ0

WQ1

WQ2

WQ3

WQ4

WQ5

WQ6

WQ7

FD3

FD2

FD1

Addr

Offset Length

Status/Cmd

PID BPID Addr

Addr

Offset Length

Status/Cmd

PID BPID Addr

Addr

Offset Length

Status/Cmd

PID BPID Addr

Data

Buffer

Data

Buffer

Data

Buffer

WQ6

WQ7

ch

an

ne

l

WQ0

WQ1

WQ2

WQ3

WQ4

WQ5

WQ6

WQ7

FD3

FD2

FD1

Addr

Offset Length

Status/Cmd

PID BPID Addr

Addr

Offset Length

Status/Cmd

PID BPID Addr

Addr

Offset Length

Status/Cmd

PID BPID Addr

Data

Buffer

Data

Buffer

Data

Buffer

Decomp

Comp

FQs

0

1

67






• DCE enqueues results to SW via Frame Queues as defined by FQ Context_B field. When buffers obtained from BMan, buffer pool ID defined by Output FQ.

• Each result is defined by a Frame Descriptor, which includes a Status field.

• DCE updates flow stream context located at Context_A as needed.

FD3

FD2

FD1

Addr

Offset Length

Status/Cmd

PID BPID Addr

Addr

Offset Length

Status/Cmd

PID BPID Addr

Addr

Offset Length

Status/Cmd

PID BPID Addr

Data

Buffer

Data

Buffer

Data

Buffer

Port

al

Decomp

Comp

DCE

Flow

Stream

Context

Context_A

Data

Buffer

Data

Buffer

Data

Buffer FD3

FD2

FD1

Addr

Offset Length

Status/Cmd

PID BPID Addr

Addr

Offset Length

Status/Cmd

PID BPID Addr

Addr

Offset Length

Status/Cmd

PID BPID Addr

Flow

Stream

Context Context_A

Status

FQs

FQs

68






• DPAA

• BMan Enablement

• FMan Enablement

• QMan Enablement

• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

69






P4080 P2040, P3, P5

Outbound

Transactions Supported Type 10 Doorbells

Type 11 Messaging

Type 5 NWRITE Type 9 Data Streaming

Type 6 SWRITE Type 10 Doorbells

Type 8 Port-Write Type 11 Messaging

Queues 1 Type 10 Doorbell

2 Type 11 Messaging Thousands of queues supporting Type 5,6,8-11

Queue Arbitration Round Robin Data Path Acceleration Architecture

• 3+3+1 SP+WRR

Segmentation Resources 2 Segmentation Units 4 Segmentation Units

Multicast Support Type 11 256B PDU to 16 Destinations Type 11 256B PDU to 32 Destinations

Inbound

Transactions Supported

Type 8 Port-Write

Type 10 Doorbells

Type 11 Messaging

Type 8 Port-Write Type 10 Doorbells

Type 9 Data Streaming Type 11 Messaging

Queues

1 Type 8 Port-Write

1 Type 10 Doorbell

2 Type 11 Messaging

1 Type 8 Port-Write

1000s Type 9-11

Classification 2 Rules (Fixed)

Type 11: [mbox]

64 Rules (Exact or Wildcards)

or

Map selected header fields to queue ID

Simultaneous Reassembly

Contexts 2 Type 11 16 Type 9, 11

Additional Features

Traffic Management N/A Type 9: End-to-end XON/XOFF Per-Queue Flow Control

70






QMan

RMan

Inbound Rule

Matching

Classification

Unit

Reassembly

Contexts

Reassembly

Unit

Segmentation

Unit

Rapid

IO I

nbound T

raffic

Rapid

IO O

utb

ound T

raffic

Classification

Unit

Classification

Unit

Reassembly

Unit

Reassembly

Unit

Segmentation

Unit

Segmentation

Unit

AR

B

WQ

0

WQ

1

WQ

2

WQ

3

WQ

4

WQ

5

WQ

6

WQ

7

Channel

Frame Manager

1GE 1GE

1GE 1GE 10GE

D$ I$

D$ I$ L2$

e500mc Core

D$ I$

SE

C

PM

E

Disassembly

Contexts

WQ

0

WQ

1

WQ

2

WQ

3

WQ

4

WQ

5

WQ

6

WQ

7

Channel

WQ

0

WQ

1

WQ

2

WQ

3

WQ

4

WQ

5

WQ

6

WQ

7

Channel

71






• Many queues allow multiple inbound/outbound queues per core

− Hardware queue management via QorIQ Data Path Architecture (DPAA)

• Supports all messaging-style transaction types

− Type 11 Messaging

− Type 10 Doorbells

− Type 9 Data Streaming

• Enables low overhead direct core-to-core communication

Core Core Core Core

10G SRIO

QorIQ or DSP

Core Core Core Core

10G SRIO

QorIQ or DSP

Type9 User PDU

Channelized CPU-

to-CPU transport Device-to-Device

Transport

MSG User PDU

72






• DPAA

• BMan Enablement

• FMan Enablement

• QMan Enablement

• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

73






• QMan 1.2 (e.g. QorIQ T42xx) supports Data Center Bridging (DCB).

• DCB refers to a series of inter-related IEEE specifications collectively designed to enhance Ethernet LAN traffic prioritization and congestion management.

• DCB can be used in:

− Between data center network nodes:

− LAN/network traffic

− Storage Area Network (SAN) (e.g. Fiber Channel (loss sensitive )) and,

− IPC traffic (e.g. Infiniband (low latency))

• The DPAA is compliant with the following DCB specifications (traffic management related) :

− IEEE Std. 802.1Qbb: Priority-based flow control (PFC)

To avoid frame loss, a PFC Pause frames can be sent autonomously by HW.

− IEEE Std. 802.1Qaz: Enhanced transmission selection (ETS)

Support Weighted bandwidth fairness.

− IEEE 802.1Qau: Quantized Congestion Notification (QCN)

end-to end congestion control mechanism.

74






• Enables lossless behavior for each class of service

• PAUSE sent per virtual lane when buffers limit exceeded

− FQ congestion groups state (on/off) from QMan;

Priority vector (8 bits) is assigned to each FQ congestion group

FQ congestion group(s) are assigned to each port;

Upon receipt of a congestion group state “on” message, for each Rx port associated with this congestion group, a PFC Pause frame is transmitted with priority level(s) configured for that group.

− Buffer pool depletion

Priority level configured on per port (shared by all buffer pools used on that port)

− Near FMan Rx FIFO full

There is a single Rx FIFO per port for all priorities, the PFC Pause frame is sent on all priorities.

• PFC Pause frame reception

− QMan provides the ability to flow control 8 different traffic classes; in CEETM each of the 16 class queues within a class queue channel can be mapped to one of the 8 traffic classes & this mapping applies to all channels assigned to the link

Transmit Queues Ethernet Link

Receive Buffers

Zero Zero

One One

Two Two

Five Five

Four Four

Six Six

Seven Seven

Three Three STOP PAUSE Eight

Virtual

Lanes

75






• Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth

• Best supported through QMan CEETM

− 8 independent classes and 8 grouped classes

− Strict priority scheduling of the 8 independent classes

− Weighted bandwidth fairness within 8 grouped classes

− Priority of the class group can be independently configured to be immediately below any of the independent classes

• Meets performance requirement for ETS: bandwidth granularity of 1% and +/-10% accuracy

• Supports 32 channels available for allocation across a single FMan;

− e.g. for two10G links, could allocate 16 channels (virtual links) per link

− Supports weighted bandwidth fairness amongst channels

− Shaping is supporting on per channel basis

10 GE Realized Traffic Utilization

3G/s HPC Traffic

3G/s

2G/s

3G/s Storage Traffic

3G/s

3G/s

LAN Traffic

4G/s

5G/s 3G/s

t1 t2 t3

Offered Traffic

t1 t2 t3

3G/s 3G/s

3G/s 3G/s 3G/s

2G/s

3G/s 4G/s 6G/s

76






• Reaction Point

− Reaction point state machine and congestion notification message (CNM) receipt and

processing must be implemented in software

− Can make use of CEETM channel shaping function to implement the rate limiter

function;

One class queue per channel; limited scalability especially if virtualization is too be supported (total

of 32 channels per FMan instance)

Shaped queue cannot be serviced using strict priority

• Congestion Point

− No hardware support

77






• DPAA

• BMan Enablement

• FMan Enablement

• QMan Enablement

• SEC Enablement

• PME Enablement


• RMan Enablement


• Conclusion

78






Tweeting? Please use hashtag

#FTF2012

Facebook.com/Freescale Tag yourself in photos

and upload your own!

Session materials will be posted @ www.freescale.com/FTF Look for announcements in the FTF Group on LinkedIn or follow Freescale on Twitter

• QorIQ is all about Accelerators and Integration.

• The Data Path Acceleration Architecture Accelerators included:

− Queue Manager

− Buffer Manager

− Frame Manager

− Hardware accelerators such as SEC, PME, DCE, and RMan

− Power Architecture Cores

• Seamless Integration of these components address multicore requirements:

− Load spreading

− Packet ordering

− Device virtualization

− Inter-core communication

− HW buffer management

http://www.freescale.com/FTF

Multicore Expert Series: QorIQ Platform Data Path Acceleration … · 2016. 3. 12. · 10Gbps...

Documents

Transcript of Multicore Expert Series: QorIQ Platform Data Path Acceleration … · 2016. 3. 12. · 10Gbps...