HSLB board User Guide

Belle II note XXX

Belle2link Design

Ver. 0.1

Date: Feb. 25 2011

By Zhen-An Liu, Dehui Sun, Jingzhou Zhao, IHEP

Note:

This document will be a full description of Belle2link system to be used in Belle II experiment. This file also functions a guidance in early stage to the subsystems in designing the interfacing to Belle2link in Front-End electronics.

But still it is in the very basic stage, we are sorry about this. You suggestions and comments are welcome.

0. Table of Contents

i. Introduction

ii. BL_RSVR,FINESSE card

iii. BL_SNDR

iv. Demo system for CDC

v. supplements

Introduction

The Belle II collaboration has decided to use optical fibers and RocketIO to make the connection between Front-End Electronics (FEE)/Trigger system and DAQ system for signal and data transmission. For system simplicity and reliability, a so-called Belle2Link, an unified high speed link system, has been defined for use in overall connection between DAQ and each of the subsystems. It features as:

1. unification in hardware design

2. unification in firmware design

3. provides electrical isolation

4. provides high speed transmission rate

5. versatile for different input data rate

6. home brew transmission protocol

Figure 1-1 the relation of Belle2Link with FEE and DAQ

As Belle II DAQ will based on the COPPER structure, Belle2Link comprise mainly RocketIO and optical fiber based high speed transmission lines and interfaces to FEE(IF FE), to global trigger and timing distribution(IF TTC) and to PCI bridge on COPPER (IF CO )(Figure 1-1), i.d. it accepts data in FIFO FE via IF FE from FEE for transmission in the sender part (BLSNDR), gets trigger and timing signals from timing and trigger system via IF TTC, and at the receiver part (BLRSVR) save the received data in the FIFO BL and moves data after necessary processing to the FIFO CO in IF CO which is actually on the COPPER(Figure 1-2).

As a unified link for all sub-detector system, Belle2Link provides same or limited type of receiver FINESSE boards, necessary merger board, and all the unified firmware for sender part and receiver part.

Figure 1-2 Block diagram of the Belle2Link

1.1 Unification in hardware

Not as a separate part, the BLSNDR will rather share the resources of and be implemented in a Virtex-5 FPGA in the most FEE readout board. The FEE readout of each sub detector systems provide the V5 GPGA and the BLSNDR part of the unified Belle2Link firmware will be integrated with the readout firmware. The requirement to the FEE from Belle2link includes:

a. An SFF optical transceiver connected to FPGA GTPs

b. On board crystal of 125MHz

c. RJ-45 connector receiving system clock

For some system where necessary a merger is added before the BLSNDR. The high precision clock CLK LK for the high speed transmission via RocketIO will be same for all Belle2Link. This CLK LK is preliminarily defined as 125MHz to get a line rate of 3.125Gbps and be produced by a high precision crystal both on the FEE board and FINESSE receiver board(BLRB). BLRD board is FINESSE board which is fit the general requirement of high speed transmission and also some data processing required and fits to the COPPER board electrically and mechanically.

1.2 Unification in firmware and protocol

Most of the circuitry of the Belle2Link will be implemented by firmware in the Xilinx FPGA. The development of the link will be based on Xilinx ISE 11 which supports the Virtex-5 and Spartan-6. The firmware for BLSNDR will be unified and provided to all system for integration to the system. IHEP group will provide as an example the full firmware for CDC subsystem which can be directly used in other system as long as the version regulation is followed. The working status can be checked by registers and displayed on some LED on the front panel. The interface with FEE is a FIFO(FIFO-FE). The FIFO FE is a 32bit write and 16bit read FIFO and will be filled with data by FEE readout under control of clock CLK FE and then be moved to GTP under sender clock CLK LN after adding some control and error checking data by BLSNDR state machine. Some of the slow control like function has been added for parameter setting and status/error checking.

In the BLRSVR, another state machine checks for the GTP line . The data on the line will be first received and stored in a FIFO-LN under control of clock CLK LN. If it is error free this data will be moved to a FIFO LN (N EV will be incremented). If there is a error occurred on the transmission, an error flag is added in the header part of data frame for later use in the online data monitoring. The data will then be check again normal data or slow control data. If it is slow control data it will be send back to the requesting task. Otherwise the data in FIFO-LN will be direct moved to another fifo FIFO WR, but for systems like B-PID the date must be first moved to a pipeline(Figure ??), and part of the data in a time window will be fetch after receives a trigger and be processed accordingly, then be stored in FIGO WR in the data form showed in table ??. A special error bits has been allocated which indicates if there is a data transmission error occurred.

table ??

Another state machine(Figure ??) checks the NEV WR, if there is an event in the FIFO WR, then send the data for this event to FIFO CO under control of clock CLK CO and decrease the NEV WR if the COPPER is ready(No FFUL signal) to read. At the same time if the number of event NEV WR exceeds a predefined number a busy signal is produced for TTRX usage in blocking the further triggers and be removed two CLK CO clocks after data has been moved to FIFO CO. The error checking scheme will be kept for the online checking of the transmission between Belle2Link and COPPER.

1.3 Clocks

Three key clocks will be needed by Belle2Link, CLK FE which is converted from central trigger and timing distribution, CLK LK for the high speed transmission which is a high precision clock from a crystal on the board. CLK_LK is connected to on board crystal as default, and then switched to the RJ-45 system clock later in normal running state. CLK CO is received from TTRX via COPPER connector. The clocks for data transmission on the RocketIO will be composed by the DCS in Virtex-5 core. For the prototype system, we have used a CLK LK of 200MHz and get successfully a line rate of 2Gbps.

2. BL_RSVR: receiver card

A receiver FINESSE card, or High Speed Link Board( HSLB) has been designed for use for a high speed data receiving on COPPER board. It uses the GTP and optical fiber to realize the Gigabit speed transmission. The rate can reach up to 3.125Gbps. Figure 2-1 shows the block diagram of HSLB board.

H

S

L

B

I

N

T

E

R

F

A

C

E

FPGA

XC5VLX50T

FLASH

XCF16P

CPLD

XC2C64A

Optical transceiver

Optical transceiver

RJ45

RocketIO

Clock

Control Signal

Data Bus

Address Bus

Trigger Signal

Figure 2-1 block diagram of HSLB board.

2.1 Key Components and Features

Figure 2-2 shows the HSLB board, which includes the following components and features:

XCLX50T-1FF1136C FPGA

2x SFP module ,

LED for SFP which can be redefined by user.

LED for indicate the status of the SFP

RJ45 port

LVDS OSC for debugging the board

MC100LVEP14DTR2 for distributing the system clock

MC100EPT21D for transforming the LVDS Clock to single Clock signal.

200M LVDS OSC for GTP

16Mbit Xilinx XCF16PVOG48C Platform Flash, in-system programmable configuration PROM

Xilinx XC2C64A-7VQG100C CPLD for controlling and configuring the PROM and FPGA

JTAG interface

HSLB board to COPPER board interface A

HSLB board to COPPER board interface B

HSLB board to COPPER board interface C

Eight individual LED

Eight individual test pins and ground pins

P/N signal test pins

3.3V Power-on indicator LED and FPGA configuration DONE LED

On-board 2.5Vregulators. , 1.2VTT , 1.2VPLL, 1.0VMGT, 1.0VINT, 1.8V

Fuze of 3.3V and 5V

27

13

12

33

11

10

987

6

5

4

3

2

1

14

16

15

17

19

18

29

30

31

28

26

25

20

21

22

23

24

12

14

13

32

Figure 2-2 shows the component location.

2.2 SFP interface and LED

HSLB board uses the 112 GTP dual(X0Y3) and 114 GTP dual(X0Y2) for SFP. The table 2-1 shows the SFP and SFP LED control signals.

112GTP (X0Y3)

114 GTP (X0Y2)

Signal

Pin#

Signal

Pin#

TXFAULT

J6

TXFAULT

N8

TXDISABLE

E6

TXDISABLE

K6

LOS

G5

LOS

N5

MOD0

F6

MOD0

M6

MOD1

E7

MOD1

L5

MOD2

F5

MOD2

L4

RATESEL

J5

RATESEL

N7

LED0

E8

LED2

K8

LED1

F8

LED3

K9

2.3 RJ45 Port

The board can bring in external clock signal, trigger signal and other control signal though RJ45. Table 2-2 shows the pin distribution.

RJ45 signal distribution

Signal

RJ45 Pin#

FPGA Pin#

SYSCLK_N

P1

AD5

SYSCLK_P

P2

AD4

INI_N

P3

AK7

INI_P

P6

AK6

RSVCLK_N

P4

AC7

RSVCLK_P

P5

AD7

SYSDAT_N

P7

AE6

SYSDAT_P

P8

AD6

LED1

P9

AC4

LED2

P12

AB6

2.4 LED and Test Pins

The HSLB board has eight individual LEDs and test pins. They are directly connected to the FPGA Pin. Table 2-3 shows the pin distribution.

Test pin P2,P4,P6,P8,P10,P12,P14,P16 are joined to the ground.

LED

FPGA Pin#

TEST Signal

FPGA Pin#

DS705

D10

P1

AM11

DS706

D11

P3

AK11

DS707

B12

P5

AL11

DS708

C12

P7

AL10

DS709

D12

P9

AJ11

DS710

A13

P11

AJ10

DS711

B13

P13

AH10

DS712

C13

P15

AG10

2.5 Clock Sources

HSLB board has four LVDS clock signals.

The clock comes form the HSLB to COPPER interface C, SCK+/-. It is used for the system clock, and synchronous with the copper read and write HSLB clock.

The clock comes form the RJ45, the external clock SYSCLK_N/P.Pin distribution see table 2-1.

The OSC clock on the back of the board is high accuracy. It is used for the GPT.

The clock on board, is used for testing clock.

Table 2-4 shows the FPGA pin distribution for clock signal.

GTP clock

P

P4

N

P3

Testing clock

P

AH20

N

AH19

System clock

P

AG22

N

AH22

2.6 JTAG Programme

The HSLB board includes a JTAG programming and debugging chain. All the Virtex-5 FPGA, the Platform Flash and CPLD devices are part of the JTAG chain, as shown in Figure 2-3.

XCLX50TFLASH

6

8

4

10

TMS

TDI

TDO

TCK

TDI

TDI

TDO

TDO

TMS

TMS

TCK

TCK

J1

CPLD

TDI

TD0

TMS

TCK

2.7 Platform Flash Configuration Storage

The HSLB board has an XCF16PVOG48C configuration Flash PROM to store FPGA configuration data. The Flash provide serial and parallel configuration. And the three signal M0,M1,M2 which decide the configuration should be assigned in the CPLD. Figure 2-4 show the way Flash connect to FPGA and CPLD.

FLASH

XCF16P

CPLD

XC2C64A

FPGA

XC5VLX50T

Address Bus

Data Bus C_LD[0:7]

C_LA[0:6]

C_CCLK

C_CSB

C_INIT_B

C_BUSY

C_CE_B

C_LWR

C_RSV[0:3]

C_M[0:2]

C_RDWR_B

C_PRGM_B

C_DIN

C_DIN

C_BUSY

C_CS_B

C_INIT_B

2.8 CPLD

CPLD is used as the interface chip in the HSLB board. Software can be download to CPLD, then the system can configure the FPGA and Flash online. Table 2-5 show the detail signals and the pin distribution of the CPLD.

Signal

CPLD Pin#

Signal

CPLD Pin#

C_LD0

P64

F_LD0

P27

C_LD1

P61

F_LD1

P24

C_LD2

P60

F_LD2

P22

C_LD3

P58

F_LD3

P19

C_LD4

P56

F_LD4

P18

C_LD5

P55

F_LD5

P17

C_LD6

P53

F_LD6

P16

C_LD7

P52

F_LD7

P15

C_LA0

P67

F_LA0

P14

C_LA1

P68

F_LA1

P13

C_LA2

P70

F_LA2

P12

C_LA3

P71

F_LA3

P11

C_LA4

P72

F_LA4

P10

C_LA5

P74

F_LA5

P9

C_LA6

P76

F_LA6

P8

C_INIT_B

P29

C_SCK

P23

C_CCLK

P30

C_PRGM_B

P32

C_DIN

P33

C_DONE

P34

C_CS_B

P35

C_RDWR_B

P36

C_BUSY

P37

C_M0

P39

C_M1

P41

C_M2

P40

C_CE_B

P42

C_CSB

P78

C_LWR

P77

F_LWR

P7

F_CSB

P6

C_RSV0

P92

C_RSV1

P91

C_RSV2

P90

C_RSV3

P89

2.9 Power Distribution

The HSLB board use TPS74401 as the 2.5V, 1.8V, 1.2VPLL, 1.2VTT, 1.0VMGT and 1.0VINT power regulater. TPS74401 is 3.0A Ultra-Low Dropout Linear Regulator. At the output of every power regulater circuit, 0R resistance is used to as the isolating resistance.

2.10 HSLB to COPPER interface

On the HSLB there are three socket for connecting COPPER, A is for transiting the mass data; B is for the power; and C is for trigger signals, clock signals, and other control signals. Table 2-6 shows detail of the HSLB to COPPER interface signals.

A

B

C

Pin#

Signal

FPGA Pin

Pin#

Signal

Pin#

Signal

FPGA/CPLD Pin

A1

FF00

B32

B1

-3.3V

C1

IRSTB

FPGA_E31

A2

FF01

A33

B2

-3.3V

C2

IENA

FPGA_F30

A3

FF02

B33

B3

-3.3V

C3

IO

FPGA_F31

A4

FF03

C32

B4

-3.3V

C4

TYP0

FPGA_G30

A5

FF04

C33

B5

-3.3V

C5

TYP1

FPGA_G31

A6

FF05

C34

B6

-3.3V

C6

TYP2

FPGA_H30

A7

FF06

D32

B7

GND

C7

TYP3

FPGA_J30

A8

FF07

D34

B8

GND

C8

TAG0

FPGA_J31

A9

FF08

E32

B9

GND

C9

TAG1

FPGA_L30

A10

FF09

E33

B10

GND

C10

TAG2

FPGA_L31

A11

FF10

E34

B11

GND

C11

TAG3

FPGA_M30

A12

FF11

F33

B12

GND

C12

TAG4

FPGA_M31

A13

FF12

F34

B13

+3.3V

C13

TAG5

FPGA_N30

A14

FF13

G32

B14

+3.3V

C14

TAG6

FPGA_P30

A15

FF14

G33

B15

+3.3V

C15

TAG7

FPGA_P31

A16

FF15

H32

B16

+3.3V

C16

LD0

CPLD_P27

A17

FF16

H33

B17

+3.3V

C17

LD1

CPLD_P24

A18

FF17

H34

B18

+3.3V

C18

LD2

CPLD_P22

A19

FF18

J32

B19

GND

C19

LD3

CPLD_P19

A20

FF19

J34

B20

GND

C20

LD4

CPLD_P18

A21

FF20

K32

B21

GND

C21

LD5

CPLD_P17

A22

FF21

K33

B22

GND

C22

LD6

CPLD_P16

A23

FF22

K34

B23

GND

C23

LD7

CPLD_P15

A24

FF23

L33

B24

GND

C24

LA0

CPLD_P14

A25

FF24

L34

B25

+5V

C25

LA1

CPLD_P13

A26

FF25

M32

B26

+5V

C26

LA2

CPLD_P12

A27

FF26

M33

B27

GND

C27

LA3

CPLD_P11

A28

FF27

N32

B28

GND

C28

LA4

CPLD_P10

A29

FF28

N33

B29

-5V

C29

LA5

CPLD_P9

A30

FF29

N34

B30

-5V

C30

LA6

CPLD_P8

A31

FF30

P32

B31

GND

C31

LWR

CPLD_P7

A32

FF31

P34

B32

GND

C32

CSB

CPLD_P6

A33

GATE

R33

B33

+12V

C33

TRG+

FPGA_AJ31

A34

FRSTB

R32

B34

GND

C34

TRG-

FPGA_AK31

A35

FWENB

T34

B35

-12V

C35

REV+

FPGA_AJ30

A36

FWCLK

T33

B36

GND

C36

REV-

FPGA_AH30

A37

FFUL

R34

C37

RCK+

FPGA_AH29

A38

NWFF

U31

C38

RCK-

FPGA_AG30

A39

ABRT

U32

C39

SCK+

to MC100LVEP14DTR2

A40

BSYB

U33

C40

SCK-

to MC100LVEP14DTR2

3. BL_SNDR

3.1 Physical interface at the fronted readout board

Each of the FEE readout board must provide one SFP transceiver which is linked to GTP_Dual Tx/Rx port of Virtex 5 FPGA as shown in Fig 3.2.

The design of power supply for GTP must fulfill the requirements expressed in RocketIO GTP Transceiver part of the datasheet.

Fig.3-1 Peripheral signals at the CDC readout board

3.2 Clocks

For a proper high-speed operation, the GTP transceiver requires a high-quality, low-jitter, reference clock. Belle2Link need a special external oscillator which must fulfills the requirements given in the GTP Transceiver section of the Virtex-5 FPGA Data Sheet. Two external Clock must be provided on the FEE readout board.

1. on board oscillator input

Each GTP_DUAL tile has a pair of dedicated pins that can be connected to the external on board clock source. We recommend using AC coupling between the oscillator output pins and the dedicated GTP_DUAL clock input pins.

2. RJ-45 system clock input

We recommend to connect the RJ-45 system clock via another nearby dedicated GTP_DUAL clock input pins to have good clock quality. As this clock cannot be used by other parts of FPGA before embedded PLL in GTP IPcore, RJ-45 clock could be connected via a global clock pin if you have to.

The internal clock nets of the FPGA provide the reference clock for the GTP_DUAL tile by connecting the output of a global clock buffer (BUFG) to the CLKIN port. But this type of clocking, called GREFCLK clocking, has the lower performance with more jitter. GREFCLK clocking should be avoided, if possible. See the Virtex-5 FPGA Data Sheet for the jitter margins at different speeds.

Fig.3-2 BL_SNDR Clocking

3.3 Structure of Belle2Link firmware

The BL_SNDR firmware in FEE readout board FPGA has three main functional components. The slow control module configures or reads the FEE registers. The data link module receives the data from FEE and packs them. The protocol module receives the package from GTP module, check the packages and unpack them. It also deals with the data transmission for CDC and arbitrate the priority between slow control module and data link module.

Fig.3-3 Structure of BL_SNDR firmware in CDC readout board

The firmware of BL_RCVR has the same function components. The protocol module receives the package from GTP module, check the packages and unpack them, separate the data and transfer them to slow control module or data link module. Data Link module receive the data of FEE, transmit the data and the clock to the COPPER FIFO. Slow Control module receives or transfer the slow control message from Local bus.

Fig.3-3 Structure of HSLB firmware in CDC readout board

3.3.1 The definition of the interface between FEE and BL_SNDR

The BL_SNDR should be designed to be transparent to COPPER. So the interfaces between FEE and BL_SNDR are designed to be similar as BL_RCVR to COPPER. The interface for FEE data exchange looks same as the COPPER FIFO Interface (Fig.3-4). The FEE transmit the data with an synchronous clock signal. I recommend keeping 32-bit width for FEE data. The existence of an enable signal indicates the length of an event.

Fig.3-4 Timing chart for Data interface

The interface for slow control, called parameter bus, is similar to the Local Bus on COPPPER (Fig.3-6). If COPPER writes a value on address 0x41 of the Local bus, the value will be send to the same address on parameter bus after a few nanoseconds.

The time charts of Belle2Link slow control at FEE are as follow (Fig.3-5).

Fig.3-5 timing chart for Parameter bus

3.3.2. Implementation of Slow Control

Fig.3-6 structure of slow control module in BL_RCVR firmware

Fig.3-7 structure of slow control module in BL_SNDR

There are two approaches for the implementation of slow control.

For the large stream, such as DSP ROM file and configure file, the system will write the file on a special address 0x00 serially. The Slow control module processes the stream, and then transfers the file to protocol module in form of package. The file is transferred to FEE by the Belle2Link，and written on the same address 0x00 on parameter bus on FEE board serially. There is no buffer in Belle2Link. The definition and structure of the file absolutely depends on the subsystems.

For individual setting (less than 128 bytes), the Belle2Link uses two dual-port RAM for the reading and writing separately (Fig.3-6). How to configure the register of FEE is straightforward, the system writes the value on some address, the values are recorded in a RAM, the addresses are recorded in a FIFO. Slow control module organizes the configure message and transfer them to the protocol module. Because of the latency of the Belle2Link, the FEE register reading cannot works like configuration. But the reading is no so straightforward for the system. The COPPER must read an address twice to complete one reading action. The first reading informs the Belle2Link which addresses need to be read. The second reading will get the value which was read from FEE board by Belle2Link. The reading works like a handshake processing with the help of a CSR (control and status register).

3.3.3 Arbitration of data sending

For the slow control message and the event of FEE are transmitted in one line, the protocol module of BL_SNDR watches the status of two links，arbitrates the priority level and processes the message and date into different package. There are 3 priority levels. The state machine of the protocol is as follow (Fig.3-8).

Fig.3-8 state machine of BL_SNDR protocol module

3.3.4. Processing of data in BL_RCVR

The receiving data are processed separately by different f of packages based on the package header. The slow control stream is recovered to the value and address. The normal data stream is processed to satisfy the format of COPPER. (Fig.3-9)

Fig.3-9 state machine of BL_RCVR protocol module

4. Demo System for CDC

The demo system for CDC is a sample witch realized all the function of Belle2Link

4.1 Hardware

The demo system consists the following modules (Fig.4-1).

· COPPER board (mother board, 9U size)

· High Speed Link Board (Plugged to COPPER)

· CDC readout board (Linked to HSLBs with optic fibers)

· Server PC (file system server)

Fig. 4-1 Hardware components for CDC setup

4.2 Firmware

HSLB main FPGA firmware ( Xilinx Virtex 5 XC5VLX50t-1ff1136 )

HSLB CPLD firmware (online downloading)

CDC FPGA firmware

4.3 Application Software and HSLB Driver

The Software

Data processing

· Readhslb : read the data of FEE from the COPPER;

Slow control

· paraconf : read or configure the register of HSLB and FEE

· fileconf : transfer file to FEE, such as a DSP ROM file

Driver

Inherit from Higuchi-san’ Finesse general driver

5. Supplements

_1234567890.vsd

�

�

�

�

�

HSLB INTERFACE

FPGA

XC5VLX50T

FLASHXCF16P

CPLD

XC2C64A

Optical transceiver

Optical transceiver

RJ45

RocketIO

Clock

Control Signal

Data Bus

Address Bus

Trigger Signal