Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

Operational experience with CCUin the RPC Trigger

University of Warsaw

for RPC Trigger Group

20 January 2010

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

Introduction

Original concept for TRACKER needs

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

Architecture:- 18 CCU token-rings (10 barrel + 8 endcaps).

- 3 CCS boards hosting 6 mFECs each.

Present tasks:1. LB/CB configuration from Flash (power on, etc).

2. Masking noisy strips from DB on LB.

3. Opto synchronization of Master LB links.

4. Collecting LB/CB monitorables.

5. Updating LB/CB firmware/params stored in Flash.

6. Setting up FEB thresholds, FEB monitoring.

Eventually: tasks (1)-(3) will be done via TTC command.

CCU for the RPC Link System

Eth cables

Opto fibersDOH

Eth cables

1 CCU-ring = 6 LBBs = 12 CBsFEC

Topology of a CCU ring

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

CCU for the RPC Link System

1 ring = 6 LBBs (3 in YE3)

1 LBB = 2 half-boxes1 half-box = 1 CB + up to 9 LBs

1 ring = 12 CBs (6 in YE3)

Control Board

Link Boards

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

CB: Too short CS strobes in CCU block transfer

It appeared, that in block transfer the CS low pulse length was always the same as at the highest speed (10MHz pulse length ca. 50 ns), even when we selected the 4MHz speed.Such short pulses were too short for our logic do decode address, to send control signals via Control Bus and/or to receive data

Delayed readEach read access triggers the real read on our control busThe read value is returned as the result of the next accessIn result the first read in the block returns random data and the lest read is not performed

Delayed writeEach write access latches the address and the data.The real write access is performed between two consecutive write accesses

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

CB: Limitations of embedded CCU I2C controller

The I2C controller built into the CCU does not support multiple I2C accesses (writes of multiple bytes to the same address, or reads of multiple bytes from the same address) only limited support for 4-bytes transfer was availableUnfortunately the chips used in RPC FEBs require such multiple accessesAdditional I2C controller had to be implemented in the CBPC FPGA to provide access to FEB chips

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

CB: Problem with PIA lines induced by spontaneous CCU resets

Sometimes errors in the CCU ring induced glitches on the PIA CCU lines. Sometimes the lines became locked on the high level (probably due to switching to the input mode and pull-up resistors set to the high level)

As a workaround we had to avoid using PIA lines to control our chips and, instead, to use internal registers implemented in FPGA, accessed via memory port for this purpose.

Sometimes errors in the CCU ring trigger random writes and reads on the memory port, which (in case of writes) corrupt the vital data in our chips.

- the protected block in FPGA is accessible only when this register is set to the

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

Two LBBs in the same rack:

Two LBBs on different UXC levels:

GND_pinsGND_shield

CCU cable

CB

LBB

GND_cbGND_lbb

CB

LBB

GND_cbGND_lbb

LVDS transmitter LVDS receiver

Signal pins are not shownfor simplicity

Noise: Improved grounding

GND_pinsGND_shield

CCU cableGND_outGND_in

LAN-trafo

CB

LBB

GND_cbGND_lbb

CB

LBB

GND_cbGND_lbb

LVDS transmitter LVDS receiver

Signal pins are not shownfor simplicity

Reduced EMC pickupsafter direct couplingof the grounds(i.e. CAT 6 cable used according to its specs!)

Reduced EMC pickupsafter capacitive couplingof the grounds

Also installed adequate magnetic shielding to prevent saturation of LAN-transformers when solenoid is ON

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

Noise: Improved CB inputs

Two modifications applied to each of 192 Control Bords in order to filter signals for both channel A and B in the electrical part of a CCU-token ring:

Before LVDS receiver for CLOCK, DATA and RESET signals:

After LVDS receiver for RESET signal:

2 x 50

nF

nF

100

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

CCU speed

1.With our present XDAQ software (based on FecSoftware V3.0) we

measure for a single CCU-ring speed of:

about 4100 8-bit FEC operations per second (read or write).

2.In practice it means that:

the reset and configuration of all LBs in the system (1232

boards) takes about 1.5 minutes.

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

Ring B

1.

2. mFEC and CBs has been used for daily

operations.

3.For two CCU-rings successfully tested bypassing of selected LBB

half-boxes using generic FEC software (ProgramTest). Results were

not 100% reproducible and sometimes commands had to be repeated

many times.

4.

support for chanel

next reset FEC command.

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

Summary

Operational issues with CCU over past years:

1.Electrical part of CCU-token rings subjected to EMC noise in the

UXC cavern.

2.Several workarounds were necessary in CB firmware to mitigate

CCU25 bugs and noise on the signal cables.

3.Limited bandwidth of VME/CCS/FEC chain increases time needed

to configure the RPC link system.

4.Loose mFEC optical ribbon connector in 1 of 18 rings (RB+1 FAR).

Overall smooth operation with the L H C beam in December 2009 (i.e. people not allowed inside UXC , uninterrupted LV supply, full B field)

FIXED

FIXED

FIXED

Pending

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

Backup slides

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

CCU rings for RPCs

Example for Barreland YE1

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

CCU rings for RPCs

Optical part:

Connects FEC with DOH on the CB

Each path uses 4 fibres: CLOCK_in, DATA_in, CLOCK_out, DATA_outTwo CBs are equipped with DOH mezzanines

Electrical part:

Interconnects: 6 LBBs (12 CBs) in barrel wheels and YE1 disks or 3 LBBs (6 CBs) in YE3 disks.3 differential signals: CLOCK, DATA, RESETCAT6 Ethernet cables having: 4 individually shielded wire pairs,

common outer shielding grounded at one end4 LAN transformers between UXC levels

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

Electrical part

CCU copper signal cable:

CAT6 Ethernet cable as a medium.8 signal wires (4 twisted pairs).Common shielding + aluminium foil for

each signal pair.

In our case: 3 pairs are used for CLOCK, DATA & RESET signals, 1 pair is reserved for GND.It was decided some years ago that:

Ground pins and shielding should be connected to the CB ground only at one end (LVDS transmitter side)Between UXC levels signal pinsshould be galvanically insulatedby means of LAN-transformers (LVDS receiver side).

Maciek Kudla, 20 January 2010 CERN, Electronics Week, Front End Control Workshop

Grounding before improvements

GND_pinsGND_shield

CCU cableGND_outGND_in

LAN-trafo

CB

LBB

GND_cbGND_lbb

CB

LBB

GND_cbGND_lbb

LVDS transmitter LVDS receiver

Signal pins are not shownfor simplicity

LBBs in the same rack:

LBBs in different racks on different floors:

We are avoiding ground loops... ...but cables are susceptible to EMC pickups

GND_pinsGND_shield

CCU cable

CB

LBB

GND_cbGND_lbb

CB

LBB

GND_cbGND_lbb

LVDS transmitter LVDS receiver

Signal pins are not shownfor simplicity