Tracker Strip and Pixel FEDs

John Coughlan

Tracker Readout Upgrade Meeting

September 12th 2007

Strip FED

9U VME64x

96 optical fibres

Analogue ORx

96 ADC channels

Limited by I/O Component density

1 nsec timing

Board mounted

Digital FPGAs Xilinx

Cluster Algorithms

Raw input data rate >3 GB/s.

Processed Output rate < 200 MB/s

VME FPGA

Front-End data processing FPGA

Power DC-DCs on board

Output to DAQ

Event Builder FPGA

ModularFE Unit

96 ADC channels

Double Sided Board

Opto Receivers

TTC rx

450 x boards + Slinks

Synch APV

Cost Driven

Strip FED Board Manufacture

Board parameters:- 9U x 440 mm VME64x form factor- Optical/Analogue/Digital logic ; 96 ADC channels- Double-sided (secondary side with half of analogue channels)- 6,000 components (majority of passives 0402) (finest pitch < 20 thou)- 25,000 tracks- 37 BGAs (typical FPGA 676 pins on 1mm pitch). All BGAs located on primary side.- 14 layer PCB (incl. 6 power & gnd)- controlled impedance

High density components.Close up of analogue section on primary sideAlmost all components on board are Surface Mount.

VALUE is in the COMPONENTS

Design for TEST

Must have HIGH YIELD

Pixel FED9U VME64x

36 optical fibres

Analogue ORx

36 ADC channels

Limited by SLINK output rate

1 nsec timing

Digital FPGAs Altera

Inputs are ZS

Unsynchronised Events on fibres

ROC 16 events

Build DAQ Events

Timestamp TTC

Encoded header

Pixel FED

9U VME64x boards

+ Slink

Mezzanine cards 9 Analogue,5 FPGA Altera cards(not ORx)

Common Electrical SLINK

P3 64 bits @ 80 MHz

+ TTS throttle FMMs

40 x 9U boards + Slinks

Reduces Motherboard complexity 10 layer(Micro vias for Alteras on mezz)Less risk. Testing, repairing easier. Spares

Takes more spaceConnector reliabilitySignal integrity. Terminations

Power

Strips and Pixel FEDs–Same Form Factor 9U VME . 450 vs 32 boards

–Both Analogue & Digital designs

–Same Interfaces ORx, ADCs

–Different Constraints on Channel Count ; I/O vs Output rate

–Same TTC, SLINK

–Same fine clock skew 1 nsec

–One board type in each system.

–No trigger functions.

Minor differences, power supply, slink connectors, fpga config, vme monitor bus

Different PCB implementations

- Motherboard vs Motherboard + Mezzanines

Main Difference Digital Processing. (Altera vs Xilinx FPGAs)

- Data Inputs. Raw & Synch vs ZS & Unsynch

FED Lessons“Good Design and Robust Manufacturing”.

–Design evolves in response to technology (ASICs in TDR) –Invested a lot to get design right first time (2 board versions.)

–Protoype to production was a long process.

–Started at cutting edge finished with v. mature technology FPGAs (support)

–Cost estimates evolve ADCs, FPGAs, Manufacture. Got FPGAs right.

–Support of detector development v. underestimated , PMCs cf 9U FEDs

–Prototype support underestimated ~ 50 proto + pre-prodn 9U FEDs

–Tender process takes a long time. (not needed for pxFED)

–Both ended up with “local” manufacturers. Good relationship essential.

–Quality Control critical. Early BGA failures. V. High production yield. (also pxFED) No problems ORx assembly

–Doing board testing at Assembly Plant took effort but paid off

- Need for Custom Test equipment Optical Testers

FED Lessons–System interfaces came late. Electrical SLINK cards, FMMs. More elegant solutions.

–Choice of industry standard VME for mechanics, power was good one

–VME bus as monitoring is poor , better Ethernet (late design change?)

–VME64x features not all used. (plug & play cost pxFED one unnecessary board iteration). Keep it simple.

–Think about practical issues cf prototypes with final system , FPGA configuration.

–Design issues with cooling. Tests in final configurations necessary.

–(Un)expected effects (temperature variation of optical inputs for pxFED encoding)

-Boards are built, debugged and delivered. But Firmware is never finished.

–Custom protocols , SLINK, TTC, -> custom Firmware. Duplication of Firmware blocks (across CMS), SLINK, TTC, I2C, VME.

– Firmware Libraries? Commercial/Vendor

–Duplication of test software? Software effort ? (costings)

Future FED Possible Implementation

FPGASwitch

Off Detector sFED

SDRAMBuffer

SNAP 12

Rear Transition Module

STTC

10G SerialBackplaneDAQCrate Event Builder

12 x 3 Gbps

FE

ATCA Crate 8U

Cntrl/Mon

PowerORx

FPGAsMGBTsGBT PHYMACSparsification1 per ORx

ORx and FPGA on Mezzanine?

ORx

ORx

ORx

ORx

ORx

ORx

ORx

Mezzanine Prototype

GBT

Industry StandardsATCA cratesIndustry Protocols Data TransmissionSerial PCIe …More use of Commercial Firmware cores. Data protocols, memory

COTS Carrier Motherboards+ CMS Mezzanines,Transition cards

Future FED-FEC

Integrate FEC functions on FED

FPGASwitch

Off Detector sFED

SDRAMBuffer

MPT 12

Rear Transition Module

STTCTriggerThrottle

10G SerialBackplaneDAQCrate Event Builder

12 x 3 Gbps

sTTC

FPGAsMGBTsGBT RecvMACZS1 per ORx

sAPV

ORx

ORx

ORx

ORx

ORx

OTx

ATCA Crate 8U

Cntrl/MonEthernet

PowerORx

ORx and FPGA on Mezzanine?

Digital Inputs

Zero Suppressed FE Data Inputs?Constraint Data Volumes on output

Final System Ideas New sDAQ (sFEDs connected direct to Filter via Super Event

Builder Network) New sTTC (Broadcasting Filter Addresses to FEDs) Crates

Just Mechanics, Power, Cooling. -> Control/Monitoring via Ethernet.

Serial Backplane based crates (Telecom ATCA , VME46?). Less Slots (but wider) Better Power & Cooling ? Better control & monitoring ? FED Event Builder Crate

Future FEDs–Common Tracker FED h/w probably technically feasible. Practical?

–Common CMS FED ? Common SLINK, Common FEC

–Go to Digital Input Data

–Zero Suppressed at FE tbd

–Large systems channel counts. Large form factors (cost driven).

–Constraints on Channel count?

–Not considered Tracker Trigger

–Use of Emerging Industry Crate Standards e.g. Telecomms ATCA (VME)

–Exploit Industry Data Protocols e.g. Serial PCIe (SLINK)

–More use COTS Vendor Firmware cores, Industry standards

–Use common COTS ATCA Carrier boards

– with custom CMS Mezzanines (cost effective)

Spare Slides