Project Overview

Ted Liu

Fermilab

Sept. 27th, 2004

L2 Pulsar upgrade IRR Review

http://hep.uchicago.edu/~thliu/projects/Pulsar/L2_upgrade_meeting.html

Materials for the review can be found at:

Q&A

Are we ready for operation? No

Why are we having this review then? To help us getting ready for final operation. To review

- what we have achieved so far

- our plans for final commissioning

- is there anything missing on our to-do-list?

- is manpower enough?

- Is the schedule realistic?

- …

Will have another review early next year

Qutline

Baseline design for operation and overall status Project Organization: who is working on what General testing&commissioning methodology Firmware Overview and necessary improvements Firmware Version Control Comments on possible future system optimization Manpower issues Summary

Baseline for operation

Pulsar-based system is very flexible Should only focus on the baseline for operation

single algorithm node deliver non-SVT and SVT data on separate PCI path simple algorithm in pre-processor Pulsars simple data merging use S32PCI64 cards instead of FILAR …

Will comment on some system flexibility later further possible improvements/optimization depend on

initial operation experience, not required for operation

L1 muon

L1 XTRPL1 trigger

L2 CAL(CLIST/Iso)

PreFred

ShowMax

(RECES)

PC

SLINK

TS

SVT

Muon

Cluster

Electron

Merger

Merger

SVT

L2toTS

Commissioning strategy:(1) Use Pulsar Tx Rx in teststand(2) Split all input signals to Run IIa

system, run parasitically with detector and beam

Baseline for operationPulsar pre-processors

Main tasks for the project

Hardware: custom: Pulsar,mezzanine&AUX, LVDS & fiber active splitter commercial: fiber passive splitter, SLINK mezzanine, SLINK-PCI cards, Linux processors Firmware: transmitters and receivers for all data paths SLINK merger and L2 to trigger supervisor interfaces about 14 different types of Pulsar firmware Software: board testing, VME DAQ readout, online/offline monitoring, software for algorithm&control node, interface with rest of world …

Overall Status Hardware:

All custom hardware in hand, and all fully tested

except fiber active splitter (see Chris’s talk)

need to order two Algorithm PCs soon, one in hand Firmware:

All firmware in place and tested in beam…some needs

fine tuning for either performance optimization or robustness

more on firmware later Software:

main work left is related to algorithm and control node

see later talks for details

Who is working on what (I)

Muon/XTRP/L1 + SVT + Merger + RECES + L2TS Tx Firmware: Tomi (work done, left) Rx Firmware: Sakari Testing: Burkard, Frans (left), Cheng-Ju (L2TS) + Sakari Commissioning: Burkard + Cheng-Ju Monitoring for Muon/XTRP/L1/SVT/Merger: Shawn Monitoring for L2TS: Cheng-Ju Monitoring for RECES: Chris with Daniel’s help earlier Active Splitter for RECES: Chris LVDS Splitter Board: Wojtek

(only the main soldiers are shown here)

Who is working on what (II)

Cluster/PreFred Tx Firmware: Tomi (work done, left) Rx Firmware:

DataIO FPGA: Vadim

Control FPGA: Sakari (similar to other cases) Testing/commissioning/monitoring: Vadim and Wojtek

Algorithm node software: Kristian Control node software: Daniel

(only the main soldiers are shown here)

Who is working on what (III)

General Pulsar Hardware: Burkard VME readout software: Jane from DAQ + Burkard’s help PulsarMon design: Vadim RunCTRL/TriggerDB etc interfaces: Daniel Firmware coordination: TL

Commissioning coordination:

in the past: TL

in the future: Trigger SPLs

(See Cheng-Ju’s talk on plan and schedule)

Testing and Commissioning Methodology

Pulsar is self-testable, at both board and system level This design feature is the key to success

Pulsar Hardware testing all done with self-testing capability Pulsar firmware testing mostly done with self-testing capability Software development all done with Pulsar self-testing capability Even at system commissioning, a few Tx boards are used This has made the commissioning work much easier

System commissioning relies heavily on input signal splitting Saved us LOTs of dedicated beam time So far, only used a few hours dedicated beam time Input signal splitting has been and will be extremely helpful Will keep input signals split even after final commissioning to have a “copy” of the new system running parasitically for hot spares, for firmware/software improvements

Pulsar Firmware: BIG effort

Pulsar has three main FPGAs L2 decision upgrade:14 different type Pulsars (Tx/Rx) Key to success:

Design: common re-usable modular design Simulation:

no simulation no testing Dedicated testing support Version Control Careful firmware

update procedures …

081          L2 Pulsar Muon/XTRP Rx IIa083          L2 Pulsar SVT Road Warrior085          L2 Pulsar Muon/XTRP/L1 Tx or SVT XTRP-emu086          L2 Pulsar Muon/XTRP/L1 Rx IIb087          L2 Pulsar RECES Tx088          L2 Pulsar RECES Rx089          L2 Pulsar Cluster/PreFred Tx090          L2 Pulsar Cluster/PreFred Rx091          L2 Pulsar SVT Tx092          L2 Pulsar SVT Rx093          L2 Pulsar Merger Tx094          L2 Pulsar Merger Rx095          L2 Pulsar L2TS “Tx” (reserved spare ID)

096          L2 Pulsar L2TS097          L2 Pulsar L1 Scaler098          L2 Pulsar SVT Track Fitter

099          L2 Pulsar test Tx100          L2 Pulsar test Rx101          L2 Pulsar XFT Stereo Tx102          L2 Pulsar XFT Stereo Rx

103 L2 Pulsar SVT Hit Buffer 104 L2 Pulsar SVT AMSRW Board ID

On

e H

ard

ware

-- many

ap

plica

tion

s

14 Pulsar types/firmware for L2 decision upgradeSakari

TomiSakariTomiSakariTomiVadim/SakariTomiSakariTomiSakari

Sakari

SakariSakari

people

other types for SVT/XFTor future improvement

Interfaces (Tx/Rx):SVT/XTRP/L1/PreFred

TS & SLINK (P3)

Interfaces(Tx/Rx):

MuonRecesCluster

IsoSlink

FPGAs and data interfaces

General simplified data flow

Mezzinterface

DAQbuffers

algoslink

formatter

Same as above

VMEresponse

CDFCTRL DataIO

DataIO

DAQbuffers

merge slinkformatter

DAQbuffersVME

response

CDFCTRL

DAQbuffers

XTRP/SVT

Well organized firmware effort is the key: clean/easy-to-understand/robust with minimal

maintenance effort

Inputinterface

DAQbuffers

algoOutputFIFO

Slinkformatter

VMEresponse

CDFCTRL

DAQbuffers

• all common parts are shared by all firmware• also share many low level routines (latency timer, bunch counter etc)

DataIO or Control FPGA

VME responsesSLINK formatter

DAQ buffersCDFCtrl responses

Mezzanine interfacesLatency timerBunch counter

…

Common core firmware (library)

Knowing one data path Pulsar knows all PulsarsData specific algorithm difference is just minor details

hotlink

muon

cluster

Taxi Iso

showermax

LVDS+ ext. FIFO

SVT

track

LVDSL1

TS

core firmware developed by people dedicated to the task

Formatversion

Datasource

RegionID

ReservedBunchcount

Buffer#

8 4 2 8 28

Data elements

Latency (previous) Latency (current)

16 16

Data size Error flags

1616

Keep it simple

SLINK data format

was defined asdata size before,which means onehas to wait untilall data arrivesbefore sending outSLINK package.

will remove it,and use it aslatency stampinstead.

SLINK Merger fragments into event package

Slink inputs

AUX cardSlink output

Four Slinkmezz cards

Merger header 2

Merger trailer

Input 1 header 1

Input 1 trailer

Input 2 header 1

Input 2 trailer

Input 3 header 1

Input 3 trailer

Input 4 header 1

Input 4 trailer

Slink input 1

data

Slink input 2

data

Slink input 3

data

Slink input 4

data

Input 1 header 2

End of fragm

ent

Input 2 header 2

Slink merger output

End of fragm

ent

Input 3 header 2

End of fragm

ent

Input 4 header 2

End of fragm

ent

0xE0F00000

Merger header 1

32-bit SLINK to 64 bit PCI interface card: S32PCI64

• highly autonomous data reception• 32-bit SLINK, 64-bit PCI bus• 33MHz and 66 MHz PCI clock speed

Slinkto PCI

Slinkto PCI

mem

CPUPCI to Slink

data

data

L2 decision

General comments on Firmware

Firmware is not hard if it just has to be “sort of working” The hard part is to get it “rock solid” Firmware is not a place to show how smart one is rather to show how many mistakes one could make

Firmware testing is as important as writing/simulating Firmware testing is often much more time consuming

than the actual writing How good the firmware depends on how well it is tested

Our general strategy:

firmware developer supported by people dedicated to testing

• Files kept• VHDL source code• Files necessary for recreating this version• Readme with revision history• FPGA configuration files the FPGAs

• Source code (working version) under CVS control

• Firmware archive updated on different PCs

How to keep Firmware in order

We have so many different types of firmware Keeping them in order is non-trivial:

under CVS control: source VHDL code, pin files etc Firmware version ID VME register per FPGA

(has to update by hand) README file update Follow careful firmware update procedures

There has been no confusion about firmware versions when the procedure has been followed

te

slot 05: IDPROM: 0016 086 PULSAR MUON RX CTRL: 04408060,DataIO1: 08407020, DataIO2: 08407020 slot 13: IDPROM: 0004 092 PULSAR SVT RX CTRL: 05408060, DataIO1: 09407280, DataIO2: 09407280 slot 15: IDPROM: 0028 093 PULSAR SLINK TX CTRL: a0406300, DataIO1: 08407230, DataIO2: 08407230 slot 17: IDPROM: 0061 096 PULSAR TS INTERFACE CTRL: 0a407280, DataIO1: 09408030, DataIO2: 09406040 slot 19: IDPROM: 0012 094 PULSAR SLINK MERGER CTRL: 02408080, DataIO1: 01407220, DataIO2: 01407220 slot 21: IDPROM: 0014 094 PULSAR SLINK MERGER CTRL: 02408080, DataIO1: 01407220, DataIO2: 01407220

Example of Pulsar crate ID&version scan by Burkard on Aug 19th after Beam test:

Not all boards are shown due to limited spaceAllow us to go back to the same configuration after shutdownHow can we check these against database when in operation mode?

Necessary firmware optimization

before operation

Reces: zero-suppression + latency optimization Reces data comes into Pulsar 8 us after L1A current firmware is not optimized needed to reduce Reces path latency by a few us Status: in progress, estimated time: ~ one month

Slink formatting: sending data words out

upon arrival (not waiting for all data to arrive) improving Slink merging latency (~ a few us) estimated time: ~ one month (including testing)

More timing measurements add latency timer in a few other places (relatively easy)

Cluster path improvement (see Vadim’s talk)

Future possible system optimization after operation

Use FILAR cards instead of S32PCI64 Remove the final merger, data directly goes into FILAR Possibly also remove Reces merger with another FILAR Also allow us to run SLINK faster: 40 MHz ~60 MHz Should reduce the latency by ~ few us Need PCI software modification (no firmware change)

Move L1 scaler into firmware if needed Move to 4 PCs instead of single PC (how much do we gain?)

Additional firmware (L2TS) work and some software work

Muon and track matching in firmware Bring upgrade XFT stereo info into L2 (this will happen) …much more, depends on what we learn&need by then

FILAR can improve the system latency Higher bandwidth, less latency overhead

64-bit/66MHz PCI bus, four 2 Gbit/s S-LINK channelsReduced PCI protocol overhead (compare to S32PCI64)

L1 muon

L1 XTRPL1 trigger

L2 CAL(CLIST/Iso)

PreFred

ShowMax

(RECES)

PC

SLINK

TS

SVT

Muon

Cluster

Electron

Merger

Merger

SVT

L2toTS

System with FILARs

Pulsar pre-processors

Send all data pathsdirectly into 2 FILARs…

Manpower Issues

Why should we worry? Tomi and Frans left (two key people on firmware/testing) Only10% Burkard’s time available from now on

(key person on hardware/VME software/firmware testing/system commissioning need replacement SOON)

Kristian needs to graduate …

(key person on algorithm/PCI software need

replacement SOON …) Need new L2 software coordinator (Daniel?) … anyone else will be full time on the project?

This is very serious …need this committee to help!

Summary Review should focus on baseline for operation Keep in mind that system is flexible Hardware is in good shape

(still need to fully test the active splitters) Firmware is in reasonable shape, needs fine tuning/testing in

some cases

key is to follow our testing methodology Software needs more work, no show stoppers

clear spec is the key Manpower issues serious (in a few areas): need new people! Details are in the rest of talks …