STAR Forward GEM Tracker Readout/DAQ/Controls Subsystem

FGT Review – 6/14/2010 BNL J. T. AndersonG. VisserJ. T. AndersonG. Visser

1

STAR Forward GEM Tracker Readout/DAQ/Controls Subsystem

Renee FatemiUniversity of Kentucky

John T. Anderson, Dave UnderwoodArgonne National Laboratory

Gerard VisserIndiana University Cyclotron Facility

Paul NordValparaiso University

(and the STAR DAQ group, T. Ljubicic, J. Landgraf, BNL)6/14/2010


2

Simplified outline of readout/DAQ… and this presentation…

FEE

Detector

HVPS ARM ARC

Crate

Cables, connectors, patch panel

PC (FGT DAQ)

STAR Trigger (TCD)

(DAQ room)

(electronics platform (south))

(in WSC)

You will hear about:

• Detailed design choices in most of these labeled elements• How we move the stored analog data

from APV chips• How we digitize it• How we zero suppress, buffer, and

deliver it to DAQ software• Care & feeding of FEE (power, clock,

trigger, I2C)• Safety, reliability, serviceability

• Current status• Schedule of remaining work• Budget and costs

HV


3

Transporting the APV readout signals

This cable diagram shows only the one analog output and the ground reference (from TPC ground to the detector). Actual connections include of course 10 outputs, control signals and power.

(10x scale)

0.5V peak

∆VGND

GND (TPC)

Actual ground noise at STAR (TPC west w.r.t. platform)

17 m (73 ns)3.3 m (16 ns)

Isig

FGT FEE Assy.

twisted pair twisted pair

1 1 1 0 1 0 1 0 1 0 0 1

APV ONLY – 110 Ω LOAD

The APV “digital” header provides a convenient test pattern! We expect to actively use this for calibration, and at least as a noise check.

If the APV analog signals are seen with noise >> “digital” signal noise, then the cable / readout system is not contributing any significant noise. This goal is met!

-/+ 4mA

+/- 4mA


4

Patch panel Approximate radius of WSC

Approximate location / size of FGT patch panel (2 ea.)

Typical FGT cable route. With (mainly TPC) cables in sector gap “channels”.


5

Clocking scheme

• APV sample clock should be synchronous to bunch crossing clock (“RHIC strobe”). This gives best amplitude measurement w/o need for a timing fit and correction.

• 4× RS is close to APV nominal sample frequency (40 MHz) so we use that. 3× RS would also be a reasonable option.

• APV offers choice of readout clock = sample clock or sample clock/2

• The half-rate output (2× RS) is quite sufficient for FGT and other APV applications in STAR: 3-pt readout requires 420/(18.766 MHz) = 22.4 µs which fits very well with the calorimeter deadtime (~27 µs except BSMD).

• In the half-rate mode the half-cycle readout phase is defined by the I2C address of the APV chip. We are careful to choose address values to keep all readouts aligned, this allows maximal settling of crosstalk in the signal cable prior to ADC sampling.

• Clock adjustment/distribution:

• incoming clock from trigger (TCD) common to all FGT. This clock has adjustable phase, useful for global timing scan.

• multiplied (4×) clock with phase control per FEE group (10(12) chips) (most likely will use identical settings but it is convenient to provide individual PS circuitry)

• the 4× clock also to ADC’s with phase control per FEE group. This phase control is tuned to compensate for cable length.

• changes in beam timing, etc., are compensated globally at STAR, or if necessary with FGT TCD phase control


6

Choosing the ADC

Requirements

• ≥40 MSPS

• 12 bits (for >10 bits effective resolution)

• input bandwidth sufficient for 20 MHz stepped waveform

Goals

• high density packaging and appropriate pinout

• single supply

• low power

• serial interface preferable

AD AD9222BCPZ-40AD AD9228BCPZ-40Linear LTC2292CUPLinear LTC2171CUKG-12TI ADS5240TI ADS5281IRGCNational ADC12DL040

Best options at present

Selected

Quad 40MSPS, 11.7 ENOB @ 30 MHz, 50 mW/ch

Newly available (Q1 2010)


7

Driving the ADC • At higher supply voltages, one can get away with single-ended drive. Not here, with a 1.8 V supply ADC.

• Transformer coupling with digital DC restoration was considered, but this is too hard. LF cutoff is nonlinear and temperature sensitive. Rejected.

• Driver requirement: Faster than filter response, low power, differential output. DC accuracy is not that significant. Prefer to use existing power rails i.e. +1.8, +/-5.5 for improved layout/performance.

• Filter requirement: Minimum bandwidth while providing 6 ns flat sampling window to allow for cable skew and clock jitter/drift.

16 ns flat (<0.1%) sampling windowDifferential driver alone, response and imbalance

Fast test pulse

Out +

Out -

Out CM53.3 ns APV read cycle

+/- 5.5V power, 38 mW

From line receiver

To ADCImplementation:(Several other low power drivers prototyped. There exist many others not low power, not considered.)

+1.8 (ADC)

(driver/filter test pulse response)


8

Driving clock and trigger to the FEE

D

C 2 1 .0 U F

D 3 . 3 V

T1

A D T1 -1 W T

4

2

63

1

5

T2

A D T1 -1 W T

4

2

63

1

5

D

C 3 1 .0 U F

R 6 1 0 0

C 4 1 1 . 0 U F

U 2

S N 6 5 L V D M 1 7 6

R1

R E2 D E3

D4

G N D5

A6

B7

V C C8

U 3

S N 6 5 L V D M 1 7 6

R1

R E2 D E3

D4

G N D5

A6

B7

V C C8

R 1 2 1 0 0

• LVDS with transformer coupling (at ARM) to maintain high common-mode impedance to the line

• precision line receiver buffers signals before presenting to APV

• DC offset applied to TRIG line, which works since duty cycle is guaranteed low (BUSY >> 25 ns)

33 m cable (1424A), P6247 probe


9

Power to FEE• Floating output low noise push-pull converter 5V to ≈3V

• Remote regulation to 1.8V

• This is a small modification of circuit successfully used on another project (GlueX ADC)

• Test board will be fabricated


10

APV / cable / ARM frontend prototype results

APV header data w/ equalization & shaping filter Last header bit

2nd analog level

Cable & ARM do not limit resolution

For reference: unequalized response


11

ARM and ARC dataflow

32 kB buffer

4096 buffers

(128 MB DDR SDRAM)

1024 × 33

Output fifo

SIU8192 × 33

Buffer write fifo

4096 × 12

Token fifo

1024 ×33

Backplane

Channel output fifo

512 × 12

Acquisition buffer

Take table

≥ ?

Threshold table

ADC

Other channels

Never busy (by construction)Covered by deterministic busy algorithm in TCD


12

Readout system integration schematic (simplified)

ethernet trig/clkDDL fiber

Wiener MPOD controller (1×)

ISEG 8 ch HV module (3×) −5 kV

ARM (12×) APV Readout Module2×(12×APV) each [IUCF]

ARC (2×) APV Readout Controller6×ARM each [ANL]

208 V 1φ

Wiener crate (1×) (FGT custom) 6U × 220mm cards

8 2+2

FGT Cables24 FEE signal24 power & remote sense24 HV coax

Cable break patch panel located just outside of west support cylinder, on TPC wheel. This point also serves as FGT detector ground tie (to TPC wheel).

Internal FEE signal cables: custom low-mass aluminum design with silicone & FEP extruded insulation

Internal HV cables: Thin coax (5kV per CERN/DESY spec)

Measured cable run 55 feet[ WSC → TPC sec 2/3 boundary → sec 2/3 tray out of magnet → 2nd level platform ceiling tray → 2C7/8/9 ]

SGIS (to kill all FGT power)

FEE cablesRear entry via 6U × 80mm transition board


13

Readout / HV crate• three HV slots, one HV/crate controller (MPOD)• two readout segments each with 7-slot CPCI backplane (custom bus usage not PCI)

• six readout board (ARM) slots• one readout controller (ARC) slot

• all modules 6U × 220 mm• PS ratings 24 V @ 23 A for HV modules, 2 × ( 5 V @ 115 A ) for readout

• we expect to run about 55% of this• controls via ethernet/SNMP w/ EPICS support• ordered/received 1 + 1 spare, presently at ANL and IUCF


14

ARM (APV Readout Module)

P2 96-pin passthroughFEE signal/power connections

“P1” readout backplane to ARC (CPCI)

ADC

Driver

Filter

Line RX

FEE power1.8V isolated remote-regulating supply

Currently working on ADC footprint & support component placement

FPGA (3×)

Handles 2 groups of up to 12 APV each

FGT will use 2 groups of 10 APV


15

ARC (APV Readout Controller)

FPGA (XC3S700A)

LEDs

SIU (optical link)

Ethernet

JTAG

Trigger

Auxiliary NIM I/O

“P1” readout backplane to 6× ARM (CPCI)

DDR memory128 MB(main data buffer)


16

Data size and rates

• APV data frame consists of 140 datapoints (12 APV-header + 128 channels)• For the typical 3-pt (per channel) readout, that’s 420 datapoints• Take 10 pre, 10 post to monitor baseline / eq. 440 datapoints• NZS fixed-array format, 2 bytes each (1.5 significant, could be packed)• ARM supports 24 APV 21120 bytes (raw) per event per ARM + 32 byte header• ARC 126912 bytes (raw) per event +64 byte header = 126976 bytes• SIU bandwidth 200 MB/s @ 50 MHz max1.58 kHz trigger rate non-ZS• Crate backplane bandwidth 120 MB/s @ 30 MHz max 945 Hz trigger rate non-ZS

• or by packing the data over backplane, then max 1.26 kHz non-ZS• for 1-pt readout, take 160 points, then SIU limit 4.3 kHz, backplane limit 2.59 kHz• To go beyond these rates, we will do ZS onboard ARM. Arbitrary neighbors included

(defined by table).• ZS event size limited to 32 kB (occupancy limited to ≈25%). For 4096-event buffer use

128 MB (SDRAM).


17

Power dissipation and cable voltage dropsSTAR Forward GEM TrackerTable of Power Dissipation and Voltage Drops

G. Visser / Indiana UniversityLast revised 3/23/2010

Item Quantity (full FGT) R (if applicable) V A each W each W (full FGT) W

APV chip (+) 240 1.25 0.09 0.113 27.00

inside

169.59

APV chip (−) 240 1.25 0.155 0.194 46.50CLK & TRIG receivers (2x ADCMP604) 24 2.5 0.034 0.085 2.04regulator drop (+) 24 0.65 0.934 0.607 14.57regulator drop (−) 24 0.65 1.584 1.030 24.71regulator control power 24 3.6 0.025 0.090 2.16HV divider chain 24 5000 0.0003 1.500 36.00internal cable (+) #22 CCAW 11 ft 24 0.271 0.253114 0.934 0.236 5.67 cable power internal cable (RET) #22 CCAW 11 ft 24 0.271 0.17615 0.65 0.114 2.75 0.063W per ft per cableinternal cable (-) 2x #22 CCAW 11 ft 24 0.136 0.215424 1.584 0.341 8.19 external cable (+) #18 50 ft 24 0.325 0.30355 0.934 0.284 6.80

outside

330.45

external cable (RET) #18 50 ft 24 0.325 0.21125 0.65 0.137 3.30 cable power external cable (-) 2x #18 50 ft 24 0.163 0.258192 1.584 0.409 9.82 0.017W per ft per cablereceiver AD8129 @ 200 mV in 288 11 0.016 0.176 50.69adc driver LTC6403-1 288 5 0.011 0.055 15.84adc LTC2171-12 w/ linear supply 72 5 0.118 0.590 42.48FPGA (roughly) w/ linear supply 36 5 0.3 1.500 54.00ARM FEE PS (+) losses @ 72% eff. 24 0.752 18.04ARM FEE PS (−) losses @ 72% eff. 24 1.701 40.82receiver power losses @ 72% eff 12 1.643 19.71other stuff in ARM 12 5 0.75 3.750 45.00trig receiver 2 5 0.096 0.480 0.96FPGA (roughly) 2 5 0.5 2.500 5.00SIU 2 5 1.5 7.500 15.00other stuff in ARC 2 5 0.3 1.500 3.00

NOTE: Crate AC mains power supply losses not included here. Crate HV supply module losses also not included here.


18

DAQ hardware and software

• Hardware: 2 DDL SIU, 1 DDL D-RORC, 1 linux PC w/ PCI-X, 2 duplex fiber runs between 2C9 and DAQ room.

• We have all this hardware, installation is nearly complete, will be finished this summer

• Software:• FGT is present in STAR run control• FGT is present in DAQ monitoring• FGT DAQ front-end software is in place• Generic “DAQ Reader” handles FGT• Specific FGT unpacking software and online monitoring software to be

completed once the readout firmware and FGT channel mapping and geometry are finalized.


19

FGT slow controls – general overview and integration with STAR

EPICS-based control system

Standard in STAR

Integrates with STAR Alarms system and Logger

Variable naming convention FGT_*

IOC setup for Slow Controls on dedicated computer

FEE (controls & power) – will be set and queried through standard EPICS calls

Communication via ethernet through EPICS CA get/set calls

DAQ system software provides interface to FEE/readout hardware via DDL optical link

During run, readout hardware generates periodic status “events” rather than accepting control transactions

Setup parameters, e.g. APV registers, timing controls, are handled (and archived) through STAR run control framework

GEM HV & Readout crate LV

Controlled via ethernet through standard SNMP calls

SNMP library provided by Wiener

Using EPICS code developed by STAR TOF group

Prototype FGT controls GUI is operational with crate/HV system at IUCF


20

Safety…is taken into account in the design process

• Materials choices (e.g., cable to UL-1581 vertical tray requirements).

• Voltages and currents within connector and wire/cable ratings.

• Incoming power fused on all boards where source may deliver a larger fault current. Fuses of course are rated to interrupt maximum expected fault.

• Outgoing power (to cables) fused or inherently limited on all boards.

• FEE power and detector bias supplies sized appropriately, not excessive power

• Where ground break is needed for (non-floating) HV supplies, a safety resistor tie is used. Generally meaning 100 Ω leaded carbon-composition resistor.

• Cables will be routed neatly and not to present hazards to personnel or other equipment (snagging, tripping, etc.).

The FGT had a preliminary C-AD Experimental Safety Review on 9/2/2009. Relevant (electronics) action items were:

• What is the expected maximum temperature inside the [WSC] when the air flow is off? Assess if an interlock is required.

• Test the high voltage distribution system to 2x (operating voltage) + 1000 V

• Check the Weiner HV and crate and secure [C-AD] approval in lieu of NRTL.

• The Wiener HV supplies will be voltage limited to 5 kV. Add a sticker on each unit prior to use to assure compliance with the SHV connector [rating].

• Provide a sample of the custom signal and power cable as well as the special HV cable to J. Levesque to approve for flammability concerns.

• Review the ARC and ARM modules after design is completed.

All good / anticipated points.

All being addressed.


21

Budget, costs to date, current projections (equipment & materials)

Item Proposal budget

Cost to date (AY $)

Remaining (current projection)

Delta (actual + remaining – budget)

External cables $ 8,815 $ 5,623 $ 1,896 $ −1,296

Aluminum inner cable $ 0 $ 7,582 $ 0 $ 7,582

Connectors & patch panel $ 1,692 $ 261 $ 4,532 $ 3,101

ARM (qty. 12 + 6 spares) $ 48,477 $ 2,360 $ 43,428 $ −2,689

ARC (qty. 2 + 3 spares) $ 8,075 $ 6,633 $1,800 $ 358

Crate (qty. 1 + 1 spare) $ 14,308 $ 24,248 $ 0 $ 9,940

DAQ computer & DDL hardware $ 10,807 $ 8,356 $ 0 $ −2,451

GEM bias HVPS (qty. 3 + 1 spare) $ 76,549 $ 14,308 $ 0 $ −62,241

Shroud bias HVPS (qty. 1 + 1 spare) $ 0 $ 0 $ 4,356 $ 4,356

Shroud bias splitter & cables $ 0 $ 0 $ 3,500 $ 3,500

TOTAL FGT Readout/DAQ $ 168,723 $ 69,163 $ 59,512 $ −39,840


22

Completed tasks

• APV long cable readout test

• Rack / cable route assignment at STAR

• Crate specification, procurement, received

• ARM – ARC interface defined

• GEM HV PS selection, procurement, received

• Custom cable specification, procurement, received 1 of 2 types

• Cable routing at detector, power budget at detector; mechanical design is proceeding

• Service installation at STAR: fibers, ethernet, AC (nearly done), smoke det., water/air heat exchanger

• DAQ PC, D-RORC, DAQ & run control software is installed and ready to run

• Event buffering & transmission scheme (1st deployed in BSMD)

• ARC design, bare board fabrication ~now


23

Highlight of tasks and schedule to completion

1. Evaluation & revisions if required for ARC printed circuit board.

2. Firmware design for ARC.

3. FEE power supply & line interface test pcb & evaluation.

4. Complete detailed schematic and layout of ARM printed circuit board.

5. Fabricate and assemble 3 units.

6. Design ARM rear transition board and patch panel printed circuit boards. Includes final selection of connectors and mechanical integration drawings & hardware. Fabricate all.

7. Assemble cables and connectors.

8. Evaluation & revisions if required for ARM printed circuit board.

9. Fabricate & assemble 16 – 18 final ARM.

10.Firmware design for ARM.

11. Rack & crate installation at STAR. Partial cable installation.

12.Standalone STAR DAQ functional with ARC/ARM.

13.FGT beam test activities.

14.Specific data unpacking, online plots, slow controls milestone.

15.FGT final C-AD safety review.

16. Installation at 1006 Assembly Bldg to support FGT/WSC tests.

17.Complete installation at STAR.

18.Commissioning.

7/10 – 9/107/10 – 11/10

7/10

8/10

9/10

8/10

9/10 – 1/118/10 – 1/1110/10 – 3/118/10 – 1/11

8/10 – 11/1010/10

11/102/114/11

4/11

8/11

11/11 – 2/12


24

Closing remarks

• The Readout / DAQ / Controls subsystem provides a modern, fast, reliable, safe, efficient APV chip readout capability for the Forward GEM Tracker in STAR. (With potential other STAR applications to IST, PP2PP.)

• We are leveraging ideas and experience from other STAR detectors particularly the EEMC and BEMC, and from other work.

• Although there are some detail design tasks to be completed before production, all concepts and interfaces are worked out and expected to be stable. Many have been prototyped.

• Crates, custom cables, and ARC module are already fabricated or in fabrication now.

• We will be in a position to fabricate the remaining hardware soon.

• We will be ready for the test beam and for summer 2011 installation, within budget.

Thank you…


25

BACKUP SLIDES


26

8 mA p-p

Drifting electron cloud

-HV (GEM3 bottom)

APV25-S1 ASIC

About 55 feet of cable

Sample clock4x RHIC strobe 37.532 MHz

AD8129High impedance differential receiver

Sampling filter 12 bit ADCFPGA w/ SRAM

ARC module(to DAQ)

Minimize common-mode currents

Cable frequency response equalization circuit

Crate backplane

-1.25 V

+1.25 VOverview I: signal path schematic


27

Overview II: GEM quadrant assembly and electrical connections

Bias divider boardFR-4, 20 x 275 x 0.8 mm

“Connector” boardFR-4, 65 x 45 x 0.8 mm

FEE board (2x)FR-4, 259 x 53 x 1.6 mm

Terminator boardFR-4, 32 x 38 x 0.8 mm

Anode strip board and ground plane

STA

R T

PC

alu

min

um s

truct

ural

par

ts(S

TAR

sys

tem

gro

und)

GEM bias patchSHV cable connectors

LV & signal patchBud #PN-1320-C64 x 58 x 35 mmPolycarbonate, UL 94HB

Ground wire18AWG stranded

FGT

cabl

es to

pla

tform

1

2

3 4

2

Total power 6.9 W per quadrant

Slide from safety review (BNL 9/2/2009)


28

FEE Service Cable (Copper-Clad Aluminum / FEP / Al Foil)


29

Cables, patch points, and connectors

Item Status Description Length Conn 1 Conn 2 Conn 3

Inner cable Custom, on order

8c+14pr CCA/FEP

Various ~11 ft

None Nicomatic or similar micro-D

-

Inner HV cable

On hand?? coax Various ~11 ft

None SHV -

Patch panel On TPC or on WSC ring??

- Nicomatic or similar micro-D

3M MDR 36 pin

Nicomatic

Outer signal cable

Stock; have 500 ft

12pr (foam PE) 55 ft 3M MDR 36 pin

3M MDR 36 pin

-

Outer power cable

Custom, received

4c+4pr (PVC) 55 ft Nicomatic 3.5mm screw header

-

Outer HV cable

On hand?? coax (same) 55 ft SHV (F) SHV -

ARM BOC In design 3M MDR 36 pin

3.5mm screw header

DIN 41612 (VME)


30

Developed and extensively tested by CERN for ALICE

Commercially available (CERNTech)

Event-driven data transfers

Bidirectional control transfers

200 MB/s per SIU

Simple protocol, no CPU is necessary to run SIU at the detector

D-RORC is PCI-X master, writes data directly to a (pre-allocated locked down) main memory buffer

Now in use at STAR for• TPX (TPC upgrade for

DAQ1000)• BTOW †

• ETOW †

• TOF• ESMD †

• BSMD †

Standard data link hardware allows STAR operations to maintain a pool of spares

STAR-standard API exists and is being maintained by Tonko Ljubicic for integration of D-RORC into STAR DAQ

Works with commodity Intel-architecture PC’s running linux

SIU

D-RORC

ALICE Detector Data Link – SIU & D-RORC

( † By FGT personnel )


31

“Protection: Automatic current regulation protects the power supply against all overload conditions, including arcs and short circuits. Fuses, surge limiting resistors, and low energy components provide the ultimate protection.

Remote Controls: Common, +10 volt reference, interlock, current monitor, current program, voltage monitor, voltage program, HV enable/disable, and ground provided on a rear panel terminal block.

External Interlock: Open off, close on.”

Glassman EH10N10, -10 kV w/ current limit

WSC field-shaping bias system

Becomes a new permit to TPC cathode

Fixed operating point tbd, expect approximately -5.5 kV

Reynolds HV cable with series 531 connectors. Rated 10 kV operation, 100% tested to 15 kV, commercially assembled. Length approximately 50 feet, tbd.

Mating connector (Reynolds series 531) is an integral bonded part of WSC – TPC mechanical interface structure. Bias voltage runs in from there to shroud electrode on 20kV silicone high voltage wire.

Slide from safety review (BNL 9/2/2009)

STAR Forward GEM Tracker Readout/DAQ/Controls Subsystem

Documents

Transcript of STAR Forward GEM Tracker Readout/DAQ/Controls Subsystem