Rods Schedule and Issues

J. Incandela

University of California Santa Barbara

for the TOB Rod Group

Tracker Integration Meeting October 27, 2005 Slides courtesy of D. Abbaneo, C. Campagnari, A. Dierlamm, Y. Gotra,

J. Lamb, others…

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

2

Rods Overview

• Basic components procured

• Assembly of frames is DONE (modulo ICC)

• Integration of modules on rods is understood• Rate of ~ 8/d in the US is not a problem …

• Testing was problematic and rod production was halted• I2C communication errors seen on 30-50% of rods

• Very substantial effort to understand and remedy

• Solution required making new Interconnect cards (ICC).

• Now in production.

• First 25 sets arrive in US today (?)

• Larger batches arrive at CERN starting early November (?)

• Rod production scheduled to restart this week.

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

3

Recent rod production

• Meanwhile we implemented a simple fix and built ~75 rods• Simple fix allows long term testing in Multi-Rod Test (MRT) stands

• It is not robust enough to be a final solution

• We shipped rods to CERN for integration studies• For integration studies

• Develop/document procedures for installation/removal of rods

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

4

Mechanical

• Assembly procedures well developed

• Trained personnel (including backups) at both sites

• Takes ~ 1-1.5 hours to assemble a rod• Add 0.5 hour for ICC swap

• Not a bottleneck• Long term testing ~6 rods/2 days or 4/day if necessary

No issues – everything is in place

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

5

Electrical testing

• General idea• Assemble rod

• Quick test of single rod

• Single Rod Test stand (SRT)

• Long term test

• long is really not that long (~ 1-2 days)

• Multi Rod Test stand (MRT)

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

6

Single Rod Test (SRT)

•Tests of

• Hardwired thermistors and humidity sensors

• CCU ring redundancy

• DCU readouts

• Leakage current

• Pedestals & noise

• Just looking for gross problems

• No channel level characterization

• At room temperature

FNAL

UCSB

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

7

SRT (continued)

• Most tests done using LtStruct • currently V30

• Hardwired sensors tested with simple custom box

• Root macro similar to module test macro generates plots

• Plots etc. uploaded to e-log

• Output root and text files saved for ALL tests• including version number, if tested more than once

• Only piece missing: protocol for testing digital functionality, experts to provide (?)

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

8

Sample Rod output

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

9

Multi Rod Test-stand (MRT)

(Semi) long term test

~ 1-2 days

6-8 SS rods/load

Thermal cycle

Full channel level characterization

FNAL

UCSB

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

10

Original MRT test cycle -have since developed shorter one

Warm

Test

Cold

Warm Cold Test Warm

Test

Cold

TestWarm Test

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

11

MRT status

• Hardware: all OK, robust• Several different coolant leaks at UCSB in last few

weeks – hopefully now solved

• DAQ software (LtStruct)• Much progress in the last months

• Thanks to great cooperation from Wim et al.

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

12

MRT status (cont.)

• Analysis SW• DefectAnalyzer

• Makes plots and XML file

• First pass cuts defined

• Will iterate using data from early production

• Custom perl script

• Compares bad channels on rod with bad channel lists from module tests

• Also flags DCU problems, etc

• Custom root macro

• To make plots similar to defect analyzer

• OK, but will be retired

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

13

Workshop in August

• We had an internal workshop in August• Try to converge on remaining loose ends

• Minutes http://hep.ucsb.edu/cms/rod/workshopAug2005.html

• Some of the issues discussed

• Original default long term standard scenario

• 2 day, go cold 4 times

• "Finalize" cuts to flag bad channels

• Module grading - same as ARCS

A: < 1% bad channels

B: < 2% bad channels

C: > 2% bad channels

F: bad HV

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

14

Workshop (continued)

• Rod grading:

• Pass: only A or B modules

• ≥ 1 out of 3 non-DCU thermistors OK

• I(leak) for all groups < 20 A

• Fail: otherwise

• XML file for DB

• Went over it line-by-line

• Decided on several changes additions

• e.g., added thermistor information

• For details see minutes of workshop

• Now implemented

• Still a few problems uploading

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

15

Towards production: what was missing

• Demonstration that we can sustain production rates on a daily basis for a long time

• To this end, we were thinking of going through a production exercise, starting asap

• ~ 40 "type 6" rods/site

• ICC Swap – where will this be done?

• Definition of digital tests• May have already been completed ?

• Some DB details• Should be OK

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

16

Rod Status Now - UCSB and FNAL

• While waiting for ICC’s to start production, the two sites have focused on slightly different issues

• FNAL: testing of new SW (FEC V3, Scientific Linux, Lt V30)

• UCSB: production exercise to shake down remaining operational issues

• Fine-tuned scenario to match 4 rods/day rate• Freezer takes max 8 SS4, 6 SS6, 4 DS need one day

scenario (so much for long term )

• Only two thermal cycles WCWCW

• CalProfRuns only with inverter-on to save time

• Added HV cycling at each temperature

• This is not working reliably, may have to ditch

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

17

General

• 4 sets (2/site) of OFECs in transit from CERN• FNAL will "commission" this system first; UCSB will

follow when it works

• First batch of ICC cards for 25 rods also in transit• Held up at Memphis, expected delivery - today

• ½ FNAL, ½ UCSB

• Will mount on existing rods, then put modules on rods, then test, then ship. ASAP

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

18

FNAL Status (1)

• Over 30 production scenarios run on MRT since July 2005

• 50 SS6 rods and four DS4 rods were assembled • (using jumpers on pre-production ICC cards), tested and burned-in

• Migration of rod test software running to Scientific Linux 3.0.5 with FEC software V3 was successful

• Long term testing with latest release of the FEC Software showed good stability of software and the control ring

• 1 upload/download error in 24 hours with 4 DS4 rods

• Latest release of the LtStruct (v30) was extensively tested

• Number of production scenarios with Single and Double Sided rods

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

19

UCSB Production Exercise: Mechanical

• Built 24 rods last week (22 SS6, 2 DS)

• Warmup 1st two days, then built 4 rods/day

• Bottom line: OK

• Some Issues:• Dirty rod boxes, Al dust

• Tail bending a la FNAL: OK.

• ICC cards swap: adds 20-30 minutes/rod

• DS rods: worry about exposed non-encapsulated bonds

• Designing cover with cutouts for ICC cards to enable handling of connectors and cables with minimal risk

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

20

UCSB Production Exercise: Testing

• 20/24 rods through SRT

• Mounted some modules with known pinholes• Flagged them without a problem

• 1st MRT scenario OK, 2nd failed• Failure in newly introduced HV cycle.

• CAEN got stuck. Not a new problem.

• A couple of laser failures (?) – Worrisome…

• Currently under investigation – expect info later this week.

• Some noise oddities to be investigated

• Some ICC/header errors.

• Hopefully will be better with new SW, new OFEC

• (Almost) successful 1st upload of rod XML file to DB

• Need some small name changes

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

21

UCSB Production Exercise: Short Term Plans

• Stop making rods• We learned what we needed to learn

• Continue exercising testing cycle• HV cycle

• XML uploads

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

22

Rod Reception Status

• Preparing rod reception testing for high rate• Automating flagging of bad channels and validation

comparison to US

• Installed/running DefectAnalyzer with the same cuts as US

• Bad channels agree so far

• Mostly concerned with making sure there is no shipping damage or other exceptional problems

• All but two of the forty “production” rods from the US have been tested

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

23

Rod Reception Status

• High current encountered on some modules• Annoying but not fatal

• Few modules on rods from US exhibit high current - humidity effect with a longer “memory” than usual

• A module that has been exposed to room air for several hours will show a current ramp-up to up to 1mA, a few minutes after bias turned on

• Then the current falls back to normal levels over a time of about an hour

• If the module is kept in dry air the high current doesn't reappear if the bias is turned on and off

• Seems to be correlated with a problem in mixing the Sylgard 186 encapsulant, which left a “stain” on the sensors.

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

24

Rod Reception Status

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

25

Rod Reception CERN Typical Noise

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

26

“Straw” Rod Production Schedule (Sep. 05)

Month TOB+ TOB- (3/d) (4/d) Integral (4/d)

Oct 0 20     0

Nov 0 60 40 70 70

Dec 80 0 20 50 120

Jan 70 0 30 60 180

Feb 120 70 -90 -60 120

Mar 100 120 -120 -90 30

Apr 0 100 0 30 60

Totals 370 370 -120 60 60

TO

B R

od

s- O

cto

ber

27,

200

5 –

Tra

cker

Inte

gra

tio

n R

evie

w s

lide

27

Summary

• Lots of progress • Can build and test rods in the US

• Some problems• Production startup will be a little rocky

• Hope to converge to stability by end of year, prior to peak production rates

• Robustness of stands has not yet been fully tested, and we will likely encounter some problems and short delays over time.

• Schedule• On paper, we can get all rods here in time

• In practice this looks extremely challenging.

• I would expect that a more realistic schedule would have 1-2 months contingency