GLAST LAT Project Managers F2F 6/15/05 4.1.9 - Integration and Test 1 GLAST Large Area Telescope:...

GLAST LAT Project Managers F2F 6/15/05

4.1.9 - Integration and Test 1

GLAST Large Area Telescope: GLAST Large Area Telescope: Face to Face Managers MeetingFace to Face Managers Meeting

LAT Integration & Test LAT Integration & Test June 15, 2005 2:00 PMJune 15, 2005 2:00 PM

Integration and Test Integration and Test 6 Towers6 Towers

Elliott BloomElliott BloomI&T ManagerI&T Manager

Ken FoutsKen FoutsI&T Engineering ManagerI&T Engineering Manager

SLACSLAC

Eduardo do Couto e SilvaEduardo do Couto e SilvaSVAC ManagerSVAC Manager

Gamma-ray Large Gamma-ray Large Area Space Area Space TelescopeTelescope



Last Month's Accomplishments (1 of 2)Last Month's Accomplishments (1 of 2)

• Management– Tracking hardware shortages for LAT integration.– Procedure Status

– 68 of 70 submitted for sign-off or released

• IFCT – Completed 4 tower tests.– Completed I&T receiving tests on 3 TKR’s, 3 CAL’s and 3

TEM’s.– Completed assembly and installation of the grid cooler (aka

the Chilla from Manila).– Completed installation of 2 additional towers for a total of 6

towers installed in the grid.– Preparations complete for 6 tower tests.



Last Month's Accomplishments (2 of 2)Last Month's Accomplishments (2 of 2)

• Online– LATTE

• Supported VSC software Python interface implementation• Exploratory work in progress to demonstrate interaction with

the VSC and its software interface• 4.9.0 released

– Initiated by ACD-discovered bug – Bug fixes, Trigger GUI, GPIB support– Updated Hippodraw addressing some memory leaks– Updated trigger GUI to address concerns raised in trigger meeting– Bug fix for incorrectly initialized TEM BUSY_LRS_MASK register

(measures TEM deadtime)

• Housekeeping multi-pen strip-chart GUI V2.0 deployed to cleanroom for real-time monitoring.

• E-logbook Version 3.4.1 released (minor bug fixes with corresponding tutorial and documentation updates)



I&T ActivitiesI&T Activities

Grid Chiller Unit Installation Grid Cooling

Six CAL/TEM’s Installed Six Tower’s Integration



Procedure StatusProcedure Status• 70 documents total• 68 latest versions submitted for sign-off• 65 are released

19 19 19 19 18 18 18 18 18 18 18 18 18

15 15 15 15 15 15 15 15 15 15 15 15 15

7 7 7 7 7 6 6 6 4 4 4 4 4

4 4 4 4 4 4 4 43 3 3 3 3

12 12 12 12 12 12 12 1212 12 12 12 12

2 2 2 2 2 2 2 22 2 2 2 2

11 11 11 11 11 11 11 1111 11 11 11 11

0

10

20

30

40

50

60

70

80

Co

un

t o

f D

ocu

men

t #

Co

un

t o

f D

ocu

men

tN

um

ber

Ass

ign

ed

Co

un

t o

f D

raft

Sta

rted

Co

un

t o

f D

raft

Co

mp

lete

Co

un

t o

f In

tern

alR

evie

w C

om

ple

te

Co

un

t o

f su

bm

itte

dfo

r si

gn

off

Co

un

t o

f D

CN

wri

tten

Co

un

t o

f D

CN

an

dD

oc

sen

t fo

r si

gn

off

Co

un

t o

f S

ign

ed b

yA

uth

or

Co

un

t o

f S

ign

ed b

yE

llio

tt B

loo

m

Co

un

t o

f S

ign

ed b

yQ

A

Co

un

t o

f S

ign

ed b

yO

ther

s

Co

un

t o

f R

elea

sed

inL

AT

Do

cs

Do

cum

ents

Online

SVAC

Particle

MGSE

Manage.

IFCT-Mech

IFCT-Elec

Depart.



Procedure Status – ManagementProcedure Status – Management



Procedure Status – IFCT ElectricalProcedure Status – IFCT Electrical



I&T NCRs by Month Open/Closed StatusI&T NCRs by Month Open/Closed Status

0

2

4

6

8

10

12

14

16

18

Sep-04 Nov-04 Dec-04 Jan-05 Feb-05 Mar-05 Apr-05 May-05 Jun-05

Open

Closed

Count of Open/Closed Status

Open/Closed Status

Month NCR Opened

Current Status

As of June 14th

0

2

4

6

8

10

12

14

16

18

Sep-04 Nov-04 Dec-04 Jan-05 Feb-05 Mar-05 Apr-05 May-05


Date by Month

Month NCR Opened

Current Status

As of May 10th

0

2

4

6

8

10

12

14

16

18

Sep-04 Nov-04 Dec-04 Jan-05 Feb-05 Mar-05 Apr-05 May-05


Date by Month

Month NCR Opened

Current Status

As of May 10th

NCR Summary

24 Open

38 Closed



Power Outage Impact on I&TPower Outage Impact on I&T

• Emergency Generator allowed work to continue in B33.• Room 103 circuit breaker/panel fault was resolved by turning

off non-essential power.• Room 104 was unaffected by the panel fault. TKR 3 was

installed on Thursday, June 19.• Root cause of the circuit breaker fault was found to be a loose

connection on the main power coming into the panel.• The building was brought back to 100% with minimal impact to

building users.



Online ProgressOnline Progress

– LATTE 5• Meetings with SE, ISOC and FSW• Requirements doc in progress • Took delivery of a Dell blade server

– Used to develop LATTE 5 concepts – Test mobile rack concepts

– Progress on parallelizing the scripts:• TKR scripts nearly complete and tested• CAL CPT and LPT nearly complete and tested. • INT (e2e, SVAC) scripts complete. • CalibGen, CalibDac and MuTrg still to go

– ACD script preparation and test support ongoing.– Risks to schedule:

• Continued ACD problems requiring one or more of the Online team to make a site visit

• LATTE 5 requirements are converging slowly.



I&T Detail Schedule (1 of 6)I&T Detail Schedule (1 of 6)

ID Task Name Start % Complete Finish

835 Tower 4 - Bay 8 2/22/05 41% 6/27/05836 FM107 Post ship test and receiving test 2/22/05 100% 2/28/05843 Tracker 4 post ship test & receiving test 6/6/05 100% 6/8/05853 TEM/TPS 4 Flight Readiness 6/6/05 100% 6/6/05854 TEM/TPS EICIT & SVT 6/6/05 100% 6/6/05855 TEM/TPS-4 I&T 6/8/05 100% 6/9/05861 Tower-4 Grid Bay 8 I&T 6/9/05 18% 6/27/05862 TKR-5 installation into grid bay 8 6/9/05 100% 6/9/05866 FM107 installation into grid bay 8 6/10/05 100% 6/13/05869 6 Tower Test 6/9/05 5% 6/27/05870 Install Chiller bars 6/9/05 100% 6/9/05871 6 Tower Survey 6/13/05 100% 6/13/05872 Install & Inject Shear plates ( Bay 0,4,5,1) 6/13/05 0% 6/14/05873 Install Cable Trays 6/14/05 0% 6/14/05874 Thermistor bonding/cure on bays 0/4 (NCR351) 6/14/05 0% 6/14/05875 Bay 8 and 9 ECITs 6/15/05 0% 6/15/05876 Bay 8 Cable EICITs 6/15/05 0% 6/15/05877 Bay 9 Cable EICITs 6/15/05 0% 6/15/05

6/6

6/9

5/22 6/12 7/3 7/24 8/14May '05 Jun '05 Jul '05 Aug '05





878 Setup EM GASU EM PDU and connect cables 6/15/05 0% 6/16/05879 Install EM GASU 6/15/05 0% 6/15/05880 Install EM PDU 6/15/05 0% 6/15/05881 Configure EM PDU/ EM GASU Primary 6/15/05 0% 6/15/05882 EM PDU / EM GASU register Tests 6/15/05 0% 6/15/05883 Reconfigure EM PDU/ EM GASU Redundant 6/15/05 0% 6/15/05884 EM PDU / EM GASU register Tests 6/15/05 0% 6/15/05885 Connect GASU and PDU cables for Bays 0,1,4,5,8,9 6/15/05 0% 6/16/05886 TEM Register Tests 0,1,4,5,8,9 6/16/05 0% 6/16/05887 TEM Register Test Bay 0 Redundant 6/16/05 0% 6/16/05888 TEM Register Test Bay 1 Redundant 6/16/05 0% 6/16/05889 TEM Regisiter Test Bay 4 Redundant 6/16/05 0% 6/16/05890 TEM Register Test Bay 5 Redundant 6/16/05 0% 6/16/05891 TEM Register Test Bay 8 Redundant 6/16/05 0% 6/16/05892 TEM Register Test Bay 9 Redundant 6/16/05 0% 6/16/05893 Set up for TEM Register Test Primary Side 6/16/05 0% 6/16/05894 TEM Register Test Bay 0 Primary 6/16/05 0% 6/16/05895 TEM Register Test Bay 1 Primary 6/16/05 0% 6/16/05

5/22 6/12 7/3 7/24 8/14May '05 Jun '05 Jul '05 Aug '05




895 TEM Register Test Bay 1 Primary 6/16/05 0% 6/16/05896 TEM Regisiter Test Bay 4 Primary 6/16/05 0% 6/16/05897 TEM Register Test Bay 5 Primary 6/16/05 0% 6/16/05898 TEM Register Test Bay 8 Primary 6/16/05 0% 6/16/05899 TEM Register Test Bay 9 Primary 6/16/05 0% 6/16/05900 Bay 8 and 9 Post Integration Testing and part of

NCR 5096/16/05 0% 6/17/05

901 Tracker CPT Bay 8 6/16/05 0% 6/16/05902 Tracker CPT Bay 9 6/16/05 0% 6/16/05903 Calorimeter CPT Bay 8 6/16/05 0% 6/16/05904 Calorimeter CPT Bay 9 6/16/05 0% 6/17/05905 Calorimeter CPT Bay 0 (NCR 509) 6/17/05 0% 6/17/05906 Calorimeter CPT Bay 4 (NCR 509) 6/17/05 0% 6/17/05907 TE702 Bay 8 6/17/05 0% 6/17/05908 TE704 Bay 8 6/17/05 0% 6/17/05909 TE702 Bay 9 6/17/05 0% 6/17/05910 TE704 Bay 9 6/17/05 0% 6/17/05911 TE702 Bay 1 (NCR 509) 6/17/05 0% 6/17/05

5/22 6/12 7/3 7/24 8/14May '05 Jun '05 Jul '05 Aug '05





911 TE702 Bay 1 (NCR 509) 6/17/05 0% 6/17/05912 TE702 Bay 5 (NCR 509) 6/17/05 0% 6/17/05913 STR 13 6/17/05 0% 6/17/05914 Tracker LPTs Bays 0,1,4,5,8,9 6/17/05 0% 6/20/05915 Tracker LPT Bay 0 6/17/05 0% 6/17/05916 Tracker LPT Bay 1 6/17/05 0% 6/20/05917 Tracker LPT Bay 4 (also covers NCR 420) 6/20/05 0% 6/20/05918 Tracker LPT Bay 5 (also covers NCR 420) 6/20/05 0% 6/20/05919 Tracker LPT Bay 8 6/20/05 0% 6/20/05920 Tracker LPT Bay 9 6/20/05 0% 6/20/05921 Calorimeter LPTs 0,1,4,5,8,9 6/20/05 0% 6/20/05922 Calorimeter LPT Bay 0 6/20/05 0% 6/20/05923 Calorimeter LPT Bay 1 6/20/05 0% 6/20/05924 Calorimeter LPT Bay 4 6/20/05 0% 6/20/05925 Calorimeter LPT Bay 5 6/20/05 0% 6/20/05926 Calorimeter LPT Bay 8 6/20/05 0% 6/20/05927 Calorimeter LPT Bay 9 6/20/05 0% 6/20/05928 Trigger Tests 6/20/05 0% 6/22/05

5/22 6/12 7/3 7/24 8/14May '05 Jun '05 Jul '05 Aug '05





928 Trigger Tests 6/20/05 0% 6/22/05929 TREQ Scan Bay 8 6/20/05 0% 6/20/05930 TREQ Scan Bay 9 6/20/05 0% 6/20/05931 TACK Scan 6/20/05 0% 6/21/05932 Efficiency Bay 8 6/21/05 0% 6/21/05933 Efficiency w/ paddle offset Bay 8 6/21/05 0% 6/21/05934 Efficiency Bay 9 6/21/05 0% 6/21/05935 Efficiency w/ paddle offset Bay 9 6/21/05 0% 6/21/05936 Efficiency Bay 1 (NCR 509) 6/21/05 0% 6/21/05937 Efficiency w/ paddle offset Bay 1 (NCR 509) 6/21/05 0% 6/21/05938 Efficiency Bay 5 (NCR 509) 6/21/05 0% 6/21/05939 Efficiency w/ paddle offset Bay 5 (NCR 509) 6/22/05 0% 6/22/05940 Calibration 6/22/05 0% 6/23/05941 CalibDac Bay 8 (w/ file extract) 6/22/05 0% 6/22/05942 CalibGen Bay 8 6/22/05 0% 6/22/05943 MuTrig Bay 8 6/22/05 0% 6/22/05944 CalibDac Bay 9 (w/ file extract) 6/22/05 0% 6/22/05945 CalibGen Bay 9 6/22/05 0% 6/22/05

5/22 6/12 7/3 7/24 8/14May '05 Jun '05 Jul '05 Aug '05





945 CalibGen Bay 9 6/22/05 0% 6/22/05946 MuTrig Bay 9 6/22/05 0% 6/23/05947 TE601/TE602/TE604 Bay 8 (pending input from Hiro) 6/23/05 0% 6/23/05

948 TE601/TE602/TE604 Bay 9 (pending input from Hiro) 6/23/05 0% 6/23/05949 SVAC Data Collection 6/23/05 0% 6/27/05950 E2E B-13 (one minute - configuration verification) 6/23/05 0% 6/23/05951 E2E B-2 (one minute - configuration verification) 6/23/05 0% 6/23/05952 E2E B-10 (one minute - configuration verification) 6/23/05 0% 6/23/05953 Confirmation from SVAC on configuration verification 6/23/05 0% 6/23/05954 E2E B-13 (1 hour) 6/23/05 0% 6/23/05955 E2E B-2 (4 hours) 6/23/05 0% 6/24/05956 E2E B-10 (15 hrs) 6/24/05 0% 6/27/05957 Tower 5, Bay 13 6/7/05 19% 7/13/05993 Tower 6, Grid Bay 12 6/7/05 7% 7/26/051032 Tower 7 -Grid Bay 10 6/7/05 1% 8/11/051070 Tower 8, Grid Bay 11 6/13/05 0% 8/19/051108 Tower 9, Grid Bay 14 6/13/05 0% 9/7/05

5/22 6/12 7/3 7/24 8/14May '05 Jun '05 Jul '05 Aug '05




Issues & ConcernsIssues & Concerns

ISSUES• NCR 497 for connector clips and screws.• Availability of flight hardware.

– Shortages identified by I&T and being worked with Subsystems.

• TKR Deliveries• SIU, EPU Deliveries from ELX• Flight Cables

CONCERNS• Continued support from subsystems for NCR closure.• TPS rework and replacement, availability for future tower

installations.• Need firm delivery dates for third layer ELX boxes • LATTE 5.0 ready Oct. 1 for use in LAT Functional testing due to

start 10/14. E2E tests start 11/1.



NCR 497 Summary (1 of 2)NCR 497 Summary (1 of 2)

Background:

• An NCR was generated for the orientation of clips and fasteners on incoming cables from Cicon.

• SLAC specifies 2 Glenair connector types on cables.

• Thread damage was found on fasteners due to a mismatched clip and fastener configuration.

Findings:

• The Glenair connector drawings are incorrect and confusing.

• Clips will damage fastener threads if installed incorrectly.

• Some clips damage fastener threads when installed correctly.

• Connectors from Glenair delivered to Cicon in wrong configuration.

• Cicon put clips on the way they were received.

Corrective Action:

• Replace clip and fasteners at SLAC with new hardware and inspect fastener threads per NCR 497 disposition.



NCR 497 SummaryNCR 497 Summary

Recess in threads should line up with square feature in clip



SAS InterfacesSAS Interfaces

• SVAC is happy with CCB process– Thanks Richard

• Code release is getting more stable and reliable– Working with SAS to sharpen validation procedures

– specially calibrations

• Pipeline issues are more infrequent– We expect to transition into a more robust architecture in a

not so distant future– Ought to be tailored to the ISOC needs



Data Analysis Issues (up to 4 tower data)Data Analysis Issues (up to 4 tower data)

• Hardware failures or problems– None so far !!! Kudos to the “royal we”

• Anomalies in offline distributions– Becoming less frequent and mostly due to

– Incorrect configurations (mostly operational procedures)

– Mistakes in the offline data analysis codes

• Understanding how the instrument works– Time sink because it requires a much broader knowledge of

– Integrated detector system functions and performance

– Details of complex offline software

– How the analysis parameter space changes as we add more towers



Feedback to Integration and TestFeedback to Integration and Test

• The Instrument Analysis effort has already provided valuable feedback to Integration in several ways – Examples will be provided in the next slides

• Improving Operations– We provided a cross check to existing procedures

– and suggested improvements

• Understanding Instrument performance– We generated discussions on a particular topic

– That led to different operational settings

– We generated new tests– To verify functionality at the integrated level



Feedback to Operations: ProceduresFeedback to Operations: Procedures

• SVAC automated reports identified that GEM was not reporting trigger information properly for some of the 4 tower tests

• Action– stopped data taking to

investigate root cause (Eduardo)

• Root cause– incorrect power up of system

– found by Horwitz and confirmed by Huffer

• Recommendation to I&T– Improve procedures– Introduce online control plots

during data taking



Understanding Performance: TEM diagnosticsUnderstanding Performance: TEM diagnostics

• We are trying to investigate 1% effects when TEM diagnostics were enabled (Luis)

– Mike pointed out that we should be careful

– TEM diag disabled in the default for on-orbit operations

– System is too complex and more measurements are needed to understand the timing behavior

• Used trigger group discussions to – Understand timing diagram with

TEM diagnostics on– How to quantify effects

• Thanks for the help – Su Dong and Martin– Mike



Understanding Performance: TKR SettingsUnderstanding Performance: TKR Settings

• April 15: Anders looked at the variable GemCondArrivalTimeTkr in the SVAC ntuple and …

– he did not understand the spike at 31

• Bill asked Anders whether one-shots were used or not

• Anders checked with Online (Lester) and the answer was no

• The trigger/ELX group was asked to chime in and investigate with all relevant parties whether we turn the one-shot on or not during data taking

– Discussion involved on-orbit operations in the presence of CNO events

• May 25: Steve chaired a discussion in which we decided to change the way we will configure the TKR for the 4 tower tests

– One shots were turned on

• June: Anders is comparing 2 tower data without one-shots with 4 tower data with one shots

– stay tuned!



Understanding Performance: CAL RetriggeringUnderstanding Performance: CAL Retriggering

• CAL FLE Retriggering identified at NRL tests (Sasha)

– Need to study what happens to FHE

• Discussion will be brought up today in the trigger meeting

– Benefits from Eric’s presence at SLAC



Understanding the Instrument: HousekeepingUnderstanding the Instrument: Housekeeping

• Checked temperature differences from CAL to TKR

– CAL is 1.5 degrees cooler than top of TKR

– Differences can be understood because of convection

– Italy checked that thermistors work fine



Understanding Performance: DeadtimeUnderstanding Performance: Deadtime

• Warren Implemented an offline variable to measure time without being affected by roll over of counter

• Measured deadtime in several configurations

– Trying to understand numbers right now

• Proposed a new test to prescale hits at the GEM level

– Similar to on-orbit operations– enable more than one

trigger engine

– However, it stresses less the TD&F

– Therefore keep current prescaling tests



Do we have enough data yet?Do we have enough data yet?

• One tower

• Two towers

– End-to-end cosmic ray data– To exercise trigger and data flow capabilities

– SVAC cosmic ray data– To perform calibrations – To do performance studies that require large statistical sample

• Four towers to Sixteen towers (prior to ACD integration)

– Only planned to collect SVAC data

– May request some tests (most likely between 6 and 8 towers) to– Configure the instrument to take data with 2 Trigger Engines

(simulate on-orbit operation) – Study nominal/high rates with TEM diagnostics OFF– Understand TKR one-shot behavior– Study TKR cable FIFO errors (caused by induced showers)



Number of Talks in the IA meetingsNumber of Talks in the IA meetings

0

5

10

15

20

25

30

Jun-7-2004

Jul-30-2004

Aug-13-2004

Sep-3-2004

Nov-5-2004

Nov-19-2004

Dec-10-2004

Jan-21-2005

Feb-18-2005

Mar-4-2005

Mar-18-2005

Apr-1-2005

Apr-15-2005

Apr-29-2005

May-13-2005

Jun-10-2005

SVAC LAT Collaboration

Workshop 1Workshop 2

Workshop 3

Workshop 4Scheduled for

July14-15, 2005

After a year of effort, here we are….



Contributions to Instrument Analysis EffortContributions to Instrument Analysis Effort



Data Analysis EffortsData Analysis Efforts

• We generated a list of 15 potential topics for– Immediate analysis with data from 1 to 6 towers – Presentations during IA workshop July 14-15

• Working with subsystems (Neil,Dave,Ronaldo,Francesco) to – Sharpen topics and ideas– Identify names for presentations

• Initiated contacts ACD– Would like to have presentation during IA workshop

– If Dave can support that given the ACD schedule

– Need to discuss details for ACD tests at SLAC– Test plans– SAS Offline software– Pipeline scripts and related infrastructue

GLAST LAT Project Managers F2F 6/15/05 4.1.9 - Integration and Test 1 GLAST Large Area Telescope:...

Documents

Transcript of GLAST LAT Project Managers F2F 6/15/05 4.1.9 - Integration and Test 1 GLAST Large Area Telescope:...