2012 DiFX user’s meeting, Sydney, Australia

USNO Software Correlator: Status Report

Outline

• USNO/WACO Background• USNO Prototype Software Correlator (UPSC)• Use of DiFX• Recent Correlator Developments/Additions

Dave Boboltz, John Spitzak, Nicole Geiger, Dave Hall, Alan Fey, Chris Dieck

2012 DiFX user’s meeting, Sydney, Australia

USNO’s Role in the Geodetic Community

• USNO is the International Earth Rotation and Reference Systems Service (IERS) Rapid Service/Prediction Center for Earth Orientation.

• USNO has several functions within the International VLBI Service for Astrometry and Geodesy (IVS). – Correlator Operations Center.– VLBI Data Analysis Center.

• USNO provides several levels of data products. – Daily EOP (rapids).– Weekly EOP updates (Bulletin A).– Long-term (periodic EOP, TRF, CRF solutions).

2012 DiFX user’s meeting, Sydney, Australia

Geodetic VLBI and the IVS

• Geodetic VLBI is coordinated through the IVS. • http://ivscc.gsfc.nasa.gov/

2012 DiFX user’s meeting, Sydney, Australia

WAshington COrrelator (WACO)

• Current MarkIV “hardware” correlator was designed and constructed by MIT Haystack Observatory and is operated in cooperation with NASA.

• 1 of 4 IVS correlators

– WACO, Haystack, Bonn, Japan.

• Only 2 MarkIV correlators still in operation.

• Experiencing end-of-life issues.

• Move to software-based correlator.

2012 DiFX user’s meeting, Sydney, Australia

WACO Workload

• Daily “Intensive” experiments– Used to measure variable rotation rate of the earth (UT1-

UTC).– 7 days/week: single-baseline, 1-hour experiment.– Occasional 3-station intensives.

• VLBA (MK-PT) intensives.– 1-hr / day, 7 days / week

• Weekly R4 experiments– Used to measure EOP (polar motion, nutation, precession).– 1 per week: 5-8 stations, 24-hr experiment.

• Other experiments– 1 per month: 2-4 stations, 24-hr CRF experiments.– 4 to 6 per year: 24-hr “R1” experiments backing up Bonn.– Occasionally: other 24-hr experiments (APSG, CONT, etc).

2012 DiFX user’s meeting, Sydney, Australia

USNO Prototype Software Correlator

• Heterogeneous cluster.

– 5 nodes (workstations)

– 32 cores

• 2 Mark5 data playback units.

• 1 Gb/s ethernet switch.

• DiFX package installed.

• Proof of concept.

– Correlates geodetic VLBI data.

– Correlates MK-PT Intensives.

• Procurement of full software correlator cluster.

2012 DiFX user’s meeting, Sydney, Australia

DiFX Processing of Geodetic Intensives

• Single baseline KK-WZ• Currently correlating Friday

intensive in parallel with h/w correlator

• E-transferred data written to Mark5 modules

• Correlated on UPSC• difx2mark4• Fringe-fitting in

HOPS/Fourfit.• Geodetic databases

created.

2012 DiFX user’s meeting, Sydney, Australia

USNO-NSF-AUI/NRAO MoA

• USNO contracting with NRAO for:

– Access to MK - PT baseline.

• 1 hr intensive.

• 7 days/week.

– Continued VLBA participation in the RDV experiments.

• Six 24-hr experiments per year.

• Vital to the CRF and TRF.

• Requires high-speed connections.

2012 DiFX user’s meeting, Sydney, Australia

Status: MK-PT Fiber Links

• Pie Town link (1 Gbps).

– Available March 2011.– Numerous transfers of VLBI data. – 100 - 300 Mbps to USNO.– Contract with NRAO.

• FY13 Contract in process.

• Mauna Kea link (1 Gbps).

– Installed and available July 2011.– Numerous transfers of VLBI data.– 100 - 300 Mbps to USNO. – Contract with Univ. of Hawaii.

• FY13 contract signed.

2012 DiFX user’s meeting, Sydney, Australia

VLBA MK-PT Intensive Experiments

• IVS Intensive4 (INT4) series– IVS Nyyddd

• Dual S/X band – 4 S-band, 12 X-band– 32 MHz/channel

• ~40 min. / experiment, ~30 scans

• 16 second scan lengths– Limits data to be transferred– Source lists from USNO, ICRF

def.

• Data fringed and analyzed.– Oct. 2011 – May 2012

2012 DiFX user’s meeting, Sydney, Australia

Leap Seconds

UT1-UTC Results

2012 DiFX user’s meeting, Sydney, Australia

UT1–UTC Results: Comparison IERS C04

• Oct. 2011 – May 2012.

• IVS 24-hr sessions (red).

• IVS Intensives (green).

– wrms = 16.1 msec.

• MkPt Intensives (blue).

– wrms = 54.5 msec.

– Un-modeled motion likely in Pie Town antenna.

2012 DiFX user’s meeting, Sydney, Australia

New USNO Software Correlator

• USNO contracted with NRAO to build correlator.

– Phases I & II

– Leverage NRAO DiFX/Cluster expertise

• Phase I (Sept. 2012).

– Replacement for current 15 station hardware correlator.

– 30 compute nodes.

– 16 cores/node = 480 cores

– Delivered to USNO Friday!

• Phase II (2013 - 2014).

– Correlator expansion.

– 30 station VLBI2010 compatible

2012 DiFX user’s meeting, Sydney, Australia

Phase I Correlator

• 2 infrastructure nodes• 1 head node• 30 compute nodes

– Diskless – Mount operating system

and DiFX from infrastructure nodes

• 10 Gbps Ethernet network– Arista 10GbE switch– 64 SFP+ ports– 40 Gbps bonded

connection to SAN

2012 DiFX user’s meeting, Sydney, Australia

Phase I at NRAO, Socorro

7 node Test Cluster will remain at NRAO

33 node Production Cluster

2012 DiFX user’s meeting, Sydney, Australia

Phase I Delivered to USNO

Mark IV correlator

Mark III correlator

Seismic rated rackCluster hardware

2012 DiFX user’s meeting, Sydney, Australia

Future Plans

• Get the new correlator up and running – James Robnett (NRAO) visit to USNO 24-28 Sept.– Unpack, rack, power-up, network connections. – Connect to SAN.– Test correlations off SAN.

• Connect Mark5 units via 10GbE– Emulex OCe10102 10 Gbps iSCSI network adapters

installed.– 6 Mark5 units available now.

• Parallel operations.– Run both correlators in parallel for ~6 months.– Compare results. – Phase out Mark IV correlator

• Go operational with MK-PT Intensives