Project CRAFT: A Test BedProject CRAFT: A Test Bedfor Demonstrating the Realfor Demonstrating the Real

Time Acquisition and AcrhivalTime Acquisition and Acrhivalof WSR-88D Base (Level II) of WSR-88D Base (Level II)

DataData

Presented to UCAR/Unidata ProgramPresented to UCAR/Unidata Program

12 October 2000 12 October 2000

Kelvin K. DroegemeierKelvin K. DroegemeierCenter for Analysis and Prediction of Storms Center for Analysis and Prediction of Storms

andandSchool of MeteorologySchool of MeteorologyUniversity of OklahomaUniversity of Oklahoma

s fn

NEXRAD Doppler Radar NetworkNEXRAD Doppler Radar Network

NEXRAD Facts and FiguresNEXRAD Facts and Figures

158 radars (141 in the Continental US)158 radars (141 in the Continental US)– 120 National Weather Service radars120 National Weather Service radars– 26 Department of Defense radars26 Department of Defense radars– 12 Federal Aviation Administration radars12 Federal Aviation Administration radars

s fn

WSR-88D ComponentsWSR-88D Components

Radar DataRadar DataAcquisitionAcquisition

s fn

WSR-88D ComponentsWSR-88D Components

Radar ProductRadar ProductGeneratorGenerator

s fn

WSR-88D ComponentsWSR-88D Components

Principal UserPrincipal UserProcessorProcessor

s fn

WSR-88D ComponentsWSR-88D Components

Principal UserPrincipal UserProcessorProcessor

NEXRAD Data TypesNEXRAD Data Types

Archive Level I (analog, time domain output)Archive Level I (analog, time domain output) Level II data (digital base data at full resolution)Level II data (digital base data at full resolution) Archive Level III (“NIDS” digital products from the Radar Product Archive Level III (“NIDS” digital products from the Radar Product

Generator)Generator) Archive Level IV (at the PUP)Archive Level IV (at the PUP)

NEXRAD Data TypesNEXRAD Data Types

Archive Level I (analog, time domain output)Archive Level I (analog, time domain output) Level II data (digital base data at full resolution)Level II data (digital base data at full resolution) Archive Level III (“NIDS” digital products from the Radar Product Archive Level III (“NIDS” digital products from the Radar Product

Generator)Generator) Archive Level IV (at the PUP)Archive Level IV (at the PUP)

s fn

NEXRAD Level III (NIDS) NEXRAD Level III (NIDS) DataData

24 products available from all CONUS 24 products available from all CONUS radars in real timeradars in real time

Provided by 3 commercial vendorsProvided by 3 commercial vendors Lowest 4 elevation angles of the radial Lowest 4 elevation angles of the radial

velocity and reflectivityvelocity and reflectivity– 1 degree x 1 km resolution1 degree x 1 km resolution– Values are quantized (e.g., 0-5, 5-10, 10-Values are quantized (e.g., 0-5, 5-10, 10-

15)15)

s fn

NEXRAD Data TypesNEXRAD Data Types

Archive Level I (analog, time domain output)Archive Level I (analog, time domain output) Level II data (digital base data at full resolution)Level II data (digital base data at full resolution) Archive Level III (“NIDS” digital products from the Radar Product Archive Level III (“NIDS” digital products from the Radar Product

Generator)Generator) Archive Level IV (at the PUP)Archive Level IV (at the PUP)

s fn

NEXRAD Base (Level II) NEXRAD Base (Level II) DataData

Full resolution digital dataFull resolution digital data– Full data precisionFull data precision– All tiltsAll tilts– Not quality controlledNot quality controlled

1 degree x 1 km resolution for 1 degree x 1 km resolution for reflectivityreflectivity

1 degree by 0.25 km resolution for radial 1 degree by 0.25 km resolution for radial windwind

Not available in real time except for Not available in real time except for selected sites (more on that later)selected sites (more on that later)

s fn

Base Data Usage: NSSL Warning Base Data Usage: NSSL Warning Decision Support System on 3 May Decision Support System on 3 May

19991999

Courtesy National Severe Storms Laboratory

Trimmed Detections and Ground Truth Damage Paths

Hits (142)

Misses (25)

FAs (21) Courtesy D. Zittel

5 April 1999 - Impact of Base Data5 April 1999 - Impact of Base Data

s fn

12 Z Reflectivity

Initial 700 mb VerticalVelocity Using NIDS

Initial 700 mb VerticalVelocity Using Level II

Data and SDVR

Courtesy S. Weygandt and J. Levit

5 April 1999 - Impact of Base Data5 April 1999 - Impact of Base Data

s fn

15 Z Reflectivity

3 hr ARPS CREF Forecast (9 km) Using Level II

Data and SDVRValid 15Z

3 hr ARPS CREF Forecast (9 km) Using

NIDS DataValid 15Z

Courtesy S. Weygandt and J. Levit

s fn

Recording Level III WSR-Recording Level III WSR-88D Data88D Data

Limited set of products recorded on Limited set of products recorded on CD-ROM at the radar siteCD-ROM at the radar site

s fn

WSR-88D Level III Data WSR-88D Level III Data Archival at NCDCArchival at NCDC

StationsStations 120120MediaMedia CDCDBytes/mediaBytes/media 600 MB600 MBDaily ReceiptsDaily Receipts 4 GB4 GBAnnual ReceiptsAnnual Receipts 60 TB60 TB

Courtesy Steve DelGreco (NCDC)

NWS

RecordDisks

Disks

To NCDC

PUP

Archive

Disk Storage

PC

Disk DriveCopy Files to HDSS

HDSS

WSR-88D Level III Data WSR-88D Level III Data ProcessingProcessing

s fn

Level III Requests at NCDCLevel III Requests at NCDC

s fn

Recording WSR-88D Level Recording WSR-88D Level II DataII Data

Not in original plansNot in original plans Exabyte 8 mm tape recorders added Exabyte 8 mm tape recorders added

to all radars (1994) as an interim to all radars (1994) as an interim solutionsolution

s fn

WSR-88D Level II Data WSR-88D Level II Data Archival at NCDCArchival at NCDC

StationsStations 158158MediaMedia 8 mm tape*8 mm tape*Bytes/mediaBytes/media 4.8 GB4.8 GBDaily ReceiptsDaily Receipts 165 GB165 GBAnnual ReceiptsAnnual Receipts 60 TB60 TB

*6-step process with NCDC operating 92 Exabyte *6-step process with NCDC operating 92 Exabyte tape drivestape drives

Courtesy Steve DelGreco (NCDC)

WSR-88D Level II Data WSR-88D Level II Data ProcessingProcessing

s fn

Level II Requests at NCDCLevel II Requests at NCDC

s fn

WSR-88D Data Archival at WSR-88D Data Archival at NCDCNCDC

% Rcpts % Rcpts #Sites#Sites

Level II NWSLevel II NWS 64.964.9 120120Level II DoDLevel II DoD 35.735.7 2626Level II FAALevel II FAA 9.59.5 1212

Total 88D NetworkTotal 88D Network 55.755.7 158158

Level III NWSLevel III NWS 89.589.5 120120

s fn

Currently 42,000 8-mm tapes in archiveCurrently 42,000 8-mm tapes in archive 9,500+ copies distributed to date9,500+ copies distributed to date Cost per tape is nominal ($100 for a radar day) Cost per tape is nominal ($100 for a radar day)

but high for large (multi-year) data setsbut high for large (multi-year) data sets Major time delays (2-4 months; human Major time delays (2-4 months; human

resources)resources) Archive incompleteArchive incomplete No mechanism for perusing the data baseNo mechanism for perusing the data base NOAA spent $768,000 on Level 2 recording NOAA spent $768,000 on Level 2 recording

system repairs from 6/1/98-6/14/99system repairs from 6/1/98-6/14/99 NCDC maintains 92 Exabyte tape systemsNCDC maintains 92 Exabyte tape systems

WSR-88D Base Data Archival WSR-88D Base Data Archival at NCDCat NCDC

s fn

Summary: WSR-88D Radar DataSummary: WSR-88D Radar Data The scientific and operational communities need The scientific and operational communities need

base data (real time and archived)base data (real time and archived) Although NIDS data are available in real time from Although NIDS data are available in real time from

allall WSR-88D radars, they are insufficient for many WSR-88D radars, they are insufficient for many applications (NWP, hydrology)applications (NWP, hydrology)– Degradation of precisionDegradation of precision– Only the lowest 4 tilts are transmittedOnly the lowest 4 tilts are transmitted

Base data currently are Base data currently are notnot available in real time available in real time– Originally would have been expensive Originally would have been expensive – Presumed large volume of data (10 mbytes/5 min/radar)Presumed large volume of data (10 mbytes/5 min/radar)– Need wasn’t there 10 years agoNeed wasn’t there 10 years ago

The technology and need now exist to prototype the The technology and need now exist to prototype the direct acquisition, use, and archival of base data in direct acquisition, use, and archival of base data in real timereal time

s fn

The Collaborative Radar The Collaborative Radar Acquisition Field Test Acquisition Field Test

(CRAFT)(CRAFT) Establish a prototype real time WSR-88D base Establish a prototype real time WSR-88D base

data acquisition test bed todata acquisition test bed to– Evaluate strategies for Evaluate strategies for compressing and transmittingcompressing and transmitting

base data in real timebase data in real time– Develop efficient and cost-effective strategies for Develop efficient and cost-effective strategies for

direct digital ingest, archive, and retrievaldirect digital ingest, archive, and retrieval at NCDC at NCDC– Assess the value of base data in Assess the value of base data in numerical weather numerical weather

predictionprediction– Test web-based Test web-based data miningdata mining techniques for rapid techniques for rapid

perusal/access of base data by the scientific perusal/access of base data by the scientific communitycommunity

CRAFT was established by the OU Center for CRAFT was established by the OU Center for Analysis and Prediction of Storms (CAPS) in Analysis and Prediction of Storms (CAPS) in collaboration with the collaboration with the – UCAR Unidata Program OfficeUCAR Unidata Program Office– NOAA WSR-88D Operational Support FacilityNOAA WSR-88D Operational Support Facility– University of WashingtonUniversity of Washington

CAPS and the NOAA National Severe Storms CAPS and the NOAA National Severe Storms Laboratory are now jointly coordinating the Laboratory are now jointly coordinating the effort in conjunction with the NEXRAD OSF effort in conjunction with the NEXRAD OSF and UCARand UCAR

s fn

ParticipantsParticipants

RIDDS: The Foundation of RIDDS: The Foundation of CRAFTCRAFT

s fn

s fn

Technical StrategyTechnical Strategy

Repeater Hub

RIDDS Linux PC

Unidata LDM ($1500)

Server

Dedicated 56K line

($2000 - $6000/year)Cisco 1600

Series Router($2000)

WSR-88D

At the radar site

Users

Internet

s fn

Original CRAFT NetworkOriginal CRAFT Network

Sharing Data from Multiple Sharing Data from Multiple Sources via Cooperating Sources via Cooperating

LDMsLDMs

Source

LDM

Source

Source

LDM LDM

LDMLDM

LDM LDM

LDM

LDM

Internet

Courtesy D. Fulker, Unidatas fn

– We added to LDM in-line We added to LDM in-line compression/decompression (BZIP2 off-the- compression/decompression (BZIP2 off-the- shelf algorithm)shelf algorithm)

maximum 16:1 in clear airmaximum 16:1 in clear air about 8:1 in the worst caseabout 8:1 in the worst case average of about 12:1average of about 12:1

– Maximum aggregate data rate is 15 Maximum aggregate data rate is 15 Mbytes/radar/5 min uncompressed = 48 Mbytes/radar/5 min uncompressed = 48 Mbits/sec compressed for Mbits/sec compressed for entire national entire national networknetwork

bandwidth isn’t the issue, but rather topology, bandwidth isn’t the issue, but rather topology, quality of service, and data storagequality of service, and data storage

one year’s worth of compressed data from NWS one year’s worth of compressed data from NWS radars is about 10 petabytesradars is about 10 petabytes

s fn

Technical HighlightsTechnical Highlights

– Compression now being improved (hope Compression now being improved (hope for > 20:1)for > 20:1)

– LDM relays data to other nodesLDM relays data to other nodes unlimited system growthunlimited system growth redundancyredundancy

– Simultaneous data transmission to multiple Simultaneous data transmission to multiple usersusers

– Single point of failure (modem is backup)Single point of failure (modem is backup)

s fn

Technical HighlightsTechnical Highlights

s fn

Real Time Management and Reliability: Real Time Management and Reliability: >95%>95%

http://traffic.onenet.net/

Real Time Management and Reliability: Real Time Management and Reliability: >98%>98%

Steve Goodman, NASA/MSFC

s fn

CRAFT Phase I: Proof-of-CRAFT Phase I: Proof-of-ConceptConcept

Internet2 is ...Internet2 is ...– a project, led by universities, to develop a project, led by universities, to develop

advanced internet technology and advanced internet technology and applications vital to the research and applications vital to the research and education missions of higher educationeducation missions of higher education

– big commitmentbig commitment member universities have committed over $70 member universities have committed over $70

million per year in new investment on their million per year in new investment on their own campuses own campuses

corporate members have committed more corporate members have committed more than $30 million over the life of the projectthan $30 million over the life of the project

Internet2 and AbileneInternet2 and Abilene

s fn

Abilene is ...Abilene is ...– an advanced an advanced backbone network testbedbackbone network testbed

that connects regional gigaPoPsthat connects regional gigaPoPs supports Internet2 applications developmentsupports Internet2 applications development that is focused exclusively on university research and that is focused exclusively on university research and

education education that is developing next generation operational and that is developing next generation operational and

quality of service capabilitiesquality of service capabilities

Internet2 and AbileneInternet2 and Abilene

s fn

Abilene NetworkAbilene NetworkJanuary 1999 January 1999

Cleve

land

New York

Atlanta

Indianapolis

Kansas City

Houston

Denver

Los Angeles

Sacramento

Seattle

Abilene Router Node

Abilene Access Node

Operational January 1999

Planned 1999

s fn

New Concept: Abilene/Internet2 + New Concept: Abilene/Internet2 + NEXRADNEXRAD

s fn

New Concept: Abilene/Internet2 + New Concept: Abilene/Internet2 + NEXRADNEXRAD

s fn

New Concept: Abilene/Internet2 + New Concept: Abilene/Internet2 + NEXRADNEXRAD

s fn

NCEP

NCDC

OU

NCAR/FSL

U-WA

New Concept: Abilene/Internet2 + New Concept: Abilene/Internet2 + NEXRADNEXRAD

AWC

TPC

Abilene Backbone

Other LDMs(university, private,

government)

NCARLDM

NCEPLDM

FSLLDM

NCDCLDM

OULDM

U-WALDM

Via Abilene orCommodity Internet

56K PhoneLines to Abilene

Universities

Logical Network TopologyLogical Network Topology

s fn

Adapted from D. Fulker, UCAR

s fn

Alternative StrategyAlternative Strategy

Repeater Hub

RIDDS Linux PC

Unidata LDM ($1500)

Server

Internet Direct

Cisco 1600 Series

Router($2000)

WSR-88D

At the radar site

Users

Internet

Abilene Backbone

Other LDMs(university, private,

government)

NCARLDM

NCEPLDM

FSLLDM

NCDCLDM

OULDM

U-WALDM

Via Abilene orCommodity Internet

Direct Connectionto the Internet inthe NWS Office

Alternative Network TopologyAlternative Network Topology

s fn

Adapted from D. Fulker, UCAR

– Configuration completely scalableConfiguration completely scalable– EveryoneEveryone has real time access to WSR-88D has real time access to WSR-88D

base data (all radars)base data (all radars) Special quality of service for key users (NCEP, SPC)Special quality of service for key users (NCEP, SPC) Leverages existing infrastructure and serves as a Leverages existing infrastructure and serves as a

means for developing advanced networking means for developing advanced networking capabilitiescapabilities

Easy linkage with other networks (e.g., AWIPS, Easy linkage with other networks (e.g., AWIPS, NOAAPort)NOAAPort)

Could easily add other radarsCould easily add other radars Significant capacity for future growth (dual-pol, Significant capacity for future growth (dual-pol,

phased array)phased array)

The BenefitsThe Benefits

s fn

– Rapid and less expensive base data archival and Rapid and less expensive base data archival and retrieval at NCDCretrieval at NCDC Number of steps reduced to 2Number of steps reduced to 2 Archive time reduced by factor of 4Archive time reduced by factor of 4 Will improve archival rate (now 65%)Will improve archival rate (now 65%) Reduces data request turn-around timeReduces data request turn-around time Reduces data volume via compressionReduces data volume via compression Eliminates tape drive costsEliminates tape drive costs

– Possible secondary back-up/real time archives Possible secondary back-up/real time archives where advanced data perusal and web-based data where advanced data perusal and web-based data mining tools will be availablemining tools will be available

The BenefitsThe Benefits

s fn

Operational reliability and redundancy in Operational reliability and redundancy in a research networka research network

Network securityNetwork security The future of the I2/Abilene network The future of the I2/Abilene network

infrastructureinfrastructure Coordination with regional/state Coordination with regional/state

networksnetworks Capacity for growthCapacity for growth Phasing with deployment of the Open Phasing with deployment of the Open

RPGRPG

The ChallengesThe Challenges

s fn

Use of LDM could expand significantlyUse of LDM could expand significantly How to provide LDM support to private How to provide LDM support to private

sector users?sector users? Integration of LDM into NEXRAD Open Integration of LDM into NEXRAD Open

Systems Architecture (more later)Systems Architecture (more later) End-user visualization and analysis End-user visualization and analysis

software (not just a UCAR problem)software (not just a UCAR problem)

Challenges for UnidataChallenges for Unidata

s fn

CRAFT-2CRAFT-2 One of 4 National Abilene InitiativesOne of 4 National Abilene Initiatives Expand CRAFT-1 to additional 30-40 radarsExpand CRAFT-1 to additional 30-40 radars Possibly include FAA radarsPossibly include FAA radars OU and NCAR have an opportunity to OU and NCAR have an opportunity to

become joint online radar archive sites for become joint online radar archive sites for data mining, visualization, etcdata mining, visualization, etc

s fn

s fn

CRAFT-2 Radars Now Delivering CRAFT-2 Radars Now Delivering DataData

s fn

May soon begin turning offsome of the 8 mm recorders!!

WSR-88DWSR-88D

Abilene (T1Abilene (T1into NCDC)into NCDC)

NEXRAD Base DataNEXRAD Base Datafrom the 12 CRAFT Radarsfrom the 12 CRAFT Radars

LDM HDSS

MassMass Storage Storage

At NCDC TodayAt NCDC Today

s fn

– Web-based data miningWeb-based data mining– Analysis and visualization packagesAnalysis and visualization packages– Networking quality of service researchNetworking quality of service research– Digital libraries and distributed storage Digital libraries and distributed storage

technology researchtechnology research– Storm- and meso-scale data assimilation and Storm- and meso-scale data assimilation and

predictionprediction– Mesoscale process studiesMesoscale process studies– Small-scale climatologiesSmall-scale climatologies

OpportunitiesOpportunities

s fn

http://geosciences.ou.edu/~jsnow/Research/Meso/http://geosciences.ou.edu/~jsnow/Research/Meso/

MS Thesis by Brad Pugh (1999)MS Thesis by Brad Pugh (1999)Oklahoma Climatological SurveyOklahoma Climatological Survey

– NOAA ESDIM Grant funded (CAPS+NSSL+OSF+NCDC)NOAA ESDIM Grant funded (CAPS+NSSL+OSF+NCDC) $540K/3 years$540K/3 years Research ThrustsResearch Thrusts

– Test of direct ingest/archival at NCDCTest of direct ingest/archival at NCDC– Improve compression algorithmsImprove compression algorithms– Initial work on web-based data miningInitial work on web-based data mining

– NOAA earmark funding to OUNOAA earmark funding to OU $474K for 1 year$474K for 1 year Expand CRAFT to 30 radars (CRAFT-2)Expand CRAFT to 30 radars (CRAFT-2) Develop data assimilation capabilities for the WRF modelDevelop data assimilation capabilities for the WRF model Kelvin doing a mini-sabbatical at the NSSL this fallKelvin doing a mini-sabbatical at the NSSL this fall

– HPCC Proposal ($150K for 1 year, about to be funded)HPCC Proposal ($150K for 1 year, about to be funded) Data miningData mining Network quality of service researchNetwork quality of service research Hardware for additional radarsHardware for additional radars

Funding StatusFunding Status

s fn

The Future: FAA RadarsThe Future: FAA Radars Initial efforts are now underway to Initial efforts are now underway to

explore extending the CRAFT explore extending the CRAFT infrastructure to FAA radarsinfrastructure to FAA radars– TDWR (terminal Doppler weather radar)TDWR (terminal Doppler weather radar)– ASR (airport surveillance radar)ASR (airport surveillance radar)– ARSR (air route surveillance radar)ARSR (air route surveillance radar)

s fn

TDWRTDWR

s fn

Weber (2000)

Airport Surveillance Radars Airport Surveillance Radars (ASR-9)(ASR-9)

s fn

Weber (2000)

Airport Surveillance Radars Airport Surveillance Radars (ASR-11)(ASR-11)

s fn

Weber (2000)

Air Route Surveillance Radars Air Route Surveillance Radars (ARSR-4)(ARSR-4)

s fn

Weber (2000)

– NEXRAD is implementing an open-systems NEXRAD is implementing an open-systems software standard based on UNIX and software standard based on UNIX and other widely-used technology (e.g., TCP/IP)other widely-used technology (e.g., TCP/IP)

– The temporary RIDDS system becomes the The temporary RIDDS system becomes the Base Data Distribution System (BDDS)Base Data Distribution System (BDDS)

– Will require moving LDM and compression Will require moving LDM and compression into BDDSinto BDDS

Open RPGOpen RPG

s fn

Open RPGOpen RPG

s fn

Phase I: Current StrategyPhase I: Current Strategy

Repeater Hub

RIDDS Linux PC

Unidata LDM ($1500)

Internet Direct or 56K

Phone Lines

WSR-88D

At the radar site

Users

Internet2/Abilene

s fn

Phase II: BDDS + LDM/PCPhase II: BDDS + LDM/PC

Repeater Hub

BDDS Linux PC

Unidata LDM ($1500)

Internet Direct or 56K

Phone Lines

WSR-88D

At the radar site

Users

Internet2/Abilene

s fn

Phase III: Full BDDSPhase III: Full BDDS

Repeater Hub

BDDS + LDM

Internet Direct or 56K

Phone Lines

WSR-88D

At the radar site

Users

Internet2/Abilene

Efforts are now underway to create a national Efforts are now underway to create a national WSR-88D Level II (base) data ingest systemWSR-88D Level II (base) data ingest system

Real-time access for all users by subscribing to Real-time access for all users by subscribing to desired radars via LDM – LDM is a KEY component!desired radars via LDM – LDM is a KEY component!

ESDIM project and CRAFT-2 are investigating key ESDIM project and CRAFT-2 are investigating key issues (reliability, scalability, security)issues (reliability, scalability, security)

Hope to have an online digital library in Norman Hope to have an online digital library in Norman for data mining and research with another archive for data mining and research with another archive at NCARat NCAR

Pacing item for implementation is Open RPG/BDDS Pacing item for implementation is Open RPG/BDDS deployment (scheduled June ’01 – Dec ‘02)deployment (scheduled June ’01 – Dec ‘02)

New build for BDDS will be phased in New build for BDDS will be phased in simultaneouslysimultaneously

Tremendous opportunities for researchTremendous opportunities for research

SummarySummary

s fn