1

National Weather Service

Jason TuellJason Tuell

Office of Science and TechnologyOffice of Science and Technology

The Evolution of AWIPSThe Evolution of AWIPSNURONURO

5 September 20075 September 2007

2

OverviewOverview

• The Why and What of AWIPS Evolution

• Objectives

• Re-architecture Approach

• Roadmap

• What does AWIPS II mean to ensembles?

• Summary

3

Why Change?Why Change?

• Case for change briefed to NWS Corporate Board – Nov 2004Case for change briefed to NWS Corporate Board – Nov 2004• AWIPS Present State SummaryAWIPS Present State Summary

• HardwareHardware– AWIPS hardware was in good shapeAWIPS hardware was in good shape

• Communications InfrastructureCommunications Infrastructure– AWIPS communications infrastructure was in OK shapeAWIPS communications infrastructure was in OK shape

• DataData– AWIPS Data was in need of improvementsAWIPS Data was in need of improvements

• SoftwareSoftware– AWIPS software was in critical need of improvementsAWIPS software was in critical need of improvements– Costly software development, maintenance and inability to meet NWS and customer Costly software development, maintenance and inability to meet NWS and customer

needsneeds

• Corporate board direction to focus on addressing software shortcomingsCorporate board direction to focus on addressing software shortcomings• Plan and requirements developed Plan and requirements developed • Shaped portions of the AWIPS O&M re-compete activityShaped portions of the AWIPS O&M re-compete activity

4

What is AWIPS Evolution?What is AWIPS Evolution?

• AWIPS EvolutionAWIPS Evolution• A long-term project which delivers a modern, robust software infrastructure that

provides the foundation for future system level enhancements• AWIPS IIAWIPS II

• Implements a modern Services Oriented Architecture (SOA) infrastructure• First output of AWIPS Evolution and provides the foundation for all subsequent

improvements• AWIPS II Extended - Creation of a seamless weather enterprise spanning AWIPS II Extended - Creation of a seamless weather enterprise spanning

NWS operationsNWS operations• Migration of NAWIPS into the AWIPS II SOA• Delivery of thin client

– Support to the Weather Service Offices and Center Weather Support Units• Integration of “orphan” systems (e.g., Weather Even Simulator)

• AWIPS II Enterprise EnhancementsAWIPS II Enterprise Enhancements• Data Delivery Enhancements

– “Smart push-smart pull” data access– Katrina satellite WAN back up

• Integrated visual collaboration• Visualization Enhancements• Information Generation Enhancements

5

AWIPS EvolutionAWIPS EvolutionObjectivesObjectives

• Establish Service Oriented Architecture for AWIPS and NAWIPS• Create a seamless weather enterprise that supports all levels of NWS

operations from National Centers to WSOs• Build a common development environment that will be used by all

developers• Establish infrastructure for GIS integration

• Enable access to data independent of its location, i.e., provide access to data not resident locally at the WFO or RFC.

• Provide infrastructure for real time graphical collaboration between• WFOs, RFCs and centers for enhanced internal collaboration• Other NOAA entities and• Trusted partners, e.g., Emergency Managers

• Implement a Common AWIPS visualization environment (CAVE) used by all applications

• Standardize generation of NWS products and services

6

AWIPS IIAWIPS IIRe-Architecture ApproachRe-Architecture Approach

• Perform “black-box” conversion• Preserve existing functionality, look and feel on top of new

infrastructure

• Thorough field validation and acceptance before deployment

• No loss of functionality• Deployed system current with deployed AWIPS capability (i.e., OB9)

• Use open source projects - No proprietary code• JAVA and open source projects enable AWIPS II to be platform and

OS independent– No plans to move from Linux

• ADE enables collaborative development• OS, Platform independence allows non-Linux based research to be

easily integrated into AWIPS II

7

AWIPS II AWIPS II FeaturesFeatures

• AWIPS Development Environment (ADE)• Used by all AWIPS developers (National, Regional, & Local)

• Developers concentrate on new capabilities, not re-implementing existing ones (i.e. screen I/O, communications protocols, data access routines, logging routines, or other previously developed capabilities)

• Software can be developed on a variety of platforms • Robust infrastructure for improved software O&M

• Use of plug-ins: visualization extensions; new data types and transforms

• System level, remediation, core services reduce system complexity

• Improved support for local requirements (e.g., local apps, scripts, plug-ins)• Common AWIPS Visualization Environment (CAVE)

• Provides a common development and execution environment for AWIPS GUIs (e.g. D2D, NMAP, GFE, etc.)

• Ability to pan/zoom large data sets (Raster & Vector) with flexibility over data rendering

• GIS tools• Thin Client (Web Browser) enabled

• Dynamic Load balancing• Processing dynamically allocated among available CPUs

8

AWIPS IIAWIPS IIRoadmapRoadmap Migration

Strategy

2007 2008 201020092006

SW CTR (AWIPS II)

AE OSIP Gates 2 3 4a 4b

= Fiscal Year

= Calendar Year

MPLS Meshed Topology

9/05/07

OBx 7 8 9 10

PIP

Analysis

Migration Planning

RTS IRAD ADE Development

“User” Functional Tests

Baseline Application Migration

8 8.3

ADE Training

Field Ops Training-- ITO, ESA

DeploymentOTE

C & A

Deployment Planning

Local App Migration

ADE Local App Training

New Release Paradigm

O & M Transition

NWS New Capability Development in ADE

OB 9 Dev & Test Deployment

O & M Transition Prep & Coordination

ADE Training Development

OTE / Deployment Support

AWIPS II 1.0

Note: Task bar colors are For speaker reference only

9

2007 2008 2009 2010 2011 2012 2013 2014

AWIPS EvolutionAWIPS EvolutionRoadmapRoadmap

AWIPS IIExtended

= Fiscal Year

= Calendar Year

OTE / Deployment

AWIPS II

AWIPS II

Baseline Application Migration

Governance Model

NAWIPS Migration

Thin Client

WES Integration

AWIPS IIEnhancements

Data Delivery IOC FOC

Phase 1 Collaboration Phase 2 Phase 3

Information Generation IOC FOC

Visualization IOC

CHPS

10

AWIPS EvolutionAWIPS EvolutionGovernance ModelGovernance Model

• What is it?What is it?• Governance model controls the development, test, integration, Governance model controls the development, test, integration,

configuration management, deployment and support of the new configuration management, deployment and support of the new system -- both hardware and softwaresystem -- both hardware and software

• Why?Why?• AWIPS II offers new levels of flexibility and extensibilityAWIPS II offers new levels of flexibility and extensibility• New rules needed to take advantage of system capabilities and also New rules needed to take advantage of system capabilities and also

define limitsdefine limits• Tension between unlimited modifications and ability to support the Tension between unlimited modifications and ability to support the

systemsystem

• Sample issues for considerationSample issues for consideration• Monolithic configurations no longer required -- how do we manage site Monolithic configurations no longer required -- how do we manage site

specific configurations?specific configurations?• Plug ins can be down loaded and installed on demandPlug ins can be down loaded and installed on demand• Scripting that modifies AWIPS menus, functionsScripting that modifies AWIPS menus, functions

• Flexible Governance Model needed for limited open source Flexible Governance Model needed for limited open source implementationimplementation

11

AWIPS IIAWIPS IIWhat does it mean to ensembles?What does it mean to ensembles?

• NAWIPS migration• Same infrastructure supporting both NAWIPS and AWIPS

applications• Algorithms

• Common library for meteorological and hydrological calculations and tools

• Data delivery• Smart push-smart pull may provide more flexibility for access

to large data sets• Visualization

• Supports new tools and techniques for visualizing ensemble data

• Information Generation• New architecture for product generation and delivery will

allow tailored products capturing uncertainty

12

AWIPS IIAWIPS IITrainingTraining

• Strategic Training Plan developed• Training targeted for the following groups:

• ESAs• ITOs• AWIPS and application focal points• Developers (both baseline and local)• NCF• SST

• Training Organizations involved in planning, developing and implementing courses

13

AWIPS IIAWIPS IITestingTesting

• Layered testing strategy• Different testing phases overlap – functionality tested multiple times in different

settings• ADE usage

• Not formal testing, but expect to get feedback as developers use the ADE• Algorithm V&V –

• Assist algorithm choice• Verify port of algorithms

• V&V• Task order validation• Executed by dev orgs in controlled setting• Performance testing – Evaluate system against known and developing

benchmarks• Acceptance testing – part of the task order acceptance process

• Risk reduction• Side by side testing of new system by field personnel in a “lab” environment• Testing at limited number sites in manner that doesn’t impact operations

• Operational Test & Evaluation (OT&E)• Formal testing of the entire system• Tests system interfaces, operations, support, training, etc.

14

AWIPS IIAWIPS IIRisks and ChallengesRisks and Challenges

• Organizational challenges• Significant off contract workload on Government• Matrixed structure increases project complexity

• Performance• Supporting the short fuse warning mission• Handling large global data sets

• Schedule• Completing the migration and testing

• Migration of local applications• Local applications outside the baseline• Not a Raytheon responsibility

15

SummarySummary

• New architecture defined and infrastructure delivered• ADE 1.0 delivered June 14, 2007

• Application migration underway• Migration Plan delivered June 2007

• AWIPS baseline migration to be completed FY09

• Local applications migration planning in progress

• Incremental approach will extend AWIPS II architecture throughout NWS

• AWIPS II and subsequent enhancements will provide more flexible foundation for delivering products that capture uncertainty

16

National Weather Service

Back UpBack Up

17

AWIPS IIAWIPS IILocal ApplicationsLocal Applications

• Survey – completed August 31, 2007• 4191 local applications identified• 37% critical, 44 % important, 19% less important

• Migration plan to address approach based on survey results• Training requirements and approach to be refined based on

survey• Raytheon to provide sample migration and code samples for

approach• “Integrate with” vice “interface to”

• Level of effort required uncertain• Raytheon estimate that 80% of local apps will be able to be

rewritten in Javascript, without extensive programming in the ADE

18

AWIPS IIAWIPS IIWhat gets us excited so far…What gets us excited so far…

• Dynamic load balancingDynamic load balancing• Failover handled automaticallyFailover handled automatically• Enables consideration of tailored hardware configurationsEnables consideration of tailored hardware configurations

• Mathematically intensive calculations handed off to the graphics cardMathematically intensive calculations handed off to the graphics card• Significant performance improvementsSignificant performance improvements

• Progressive disclosure of all dataProgressive disclosure of all data• Imagery via quad tree tiling, grids and observationsImagery via quad tree tiling, grids and observations

• Integrated thin clientIntegrated thin client• Allows baseline solution to be extended to CWSUs, WSOs, and IMETsAllows baseline solution to be extended to CWSUs, WSOs, and IMETs

• Integrated drawing and graphical collaborationIntegrated drawing and graphical collaboration• Tools built into the infrastructure, implemented in 2011Tools built into the infrastructure, implemented in 2011

• Built in GIS via geotools libraryBuilt in GIS via geotools library• Scripting level access to practically all system level services and Scripting level access to practically all system level services and

functionsfunctions• LESS CODE LESS CODE

• Potential order of magnitude reduction in amount of software with increase in Potential order of magnitude reduction in amount of software with increase in functionalityfunctionality

19

AWIPS IIAWIPS IIGIS CapabilitiesGIS Capabilities

• GIS functionality integrated into AWIPS II architecture via GIS functionality integrated into AWIPS II architecture via GeoTools GeoTools

• GeoTools GeoTools • Open Source Java code libraryOpen Source Java code library• Provides OGC compliant methods for manipulation of geospatial dataProvides OGC compliant methods for manipulation of geospatial data• Equivalent to ESRI’s arc engineEquivalent to ESRI’s arc engine

• AWIPS II will support JavaScript access to GeoTools libraryAWIPS II will support JavaScript access to GeoTools library• ADE 1.0 supports limited accessADE 1.0 supports limited access

– Read, write shape filesRead, write shape files– Create GeoTiffsCreate GeoTiffs– Spatial queries of ingested dataSpatial queries of ingested data

• Task Order 10 will deliver full JavaScript accessTask Order 10 will deliver full JavaScript access• GeoTools does GeoTools does notnot make AWIPS a stand alone GIS system make AWIPS a stand alone GIS system• Reference:Reference:

• GeoTools home: GeoTools home: http://geotools.codehaus.orghttp://geotools.codehaus.org• GeoTools & Eclipse: GeoTools & Eclipse:

http://docs.codehaus.org/display/GEOT/Eclipse+Developers+Guidehttp://docs.codehaus.org/display/GEOT/Eclipse+Developers+Guide

20

AWIPS EvolutionAWIPS EvolutionOutcomesOutcomes

• Short-term (1-3 years)• Shorten transition of research to operations

• Improve software O&M and technology refresh– Fewer DRs and TTs

– Focus on hardening and productionizing for life cycle support

• Minimize adverse impacts on operations from software and hardware upgrades

• Long-term (3-10 years)Long-term (3-10 years)• Increase integration of AWIPS and National Center AWIPS

• Improve performance and functionality of AWIPS

• Improve collaboration at all levels of NWS operations

• Increase access to all environmental data for decision making

21

AWIPS EvolutionAWIPS EvolutionData DeliveryData Delivery

• OSIP Project 05-040OSIP Project 05-040• Enables “smart push - smart pull” data deliveryEnables “smart push - smart pull” data delivery

• Implements a discovery service within the SOAImplements a discovery service within the SOA• Access to data not available locallyAccess to data not available locally• Freedom from the tyranny of the SBNFreedom from the tyranny of the SBN• Enables consideration of new data delivery architectureEnables consideration of new data delivery architecture

– What data to you broadcast over SBN?What data to you broadcast over SBN?– What data do you make available on servers? What data do you make available on servers?

• ScheduleSchedule• IWT starting Q4 2007 to define concept of operations and IWT starting Q4 2007 to define concept of operations and

operational requirementsoperational requirements• IOC - 2011 - software implementation for remote data accessIOC - 2011 - software implementation for remote data access• FOC 2012 - enterprise configuration (servers, comms, etc.) that FOC 2012 - enterprise configuration (servers, comms, etc.) that

enables remote data accessenables remote data access

22

AWIPS EvolutionAWIPS EvolutionCollaborationCollaboration

• OSIP Project 05-041OSIP Project 05-041• ObjectiveObjective

• Integrated graphical collaboration throughout the NWS Integrated graphical collaboration throughout the NWS Weather Enterprise and beyondWeather Enterprise and beyond

• Phase 1 - Integrated collaboration between all levels of NWS Phase 1 - Integrated collaboration between all levels of NWS operationsoperations

• Phase 2 - Collaboration between NWS offices and other NOAA Phase 2 - Collaboration between NWS offices and other NOAA entitiesentities

• Phase 3 -Collaboration between NWS offices and trusted Phase 3 -Collaboration between NWS offices and trusted external partners, e.g., Emergency Managersexternal partners, e.g., Emergency Managers

• ScheduleSchedule• Phase 1 IOC - 2011Phase 1 IOC - 2011• Phase 2 IOC - 2012Phase 2 IOC - 2012• Phase 2 IOC - 2013Phase 2 IOC - 2013

23

AWIPS EvolutionAWIPS EvolutionInformation Generation & VisualizationInformation Generation & Visualization

• OSIP Projects 05-042 (IG) and 05-021 (Vis)OSIP Projects 05-042 (IG) and 05-021 (Vis)• Information Generation objectiveInformation Generation objective

• Re-architect generation of all NWS products and servicesRe-architect generation of all NWS products and services• Separation of content generation from formatting and Separation of content generation from formatting and

disseminationdissemination• Enable faster response to emerging customer demandsEnable faster response to emerging customer demands

• Visualization objectiveVisualization objective• Common user interface - standardize User Interfaces across Common user interface - standardize User Interfaces across

applicationsapplications• 3-D visualization3-D visualization• Improve user interfaces based on latest principles and Improve user interfaces based on latest principles and

researchresearch

24

What is AWIPS?What is AWIPS?

NEXRAD

AWIPS Workstations andServers

GOES/POES/NPOESS

ASOS

NCEPModels

Buoys,River Gauges

Forecasts

Warnings

Advisories

Watches

AWIPSCommunications

Service provided to3066 US Counties

24 hrs/day, 365 days/yr.

169 separate AWIPS systems at137 geographical locations

~900 Workstations (total)~1200 Servers (total)

25

Environment Environment

Environment

LightningDetectionCooperative

Observers

AWIPS

LOCAL EXTERNALUSERS;

COMPUTERSYSTEMS;

OBS PLATFORMSAND NETWORKS

NESDIS

NWSTG

WALLOPSCDA

GOES I-M, PolarOrbiter, and

Other Satellite Data

- Satellite Product Gen- Satellite Operation

GOESI-M

Spacecraft

Data CollectionPlatforms (DCP)

Official UserProducts

USERSUSERS

PROFILER

MICROART

NEXRAD

(May Interface Locallyat Site or Centrally

VIA NWSTG

Upper AirObservations

Weather RadarProducts

Class 4NIDS

USERS

DCPData GOES

Data

USERS

Forecaster

- Product Formulation- Data Integration & Analysis- Forecasting- Services Dissemination

GTSUpper AirObserving

(ServicesDissemination)

SurfaceObservations

World-Wide DataCollection &Distribution

- Numerical Predictions- Large-scale

Forecaster Guidance

Gatewayfor International & Non-

NOAA Domestic products

-Surface Observing

Remote SurfaceObserving

- Radiance Measurements- DCP Data Collection &

Distribution

SatelliteDistribution &

Processing

AtmosphericProfiles

Official UserProducts

- RADAR ReflectivityMeasurement

- RADAR Product Gen NWWSNWR

ASOS

POESSpacecraft

POESData

LightningData

OceanObs

NCDC

(NCF)

MDCRSCommercial Aircraft

Data

NCEPRetrospective

Users

Official UserProducts,

Observations, &Graphics

NCF

-Archiving

LDAD

USERS

FAA Data

Level 2 &Level 3 Data

Data- Calibration Data

Modeling- Interactive Data

Analysis

POESData

NCEPGuidance,

Observations,& Offical User

Products

Enterprise ArchitectureEnterprise Architecture

Dissemination

Modeling & Product Generation

Observing

Systems Legend

* Note some systems perform multiple functions, e.g. LDAD disseminates information and receives observations

“AWIPS is the Integrating Element of the

Modernized Weather Service”

Figure 6-8 of NOAA EA v 1.6

26

Architectural ComplianceArchitectural Compliance

• Collaboration on Achieving Enterprise SolutionsCollaboration on Achieving Enterprise Solutions

• AWIPS is the Integrating Element of the Modernized Weather ServiceAWIPS is the Integrating Element of the Modernized Weather Service

• AWIPS Uses Enterprise Red Hat Linux support contractAWIPS Uses Enterprise Red Hat Linux support contract

• AWIPS Contract supports Climate, Weather and Water, Transportation, and AWIPS Contract supports Climate, Weather and Water, Transportation, and extensible to Coastal and Ocean Resources extensible to Coastal and Ocean Resources

• Reuse of existing assetsReuse of existing assets

• HazCollect uses AWIPS IT infrastructure for message handling/creation and HazCollect uses AWIPS IT infrastructure for message handling/creation and routing to dissemination systemsrouting to dissemination systems

• Shared use of NOAA-Net for Wide Area Network in progressShared use of NOAA-Net for Wide Area Network in progress

• AWIPS provides NEXRAD visualization and data collection functionsAWIPS provides NEXRAD visualization and data collection functions

• AWIPS software used by other Government projects - RSA II, WES, FX-NetAWIPS software used by other Government projects - RSA II, WES, FX-Net

• Usage of StandardsUsage of Standards

• Complies with statutory requirementsComplies with statutory requirements

– Information Quality (Section 515) & Accessibility (Section 508)Information Quality (Section 515) & Accessibility (Section 508)

– Government Paperwork Elimination Act (GPEA)Government Paperwork Elimination Act (GPEA)

– FISMA/NIST Standards and GuidanceFISMA/NIST Standards and Guidance

27

AWIPS EvolutionAWIPS Evolution An EA PerspectiveAn EA Perspective

• EA perspective focuses on hydrological and meteorological mission operations• Administrative functions not part of AWIPS

• Supercomputing functions outside of AWIPS

• AWIPS II compliant with current DMIT data standards

• Some services, e.g., discovery service, not implemented as part of AWIPS II• Will be implemented in the Data Delivery system enhancement

• Exploring partnerships with other agencies within and outside of NOAA• NESDIS and NPOES NDE

• NASA SPORT and NASA Goddard

28

Open Source ProjectsOpen Source Projects Used in ADEUsed in ADE

OS ProjectOS Project FunctionalityFunctionality ADE RoleADE Role

ANTANT Build scriptingBuild scripting Build systemBuild system

Mule + SpringMule + Spring Enterprise Service Bus + ContainerEnterprise Service Bus + Container Runtime EnvironmentRuntime Environment

ActiveMQActiveMQ Java Messaging System BrokerJava Messaging System Broker JMS BrokerJMS Broker

JibxJibx XML to Object SerializationXML to Object Serialization Canonical XML MessageCanonical XML Message

GeoTools + JTSGeoTools + JTS GIS capabilitesGIS capabilites GIS primitivesGIS primitives

TomcatTomcat Web ServerWeb Server Test Client ServerTest Client Server

BaltikBaltik Scalable Vector Graphics ToolsScalable Vector Graphics Tools SVG toolsSVG tools

EhcacheEhcache Event Driven Clusterable CacheEvent Driven Clusterable Cache Cache FrameworkCache Framework

Log4jLog4j Java Logging APIJava Logging API Log managerLog manager

JoglJogl Java API to OpenGLJava API to OpenGL CAVE rendering interfaceCAVE rendering interface

Eclipse RCPEclipse RCP GUI plug-in based frameworkGUI plug-in based framework CAVE frameworkCAVE framework

Eclipse IDEEclipse IDE Java Integrated DevelopmentJava Integrated Development Development environmentDevelopment environment

MC4J ConsoleMC4J Console JMX Management ConsoleJMX Management Console Remote management consoleRemote management console

29

SOA Architecture – Logical Layered ViewSOA Architecture – Logical Layered ViewLayers Separated By Simple APIsLayers Separated By Simple APIs

Client/Presentation Services

Platform Layer

Mission Services Layer

Data Access Layer

Metadata Index

Data PersistenceStore

En

terp

ris

e S

erv

ice

Bu

s -

Co

mm

un

ica

tio

n

Se

cu

rity

Se

rvic

es

/D

em

ilit

ari

zed

Zo

ne

(D

MZ

)

SpatialIndex

Hydro Models

LAPS

FORTRAN/C/C++Command Line

Programs

External Programs

JMX

<<Java>>

DataLayer

PostgreSQLHDF5

<<abstract>>

BaseDao

Hibernate

<<Java>>

HDF5DataStore

HDF5 API

IngestSrv

PersistSrv

IndexSrv

ProductSrv

AdapterSrv

NotifySrv

SubscribeSrv

AutoBldSrv

PurgeSrv

Mbean

CAVE

StagingSrv

UtilitySrv

LocalizationStore

30

AWIPS-II ADE High Level System ServicesAWIPS-II ADE High Level System Services SOA Services Running in an ESB ContainerSOA Services Running in an ESB Container

IngestSrvStagingSrv

PersistSrvPurgeSrv

IndexSrv ProductSrv

NotifySrv AutoBldSrv Collaboration

CAVEVisualization

Client

CORE SOA Services

Mbean Mbean Mbean Mbean

Mbean Mbean

JMX RemoteService Management

Client

METAR Plug InRadar Plug InSatellite Plug In

FileSystemRDBMS via JDBC

Meta Data IndexSpatial Data Base

uEngineManage Subscription

Subscription Notify

Data RenderingCAVE Bundles

CAVE ProceduresWx DrawingWx Warning

Data Interrogation

Enterprise Service Bus – HTTP, JMS, Virtual Memory, File Endpoints

AdapterSrv

Mbean

Exec AdapterJNI Adapter

Radar All Tilts

ADE 1.0

uEngine

Services Independent of End Points

Enterprise Service Bus – HTTP, JMS, Virtual Memory, File Endpoints

HDF5 persistence

XMPP

Updated T06

Hydro Visualization

WAN Visible

GRIB Plug In

UtilitySrv

Mbean

Localization Data

GFE IFPServer

Color TablesMaps and Topo