FME and FEMA's National Flood Hazard Layer

FME & FEMA’s National Flood Hazard Layer (NFHL) Rob Gaines and Eliza Ledwell

About Us

Eliza Ledwell Rob Gaines

IBM – FEMA Risk MAP Customer and Data Services,

Managing Consultant [email protected]

IBM – FEMA Risk MAP Customer and Data Services,

Data Services Manager [email protected]

FEMA Risk MAP

Risk MAP Vision: •  Work collaboratively with state & local entities •  Deliver quality data •  Increase public awareness •  Reduce risk to life & property

Mapping, Analysis, and Planning

FEMA Risk MAP

Customer & Data Services (CDS): •  IT Hosting •  Application development •  Communications & user support •  Consulting

Mapping, Analysis, and Planning

FEMA’s National Flood Hazard Layer (NFHL)

•  Widespread: all effective Flood Insurance Rate Maps for areas covered by digital data

•  Updated daily: the single most up-to-date source of FEMA regulatory flood hazard information

•  Public-facing: NFHL web services receive over 20 million hits per month

Organization and Display The NFHL is organized into more than 50 data layers

The NFHL layers include:

•  Flood hazard zones and labels •  Base Flood Elevations (BFEs) •  Cross-sections and coastal transects •  Revision information such as LOMR and

FIRM Panel boundaries •  Community boundaries and names •  Structures such as levees, hydraulic, and

others involved in flood control

Ways to Access NFHL Data

WFS

NFHL Status Page

Google Earth TM

WMS

REST

FEMA GeoPlatform

MSC

NFHL on FEMA GeoPlatform

Data Flow

Flood Study

Revisions

PTS CDS NFHL Users

Data Revision (PTS)

Data Production (Mapping Partners)

Application and Data Hosting (CDS)

Data Flow

Flood Study

Revisions

PTS CDS NFHL Users

Application and Data Hosting (CDS)

How Does the NFHL Get Updated?

Receive Data Track Data QC Data Stage Data Incorporate

Data Extract Data


Receive Data Track Data QC Data Stage Data Incorporate

Data Extract Data

FME


Receive Data Track Data QC Data Stage Data Incorporate Data Extract Data

Manual data movement •  Data is received through download links in an email •  Must be manually transferred to secure DHS hosting environment



FME

Tracking – the data undergoes basic integrity checks and is logged into the NFHL Tracking database

NFHL Tracking Database



FME

QC – the data is checked for quality issues and rejected if issues are found



FME

Staging – data is loaded to the NFHL Staging Database (an offline file geodatabase) once it has passed QC



FME

Incorporation – data is published to the live NFHL database once it becomes effective



FME

Extraction – data is extracted out of the live NFHL database to jurisdictional and state datasets and made available on www.msc.fema.gov

Current Process

Job Management

Current Process Overview

Detect new submission

files

File Repository

Track submission information

Tracking Database

QC submission

data

Stage submission

data

Publish datasets

Extract datasets

File Repository

NFHL DB

Old System vs. Current Old •  Manual input of

parameters •  Manual running of

workspaces

Current •  Automatic detection of

new submission files •  Automatic job triggering

and population of parameters

•  Increased tracking visibility

Automatic File Detection

Requirements Predefined file structure: L:\NFHL_Staging\TransferData\<BatchID>\<Submission Data>

Requirements Standardized file formats: •  ESRI Personal Geodatabase (.mdb) containing submission data •  Excel sheet containing submission manifest •  Matching filenames

Automatic File Detection 1.  Read in relevant files 2.  Query log to see if files have been encountered 3.  If new, record in log

Tracking DB FME Read-In

Model File Repository Downstream

Models

Automatic File Detection Step 1: Read in relevant files using Directory and File Pathnames Reader pointed to base file repository

•  Identify relevant filetypes

Automatic File Detection Step 2: Query Tracking database to identify any datasets that have already been ‘seen’ by the system

•  Disregard any seen datasets •  Record any new datasets

Automatic File Detection Step 3: File information can now be used to run downstream processes and associated with tracking entries

Job Management Framework

Job Management Basics •  Very useful for automating multi-stage data processes

• Tracking database captures information about each submission and records the runtimes and results of each processing phase

•  Parent jobs query tracking database to determine what work to perform, then spawn child jobs to handle each dataset

•  Child jobs perform work and write results back to tracking database

Tracking DB

Parent Job Child Jobs Child Jobs Child Jobs

read

run

write

Submission Data

Job Management



files

File Repository


Tracking Database

QC submission

data

Stage submission

data

Publish datasets

Extract datasets

File Repository

NFHL DB

Job Management Example: QC Step 1: Parent job queries tracking DB to determine datasets in need of QC

•  Can use in-model logic or DB view/query

Query for datasets that have been tracked but not QCed

Job Management Example: QC Step 2: Parent job runs child job for each dataset via WorkspaceRunner

•  Use data elements from Tracking DB to set parameters for each job

Use data elements to set parameters

Job Management Example: QC Step 3: Write results back to Tracking DB with child job

•  Write timestamped status to master table so that future QC jobs will not re-check the same dataset

QC timestamp

New Process: FME Server

Current System vs. New Current (FME Desktop) •  Manual file download and

upload •  Transmission by email /

http download •  Manual resolution of QC

issues

New (FME Server) •  Web-based file upload •  Realtime QC results •  Fully automated

Job Management



files

File Repository


Tracking Database

QC submission

data

Stage submission

data

Publish datasets

Extract datasets

File Repository

NFHL DB

Extraction Manager

Replication Manager

Submission Manager (FME Server)

New Process Overview

Log Submission

User Upload

Tracking Database

Publish datasets

Extract datasets

File Repository

NFHL DB


QC submission

data

QC Results

File Repository

New Process Lessons Learned •  FME Server plays very nice with parent/child job approach.

•  It’s difficult to pass information between FME Server workspaces.

•  Tracking Database is more important than ever with FME Server.

•  It can be difficult to provide output reporting with the out-of-the-box FME Server UI.

•  Error handling is important.

•  The Schema Reader is awesome for quality control.

SubmissionManager.fmw

The Future •  Expanded self-serve reporting options

•  Additional data submission / QC processes

•  Data integration web services

Thank you! Robert Gaines IBM – FEMA Risk MAP Customer and Data Services, Data Services Manager [email protected]

Eliza Ledwell IBM – FEMA Risk MAP Customer and Data Services, Managing Consultant [email protected]

FME and FEMA's National Flood Hazard Layer

Technology

Transcript of FME and FEMA's National Flood Hazard Layer