portland2003_animated
-
Upload
tammy-kobliuk -
Category
Documents
-
view
13 -
download
0
Transcript of portland2003_animated
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Lessons Learned
From Handling More Data
Than
Grains of Sand on the Earth
Presenter: Tammy Kobliuk
Resource Analysis Section
Forest Management Branch
Public Lands and Forest Division
Alberta SRD
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Brief Overview • A little bit about Alberta
• A brief history of GIS in SRD
• Our technical setup
• Data, data, data…
• Analysis challenges
• Technical challenges
• Lessons learned
• What the future holds
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
An Alberta Primer
• We’re big.
• We have oil & gas, cows, trees, and grain.
• We are considered a “Prairie” province.
• We are where the prairies meet the mountains
(those little things we call the Rockies).
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Where are we?
ÊÚ
ÊÚ
BritishColumbia
Alberta Saskatchewan
Manitoba
Ontario
Washington
OregonIdaho
Montana
Wyoming
North Dakota
South Dakota
Nebraska
Minnesota
Iowa
Wisconsin
Michigan
California Nevada UtahColorado Kansas Missouri
Illinois
Alaska
Edmonton
Portland
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
How big are we?
• 66,258,096 Hectares
• 662,581 Square Km
• 255,824 Square Miles
• 163,726,700 Acres
• 2 UTM zones
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
A Size Comparison
Washington +
Oregon +
Nevada =
ALBERTA
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Tiling the province
• 50 1:250 000 NTS mapsheets
• 764 1:50 000 NTS mapsheets
• 7204 Townships
• > 1 million quartersections
• 4400+ Phase 3 mapsheets
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Ecological
Regions
Prairie
Parkland
Forest
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Forest
Management
Agreements
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Who Are the Resource Analysis Section?
Forest Protection
Resource Analysis Section
Forest Health
Forest Planning
Harvest and Renewal
Forest Management Branch
Forest Operations
Forest Business
Public Lands and Forest Fish & Wildlife PLFD Regions
Sustainable Resource Development
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
What do we do?
• Review timber supply and technical analysis of company management plans.
• Timber supply analysis and technical analysis for Crown management plans.
• Provincial-scale reporting to federal govt.
• Development of modeling tools and analysis procedures.
• Development/maintenance of specialized databases.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
What else do we do?
• Technical support for departmental
initiatives.
• On-demand analysis for the
Minister’s office.
• Answer Freedom-of-Information
(FOIP) requests.
• Any other priorities-of-the-day.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
A Brief History of GIS in the
Forest Management Branch • In the beginning…there were dot grids and
colored pencil crayons.
• Terrasoft was purchased in the late 1980’s to early 1990’s.
• Terrasoft was retired in 1995 when ArcInfo and ArcView were first purchased.
• Widespread ArcView training commences in 2001 with the implementation of the Citrix ArcView project. Forest Management is the primary driver.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
…
• Base data has always been managed by the
Resource Data Branch.
• The original capture of digital elevation data was
in the 1980’s for the purpose of contour creation.
This data is still in use today.
• The original capture of the base data was by CAD
systems. This has been improved and updated to
attributed ArcInfo-ready files.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
…
• Resource Data Branch was de-centralized in the mid-to-late 1990’s as “GIS” staff were sent out to colonize the regions. These staff were primarily CAD users and had little GIS training.
• Forest Management GIS staff are located only in Edmonton.
• Edmonton no longer has control over regional GIS staff.
• Control over computer equipment purchases has been returned to Edmonton.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
General Setup
PC SunBlade workstation
Data drive
Data drive
Data drive
Data drive
PC PC
PC PC
Citrix Application Server
Data Server
Citrix
Server
Farm
Server Room
FMB
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Basic Hardware
• Sunblade 1000 dual CPU workstation with
500 GB storage space.
• Compaq workstations: dual CPU, dual
harddrive, CD writer, DVD, 21” monitors
• HP2500 Plotter (soon to be replaced)
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
GIS Software
• ArcInfo 7.2.1 and 8.2 Workstation (Solaris)
– 2 Node locks
– 2 Floating
• ArcView 3.2 (Windows 2000)
• ArcView 8.2
• ArcIMS and ArcExplorer
• SDE (Resource Data Branch only)
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Remote Sensing Software
• ERMapper 6.3
– Image processing
– Image compression (.ECW)
• Erdas Imagine Professional 8.5
– Primarily legacy modelling
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Enabling Software
• Hummingbird Exceed for XTerming into unix workstation.
• Samba for serving out unix data drives.
• Citrix Metaframe for serving out thin client ArcView and regional data.
• Oracle for enterprise database applications. *Note: Not currently available to ArcInfo users.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Little Added Extras
• Adobe Acrobat 5
• CorelDraw 9 and Corel
PhotoPaint 9
• Microsoft Office 2000
• FoxPro
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Software Maintenance
• When you’re big, costs are high.
• $225,000 to ESRI annually for SRD.
• ArcView 3.2 to ArcView 8.x
conversion comes with a steep annual
pricetag.
• ~$68,000 annual FMB costs.
• $ 3 million just to migrate operating
systems.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Sure there’s data, but…
• Who has it?
• Can you get it?
• What scale is it?
• Where do you store it?
• What does it cover?
• How good is it?
• How old is it?
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Where’s the Data?
Data
Resource Data Branch
Spatial Data Warehouse
FMA-Holders
Federal Government
Research
Organizations
Internet
Other Provincial
Agencies
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Data Acquisition in Canada
• Very little available for free.
• Cost recovery model for governments.
• Most data is licensed.
• Can be very difficult to find out who has it.
• When you can find it, costs are often prohibitive.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Base Data in Alberta
• ESRI Digital Chart of the World
• Provincial 1:1 million
• Federal 1:250,000 and 1:50,000
• Provincial 1:20,000 (and
1:50,000)
• Local (municipalities, cities and
utilities)
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Data Storage
• Resource Data Branch is the primary data
repository and distribution arm for SRD.
• Regional/Area offices load and maintain
detailed base information for their areas.
• The Citrix data server is the only central
accessible data repository for users.
• FMB stores all data acquired for projects
for internal use only.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Vegetation Data
• Canada/Alberta Land Inventories
• Ecological and biophysical inventories
• Satellite classification
• Strategic level Forest/Vegetation Inventories
• Wetland Inventory
• Prairie Vegetation Inventory
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
More on Strategic Level Inventories
• Phase 1
• Phase 2
• Phase 3
• AVI 1.0
• AVI 2.1 * The current standard
• AVI 2.2
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Data Coverage
It’s hard to know what coverage we have for
what data. There are few accessible maps
detailing the age and coverage of base,
vegetation, and other data.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Coverage Example:
Vegetation Inventories
• Phase 1, Phase 2, Phase 3
• Alberta Vegetation Inventory: CVI, AVI
1.0, AVI 2.1, AVI 2.2.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Present
Digital
Vegetation
Inventory
Status
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Data Quality?
Reports on data quality and
acquisition specifications
are not readily available to
internal staff, let alone the
general public.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Projections, anybody?
• Alberta covers 2 UTM
zones
• A single projection
requires a custom
projection (10TM)
• Local: 3TM
• Agriculture: Albers
• Canada: Lambert
11 12
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Size of study areas…
• Extremely large areas of contiguous
industrial forest land.
• Massive size of individual tenures and
management units.
• Undergoing amalgamation of units.
• Multi-jurisdiction study areas.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
The Multi-Use Landbase
• There is a phenomenal amount of activity
modifying the landscape in some areas of
the province.
• Oil & gas, grazing, forestry and recreation
are all on some of the same areas.
• Datasets are out-of-date the moment they
are completed.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Seamless Data?
• Only recently has it been possible to order seamless base data in coverage format.
• 110+ vegetation inventory datasets to cover forested areas.
– Overlaps
– Different dates
– Differing quality
– Differing database formats
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Provincial Analysis
• No seamless spatial vegetation dataset.
• 3.85 million inventory polygons over 110+ datasets.
• How to create a provincial dataset that can incorporate data of differing dates and specifications?
• Do you need stand-level data?
• Can you use proxies?
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Change Over Time
• Keeping data for a project “up-to-date” is impossible.
• Choose an effective date for analysis. Document this.
• Make and document assumptions surrounding dated data.
• Choose a projection and datum and stick with it. Document this.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Confidentiality
• Company DFMP submissions are
confidential.
• Who should have access to what?
• What type of access should they have?
• How do you implement those access
restrictions?
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Getting Information From Your Data
• Does your data require massaging to extract
useful information?
• Can the data be enhanced by attribute
rollup?
• Can the data be enhanced by predictive
modelling?
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Case Study:
Interior Forest Analysis
• Study Area: single FMA 3.5 million Ha
• Source data: ArcInfo Library (~2.25
million polygons in 406 tiles)
• Attributes in separate .dbf file.
• Problem undefined.
• Solution undefined.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Timeline
• 2 weeks to nail down the exact problem and formulate the solution(s). Solution(s) were programmed for automation.
• 2 weeks to complete data prep and spatial analysis.
• 1 day to compile final datasets.
• 5 minutes to run Patch Analyst for final landscape metrics.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Obstacles
• Had to create input datasets tile-by-tile since the entire spatial net landbase could not be extracted.
• Could not grid off the entire FMA at 5 m.
• Difficult to partition the landscape into smaller pieces.
• Vector processes took an inordinate amount of time.
• We seemed to hit every software and hardware limit possible.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Raster vs Vector
• Raster data sets were very large.
• Raster processes were comparable to vector
processes on simpler landscapes.
• Raster processes were faster than vector
processes on complex landscapes.
• Raster outputs were easier to check than
vector.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
From Plan A to Plan D
• Plan A: Grid off the entire FMA and run
analysis.
• Plan B: Run entire FMA as vector
• Plan C: Partition the FMA and run pieces
as raster.
• Plan D: Run remaining partition as vector.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Lots of Data Means…
• Disk space issues:
– How many should you have?
– Where do you put them?
– How big should they be?
• Backup issues.
• Archiving issues.
• File management issues.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Large Datasets Mean…
• Streaming live data requires lots of
network bandwidth.
• Hardware, operating systems, and software
need to be optimally tuned.
• Unix systems may still out-perform PC’s.
• Data display needs to be managed.
• Data format needs to be carefully chosen.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Size is Relative
• What’s large for you?
• What’s large for the
software developer?
• Are software limits
documented? Can you
find limits prior to running
a process?
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
So Your Command Ran…
So What?
• ArcInfo commands may run to completion
on large datasets, but:
– Did it really complete?
– Did it run correctly?
– How do you check it?
– Did it clean up after itself?
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Plotting Large Files
• Print directly from ArcView or ArcGIS.
OR
• Export to .eps and use Image Alchemy
to convert to on-the-fly RTL.
• Export to .eps and use Adobe Acrobat
Distiller to create .PDF files.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Data Transfer
How to transfer very large files?
• Attempt to avoid ftp timeout.
• Compress and burn to CD.
• Get into burning DVD’s.
• Dig out the old tape drive.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Moving to Geodatabase
• The 2GB Personal Geodatabase limit is too
small for many datasets.
• We don’t currently have Oracle access.
• We don’t have an Oracle DBA.
• We don’t have an SDE license. Needed?
• It’s not feasible to port all of our data.
• Is Oracle Personal an option?
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
SDE?
• SDE requires a lot of up-front Oracle database tuning.
• Requires a knowledgeable administrator.
• $$$ Purchase cost and $$ annual maintenance.
• Fine for map display and browsing/querying.
• Not good for spatial analysis.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
The Basics
• Keep it simple!
• Newer doesn’t necessarily
mean better.
• Older doesn’t always mean
bad. (ie. AML)
• Don’t put all your eggs in one
basket (ie. Web deployment).
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Knowledge and Creativity
• Know your data and what it can and should
be used for.
• Don’t reinvent the wheel. Look at what
others have done.
• There’s always more than one way to skin a
cat. Always have a plan B and C for when
plan A fails.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Technology
• Know your software. Use it.
• Use the best tool for the job. No one toolset will
do everything you need.
• Be cautious. Let other people find the bugs.
• Poorly planned software upgrades can become
“downgrades”.
• Monitor software and hardware developments.
• Do performance tuning on hardware/software.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Practice Good File Management
• Get rid of unnecessary and duplicate files.
• Archive to CD, DVD, and/or tape.
• Take responsibility for managing files
and data.
• Attempt to keep documentation.
• Standardize wherever possible.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Cost Management
• Consolidate software client numbers to take advantage of secondary license pricing.
• Use your organization size or partnership arrangement(s) to negotiate a deal.
• Consider floating licenses or software servers to cut down on total license numbers.
• Keep an up-to-date inventory of licenses.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Project Risk Management
• Don’t put all your eggs in the ArcView project
or ArcGIS map document baskets.
• Sever the reliance on data availability and
location.
• Save out legend and layer files.
• Save important maps as image files and/or .pdf
files. Burn them to CD. Have hard copies.
• Document, document, document…
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Check Your Outputs
• Spatially view your data.
• Query out results.
• Do spot checks.
• Do topology checks.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
To Contract or Not To
Contract? • DO Use consultants wisely.
• DO contract out simple one-time projects that regular staff don’t have time for.
• DO ensure transfer of skills to internal staff.
• DON’T use them as a replacement for internal staff.
• DON’T get too closely tied to any one company
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Failing the Bus Test?
• Inventory your data.
• Inventory your folders.
• Standardize your directory structure.
• Keep up on metadata for data.
• Keep up on metadata for projects.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Data Access Management
• Limit data exposure.
• Use different Samba shares to share out
specific directories.
• Use different Samba shares to enforce
access control.
• Use different Unix account groups to allow
for setting Unix permissions.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Change Management
• Don’t hardcode things that will change over
time. (ie. Administrative boundaries)
• Make use of relational databases to enable
partial updates.
• Plan for scalability.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Data Models
• Develop them from the start.
• Document them and make them available.
• Don’t let non-data people control Data
Model creation.
• Ensure proper representation of end-users in
the Data Model creation process.
• Ensure that the data model meets needs.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Learn • Admit what you don’t know. Then try
to learn.
• The pace of change in both software and hardware is extraordinary. Keep tabs on it.
• You can’t be an expert on everything. Know a little about a lot and a lot about a little. Specialize.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Communicate
• Sell the technology from both ends. (ie. To management while also recruiting end-users).
• Communicate your accomplishments.
• Share what you learn through technology transfer. Generosity pays dividends.
• Network with your peers.
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
• ArcGIS 9 Suite
• XML-based metadata
• A possible move to geodatabase
• Still AML, but possibly Python
• SDE and Oracle?
• More imagery
• Possible server consolidation
Software/Data
Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Expectations
• More demand for services.
• More complex analyses requested.
• More internal expectation of basic
ArcView skills.
• Internal training switch from
ArcView 3.2 to ArcView 8.x.
• More staff?