GEOFRAME: a system for doing hydrology by computer

CUAHSI BIANNUAL MEETING - BOULDER (CO) - JULY 14-16 2008

University of Trento (IT) - HYDROLOGIS

GEOFRAME a system for doing hydrology by computer

R. Rigon, A. Antonello, S. Franceschi, D. Giacomelli, E. Cordano, S. Endrizzi, S. Simoni, M., Dall’Amico, C. Tiso, F. Zanotti


Rigon et al., JHM, 2006, Bertoldi et al., JHM, 2006, Simoni, 2007, Endrizzi, 2007www.geotop.org

GEOtop

Figu

re m

odifi

ed a

fter

Lia

ng e

t al

, 199

4


http://www.geotop.org



JGrass/uDig

http://www.jgrass.org, http://jgrasstechtips.blogspot.com/




http://jgrasstechtips.blogspot.com



MODELS

IS MODELING SCIENCE ?

Mad

emoi

selle

Ros

e, c

a. 1

820.

Eug

ène

Dela

croi

x.


DataParametersEquations

Mass, momentum and energy conservation. Chemical transformations

Forcings and observables

Equation’s constant. In time! In space they are usually heterogeneous

Models we are talking about are computer applications


In the past they were built as monolithic programs


I - Once a model, design and implemented as a monolithic software entity, has been deployed, its evolution is totally in the hands of the original developers. While this is a good thing for intellectual property rights and in a commercial environment, this is absolutely a bad thing for science and the way it is supposed to progress.

Rob

bed

from

a C

CA

pre

sent

atio

n



II - Independent revisions and third-party contributions are nearly impossible and especially when the code is not available. Models falsification (in Popper sense) is usually impossible by other scientists than the original authors.

III- Thus, model inter-comparison projects give usually unsatisfying results. Once complex models do not reproduce data it is usually very difficult to determine which process or parameterization was incorrectly implemented.



Q: HOW CAN WE BE MORE “GALILEIAN” ?

A: YES, PRODUCING AND PROMOTING OPEN SOURCE MODELS. THIS HOWEVER IS NOT ENOUGH SINCE MODELS SHOULD B E S T R U C T U R A L LY E A S Y T O UNDERSTAND, DOCUMENT, MODIFY, MAINTAIN, AND FAVOR PROCESSES ANALYSIS.


MODELLING, FOR WHO ?Which end user do you have in mind ?

Babo

on, P

apiu

s Anu

bis

SCIENTIST ARE NOT THE ONLY MODELS USERS


• Prime users: take or prepare decisions at a political level

• Technical users: prepare projects or maps for the primary users

• Other end-users: national agencies, representative groups, etc. They may take or prepare decisions at national or regional level, or represent stakeholder groups.

• Model and application developers/modellers: build models and targeted applications

Use

rs/A

ctor

s

Mod

ified

from

Riz

zoli

et a

l., 20

05



• Coders: implement models, applications and tools.

• Linkers: link existing models and applications.

• Runners: execute existing models, but they create and define scenarios.

• Players: play simulations and experiments comparing scenarios and making analyses.

• Viewers: view the players’ results, have a low level of interaction with the framework.

• Providers: provide inputs and data to all other user roles.

Rol

es

Mod

ified

from

Riz

zoli

et a

l., 20

05



RolesUsers

Hard Coders

SoftCoders

Linkers Runners Player Viewers Providers

Prime

Other End Users

Technical

Researchers

Modified from Rizzoli et al., ,2005



Thus there exists a “zoo” of users, each one with different abilities.

Bundling data, algorithms and the graphical user interface of a model in an application with the old techniques makes the models very hard to re-use out of its original context.

AN ECOLOGY OF MODELS


Object-oriented software development . O-O programming is nothing new, but it has proven to be a successful key to the design and implementation of modelling frameworks. Models and data can be seen as objects and therefore they can exploit properties such as encapsulation, polymorphism, data abstraction and inheritance.

Component-oriented software development. Objects (models and data) should be packaged in components, exposing for re-use only their most important functions. Libraries of components can then be re-used and efficiently integrated across modelling frameworks. Yet, a certain degree of dependency of the model component from the framework can actually hinder reuse.

NEW (well relatively) MODELING PARADIGMS

Mod

ified

from

Riz

zoli

et a

l., 20

05

MODELLING BY COMPONENTS


Discrete units of software which are re-usable even outside the framework, both for model components and for tools components.

Seamless and transparent access to data, which are made independent of the database layer.

A number of tools (simulation, calibration, etc.) that the modeller will be free to use (including a visual modelling environment).

A model repository to store your model (and simulations) and to share it with others.

BENEFITS



Tools for studying feedbacks among different processes.

BENEFITS FOR SCIENTISTS

Encapsulation of single processes or submodels

MUCH MORE in the field of possibilities


New educational tools and a “storage” of hydrological knowledge using appropriate onthologies


T H E R E E X I S T S U C H M O D E L I N G INFRASTRUCTURE ?

Economic modelling frameworks^. GAMS (general algebraic modelling system, http://www.gams.com) and GTAP (global trade analysis program, http://www.gtap.agecon.purdue.edu ) are some of the most used modelling systems in the agro-economic domain. They can also account for social variables, such as unemployment.

^from Rizzoli et al., (Modeling Framework (SeamFrame) Requirements 2005


http://www.gams.com

http://www.gams.com

http://www.gtap.agecon.purdue.edu






Environmental modelling frameworks. If we limit to the agricultural domain, the list is quite limited. There is no ‘real’ framework according to the definition, but APSIM, STICS and CropSyst provide some of the functionalities. In this area SEAMFRAME is an emerging technology. When we consider the water management sector, we find many examples, such as TIME (the invisible modelling environment), IMT, OpenMI, and OMS, and, to a certain respect, JUPITER-API.

^ extended from Rizzoli et al., (Modeling Framework (SeamFrame) Requirements 2005




Other modelling software environments of notable interest are SME, MMS, ICMS, Tarsier, Modcom, Simile, but they are integrated modelling environments, not frameworks. This means that they can be used to perform assessments, analyses, decision support, but they do not provide programming structures such as classes, components, objects, design patterns to be used to create end-user applications.

^from Rizzoli et al., Modeling Framework (SeamFrame) Requirements, 2005




Atmospheric Sciences: Earth Sciences Modeling Framework (ESMF) (including Earth System Curator)

High Performance Computing: Common Component Architecture (CCA)



AN EXAMPLE OF FURTHER NEEDS

Rigon et al., JHM, 2006, Bertoldi et al., JHM, 2006, Simoni, 2007, Endrizzi, 2007www.geotop.org

Figu

re m

odifi

ed a

fter

Lia

ng e

t al

, 199

4

AWARE OF HYDROLOGICAL PROCESSES




Rigon et al., JHM, 2006, Bertoldi et al., JHM, 2006, Simoni, 2007, Endrizzi, 2007

GEOtop EquationsSurface Runoff



GEOtop EquationsTurbulent Sensible and Latent Heat Exchanges





GEOtop EquationsSubsurface Flow



GEOtop EquationsSoil Temperature




Required Data- Digital Elevation Model (and derived geomorphic information - as lakes, river networks, hillslopes - and geomorphometric information - like contributing areas, slopes, curvatures)

- Sky View Factor

- Vegetation and Land Use (for interception, ET, surface roughness)

- Soil Depth, Texture and Type (for hydraulic and thermal properties)




Requires Boundary Conditions



REQUIRES ALSO:

BOUNDARY CONDITIONS

METEO FORCINGS

PARAMETERS

INITIAL CONDITIONS

METHODS OF INITIALIZATION


Forcings- Precipitation (quantity and type, spatially distributed)

- Relative humidity (spatially distributed)

- Wind Speed and direction (spatially distributed)

- Solar Radiation (spatially distributed)



Parameters

- Soil Parameters (e.g. van Genuchten)

- Surface roughness (for water celerity)

- Bulk aerodynamic properties

- Ice and snow properties

- Atmosphere radiative properties



- Soil moisture (profile, in terms of matric potential, spatially distributed)

- Soil temperature (profile, spatially distributed)

- Surface water (if present)

- Snow cover (if present)

Required Initial Conditions


DataParametersEquations

Mass, momentum and energy conservation. Chemical transformations

Forcings and observables

Equation’s constant. In time! In space they are heteorgeneous

Numerics, boundary and initial conditions

Data Assimilation. Data Models. Tools for Analysis.

Calibration, derivation from proxies

To Sum up



input tools output

models

expertsystems

scenarios

forecasting

CLIMATE forecast

field observations

human expertise

StAtistics

Besides

MODELLING BY COMPONENTS REPRISE


WeatherGenerator

Monitoreddata

WeatherForecast

SurfacesInterception

SurfacesRunoff

Subsurface Flow

Evapotranspiration

http://www.openmi.org, http://www.openmi-life.org/, http://public.wldelft.nl/display/OPENMI/Home


Models as components

http://www.jgrass.org


http://www.openmi-life.org


http://public.wldelft.nl/display/OPENMI/Home



Models as components

SurfacesRunoff

Subsurface Flow

Evapotranspiration

Channel RoutingII

Channel Routing III

Channel Routing 1

http://www.openmi.org, http://www.openmi-life.org/, http://public.wldelft.nl/display/OPENMI/Home









Decision making

EVALUATION OF STRATEGIES THROUGH MODELS

STRATEGIES FOR POLICY MAKERSDATA INTERPRETATION



DATA INTERPRETATIONDATA INTERPRETATION

STRATEGIES FOR POLICY MAKERSSTRATEGIES FOR POLICY MAKERS



GEOFRAME

AnalysisTools (UNITN/R)

Database (PostgresSQL/PostGIS/HSQL/H2/CUAHSI)

Models(UNITN/OpenMI)

Interfaces(Java/udig/JGrass/BeeGIS)

users webexternal database

DEPLOYEMENT


PREREQUISITES

Programming LANGUAGE NEUTRAL

PLATFORM NEUTRAL: Windows, Linus and Mac

OPEN SOURCE

TARGETED AT PERSONAL PRODUCTIVITY OF DIFFERENT USERS People come before program efficiency.

BUSINNES NEUTRAL: GPL would be fine, LGPL better

DEPLOYEMENT


PREREQUISITES

ALLOWS WRAPPING OF EXISTING CODES BUT PROMOTES BETTER PROGRAMMING STRATEGIES

BUILT BY OPEN SOURCE TOOLS

DATA BASE PROVIDED

OGC COMPLIANT

CUAHSI SPECIFICATIONS AWARE

DEPLOYABLE THROUGH THE WEB

DEPLOYEMENT

CAN BE ENDOWED WITH ONTOLOGIES


PostGISPostgres

Webservices

WMSWFS-TWPS

Webservices

WMSWFS-TWPS

The complete framework

OpenMI

J-Console Engine

JGrassuDig

Eclipse RCP

H2 spatial

UIBuilder

GRASS

GIS engine

The Horton Machine

Models

BeeGIS

DEPLOYEMENT


Java

JGrassuDig

Eclipse RCP

SOLIDITY: The framework bases on the solid fundaments of the Eclipse RCP framework first created by IBM.

CONNECTIVITY and USERFRIENDLYNESS: The GIS framework is based on the uDig GIS framework, specialized in accessibility and remote connections

ANALYSIS: The JGrass extentions define a layer of powerful GIS analysis tools and a straight connection to the GRASS GIS

MOBILITY: The BeeGIS extentions supply tools for digital field surveying

BeeGIS

DEPLOYEMENT


Database:PostGIS-PostgresH2 spatial

Web services

WMSWFS-T

soon WPS

Connectivity and web standards

RESTful?

DATABASE: The GIS framework is ready to connect to external relational databases as postgres, mysql or oracle. To spatial data servers like postgis, Oracle spatial and Arcsde. It also comes with an internal spatial database based on H2 (no indexing yet)‏‏.

It would be fairly easy to create connections to RESTful services to acquire data.

W E B S E R V I C E S A N D S TA N D A R D W E B PROTOCOLS: The framework supports OGC web standards like the web mapping service (WMS), the web feature service, also in transactional format (WFS-T). An efforth for the web processing service is ongoing.

DEPLOYEMENT


The analysis engine

OpenMI

J-Console Engine

GRASS

THE CONSOLE ENGINE: the console engine supp l i e s a f r amework fo r mode l i n g development and scripting environment for fast methodology testing.The engine contains already masses of modules called Horton Machine for various terrain analyses as well as a stability model and hydrologic models.Also the engine gives access to the GRASS analysis modules.

THE STANDALONE MODE: The need for usage of the modelling environment on supercomputer defined a heavily decoupled design for the console engine. The framework defines a strict interface between GUI and analysis engine, which makes it easy to exploit the console engine in standalone mode on server-side.

The Horton Machine

Models

DEPLOYEMENT


The relationship to OpenMI

OpenMI

THE OPENMI ENGINE: the console engine exposes a compiler for an OpenMI based modeling language.This gives a way to write scripts to execute openmi chained models.

THE OGC STANDARDS EXTENTION: The need for big vector and raster data forced the team to extend the OpenMI standard interfaces with two GIS OGC standards: the OGC feature modelthe OGC grid coverage service (in prototype mode)‏

OGC IN JGRASS: the OGC feature and grid coverage models are served by the geotools libraries. The coverage model is based on the Java Advanced Imaging library and supports tilecaching for processing of large dataset. Coverage data are passed to native languages as C, C++ and Fortran through the easy adoptable JNA libraries.

The Horton Machine

Models

DEPLOYEMENT


Strategic calls for cooperation

Statet: it is a small RCP plug-in coordinated by Stephan Wahlbrink for working with R scripts and documentations. R is "a language and environment for statistical computing and graphics". The autor has already been contacted.

Talend – Spatial data integrator: CampToCamp develops a psatial data integrator based on Talend. A first meeting was made to discuss architectural issues. By end of the year the developers of JGrass and SDI will meet to prototype connectivities between the the softwares.

AND TO EVERYBODY OF GOOD WILL OBVIOUSLY!

DEPLOYEMENT


SNAPSHOTS OF THE SYSTEM ...It really exists (yes in a pre-alpha version)!

DEPLOYEMENT


GEOcuencas (modified from Mantilla and Gupta, 2005)

Reservoirs and intakes

ETMeteo Forcings

Runoff and Routing modeling

Snow

DEPLOYEMENT


Snapshot of a DEM of river Adige with subbasin partition

DEPLOYEMENT


active nodes

Snapshot of a DEM of river Adige with subbasin partition

DEPLOYEMENT


reservoirs

hydrometers

meteo stations

returns

AdB Adige – CUDAM - Bilancio idrico di superficie di primo livello

Active means that they are connected to db information

JHYDRO


• Monitoring stations:

reservoirs

hydrometers

meteo stations

returns



JHYDRO


• Monitoring stations:

reservoirs

hydrometers

withdraws

meteo stations

returns



JHYDRO

CUAHSI BIANNUAL MEETING - BOULDER (CO) - JULY 14-16 2008AdB Adige – CUDAM - Difesa idrogeologica e bilancio idrico nel bacino dell’Adige

The database is based upon ArcHydro (Maidment, 2002)

DEPLOYEMENT


First Level

third level

second level

Using a modified Pfafstetter enumeration (Verdin and Verdin, 1999)

JHYDRO


J-HYDRO: STRUTTURA DELLA RETE

Livello rete 5Livello rete 4Livello rete 3Livello rete 2Livello rete 1

JHYDRO


J-HYDRO: The network structure

Livello rete 4Livello rete 3Livello rete 2Livello rete 1

JHYDRO


Livello rete 3Livello rete 2Livello rete 1

JHYDRO



Livello rete 2Livello rete 1

JHYDRO



Livello rete 1

JHYDRO



JGRASS: visualization of the DB

DATABASE CONNECTION

LAYERS VIEW

JGRASS


JGRASS: visualization of other data

JGRASS


Models’ runs

OPENMI MODELS


Models’ runs: monitoring stations

OPENMI MODELS


Rainfall Forcings

OPENMI MODELS


Models’ runs

OPENMI MODELS


Models’ runs: parameters

OPENMI MODELS


Summary of results

OPENMI MODELS


JGRASS: visualization of results

OPENMI MODELS


http

://jg

rass

tech

tips.

blog

spot

.com

/

WEB DEPLOYEMENT OF THE DEVELOPERS SETUP




PERSPECTIVES AND CONVERGENCESESMF

Earth System CURATOR

CCA OMS


PERSPECTIVES AND CONVERGENCES

These above systems are not mutually exclusive they have many features that can be integrated in each of the systems, for sure in GEOFRAME.

It i s to support cooperat ion and re inforce interoperability among these characteristics.

And we, GEOframers offer our unselfish collaboration.



For instance future components could adopt the ESMF as modeling infrastructure.

Strength: This would allow interoperability with the Atmospheric Science work.

Weaknesses: At the moment using the infrastructure is not as easy as it could be. Documentation is very FORTRANISH and more examples should be added for C/C++ programmers. An effort should me made to compile the whole thing using Eclipse, besides than using makefiles.



OMS already provides:

- Components descriptions-Tools for adding a component- Model Wizard- Descriptions of Model Attribute Connectivity- Connection to COLAB

which could be added to GEOFRAME probably almost “as is”



At the same time GEOFRAME can provide to OMS:

- a Java interconnected GIS system - a Java database for storing data information- The J-console for scripting and dealing also with OpenMI compliant models



Unfortunately OMS provides mechanism for component linking different from OpenMI. However OpenMI is a set of interfaces which is pretty much evolving and this is the right time to make efforts for interoperability.



Not OMS (which uses JNative) nor GEOFRAME* (which uses JNA) have a system for binding different languages codes. Thus there is room to adopt CCA (or other) solutions, if they result feasible.

*Nor OpenMI on which GEOFRAME is based


EPILOGUE

O U R A I M I S N O T T O M O D E L EVERYTHING*OR DO A MODEL OF EVERYTHING BUT GIVE A SPACE WERE DIFFERENT, EVEN CONTRADICTORY, IDEAS, AND DATA CAN BE EXPLOITED IN A WAY WHICH PROPELS COLLABORATIVE EFFORTS BY SCIENTISTS AND USERS.

*“Correctly interpreted, you know, pi contains the entire history of the human race.”-Dr. Irving Joshua Matrix, from M. Gardner, “The magic numbers of dr. Matrix”


Direct Contributors:

Andrea Antonello JGRASS, JHydro, JConsole, core developer and architectGiacomo Bertoldi GEOtop developer (energy budgets, vegetation)Emanuele Cordano GEOtop developer (Richards equation, I/O)Matteo Dall’Amico GEOtop developer (permafrost, GEOtop-mono)Stefano Endrizzi GEOtop developer (energy budgets, snow, permafrost) Silvia Franceschi JGRASS models developer and architectDavide Giacomelli JHydro developerAndreas Hamm JConsole developerMarco Pegoretti GEOtop developer (surface runoff and routing)John Preston JGrass ver. 1 and 2 core developerRiccardo Rigon GEOFRAME architectSilvia Simoni GEOtop developer (Richards equation), GEOtop-FS developerChristian Tiso GEOtop power user, apostle and Horton Machine developerPaolo Verardo GEOtop developer (Radiation budgets)Fabrizio Zanotti GEOtop-FS developer, GEOtop power user and apostle

Erica Ghesla, Andrea Cozzini, Silvano Pisoni and others contributed to the Horton Machine


Dan

ce, H

enry

Mat

isse

, Hot

el B

iron

early

190

9Numerics

Statistics

Geography

Hydrology

THANKS FOR YOUR ATTENTION

Analytics


J-Console is the core to open OpenMI to modellers


http

://jg

rass

tech

tips.

blog

spot

.com

/




Where does the console live ?

http

://jg

rass

tech

tips.

blog

spot

.com

/




Issuing JGRASS commands

http

://jg

rass

tech

tips.

blog

spot

.com

/




Issuing GRASS commands

http

://jg

rass

tech

tips.

blog

spot

.com

/




Issuing BEANSHELL commands

http

://jg

rass

tech

tips.

blog

spot

.com

/




BTW YOU CAN USE THE CONSOLE

WITHOUT

JGRASS

http

://jg

rass

tech

tips.

blog

spot

.com

/




IT IS EASY TO CREATE YOUR

COMMAND

http

://jg

rass

tech

tips.

blog

spot

.com

/




Write your OpenMI code like the one you can find on: http://jgrasstechtips.blogspot.com/search/label/console

http://jgrasstechtips.blogspot.com/search/label/console

http://jgrasstechtips.blogspot.com/search/label/console


CREATING A ModelAlgorithmCommandDosomething (MACAD): The above is the list of the available MACADs ... add your own


CREATING THE XML FOR THE GUI BUILDER


CREATING AN RCP ACTION TO LAUNCH THE PROGRAM


TRYING IT

http

://jg

rass

tech

tips.

blog

spot

.com

/



GEOFRAME: a system for doing hydrology by computer

Education

Transcript of GEOFRAME: a system for doing hydrology by computer