Huub Scholten and Adrie J.M. Beulens Wageningen University - Netherlands

Laxenburg, 29-31 August 2005 Huub Scholten - CSM/KSS2005 - test MoST 1

Testing ontological support for multidisciplinary model-based

problem-solving for water management

Huub Scholten and Adrie J.M. Beulens

Wageningen University - Netherlands

Laxenburg, 29-31 August 2005

Huub Scholten - CSM/KSS2005 - test MoST 2

Index Introduction Problems to solve HarmoniQuA - MoST

Project Ontological KB and Knowledge Editor Modelling Support Tool

Verification - validation Mental Modelling Model Testing: criteria and results Conclusions



Introduction

Model-based problem solving is seen as a process Integrated problem-solving (with socio-economics

aspects) multidisciplinary problems higher demands to QA

QA through Process Knowledge Base

(here for modelling, instantiated for water management) Process support tool



Problems to solve Growing interest for QA in modelling with a multitude of

reasons: ambiguous terminology; a lack of mutual understanding between key-players, malpractice in regard to input data; inadequate model set-up; insufficient calibration/validation; model use outside of its intended scope; insufficient knowledge on some processes; miscommunication of the modeller to the end-user; overselling of model capabilities; confusion on how to use model results in decision making; a lack of documentation and transparency of modelling processes.



HarmoniQuA - the project

Period 1-1-2002 – 31-12-2005 (4 years)

2.57 million Euro (1.65 million Euro financed by EC)

12 partners in 10 countries with expertise in 7 domains of model

based water management

www.HarmoniQuA.org



HarmoniQuA - partners

WU, The Netherlands GEUS, Denmark NTUA, Athens, Greece CEH-Wallingford, UK WL | Delft Hydraulics,The

Netherlands Cemagref, France BfG, Germany SMHI, Sweden VITUKI Plc, Hungary Uni Dortmund, Germany LNEC, Portugal DHI Hydroinform a.s.,

Czech Republic



HarmoniQuA - aimsDeveloping a methodology and support for modelling in water

management according to the Water Framework Directive

consisting of:

1. Knowledge Base (KB): Guidance on how to model for water management Glossary of terms

2. Modelling Support Tool (MoST): Providing guidance from the KB Monitoring what modelling teams do Helping generating fancy reports

3. Training website for professionals and students: Screen-recording movies on MoST Presentations, exercises, background information

Model StudyPlan

Data andConceptualisation

ModelSet-up

Calibrationand

Validation

Simulation and

Evaluation

Model TeamSingle/multiple domain

MoST

Reporting and client review take place in each step

User

Knowledge BaseGuidelinesSoftware capabilitiesGlossaryDomains:

GroundwaterPrecipitation-runoffHydrodynamicsFlood forecastingWater qualityBiota (ecology)Socio-economics

GuidanceGeneric + specific for:- model domain- user- job complexity

ReportingSpecific for typesof users

MonitoringGeneric + specific for:- model domain- user- job complexity

AdviseFrom previousmodel projects

ModelArchive

Model journal, Project A

Model journal, Project B

Model journal, Project C

Model journal, Project D

MoS

T (

KB

+ to

ol)

over

view



Knowledge Base Content (what to do and how to do it) in increasing

specialization Knowledge on processes (co-operating in teams)

Specialised for modelling

Instantiated for water management

Implemented in an ontology (e.g. smart database; Protégé2000, (http://protege.stanford.edu) Protégé2000 is for knowledge engineering experts and too difficult

for domain experts

HarmoniQuA’s Knowledge Base Editor act as front-end for Protégé2000 and deals with authorisation issues

Provided to users by MoST (Modelling Support Tool)



Modelling Support Tool (MoST)

Provides guidance from the KB to users

Can set-up and support the execution of multidisciplinary

modelling projects

Monitors the work of all persons involved in a modelling project

Reports to specific audiences



Verification – validation of KBS

Verification: determining that the KBS is built according to its specifications.

Validation: determining that the KBS actually fulfils the purpose for which it was intended.

Testing: more generic term, referring to verification and validation.



Mental Modelling Model If problem owner has a problem, he/she can organize a

multidisciplinary team, consisting of team members, who have different disciplinary backgrounds and fulfil different roles in that team.

The multidisciplinary team ‘solve’ the problem at hand in a process with best professional practices.

The process is carried out in a modelling project, consisting of one or more subprojects, belonging to one or more domains/disciplines.

Subprojects: are single domain subproject or multidomain subproject; consist of steps.

Steps consist of tasks. Tasks consists of activities. Tasks and activities can use methods and tools. What teams have to do can be made explicit in a knowledge base.



Structured testing

Implemented ontological KBS (1) KB structure (11) KB content (related to water management) (12)

Process decomposition (121)Content of decomposition elements (122)

MoST (13) Training material (14)

KBS technology (2)



KB structure (11)Criteria

Correctness, completeness, consistency, granularity.

Method(s) Discussions with (1) internal experts, (2) internal testers and

(3) external reviewers. Using the structure to fill KB with the knowledge.

Result(s)

Initially discussions were fruitful, later not.



KB content: process decomposition (121)Criteria

Correctness, completeness, redundancy, consistency, transparency, granularity.

Method(s) Discussions with (1) internal experts, (2) internal testers and

(3) external reviewers. Using the KB in 2x10 test cases.

Result(s) Demands for changing the process decomposition came from

project partners and from the wider modelling society. These were related to (1) order of tasks, (2) dependencies of tasks and (3) is some piece of work a task or an activity.

Achieving agreement was difficult, but implementing changes was easy and not time consuming.



KB content: decomposition elements (122)Criteria

Correctness, completeness, redundancy, consistency, meaningfulness.

Method(s)Reading and using by (1) internal testers (not involved in filling KB, (2) professional modellers in test cases, (3) external reviewers.

Result(s)Long list of errors, wishes, comments, but all appreciated the guidance from the KB.



Modelling Support Tool (13)Criteria

Correctness, reliability, according to requirements, adequate for professionals, appropriate for students.

Method(s)Using MoST by (1) internal testers (not involved in filling KB),

(2) professional modellers in test cases, (3) external reviewers.

Result(s)Incremental (prototyping) development of MoST in 3 phases, with a test after phase 1 and phase 2 long list of ‘bugs’ more support needed for multidisciplinary teams + more flexibility (working

on 2 decomposition levels), because activity-level is often too detailed.



Testing training material (14)Criteria

Correctness, usefulness professionals, usefulness students.

Method(s)Using MoST in (internal and external) workshops/seminars and university courses.

Result(s)Many small changes, but training website and especially screen-recording movies on MoST appeared to flatten the (steep) learning curve.



Testing re-use of technology (2)Criteria

Is the KB ontological structure reusable and is the tool MoST useful for other processes?

Method(s)Applying the presented technology for other processes, i.e. within the EC-funded project Aquastress (aiming to mitigate water stress problems).

Result(s) Applying the ontological KB structure for the Aquastress project,

did not show any technical problem. MoST could show and use the new KB without a problem.



Conclusion Testing is a continuous process. Peer reviewed scientific output = testing. Applying our technology for other processes in which

multidisciplinary teams have to perform a complex job, shows that the technology has a wide field of application

Public website www.HarmoniQuA.org free software training material etc.

Huub Scholten and Adrie J.M. Beulens Wageningen University - Netherlands

Documents

Transcript of Huub Scholten and Adrie J.M. Beulens Wageningen University - Netherlands