An update on BFG,The Bespoke Framework Generator

Graham Riley (& Rupert Ford, STFC)Coupling Workshop 2013

@ Boulder, Colorado - February 20th-22nd 2013

Overview• Why BFG?• What is BFG? Not a coupler!• What is new?

- New implementation in python, faster Interface to other languages (‘R’ and GAMS in IAMs)

- Component- and program-compliance- Portal and BFG utilities

the power of metadata!- Parallel model support (and Infrastructure API)- Model ‘export’ – step to interoperability...

to couple to existing models using other technologies, such as ESFM or OASIS

• References & Future

Why BFG? Community-based coupled modelling (still)

emerging:• From Centres of expertise…

- in and across science domains (e.g. ESM, IAM, Flood modelling etc.)

• Distributed across institutions• Implied: need for rapid exploration of use of

models; sharing of models etc. Requirement for flexibility to

choose appropriate models

Why BFG? This historic development of science has led to

several coupling technologies (infrastructures/frameworks):• Shared memory, MPI, TDT, MCT, OASIS,

ESMF, CESM, FMS…• (and other infrastructure, e.g. for parallel

models etc.) Requirement for flexibility to

choose appropriate supporting technology• Similar situation for the community which

led to MPI standard?

Why BFG? Choice of computing systems:

• Many options…- (Parallel) laptop/desktop, cluster, cloud,

supercomputers…• Rapid developments in hardware technology

Requirement for flexibility to use available computing systems efficiently

What is BFG? Aims… Isolation of science code (i.e. models)

• Protect science code from changes in supporting software technologies and hardware (as far as possible)

Flexibility in composition and deployment• Allow models to be put together rapidly to create new

coupled models (as science permits!)• Allow the models to be deployed into program units to

make best use of computing resources• Allow the coupled model to use the most appropriate

existing coupling technology (or technologies)

Provide this flexibility through metadata descriptions and code generation (BFG2)

See other benefits of metadata-based approach…

What is BFG2? A metadata description of a coupled model

• Capturing both its static structure and run-time, dynamic behaviour.

• The individual models described have to conform to certain, minimal, rules (see next).

A set of tools that use the metadata• BFG2 - generates ‘wrapper’ code that can be

compiled with model code to build an implementation.

- Using a choice of existing (coupling) technologies etc.• Visualise the coupled model (static composition

and dynamic – control schedule – behaviour)• Other utilities – see the BFG portal later…

What is a BFG2-compliant model1. Component compliance (fine-grain

models)• Essentially a subroutine (in Fortran)• Science code communication

- by argument or ‘in-place’ put() and get()• Control ‘outside’ the model

2. Program compliance• A model already in an existing program• Communication by put() and get()• Other BFG calls (e.g. end_of_step())• Possibly embedded control

(Support composition of fine-grain models and aggregation of program-based models

A component-compliant modelmodule atmos use bfg, only : put,get implicit none private public :: init,iteration,finalise contains subroutine init(arg1,arg2,...) end subroutine init subroutine iteration(arg1,arg2,...) call get(data3,tag3) call put(data4,tag4) end subroutine iteration subroutine finalise(arg1,arg2,...) end subroutine finalise end module atmos

• Couping by argument

• Tags generated by BFG

• Couping by in-place put()/get()• The ‘tag’ connects to the metadata!

• Key to BFG

• ‘Standard’ init, run, finalise

10

BFG2 model ‘wrapping’

Container, Control (“Driver” layer), ArgPass Data, ArgPass Wrapper

Model Code(s)

Concurrency

InPlace Calls

Target Coupling Infrastructure

- Existing code/library code- BFG-generated code

Component-compliance

Flexible Deployment

atm

sea ice

chem

ocean

bio-geo

model

program unit/deployment unit

TT

T

T transformer

A program-compliant modelprogram atmos use bfg, only : put,get, bfg_init, bfg_eos, bfg_finalise implicit none call bfg_init() call get(data1,tag1) call put(data2,tag2) do i=1,nts call get(data3,tag3) call work(...) call put(data4,tag4) call bfg_eos() end do call get(data5,tag5) call put(data6,tag6) call bfg_finalise()end program atmos

•Stand alone program

•Only put()/get() in the ‘run’ section

•Calls to tell BFG about control structure

•Generated ‘target’ technology init.

Overview of BFG2 metadata Model(s)

• Name, language• Type

- Science/transformer• Coupling data

- As args: in, out, inout- As put/get

• timestep

Composition• Coupling description

- What connects to what

• Priming data- Which connections

are first started- Controls sequential

or concurrent model execution

Deployment• Mapping of models to ‘threads of control’ and main

programs- to sequence units (“this model runs before that” in a

‘thread’)- to deployment units (-> the sequence units in a main

program)• Schedule

- Iteration control of model execution Nested loop structures allowed

• Target coupling technologyAll together, the metadata captures the run-time behaviour of the coupled model

Think of SPMD model

OutputPythonengineInput

xml

script

coupledxml

Deploymentxml

Compositionxml

model Descriptionxml

xsltTemplates

& xslt

Code andscripts

What is BFG2? - summary

Metadata + generative approach

The Power of Metadata - BFG Portal For examples of the use of metadata

• Prototype, in development• bfg.cs.man.ac.uk/• Demos available on request!

Several utilities exist• Validation• Upload metadata and run BFG• Access to some (small) examples• Visualisation of composition and control/schedule• Model stub generation (for test purposes)• Makefile generation• …

Simple ESM…Composition andSchedule views

Future?

Trend towards a common, shared infrastructure for ESM (and beyond)?• Similar situation for the community which led to MPI?

And/Or – go for Framework interoperability• BFG export is a first exploration• (we have a design for BFG import too)

Use of (and extension of) CIM metadata• Out of METAFOR project• More uses of metadata

- Including support for performance analysis, for example.

BFG as a Domain Specific Language…

Web References

BFG web page – including papers• http://cnc.cs.man.ac.uk/projects/bfg

BFG portal:• http://bfg.cs.man.ac.uk/

A BFG Wiki (from the GENIE project)• https://source.ggy.bris.ac.uk/wiki/GENIE_BFG

IS-ENES,EU project deliverables• D8.3, “Towards Flexible Construction of

ESMs using BFG”, March 2012• D8.5 – “ditto”, final report, March 2013

Thanks to our sponsors…

Towards parallel models (D8.5) Support for parallel models on quasi-uniform

grids Wider than coupling

• Looking at other infrastructure support too- E.g. halo exchange (from MCT and ESMF)

Prototype “manUgen” solution• Mainly targeting MCT

- but some ESMF and OASIS3-MCT (for coupling) tests Developed a prototype Infrastructure API

• Metadata + configuration of run-time system approach

- plus some code generation

Infrastructure API (D8.5) A minimal API for use

by model developers Metadata-driven Target coupling and

halo exchange Support

implementation of API in any of a number of existing technologies

Select ‘best of breed’

Parallel Model Support (D8.5)

Code Generation vs Configuration (D8.5)

Metadata describing

coupling and partition etc.

Per model ModelInfo structure

captures specific requirements – for halo exchange and

coupling etc.

Models use ‘tag’ in calls to

access ModelInfo

Composite models – BFG2 export (D8.3)BFG atmoscomponents

dyn rad conv

Ocean(OASIS)

Ocean(ESMF)

Atmos(OASIS)

Atmos(ESMF)

Export to ESMFExport to OASIS

Coupling via OASIS

Coupling via ESMF

Existing Oceanmodels

ERMITAGE use case

GENIE‘fast’ ESMmodel (oremulator)

LPJmL – veg./hydrology model

MAgPIE – land usemodel

TIAM/REMIND – enegy/economic models

An IAM example

‘Import’ of models

Aim: to enable models written to use different existing coupling technologies to talk to each other• Using a selected communication

mechanism…

BFG import

Ocean(OASIS)

Coupling via OASIS

BFG

Adaptor

Coupling via BFG-generated

‘target’

?Atmos(ESMF)

Coupling via BFG-generated

ESMF

Atmos(ESMF)

BFG

Adaptor

Coupling via BFG-generatedTARGET

Target could be: (almost) anything BFG will support…… MPI, GridMPI, OASIS3/4, ESMF, web services…

BFG-generated ‘adaptor’ code

E.g. web services

Import both models

BFG import…

Ocean(OASIS)

Coupling via OASIS

BFG

Adaptor

Coupling via BFG-generated

‘target’

Coupling via BFG-generated

‘target’

Atmos(BFG-gen)

Summary Working with real ESM codes takes a lot of effort

• Experience with GENIE and JULES Using ‘toy’ models to demonstrate capability

and potential Motivation from current ESMs is small

• Few models (but growing number)• Large cost of change

Seeking use cases for export and import• Examples of community collaboration (c.f. NEMO)• Case for import made in ERMITAGE project

An emerging infrastructure requiring long term support for the community