COL: data acquisition, storage and preservation

Roger Brugger.brugge@reading.ac.uk

www.met.reading.ac.uk/~brugge

Climatological Observers Link

Department of MeteorologyUniversity of Reading

Transmission, presentation and archiving of meteorological data, 5 Oct 2011

Outline

• What is COL?– What the data are used for

• Data gathering– Daily versus monthly data– Old and new methods

• Data quality – station gradings• Data storage

– What is stored?– How is it stored?

• Data preservation– For use by COL– For use by others

What is COL?• June 1950 - An advert in Weather in June 1950 by Tom Suttie

– enquiring about setting up an organisation for the exchange of weather data by amateurs

• 1969 - A chance sighting of this advert by Eric Bonsor– there followed an exchange of letters, and

• January 1970 - first edition of the Directory of Amateur Observers published– Site details, instrumentation and observing routines

• June 1970 - first issue of the Bulletin– 48 stations reporting, describing the weather of May 1970.– Nowadays more than 330 sites each month

• Aim of COL - to publish the monthly bulletin at the earliest opportunity– Achieved for over 41 years now– The COL Bulletin is now a rare source of comprehensive UK monthly

weather summaries.– Membership details added to the Directory of Amateur Observers– Almost 1300 stations are now included, together with photographs of

stations

Data gathering

• Monthly data only– A few exceptions for a daily data table– COL cannot handle the volume of daily data that would

otherwise accrue• Paper

– The original method (15% of returns)• Email

– Also used as a reminding mechanism to most members (15%)• Web (70% of returns)

– Underpinned by a database– PHP5 MyAdmin and SQL >archiving– Provides members with an early snapshot– The way of the future?

Paper form – main data features only shown

Data gathering

• Monthly data only– A few exceptions for a daily data table– The Bulletin

• Paper– The original method (15% of returns)

• Email– Also used as a reminding mechanism to most members (15%)

• Web (70% of returns)– Underpinned by a database– PHP5 MyAdmin and SQL > archiving– Provides members with an early snapshot– The way of the future?

Email reply form – the start of a message for August 2011

Data gathering

• Monthly data only– A few exceptions for a daily data table– The Bulletin

• Paper– The original method (15% of returns)

• Email– Also used as a reminding mechanism to most members (15%)

• Web (70% of returns)– Underpinned by a database– PHP5 MyAdmin and SQL > archiving– Provides members with an early snapshot– The way of the future?

Data quality

• COL welcomes observations from everyone– Multiple stations per town– Wide variety of instrumentation and exposure

– Airports, universities, institutions and the hobbyist

• Instrumentation– Originally based around Stevenson screen, mercury-in-glass

thermometers and Snowdon raingauges– Many good quality sites– Increasing use being made of AWS equipment

• Varying costs, and also performance

• Need a way to distinguish between stations that (even in the same locality) might report widely differing obs– > COL station grading system (Stephen Burt)

Buxton, Norfolk

Stratfield Mortimer

Elderslie

Tivington

Red Wharf Bay, Anglesey

Compton Martin

Data quality

• COL welcomes observations from everyone– Multiple stations per town– Wide variety of instrumentation and exposure

– Airports, universities, institutions and the hobbyist

• Instrumentation– Originally based around Stevenson screen, mercury-in-glass

thermometers and Snowdon raingauges– Increasing use being made of AWS equipment

• Varying costs, and also performance

• Need a way to distinguish between stations that (even in the same locality) might report widely differing obs– > COL station grading system (Stephen Burt)

Station grading system(as used in 2011)

Grading system development over the years has been led by Stephen Burt

COL station grades

A27%

B17%

C233%

U15%

C18%

Incl. SYNOP and CLIMAT sites

• Station grade + photograph + station details = a reasonable idea of data quality

Data storage

• What is stored?• Monthly summaries

– A subset of the data; coverage 1940s onwards• Column-aligned text files• Easily manipulated by Excel and Fortran, etc.• Used to make quick calculations and climatological values

– e.g. recently published 1981-2010 averages

• All monthly data, numbers and text– Database; coverage 2006 onwards

• Generated by SQL/webpage• Little-used

• All back issues of the bulletin available electronically – 40th anniversary CD

Members observations – where do/could they go?

• Lots of high-quality COL observations (especially grade A stations)• Effectively unused or un-archived simply because they are not reporting to

UKMO• Data from these sites are likely to be lost unless special arrangements are

made. Currently paper copies– traditionally,

• sometimes saved in local history libraries• donated to an observer living close by• sent to the local rubbish tip (the majority)

– now: they can be deposited with the Chiltern Observatory Trust• Electronic data

– Monthly – some/all retained by COL– Daily – ‘delete’ key– Either way – file formats soon become outdated

• A proper remote archiving/backup facility is preferable.• Might BADC have a role here?

• Two suggested solutions…

Preserving members observations – Possible solution (1)

• 2011: Many amateur sites are fully/partial AWS • Might consumer-AWS instrument manufacturers agree a

common ‘archive file’ export standard with BADC?– to simplify the archiving process– ensure amateur data is not lost when a disk crashes or the

observer dies.

• If this was a file that could be exported then uploaded to a BADC site, say annually, it would take only minutes.

• A mandatory metadata section should be required.• Might Davis…

– who are the market leaders– who currently provide an export file format for NOAA

• …take a lead on this?

Preserving members observations – Possible solution (2)

• BADC to propose a standard Excel spreadsheet-type template for amateur observers to use.– This would greatly simplify the transmission and storage of

amateur observations to BADC.– A mandatory metadata section should be required, perhaps

including site plan/s and photographs from cardinal points.– Perhaps pilot

• using the existing ‘COL archive’ template materials (hourly, daily, monthly templates have been created)

• and with a voluntary participation of a handful of A-grade COL stations in 2012?

• Issues:– There are as many observer-owned spreadsheet formats as

observers– However, given a lead by BADC, COL members might be

persuaded to modify or convert their formats

More information• This presentation –

– To be placed at www.met.reading.ac.uk/~brugge• Website

– www.met.reading.ac.uk/~brugge/col.html• Weather article

– Brugge, R., 2010. Forty years of the Climatological Observers Link. Weather, 65, 139-143.

• Email the author– r.brugge@reading.ac.uk

• Bulletin – some copies available today• ‘Processed data’: an example

– Burt, S.D. and R. Brugge, 2011. Climatological Averages for 1981-2010 and 2001-2010 for stations appearing in the monthly bulletin of the Climatological Observers Link. ISBN 9780956948502.