Andrew ZolnaiCambridge UKwww.zolnai.ca

Shipping and weather data CLIWOC ships captains logs 1662-1855Data points for climate research

Image: Royal Museum Greenwich

Climatological data

• Ships captains logs data from 1662 to 1855• Ships and routing details and location• Weather recording of over 120 parameters• Release 1.x (2004) and Release 2.x (2007)

Shipping data• First added to Esri file geodatabse – The complete (weather and location) dataset has

almost 290,000 points, and is almost 250Mb database– The locations were extracted into a geodatabase (70

Mb), then into a compressed file geodatabase (10 Mb)• Posted for ArcGIS Explorer upload on

original website 1750 – 1850 data from Release 1.5– George Dailey further developed this for education– and posted it on ESRI's Education Blog

• And map package shares Release 2.0– layer files showing them in 25-year time-slices– and for Captains Cook and de la Perouse

Time-enable shipping

• Daniel Schober reposted this as map package– Here layers are time-enabled through the field "Time"

which is the UTC time in the format YYYYMMDD• Re-reposted here and as a YouTube video– data are no longer cast into arbitrarily time-sliced feature-

classes– Time Slider allows to step through all 252,917 points from

1662 to 1855– it beautifully depicts the various colonial activities driven

by, for example, spice trade in the East and West Indies for the British, Dutch, French and Spanish

– in other words Time Slider greatly facilitates the investigation of this large data set

Static vs. dynamic web maps• Amazon Web Service ‘small' elastic service• Mapcentia Geocloud2 stack• Leaflet frontend w Stamen toner, postGIS backend• unlimited data, no performance hits• CLIWOC all nations by year (1750-1775)

• Use time stamps to aggregate layers• Posted as ArcGIS Online map service• But stock server has performance issues• So Filter data by nationality to limit fetches

Dynamic maps II

• Workflow– of real interest was to compare nations shipping over

time, rather then animation per se– thus create a pair of layers to step through

• Context– add Rumsey 1812 map for backdrop from ArcGIS Online– add current wind data from NOAA METAR with

continents mask to show offshore wind patterns• Posted as CLIWOC Compare – move slider to 1802-1812 to view with Rumsey backdrop– turn on Continent Mask + NOAA METAR for current wind

Restore climate data

• CLIWOC posted all shipboard data it had – from British, Dutch, French and Spanish sources– with look-up tables for each different source

• Merged wind force and direction data– table joins with lookup tables gives direction and

speed data consistent numeric values, but…– wind (class, Beaufort and speed) keeps 277,581 points– wind direction however boosts count to 471,214

points– and joining wind direction and force to graduate and

orient the symbols drops the count slightly to 431,651

Lessons learned

• Join and aggregate data in a lossless manner• Use File Geodatabases to allow mega datasets• Filter data by parameters of interest:– nationalities in pairs to compare– step-thru animation to see evolution

• Share on-line to foster re-use & collaboration:– feature template CLIWOCwinds– web map CLIWOC Wind Force+Direction– and layer package WindDutch1812

Take-aways

• Very rich dataset from 160 - 350 years ago

• Lots of data strung out along ship tracks

• Navies had standards • But issues w data

recording and capture

• Compare Old Weather NOAA ICOADS, &c.

• Many visualisations • Few share &/or re-use

• Foster online collaboration

• Further climate research