Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... ·...

23
Big Data for a Big Country: The Australian Geoscience Data Cube Dr. Stuart Minchin, Geoscience Australia

Transcript of Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... ·...

Page 1: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Big Data for a Big Country: The Australian Geoscience Data Cube

Dr. Stuart Minchin, Geoscience Australia

Page 2: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Insert document footer here

Page 3: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Traditional remote sensing process

Page 4: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Developing the Australian Geoscience Data Cube

• Earth Observation data are

calibrated to surface reflectance

observations

• Calibration of data to a single

uniform standard allows the

comparison of data through time

• In the AGDC every unique

observation is kept and included

for analysis creating very dense

time-series

Page 5: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.
Page 6: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Surface Water Menindee lakes

40 billion observations

15 minutes processing time

Page 7: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

• 27 Years of data

• 25m N ominal Pixel Resolution

• Approx. 300 000 individual scenes

in approx. 20 000 passes

• Entire 27 years of 1 312 087

ARG25 tiles => 93x1012 pixels

visited

• 0.75 PB of data

• 3 hrs at NCI

Continental Scale Water Observations from Space

Page 8: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Cubbie Station: floodplain water storages

Page 9: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Cubbie Station: floodplain water storages

Page 10: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Cubbie Station: floodplain water storages

Page 11: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Cubbie Station: floodplain water storages

Page 12: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Coastal Change Detection

Aquatic Applications on the AGDC

1988 Landsat 5 First Water Observation Anomaly 2013 Landsat 7 Last Water Observation Anomaly

Page 13: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Coastal Change Detection

Aquatic Applications on the AGDC

1988 Landsat 5 First Water Observation Anomaly 2013 Landsat 7 Last Water Observation Anomaly

Page 14: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Coastal Change Detection

Aquatic Applications on the AGDC

1988 Landsat 5 First Water Observation Anomaly 2013 Landsat 7 Last Water Observation Anomaly

Page 15: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Coastal Change Detection

Aquatic Applications on the AGDC

1988 Landsat 5 First Water Observation Anomaly 2013 Landsat 7 Last Water Observation Anomaly

Page 16: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Mangrove change detection:

Timing and

strength of

change

for any (data

cube) variable

National &

regional shoreline

erosion studies,

mangrove

change mapping

A B

Page 17: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Multiple Algorithms: Time series data extraction

Page 18: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Tracking agricultural change

Page 19: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Water quality monitoring: Lake Burley Griffin

Page 20: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Tracking development conditions

Page 21: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Some prototyped applications for the data cube:

• Vegetation change, agricultural production

• Flood inundation mapping, farm dam development

• Wetland management and characterisation

• Carbon accounting

• Seagrass and substrate mapping

• Coastal change and water quality

• Shallow water bathymetry

• Mining footprint and urban development

• Bushfire scar mapping and forestry inventory

Page 22: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Big Data for a Big Planet: a global network of regional data

cubes?

Data Cubes for: Africa, Antarctica, South

America, Europe, North

America, …

Connecting the EO, Spatial and Statistical

world to support global SDGs?

Page 23: Big Data for a Big Country - UN-GGIMggim.un.org/meetings/2016-4th_HLF/documents/1-Dr Minchin... · 2017-10-16 · •27 Years of data • 25m N ominal Pixel Resolution • Approx.

Thankyou

Open source code

Built with open data

Built with open standards