Download - Fast, Distributed Geoprocessing with Scala, Spark and GeoTrellis

@azavea

@rcheetham

21st Century Geoprocessing

with Scala and GeoTrellis

Robert Cheetham

[email protected]

B Corporation

• Civic/Social impact

• Donate share of profits

Research-Driven

• 10% Research Program

• Academic Collaborations

• Open Source

• Open Data

Use geodata to

do stuff that matters

Land

Water

People

Ian McHarg

Dana Tomlin

Idrisi

advanced

spatial analysis

on the web

3 Challenges

1. Performance & Scalability

Big Data – Cities

2. Large Data Sets – Digital City

2. Large Data Sets – Social Media

2. Large Data Sets - Science

3. User Interface

We can do better

• IO

• Geoprocessing Operations

• Distributed Processing

• Web Services

Real-time Processing

6183 x 4992 4598 x 4867

118 MB 86 MB

Cluster-style Processing

1770271 x 910139

5.8 TB

How does it work

On the shoulders of giants

LocationTech Community

Some changes coming

• Parallel operations across tiles

• Parallel execution of operations

• Basic cluster capabilities with

GeoTrellis v0.9:

+

• Sharding raster data across the cluster

• Caching operation results across cluster

• HDFS support

• Advanced Fault tolerance

• Advanced Scheduling

• ...

What's missing?

+

• Caches results in memory

• Ideal for iterative algorithms

• Significantly outperforms Hadoop

• Uses Hadoop's file system (HDFS)

What becomes possible?

Urban Forests

Simulation Modeling

Sea Level Rise

Business Siting

Streaming Data

Counting Carbon

Digital Humanities

GeoTrellis Transit

Travelsheds

Crime Analysis and Forecasting

It’s the second Monday in October

and school is in session. There were 2

burglaries and 3 assaults yesterday.

The Maple Leafs are not playing this

evening. Six bars, three take-out

stores, and a high school are in the

neighborhood. The forecast is 9°C

with a 50% chance of rain this evening.

Where do you focus your 3 vehicles?

It’s the second Monday in October

and school is in session. There were 2

burglaries and 3 assaults yesterday.

The Maples Leafs are not playing this

evening. Six bars, three take-out stores,

and a high school are in the

neighborhood. The forecast is 9°C

with 50% chance of rain.

Where do you focus your 3 vehicles?

Data Science + Geography

Faster is different…

Educational Games

New Devices and Displays

I am very excited

advanced

spatial analysis

on the web

Land

Water

People

Simulation

Modeling

Forecasting

• Multi-band

• Temporal bands (climate)

• More operations

• Tile indexes

• GeoMesa collab.

• Simpler setup

• More integration points

What’s next?

GeoTrellis.io

Get Involved

Get Involved

[email protected]

Get Involved

IRC: #geotrellis on freenode

Use geodata to

do stuff that matters

jobs.azavea.com

[email protected]

@rcheetham

[is hiring]