"Complementing Snow Cover Estimation with Analysis of Web Public Images"
-
Upload
pasinipolimi -
Category
Software
-
view
111 -
download
0
Transcript of "Complementing Snow Cover Estimation with Analysis of Web Public Images"
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
SnowWatch
Complementing Snow Cover Estimation with Analysis of Web Public Images
Roman Fedorov: [email protected]
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Introduction
Complementing Snow Cover Estimation with Analysis of Web Public Images 2
• Challenge: transform public images depicting mountain slopes into virtual snow measurement stations.
• Measurements complement existing ground and satellite sensor data
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
UGC
3
Media User Generated Content (UGC) publicly available on the Web is reaching unprecedented size nowadays:• Facebook ~ 350 MM daily photo uploads• Flickr ~ 8.000 MM photographs with 3.5 MM daily
uploads• Panoramio reached ~ 75 MM geo-tagged photograph
It contains an enormous amount of latent implicit knowledge, the challenge is to extract it.
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Project
4
SnowWatch aims at:• improving snow analysis and water management
through the acquisition and analysis of public media content depicting alpine mountains
• Creating alpine snow media dataset with open contribution of new media and access to processed media and its results for any environmental researchers
The project aims to design, implement and test a new approach to homeland security that combines the traditional measures of the Public Administration with the use of social networks as tools for monitoring and warning the population.
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Content Providers
SnowWatch Engine
SnowWatch Architecture
User Photos
5
Flickr Photos
Content Consumers
Webcams
API Media Storage
Get Image Orientation, Identify Mountain Peaks
Get Image Snow Pixel-level classification
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Flickr Photographs Crawling
6
Does an image contain a clear a mountain slope view?
- Heavily depends on thephotograph shooting point altitude
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Flickr Crawling Implementation
7
SVM Image Content-based Classifier:� Bag of Visual Words / Fisher Vector� SIFT/SSIM/GIST/HOG2x2 Descriptors� Spatial Histogram, Vertical Bands
Manually annotated 1658 samples (~70%) for training and validation and 710 samples (~30%) for testing.
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Webcam Images Aggregation
8
Identified ~2.000 webcams depicting mountains in Alpine regions.Each webcam produces an image every 1’ – 15’.• 67% of the images are not suitable due to bad weather• More than 50.000 images and 6 GB of data acquired dailyThrough the extraction of the edges we can compute the percentage of the skyline clearly visible, and discard all images with visibility lower than a certain threshold, obtaining the following ROC curve varying the threshold:
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Webcam Images Aggregation
9
Remaining small moving obstacles (such as low clouds, persons, vehicles) and varying illumination conditions are managed through the median aggregation of all daily images
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Peak Identification
10
An orientation of a geo-tagged photograph is needed.
Given a Digital Elevation Model (a 3D terrain model) we can simulate the terrain silhouettes that should be seen from the photographer point of view.
Steps� Panorama� Scale� Edges� Filter� Match
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Panorama
11
Steps� Panorama� Scale� Edges� Filter� Match
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Scaling
12
Photo must be scaled properly, so the mountains have the same pixel dimension on the photograph and on the panorama.This can be obtained by calculating the Field Of View (FOV) of the photo:
Steps� Panorama� Scale� Edges� Filter� Match
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Edge Extraction
13
Steps� Panorama� Scale� Edges� Filter� Match
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Edge Filtering
14
Too many edges on the image that do not belong to a terrain contour!• Skyline detection algorithm is employed, and all the edge
pixels above the skyline are removed, being considered obstacles or clouds.
• Weighting mechanism is applied, which assigns decreasing weights to the edge pixels as the distance from the skyline increases.
Steps� Panorama� Scale� Edges� Filter� Match
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Alignment
15
Find the best overlap maximizing the vector cross correlation score:
Steps� Panorama� Scale� Edges� Filter� Match
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Peak Alignment
16
A global overlap between the photo and the panorama is often not enough, a neighborhood vector cross correlation adjustment is applied to each identified peak:
Steps� Panorama� Scale� Edges� Filter� Match� Local Match
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Peak Identification
17
Fraction of photographs with orientation correctly identified
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Snow Pixels Classification
18
Classifying each pixel as snow covered/terrain is much more than simply checking bright/dark pixels:
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Snow Pixels Classification
19
SnowGround GroundGroundSnow Snow Snow
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Snow Pixels Classification
20
Random Forest Classifier that considers pixel and pixel neighbors color, global and local intensity of the image.Trained on 59 manuallylabeled images(~ 7 Million pixels),outperform currentmethods:
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Environmental Impact Evaluation
21
Environmental consistency evaluation of extracted snow cover indexes.� No direct ground truth, how do we
evaluate it?� Practical utility w.r.t. an environmental
modelUse case: Como Lake water managementFlooding vs Irrigation trend, what is the best solution?
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
SnowWatch 2°°°° year
22
If the physical snow estimations provided by ARPA (Italian EPA) are not available, Adding Virtual Snow Measures improves the performance by ~100%!
Complementing Snow Cover Estimation with Analysis of Web Public Images
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
SnowWatch 2°°°° year
Roman Fedorov: [email protected] 23
With some irrigation <-> flooding target solutions, using SnowWatch virtual measurements is even better than using ARPA snow estimation!
DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Current and Future Work
24
Current:• MountainWatch: real time augmented reality mobile
application• Expand tests on environmental impact
Future:• Expand area of interest and content providers• Orthophotos and new snow indexes
SnowWatch: Estimating Snow Cover from Publicly Available Images