K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Chapter 5 Cloud-Enabling Geospatial Applications

Kai Liu, Qunying Huang, and Jizhe Xia

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Learning Objectives

•Summarize the common components for geospatial applications•Summarize the general steps to deploy cloud-enabled geospatial applications, and study the detailed steps using two use cases

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Learning Materials

•Videos:o Chapter_5-Video_1.movo Chapter_5_Video_2.mp4

•Scripts, Files and others:o hpc.zip

3

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Learning Modules

1. Common Components for Geospatial Applications• Server-side scripting• Database• HPC

2. General Steps to Deploy Cloud-Enable Geospatial Applications

3. Use Cases• Database-driven web applications• Typical HPC Applications

4. Conclusion and discussions

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Common Components for Geospatial Applications:Server-side scripting

• Server-side scripting

• Executed on the server

• Popular Server-side scripting languages

• ASP

• PHP

• JSP

• Perl and Ruby

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

• Spatial database

Database with spatial plug-ins or data types to enable spatial data

analysis, e.g.

• PostGIS plug-in of PostgreSQL

• Oracle Spatial

• Spatial engine of Microsoft SQL server

Common Components for Geospatial Applications:Spatial database

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

• HPC

• High Performance Computing (HPC)

• Parallel Computing is a popular way for HPC. There are two task

decomposition methods:

• Domain decomposition

• Functional decomposition

• HPC open source solutions, e.g., • Condor

• MPICH2

• Hadoop Mapreduce

Common Components for Geospatial Applications:HPC

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Learning Modules

1. Common Components for Geospatial Applications• Server-side scripting• Database• HPC

2. General Steps to Deploy Cloud-Enable Geospatial Applications

3. Use Cases• Database-driven web applications• Typical HPC Applications

4. Conclusion and discussions

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

General Steps to Deploy Cloud-enabled Geospatial Applications

• Set up environments• Software and Libraries (e.g., HTTP server, database server and JRE)• Environment variables (e.g., JAVA_PATH)

• Deploy application• Customization of the application (e.g., database, storage services,

email services and log services)

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Learning Modules

1. Common Components for Geospatial Applications• Server-side scripting• Database• HPC

2. General Steps to Deploy Cloud-Enable Geospatial Applications

3. Use Cases• Database-driven web applications• Typical HPC Applications

4. Conclusion and discussions

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Database-driven web application

The procedure of deploying Drupal site onto EC2

2. Launch an instance with Ubuntu 12.04

3. Login to the instance

4. Set up environments, e.g., Apache HTTP, DBMSs.

7. Customization of the application

8. Create a new AMI from the running instance

5.Transfer Drupal files onto instance

6. Deploy Drupal site

1. Authorize network access

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Step 1, 2 & 3

• Port 22 for SSH and Port 80 for HTTP should be opened for login and

HTTP access

• Drupal is supported by most Linux versions but Ubuntu is highly

recommended by the Drupal community

• Linux and Mac users could use the SSH command in a terminal to

login to the instance, while Windows users could use Putty

2. Launch an instance with Ubuntu 12.04

3. Login to the instance

4. Set up environments, e.g., Apache HTTP, DBMSs.

7. Customization of the application

8. Create a new AMI from the running instance

5.Transfer Drupal files onto instance

6. Deploy Drupal site

1. Authorize network access

Video: Chapter_5-Video_1.mov0:00-4:56

Play Video(1-3)

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Step 4Set up Environments

• Install and configure the Apache HTTP server and the MySQL DBMS. The following commands can be used in the console:

$: sudo apt-get update$: sudo apt-get install apache2$: sudo apt-get install mysql-server mysql-client$: sudo tasksel install lamp-server

$: sudo mysql_secure_installation

• Secure MySQL Installation

2. Launch an instance with Ubuntu 12.04

3. Login to the instance

4. Set up environments, e.g., Apache HTTP, DBMSs.

7. Customization of the application

8. Create a new AMI from the running instance

5.Transfer Drupal files onto instance

6. Deploy Drupal site

1. Authorize network access

Video: Chapter_5-Video_1.mov4:56-10:20

Play Video(4)

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

• Enable Apache Rewrite Module

$: sudo a2enmod rewrite

$: sudo mysqladmin -u root -p create ec2drupal$: mysql -u root -pmysql> create database drupal;mysql> grant all privileges on drupal.* to ec2drupal@localhost identified by

'your_password';mysql> flush privileges;mysql> \q

• Create a MySQL Account

Step 4 Set up Environments(cont’d)

2. Launch an instance with Ubuntu 12.04

3. Login to the instance

4. Set up environments, e.g., Apache HTTP, DBMSs.

7. Customization of the application

8. Create a new AMI from the running instance

5.Transfer Drupal files onto instance

6. Deploy Drupal site

1. Authorize network access

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Step 5 Transfer files onto instance

• wget is supported by most Linux systems to download files from the web.

$: wget http://ftp.drupal.org/files/projects/drupal-7.19.zip

2. Launch an instance with Ubuntu 12.04

3. Login to the instance

4. Set up environments, e.g., Apache HTTP, DBMSs.

7. Customization of the application

8. Create a new AMI from the running instance

5.Transfer Drupal files onto instance

6. Deploy Drupal site

1. Authorize network access

Video: Chapter_5-Video_1.mov10:20-11:06

Play Video(5)

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Step 6 Deploy the application

• Install unzip utility and extract the Drupal files• Deploy the application

$: sudo apt-get install unzip$: sudo unzip drupal-7.19.zip$: sudo mv drupal-7.19 /var/www/drupal$: sudo chown www-data:www-data /var/www/drupal –R$: sudo service apache2 restart

2. Launch an instance with Ubuntu 12.04

3. Login to the instance

4. Set up environments, e.g., Apache HTTP, DBMSs.

7. Customization of the application

8. Create a new AMI from the running instance

5.Transfer Drupal files onto instance

6. Deploy Drupal site

1. Authorize network access

Video: Chapter_5-Video_1.mov11:06-12:23

Play Video(6)

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Step 7Customization

Customize Drupal’s database and administration management through its interface: http://INSTANCEIP/Drupal/install.php

INSTANCEIP should be replaced with the targeted IP address.

2. Launch an instance with Ubuntu 12.04

3. Login to the instance

4. Set up environments, e.g., Apache HTTP, DBMSs.

7. Customization of the application

8. Create a new AMI from the running instance

5.Transfer Drupal files onto instance

6. Deploy Drupal site

1. Authorize network access

Video: Chapter_5-Video_1.mov12:23-20:40

Play Video(7)

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Step 8Create a new AMI

• The last step is to create a new AMI based on the running instance. Chapter 4, Section 4.3.1 describes the creation of an AMI. The image can be reused to create multiple virtual instances.

2. Launch an instance with Ubuntu 12.04

3. Login to the instance

4. Set up environments, e.g., Apache HTTP, DBMSs.

7. Customization of the application

8. Create a new AMI from the running instance

5.Transfer Drupal files onto instance

6. Deploy Drupal site

1. Authorize network access

Video: Chapter_5-Video_1.mov20:40-end

Play Video(8)

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Learning Modules

1. Common Components for Geospatial Applications• Server-side scripting• Database• HPC

2. General Steps to Deploy Cloud-Enable Geospatial Applications

3. Use Cases• Database-driven web applications• Typical HPC Applications

4. Conclusion and discussions

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

The process of configuring an HPC system to run DEM interpolation on EC2 (Blue color indicates the additional steps for

configuring a virtual HPC environment)

HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example

2. Launch a cluster instance as the head node

1. Authorize network access

6. Configure the middleware on both nodes to enable communication

5. Start another instance from the new AMI as a computing node

3.Install the middleware packages, e.g. Condor

4. Create a new AMI from the running instance

7. Transfer the DEM data and interpolation code to the head node

8. Run the DEM interpolation

Video: Chapter_5-Video_2.mp4

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

• Port 22 for SSH and the ports for the communication managed by middleware between master node and computing nodes (e.g., 9000-9999 in this case) should be opened.

• EC2 provides cluster instances for running HPC applications. Users can also select High-CPU Instances or High Memory Cluster Instances depending on whether the geospatial application is data- or computing-intensive

HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example

Step 1 : Authorize network access

Video: Chapter_5-Video_2.mp40:00 – 1: 10

Play Video(1)

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Select instance type based on the CPU, memory, and networking

requirements of the geospatial HPC application

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example

Step 2 : Lunch an instance

Step 3. Install the middleware packagesIn this example, Condor is used as the middleware solution

$ rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm ## install additional packages

$ yum install yum-plugin-priorities $ rpm -Uvh http://repo.grid.iu.edu/osg-el6-release-latest.rpm

$ yum install condor$ touch /etc/condor/condor_config.local ## Create Condor configuration file

Video: Chapter_5-Video_2.mp41:10 – 8:46

Play Video(2-3)

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Add the following content to the configuration file

“/etc/condor/config.d/local.conf”## OSG cluster configuration# List of daemons on the node (Condor central manager requires collector and

negotiator, # schedd required to submit jobs, startd to run jobs)DAEMON_LIST = MASTER, COLLECTOR, NEGOTIATOR

Start the condor service$ service condor start ## Start the service$ service condor stop ## Stop the service

HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Condor Commands

Command Description Usage Example

condor_submit Submit a jobcondor_submit [submit file] $ condor_submit submit.file

condor_qShow status of jobs

condor_q [cluster]

$ condor_q 1170

condor_rmRemove jobs from the queue

condor_rm [cluster]

$ condor_rm 1170

condor_statusShow the status of resources

condor_status $ condor_status -all

HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Condor needs Java Development Kit (JDK)

[root@domU-12-31-39-13-DD-FF ~]# chmod a+x jdk-6u14-linux-x64.bin [root@domU-12-31-39-13-DD-FF ~]# ./jdk-6u14-linux-x64.bin

HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

This step is to store the state of configuring the HPC environment in case the master node crashes. In addition, new computing node instances can be launched directly from this AMI, and all software dependencies will already have been installed and configured.

HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example

Step 4. Create a new AMI from the running instance

This step is to launch other instances from the new AMI as a computing node as indicated in step 2 (except with a different AMI)

Step 5. Start another instance

Video: Chapter_5-Video_2.mp48:46 – 10:56 Play Video(4-5)

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Step 6. Configure the middleware on all nodes to enable communication

On the computing node, the configuration file

“/etc/condor/config.d/local.conf” should be changed to:DAEMON_LIST = MASTER, SCHEDD, STARTD

HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example

Step 7. Transfer the DEM data and interpolation code Use scp command Video: Chapter_5-Video_2.mp4

10:56 – 18:00Play Video(6-7)

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Step 8. Run the DEM interpolationAn example of the submission file:Universe = javaExecutable = interpolate.classArguments = interpolate DEMfile.txt ## interpolate is the java main program, and DEMfile.txt is the inputinitialdir = dir.$(Process) ## input directoryoutput =../interpolate.output.$(Process) ## output fileerror = interpolate.error.$(Process)log = ../interpolate.logrequirements = (Memory > 1024) # Select machine with memory size bigger than 1024Mbtransfer_input_files = MyPoint.class, PngWriter.class,interpolate.class, cutfile.txtshould_transfer_files = ALWAYSwhen_to_transfer_output = ON_EXITqueue 12 ## concurrent process numbers

MyPoint.class, PngWriter.class, and interpolate.class are the Java

programs, and DEMfile.txt is the input for each task.

HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Step 8. Run the DEM interpolation

HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example

The command “condor_submit” can be used to submit the tasks to the

cluster using a submission file named “interpolate_submit” as follows:

[root@domU-12-31-39-13-DD-FF ~]# su condor # use the condor account[root@domU-12-31-39-13-DD-FF ~]# condor_submit interpolate_submit

Video: Chapter_5-Video_2.mp4

18:00 – end Play Video(8)

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Learning Modules

1. Common Components for Geospatial Applications• Server-side scripting• Database• HPC

2. General Steps to Deploy Cloud-Enable Geospatial Applications

3. Use Cases• Database-driven web applications• Typical HPC Applications

4. Conclusion and discussions

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

Conclusions and Discussions

1. How to select the optimal cloud services and configurations? 2. Enumerate some other common components for geospatial applications

besides server-side scripting, database and HPC.3. What are the general steps to deploy a geospatial application onto cloud

services?4. What is the database service provided by Amazon AWS? How to use it?5. What is the database service provided by Windows Azure? How to use it? 6. Can you list some other geospatial applications?

K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.

References

• Clarke, K. C. 2003. Geocomputation’s future at the extremes: High performance computing and nanoclients. Parallel Computing29, no. 10: 1281–1295.

• Huang, Q. and C. Yang. 2011. Optimizing grid computing configuration and scheduling for geoscience analysis—An example with interpolating DEM. Computers & Geosciences37, no. 2:165–176.

• Mower, J. E. 1996. Developing parallel procedures for line simplification. International Journal of Geographical Information Systems10, no. 6: 699–712.

• Xie, J., C. Yang, B. Zhou and Q. Huang. 2010. High performance computing for the simulation of dust storms. Computers, Environment and Urban Systems34, no. 4: 278–290.