K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Chapter 8 Cloud-Enabling GEOSS Clearinghouse

Kai Liu, Douglas Nebert, Qunying Huang, Jizhe Xia, and Zhenlong Li

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Learning Objectives

•Study the GEOSS Clearinghouse background and challenges

•Study how to deploy and optimize the GEOSS Clearinghouse onto

cloud services

•Get familiar with GEOSS Clearinghouse

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Learning Materials

•Videos: o Chapter_8-Video_1.mp4o Chapter_8-Video_2.mp4o Chapter_8-Video_3.mp4o Chapter_8-Video_4.mp4o Chapter_8-Video_5.mp4o Chapter_8-Video_6.mp4

•Scripts, Files and others:o geonetautoscaling.jasono geonetwork-2013-08-20.dump.tar.gzo geonetwork-2013-08-20.tar.gzo geoss-2013-08-20.tar.gz

3

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Learning Modules

1. GEOSS Clearinghouse: background and challenges

2. Deployment and optimization

• General steps

• Special considerations

3. System demonstrations case

• Local Search

• Remote Search

4. Conclusion and discussions

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

GEOSS Clearinghouse: Background

• GEOSS

• Stands for Global Earth Observation System of Systems

• Supports different Societal Benefit Areas (SBAs) including Agriculture,

Biodiversity, Climate, Disasters, Ecosystems, Energy, Health, Water,

and Weather

• Three key components: GEOSS Registry, GEOSS Clearinghouse and

Geoportal

• GEOSS Clearinghouse

• Engine of the GEOSS Common Infrastructure (GCI)

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

GEOSS Clearinghouse: Challenges

• Big data: three “V”s

• Volume: Harvest metadata from various catalogs

• Velocity: Frequent updating

• Variety: Various metadata standards (FGDC CSDGM Metadata

Standards, Dublin core and ISO-19139) and web protocols

(CSW, SRU, RSS, WAF etc.)

• Spatiotemporal search and full text search

• Concurrent access

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Learning Modules

1. GEOSS Clearinghouse: background and challenges

2. Deployment and optimization

• General steps

• Special considerations

3. System demonstrations case

• Local Search

• Remote Search

4. Conclusion and discussions

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

General deployment workflow

The process of deploying GEOSS Clearinghouse onto Amazon EC2 (boxes

with blue color indicate the steps required special considerations)

8. Restore the GEOSS Clearinghousedatabase

7. Transfer the GEOSS Clearinghouse codes/data into the instance

9. Configure servlet for GEOSS Clearinghouse

10. Configure the load balance,scalability

11. Start the service and run test2. Launch an instance

3. Create an EBS volume

1. Authorize network access

4. Attach the EBS volume to the instance

5. Install package (e.g.,Postgresql,PostGis,Tomcat)

6. Mount the EBS Volume

12. Create a new AMI from the running instance

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Step 1 &2

• Step 1: Authorize network access

• port 22

• port 80

• Step 2: Launch Instance using Public AMI with PostgreSQL and

PostGIS by seaching “PostgreSQL 8.4 PostGIS 1.5” in AMI search page

8. Restore the GEOSS Clearinghousedatabase

7. Transfer the GEOSS Clearinghouse codes/data into the instance

9. Configure servlet for GEOSS Clearinghouse

10. Configure the load balance,scalability

11. Start the service and run test2. Launch an instance

3. Create an EBS volume

1. Authorize network access

4. Attach the EBS volume to the instance

5. Install package (e.g.,Postgresql,PostGis,Tomcat)

6. Mount the EBS Volume

12. Create a new AMI from the running instance

Video: Chapter_8-Video_1.mp40:00-3:04 Play Video(1-2)

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Step 3, 4, 5 & 6

Steps to customize the instance.

• Step 3, 4 and 6 are optional,

which make the system more reliable with more storage capacity.

• Step 5: Install packages

• PostgreSQL/PostGIS: (AMI contains the packages; don’t need to

install them again)

• Tomcat servlet (e.g., install tomcat 7.33 to /opt/geoss)

8. Restore the GEOSS Clearinghousedatabase

7. Transfer the GEOSS Clearinghouse codes/data into the instance

9. Configure servlet for GEOSS Clearinghouse

10. Configure the load balance,scalability

11. Start the service and run test2. Launch an instance

3. Create an EBS volume

1. Authorize network access

4. Attach the EBS volume to the instance

5. Install package (e.g.,Postgresql,PostGis,Tomcat)

6. Mount the EBS Volume

12. Create a new AMI from the running instance

Video: Chapter_8-Video_1.mp43:04-5:25

Play Video(5)

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Step 7 & 8

• Step 7: Transfer the CLH code

and data

root@ip-10-189-149-104:/mnt$ chown postgres:postgres geonetwork.dumproot@ip-10-189-149-104:/mnt$ su postgresbash-3.2$ createdb geonetworkbash-3.2$ psql geonetwork < geonetwork.dump

8. Restore the GEOSS Clearinghousedatabase

7. Transfer the GEOSS Clearinghouse codes/data into the instance

9. Configure servlet for GEOSS Clearinghouse

10. Configure the load balance,scalability

11. Start the service and run test2. Launch an instance

3. Create an EBS volume

1. Authorize network access

4. Attach the EBS volume to the instance

5. Install package (e.g.,Postgresql,PostGis,Tomcat)

6. Mount the EBS Volume

12. Create a new AMI from the running instance

• Step 8: Restore the database

Video: Chapter_8-Video_1.mp45:25-end

Video: Chapter_8-Video_2.mp4

Play Video(7)

Play Video(8)

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Step 9: Configure servlet for CLH

• Install jdk and jre on the

instance (e.g., /usr/bin/java)

8. Restore the GEOSS Clearinghousedatabase

7. Transfer the GEOSS Clearinghouse codes/data into the instance

9. Configure servlet for GEOSS Clearinghouse

10. Configure the load balance,scalability

11. Start the service and run test2. Launch an instance

3. Create an EBS volume

1. Authorize network access

4. Attach the EBS volume to the instance

5. Install package (e.g.,Postgresql,PostGis,Tomcat)

6. Mount the EBS Volume

12. Create a new AMI from the running instance

groupadd tomcatuseradd –s /sbin/nologin –g tomcat –d /opt/geoss tomcatpasswd tomcat

• For Security, use virtual user (e.g., tomcat) to run tomcat

iptables -t nat -I PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080

• Redirect port 80 to port 8080 (because Ports below 1024 can be

opened only by root.) Video: Chapter_8-Video_3.mp4 0:00-12:27

Play Video(9)

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

• Add following lines to “/etc/rc.local”: enable CLH start automatically when the system boots up

sudo –u =/opt/geoss/apache-tomcat-7.0.33/bin/startup.sh

• Enable the rules through re-booting by adding the following lines to /etc/network/interfaces

pre-up iptables-restore < /etc/iptables.rulespost-down iptables-save > /etc/iptables.rules

• Add geonetwork services to HOST in tomcat/conf/server.xml<Context path=“/geonetwork”

docBase=“/opt/geoss/apache-tomcat-7.0.33/webapps/geonetwork” crossContext=“false” relodable=“false”>

Step 9: Configure servlet for CLH(Cont’d) Video: Chapter_8-Video_3.mp4

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Step 10, 11 & 12

• Set the URL for remote search

• Load balancing and scalability

Set Remote Server Host and Port

8. Restore the GEOSS Clearinghousedatabase

7. Transfer the GEOSS Clearinghouse codes/data into the instance

9. Configure servlet for GEOSS Clearinghouse

10. Configure the load balance,scalability

11. Start the service and run test2. Launch an instance

3. Create an EBS volume

1. Authorize network access

4. Attach the EBS volume to the instance

5. Install package (e.g.,Postgresql,PostGis,Tomcat)

6. Mount the EBS Volume

12. Create a new AMI from the running instance

Video: Chapter_8-Video_3.mp412:27-end

Play Video(10-12)

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Special Considerations

• Data backup:

• Elastic Block Store (EBS) Volume

• Used to store data, log files and application from the volume in case

the current instance crashes

• Size could vary from 1GB to 1TB

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Create a new EBS volume, and attach to the instance

Step 1: Create an EBS volume from scratch with no content in web

console and make sure the select the EBS volume zone is the same as

the zone of GEOSS clearinghouse instance

Steps for Data Backup Video: Chapter_8-Video_4.mp4

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Step 2: Attach the volume to the running instance

Step 3: Mount the EBS to the file system

[root@ip-10-189-149-104~] mkfs -t ext3 /dev/sdh# make a file system[root@ip-10-189-149-104~] mkdir /mnt/datavol_1[root@ip-10-189-149-104~] mount /dev/sdh /mnt/datavol_1/

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Special Considerations

• Load balancing:

Configure load balance service

Video: Chapter_8-Video_5.mp4

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Special Considerations

• Auto-scaling

Using CloudFormation service to configure auto-scaling

capability through the web console

Video: Chapter_8-Video_6.mp4

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Or, configure auto-scaling through Command line using a template

cfn-create-stack GEOSSClearinghouse --template-file GEOSSClearinghouse --template-file GEOSSClearinghouse.template --region us-east-1 --awsaccesskey=FAKEKEY --awssecretkey=FAKEKEY2 --parameters=“KeyName=GeoNet; InstanceType=m1.large”

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Learning Modules

1. GEOSS Clearinghouse: background and challenges

2. Deployment and optimization

• General steps

• Special considerations

3. System demonstrations case

• Local Search

• Remote Search

4. Conclusion and discussions

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Url: http://ec2-50-19-223-225.compute-1.amazonaws.com/geonetwork

Main Page

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Local Search: search records through CLH interface and visualization the

map services

Search Results of Global "Rain-Use Efficiency"

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Remote Search: search records through CLH remote protocols (CSW,

SRU & RSS)

Search Results of Global "Rain-Use Efficiency" from

GEO Portal

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Learning Modules

1. GEOSS Clearinghouse: background and challenges

2. Deployment and optimization

• General steps

• Special considerations

3. System demonstrations case

• Local Search

• Remote Search

4. Conclusion and discussions

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Advantages for hosting CLH on Cloud

• Technique advantages:

• Scalability

• Highly Reliable Environments

• Economic advantages

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

Discussion Questions

1. What are the general steps of deploying GEOSS Clearinghouse onto the cloud? What are the differences from the general steps in Chapter 5?

2. How to attach and use Amazon EBS volume?3. What kind of cloud services can be used to balance the

system load? Discuss how to use them?4. What scalable services are provided by AWS? How to use

them?5. Using GEOSS Clearinghouse as an example, explain the

technical advantages of cloud-enabled Geoscience applications.

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 123-143.

References

• GEO. 2009–2011 Work Plan [online], 2009. http://www.earthobservations.org/documents/work%20plan/geo_wp0911_rev2_091210.pdf. (Accessed January 4, 2013).

• Goodchild, M. F., M. Yuan, and T. J. Cova. 2007. Towards a general theory of geographic representation in GIS. International Journal of Geographical Information Science21, no. 3:239–260.

• Huang, Q., D. Nebert, C. Yang, and K. Liu. 2011. GeoCloud Project Report—CLH [online]. http://www.fgdc.gov/initiatives/geoplatform/geocloud/reports/fgdcgeocloud-project-report-geonetwork.pdf. (Accessed March 4, 2013).

• Liu, K., C. Yang, W. Li, Z. Li, H. Wu, A. Rezgui, and J. Xia. 2011. The CLH High Performance Search Engine. The 19th International Conference on Geoinformatics, June 24–26, 2011, Shanghai, China.

• Yang, P., J. Evans, M. Cole, N. Alameh, S. Marley, and M. Bambacus. 2007. The emerging concepts and applications of the spatial Web portal. Photogrammetric Engineering and Remote Sensing73, no. 6:691.

• Yang, C. and R. Raskin. 2009. Introduction to distributed geographic information processing research. International Journal of Geographical Information Science23, no. 5:553–560