Developing an Agricultural Monitoring Developing an Agricultural Monitoring System from Remote Sensing Data System from Remote Sensing Data

Using GridRPC on Ninf-GUsing GridRPC on Ninf-G

Shamim Akther, Yann Chemin, Honda KiyoshiAsian Institute of Technology, Thailand

Itthichok Jangjaimon, Putchong UthayopasKasetsart University, ThailandAnd Thai National Grid Project

2

MotivationMotivation

• Recent advances in data assimilation of satellite imagery for the purpose of agricultural monitoring led to the increasing demand for computing power

• There is a need for technology and platform that can meet the requirement

3

SWAP Model and StructureSWAP Model and Structure

Adopted from Van Dam et al. (1997)Drawn by Teerayut Horanont (AIT)

4

SWAP-GASWAP-GA

• Know SWAP Input parameter – sowing date, soil property, Water management, etc

• Finding SWAP Crop model

Comparison of Satellite LAI and Simulated LAI

0

1

2

3

4

5

0 30 60 90 120 150 180 210 240 270 300 330 360

DOY

LA

I

LAI_sat

LAI_sim

5

SWAP Model Parameter identificationSWAP Model Parameter identification - Data Assimilation using RS and GA- Data Assimilation using RS and GA

0.00

1.00

2.00

3.00

4.00

0 45 90 135 180 225 270 315 360

Day Of Year

Eva

po

tra

nsp

iratio

n L

AI

RS ObservationRS Observation

SWAP Crop Growth ModelSWAP Crop Growth Model

SWAP Input ParametersSWAP Input Parameters

sowing date, soil property, sowing date, soil property, Water management, and etc.Water management, and etc.

LAI, LAI, EvapotranspirationEvapotranspiration

0.00

1.00

2.00

3.00

4.00

0 45 90 135 180 225 270 315 360

Day Of Year

E

avpo

tran

spira

tion

LA

I

FittingFitting

LAI, LAI, EvapotranspirationEvapotranspiration

Assimilation by Assimilation by finding Optimized finding Optimized

parametersparameters

By GABy GA

RSRS ModelModel

6

Pixel pool

Result pool

Best fit population

Sequential Version

Sequential VersionSequential Version

• Take satellite image and loop through each pixel

• For each pixel perform GA processing

7

ChallengesChallenges

• The calculation time for identify SWAP parameters only for 1 pixel (1 sq.km) takes several minutes to 30 minutes.

• Thus, a RS image of 1000 x 1000 sq.km of 1000x1000 pixels will take more than 50 years (30min x 1000 x 1000) is not acceptable.

• Solutions– Parallel Computing on

Cluster and Grid

• Longitude: 100.008133

• Latitude: 14.388195

8

Thai Grid InfrastructureThai Grid Infrastructure

Network

SPIRIT

PALM

OPTIMA

ENQUEUE

GASS

AMATA

MAEKA

WARINE

KU

ThaiGrid User

ThaiGrid PortalTgcc.cpe.ku.ac.th

Submit

Grid Job Scheduling

CU

KMUTNB

KMUTT

AIT

SUT

CAMETAINCA

9

Programming ModelProgramming Model

• Message Passing Using MPI– Widely adopted standard– Work efficiently on Cluster– Can work at some level on Grid

• Grid RPC– Remote procedure call programming model

on Grid system– GGF adopted as a standard programming on

Grid– Several Implementation : GridSolve, Ninf-G

10

GridRPC Programming ModelGridRPC Programming Model

• Application partition into client/server– Multiple client can be used to achieve

parallelism using asynchronous call

• Role– Server side

• Implement the algorithm as a remote procedure

• Using Grid for security mechanism and execution management

– Client side • Implement main workflow of the application

• Handling input/output for user

• Call Remote Functions at Server using GridRPC API

11

Ninf-G programming modelNinf-G programming model

• A software package which supports programming and execution of Grid applications using GridRPC

• Advantages– Simple programming model that work

efficiently over lossely copuple heterogenous system

– Hiding most of the grid complexity from programmer and developer

Low level Grid Middleware

Grid RPC

Grid application

Grid Fabric Layer

12

Partitioning strategy on clusters and GridPartitioning strategy on clusters and Grid

• There are 3 possible partitioning strategies– Population Distribution – Pixel Distribution– Hybrid Distribution

• Group compute nodes as virtual group• each virtual group take care one pixel• inside the group, distributing population to members

• Pixel and population distributed has been implemented using MPI and reported in our previous work

13

Population Distribution StrategyPopulation Distribution Strategy

Pixel pool Result pool

Eval Eval Eval

GA

Set of Population

fitness

14

Pixel Distribution StrategyPixel Distribution Strategy

Pixel pool Result pool

GA GA GA

Image

Pixel

Results

15

Hybrid Distribution StrategyHybrid Distribution Strategy

GA

Pixel pool Result poolImage

Pixel

Results

EvalEval Eval

population

Cluster

GA

Pixel

Results

EvalEval Eval

population

Cluster

16

Extension to the GridExtension to the Grid

• Modify and adjust code to change distributing middleware from MPI to Ninf-G

• Implement secure data-transferring method for Grid instead of using NFS for cluster environment

• Gain benefit of Grid computing– More effective– More scalable– More security

17

ArchitectureArchitecture

18

Table 4: Server Program Model with Ninf_G //GIS program extract pixels from the Modis RS image and //pixel information will be written in separate file for each pixel. for pix_no = first to last { //Grid Remote Procedure Call environment initialization. grpc_initialize(config_file); for(pixel is 1 to 16) //let one time we manipulate 16 pixels { //Each pixel create one handle grpc_function_handle_t handle[]; grpc_function_handle_init(handle[pixel],host_name,port_name,”lib_name”); } for (pixel 1 to 16) { //for the 16 pixels a asyncnous call occure grpc_call_async(handle[pixel], args….); } // Wait until every node finished grpc_wait_all(); for( pixel 1 to 16) { // all handles need to detruct grpc_function_handle_destruct(handle[pixel]); } grpc_finalize(); }

GA_SWAP(pixel_buffer, result){

Load(pixel_buffer);Run(swap);Send(result);

}

19

Results & DiscussionsResults & Discussions

• Finish only pixel distribution over cluster

• Experiment run on 3 clusters in the Grid – MAEKA 31 nodes Opteron Dual 1.4&1.8 GHz– AMATA 14 nodes Athlon XP 1 GHz– MAGI 4 Nodes Athlon XP 2500+

20

ResultResult

Pixelversion speedup

0.000

1.000

2.000

3.000

4.000

5.000

1 2 3 4 5nodes

speedup

pop10pix5

pop20pix5

21

ResultResult

Popversion speedup

0.000

2.000

4.000

6.000

8.000

10.000

12.000

14.000

16.000

18.000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

nodes

speedup

pop10pix1

pop10pix5

pop20pix1

pop20pix5

22

ResultResult

speedup 20 pop 5 pixels

0.000

2.000

4.000

6.000

8.000

10.000

12.000

14.000

16.000

18.000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

nodes

speedup

pixel-version

pop-version

23

Lesson LearnedLesson Learned

• Grid programming is still difficult – Learning curve– Debugging? – Grid reliability. May be useful to have a small

development grid separate from deployment grid

• Ninf-g application on grid is working fine. A good approach for development of grid applications.

24

Future WorkFuture Work

• Improve performance

• scheduler and node selecting (now is only round-robin)

• Test running across international Grid– collaboration with Prof. Kento Aida,Titech– PRAGMA testbed

• Deployment of ninf-g for Thai National Grid application development

Thank youThank you

Q&A