LOGO PRism project 1 Lessons of UNIX/GMT/FORTRAN Naota Hanasaki IIS,UT DRAFT.
-
Upload
edith-terry -
Category
Documents
-
view
221 -
download
0
Transcript of LOGO PRism project 1 Lessons of UNIX/GMT/FORTRAN Naota Hanasaki IIS,UT DRAFT.
1
LOGOPRism project
Lessons of UNIX/GMT/FORTRANLessons of UNIX/GMT/FORTRANNaota Hanasaki IIS,UT
DRAFT
2
YOUR SITE HERE
LOGOPRism project
AcknowledgementAcknowledgement
I thank all attendees (2002-2005): T. Yamada, Y. Suga, A. Aslam, C. Manusthiparom, C. Apirumanekul, T. Okazawa, T. Kokubo, K. Takagaki, W. Saita, K. Kubo, Y. Koiwa, Y. Ishizaki, Q. Tang, N.S. Farzin, J. Cho, M. Lin, T. Inuzuka, T. Sakimura, D. Ikari, Y. Arai and N. Utsumi
I thank all advisors (2002-2005):Dr. T. Oki, Dr. S. Kanae, Dr. Y. Agata, Dr. S. Seto, Dr. Y. Hirabayashi, Dr. S. Miyazaki, Dr. D. Komori, Dr. Y. Shen and Dr. S. Islam
3
YOUR SITE HERE
LOGOPRism project
Table of ContentsTable of Contents
Introduction Lesson 1
– Operating UNIX
– Basic UNIX commands (cp, ls, etc.) Lesson 2
– Linking command and files
– Redirect/Pipe/Wildcard/Awk Lesson 3
– Drawing global precipitation distribution
– GMT scripts (xyz2grd, grdimage, pscoast etc.) Lesson 4
– Processing grid data 1
– Arithmetic of grid data using simple programs
4
YOUR SITE HERE
LOGOPRism project
Table of ContentsTable of Contents
Lesson 5 – Reading/Writing binary– Fortran IO
Lesson 6– Processing grid data 2– Masking grid data using simple programs
Lesson 7 – Fortran and Makefile– Useful fortran tips, function/subroutine, makefile
Lesson 8– Bourne shell script– do-loop, for-loop, if-statement
Lesson 9– Setting UNIX environment– .cshrc
5
LOGOPRism project
IntroductionIntroductionWhy UNIX?
6
YOUR SITE HERE
LOGOPRism project
IntroductionIntroduction
An assignment of “Advanced Hydrology” in 2001– Read the following article
Xie and Arkin 1997: Global Precipitation: A 17-year monthly Analysis Based on Gauge Observations, Satellite Estimates and Numerical Model Outputs. BAMS, 78, 2539-2558
– Summarize their methodology– List up advantage/disadvantage– Compare the data with other data
(station observation, global dataset, etc.)– See the site below for original data ftp://
ftp.ncep.noaa.gov/pub/precip/cmap/monthly/
7
YOUR SITE HERE
LOGOPRism project
Introduction (continued)Introduction (continued)
To submit a good report, you need– Powerful computation environment
UNIX
– Skill to process large data FORTRAN (or any other language)
– Skill to create beautiful/complex figures GMT (or any graphic package)
– Skill to manage iterative process Shell script
8
YOUR SITE HERE
LOGOPRism project
Introduction (continued)Introduction (continued)
Target of this seminar– Tutorial of the assignment– Using CMAP dataset, analyze global
precipitation How different global precipitation pattern
between El Nino year, La Nina year? How many people in the world live with less
than 500 mm/year?
– Practical and applicable lessons for your bachelor/master/doctor research
9
LOGOPRism project
Lesson 1Lesson 1Basic UNIX operation (1)
Fundamental UNIX commands
10
YOUR SITE HERE
LOGOPRism project
Today’s goalToday’s goal
Where can we use UNIX? How do we log in? How do we operate UNIX?
11
YOUR SITE HERE
LOGOPRism project
Oki Lab’s UNIX EnvironmentOki Lab’s UNIX Environment
Environment– OS: Solaris (not linux)– Machines: Sun Microsystems
Servers– Rainbow– Hydro– Reservoir– Chaser
Client (PC X server)– Astec X (commercial software)
Please ask your senior for installation and initial setting. Detailed environment setting will be shown in Lesson 9.
12
YOUR SITE HERE
LOGOPRism project
Login/LogoutLogin/Logout
Go to login screen Select language
– [Options]-[languages]-[select] Type user name & password Open terminal window
– [Third button]-[Tools]-[Terminal]
13
YOUR SITE HERE
LOGOPRism project
Managing files (1/2)Managing files (1/2)
Show present directory % pwd
Show file list % ls
Change directory % cd directory
Change directory (to upper) % cd ..
Change directory (to home) % cd ~
Change directory (to current) % cd .
View file % less file% more file
(to quit, type “q”)
Edit file % mule file
Copy file % cp original copy
Move file or Rename file % mv original copy
Remove file % rm file
14
YOUR SITE HERE
LOGOPRism project
Managing files (2/2)Managing files (2/2)
Make new directory % mkdir directory
Remove directory (not empty)
% rm –r directory
Remove directory (empty)
% rmdir directory
Copy directory % cp –r directory
Change permission % chmod 744 fileUser(1st)-Group(2nd)-Other(3rd) Read(+4), Write(+2), Execute(+1)
% chmod u+x
User(u)-Group(g)-Other(o) add(+), remove(-) Read(r), Write(w), Execute(x)
Compress file % gzip file
Uncompress file % gunzip file
15
YOUR SITE HERE
LOGOPRism project
Editing file (1/2)Editing file (1/2)
Mule % mule file
Save Ctrl-x + Ctrl-s
Exit Ctrl-x + Ctrl-c
Delete Ctrl-d
Delete line (kill) Ctrl-k
Cut Ctrl-Space (starting point), Ctrl-w
Paste (yank) Ctrl-y
Forward cursor Ctrl-f
Backward cursor Ctrl-b
Upward cursor (previous) Ctrl-p
Downward cursor (next) Ctrl-n
Page down; End Ctrl-v; Esc->
Page up; Home Esc-v; Esc-<
16
YOUR SITE HERE
LOGOPRism project
Editing files (2/2)Editing files (2/2)
Kanji Ctrl-\
Cancel Ctrl-g
Find Ctrl-s
Replace Esc-Shift-%
Replace all Esc-x replace string
See Emacs/Mule text books for detail
17
YOUR SITE HERE
LOGOPRism project
Other commandsOther commands
Netscape % netscape
Ghost view(PS file viewer)
% gv file
X view(JPEG/GIF viewer)
% xv file
Convert image file % convert file.ps file.gif
% convert file.gif file.ps
Print file % lpr -Pprinter file
See printer job % lpq –Pprinter
Remove printer job % lprm –Pprinter jobID
18
YOUR SITE HERE
LOGOPRism project
In emergency…In emergency…
See manual % man command
Break process Ctrl-c
See process % top
Kill process % kill processID
Exit % exit
19
YOUR SITE HERE
LOGOPRism project
What is permission?What is permission?
Create a text file:
Save as “example1.txt” Change permission and add executable
% chmod 744 example1.txtor% chmod o+x example1.txt
Execute the file (“text” file can be executed)
% example1.txt
pwdlswho
20
YOUR SITE HERE
LOGOPRism project
ExerciseExercise
Check your current directory Go to home directory (/home/user) Make directory “CMAP” Go to directory “CMAP” Copy files from “/home/hanasaki/pub/unix_s
emi/src/”global.gmtgrad.cptdata.xyz
Change permission of file global.gmt to executable
Execute global.gmt View output file “image.ps” Print out the image Convert “image.ps” from Post Script format
to GIF format
21
LOGOPRism project
Lesson 2Lesson 2Basic UNIX operation (2)
Redirect/Pipe/Wildcard/Awk
22
YOUR SITE HERE
LOGOPRism project
Today’s goalToday’s goal
Open the file global.gmt. You will find the file consists of numbers of short commands.
If you look more carefully, you will find curious characters such as>, >>, <, <<, | …
These characters are extremely important in UNIX operation. Let’s learn about them.
23
YOUR SITE HERE
LOGOPRism project
Redirect(Write/Read files)
Redirect(Write/Read files)
Redirect is used – to write the output of a command to a file
– to read the input of a command from a file
Write % command > file
Write (add lines) % command >> file
Overwrite % command >! file
Read % command < file
Read until EOF appears
% command << EOF
strings
EOF
24
YOUR SITE HERE
LOGOPRism project
Example/ExerciseExample/Exercise
Example 1
Exercise 1– Make a file listing who are logging in our
system. (Use command “who”)– Make a file listing the calendar for 2004 to
2005. (Use command “cal”)
% date > redirect.txt% more redirect.txt% date >> redirect.txt% more redirect.txt% date >! redirect.txt% more redirect.txt
% cat << EOF> Hello.> My name is Taikan.> EOF% more redirect.txt
% cat << EOF >> redirect.txt> Hello.> My name is Taikan.> EOF% more redirect.txt
25
YOUR SITE HERE
LOGOPRism project
Pipe(Linking commands)
Pipe(Linking commands)
Pipe hands the output of a command to the successive command as a input
% command 1 | command2 Firstly, command1 is executed.The output of command1 is input to command 2.
26
YOUR SITE HERE
LOGOPRism project
Example/ExerciseExample/Exercise
Example 2
Exercise 2– Count the number of files and directories in y
our home directory. (Use command “wc”) By using redirect ( > ) By using pipe ( | )
% ls –l% ls –l | more% ls –l | tail% ls –l | head
27
YOUR SITE HERE
LOGOPRism project
Wild cardWild card
Wild card is a special character to indicate any character or string.
* Any character(s) including null
? Any one character
28
YOUR SITE HERE
LOGOPRism project
Example/ExerciseExample/Exercise
Example 3
Exercise 3– Copy again the above directory (wildcard2)– Remove files include character “1”– Remove all files in the directory “wildcard”– Copy all files from
/home/hanasaki/pub/unix_semi/lesson2/wildcard/(Do not use –r option for command “cp”)
% cp –r /home/hanasaki/pub/unix_semi/lesson2/wildcard/ .% cd wildcard% ls –l% ls –l *txt% ls –l test*% ls –l te?t1.txt
29
YOUR SITE HERE
LOGOPRism project
AwkAwk
Awk is a kind of program language of UNIX system. Its grammar is quite similar to C language. No need for compilation, directly type to command line. In this lesson, we just use a function of awk.
Print the N th column in file % awk ‘{print $N}’ file
Print the N th column if the M th column is m
% awk ‘($M==m){print $N}’ file
30
YOUR SITE HERE
LOGOPRism project
Example/ExerciseExample/Exercise
Example 4
Exercise 4– Let’s make “data.xyz”, we used in Lesson 1.
% cd % ls –l > home.txt% awk ‘{print $9}’ home.txt% awk ‘($5==512){print $9}’ home.txt
31
YOUR SITE HERE
LOGOPRism project
Create data.xyzCreate data.xyz
Visit CMAP web site(ftp://ftp.ncep.noaa.gov/pub/precip/cmap/)
Download a monthly data file 1979 Uncompress the file
% gunzip cmap_mon_v0411_79.txt.gz
Check the file
% more cmap_mon_v0411_79.txt
32
YOUR SITE HERE
LOGOPRism project
Create data.xyzCreate data.xyz
The file should be as follows
Create monthly file (for Jan 1979)
% awk ‘($2==1){print $4, $3, $5}’ cmap_mon_v0411_79.txt > data.xyz
1979 1 -88.75 1.25 0.08 60.00 0.08 60.001979 1 -88.75 3.75 0.07 60.00 0.07 60.001979 1 -88.75 6.25 0.07 60.00 0.07 60.001979 1 -88.75 8.75 0.07 60.00 0.07 60.001979 1 -88.75 11.25 0.07 60.00 0.07 60.001979 1 -88.75 13.75 0.07 60.00 0.07 60.00
yearmonth
lat londata
See the CMAP documents
33
LOGOPRism project
Lesson3Lesson3Drawing figure
<GMT>
34
YOUR SITE HERE
LOGOPRism project
Today’s goalToday’s goal
In lesson 1, we drew a nice figure by executing global.gmt.– How was the figure drawn?– How we can modify it?
By the way, what is GMT?– GMT is short for Generic Mapping Tool– Free software developed by Paul Wessel an
d Walter H. F. Smith in Hawaii University.– See: http://gmt.soest.hawaii.edu/
35
YOUR SITE HERE
LOGOPRism project
Reviewing Lesson 1Reviewing Lesson 1
% global.gmt% gv image.ps
36
YOUR SITE HERE
LOGOPRism project
Files needed for GMTFiles needed for GMT
GMT source file (GMT script file)– global.gmt
Data file– data.xyz
CPT file (Color Palette Table file)– grad.cpt
37
YOUR SITE HERE
LOGOPRism project
global.gmt (1/6)global.gmt (1/6)
#!/bin/sh#############################################################ex Input/Output File#on 31/May/2002#by nhanasaki#at IIS,UT############################################################# Define File Names############################################################DATFILE=./data.xyz #data file (input)#DATAFILE=./cmap_mon_v0203_79.txtPSFILE=./image.ps #image file (output)GRDFILE=./grd #temporary fileCPTFILE=./grad.cpt #color paltette table file
Declaration of shell script type (Bourne Shell)
Com
ments
(Starting w
ith #)
Input VALUE into VARIABLEVARIABLE=VALUE
Comments
38
YOUR SITE HERE
LOGOPRism project
global.gmt (2/6)global.gmt (2/6)
############################################################# Define Mapping Area############################################################XMIN=0.00 #Horizontal minimum [degree]XMAX=360.00 #Horizontal maximum [degree]YMIN=-90.00 #Vertical minimum [degree]YMAX=90.00 #Vertical maximum [degree]XWID=21.0 #Width of image [cm]YWID=10.5 #Height of image [cm]DXa=30.0 #a:Horizontal Anotation Interval [degree]DXf=30.0 #f:Horizontal Frame Interval [degree]DXg=10.0 #G:Horizontal Grid Interval [degree]DYa=30.0 #a:Vertical Anotation Interval [degree]DYf=30.0 #f:Vertical Frame Interval [degree]DYg=10.0 #g:Vertical Grid Interval [degree]D=2.5 #grid size############################################################## Short Cuts for Options############################################################RFLAG="-R${XMIN}/${XMAX}/${YMIN}/${YMAX}"JFLAG="-JX${XWID}/${YWID}d"BFLAG=“-Ba${DXa}f${DXf}g${DXg}:Longitude:/a${DYa}f${DYf}g${DYg}:Latitude:neWS
Call variables${VARIABLES}
39
YOUR SITE HERE
LOGOPRism project
global.gmt (3/6)global.gmt (3/6)
XMIN [deg] XMAX [deg]
YMIN [deg]
YMAX [deg]
YW
ID
[cm]
XWID [cm]
Annotation
Frame
Gridcannot see
D
each grid
40
YOUR SITE HERE
LOGOPRism project
global.gmt (4/6)global.gmt (4/6)
############################################################---Jobs#############################################################awk '($2==1){print $4, $3, $5}' $DATAFILE | \#xyz2grd $RFLAG -G$GRDFILE -I${D}/${D} -Fxyz2grd $RFLAG -G$GRDFILE -I${D}/${D} -F $DATFILE#psbasemap $RFLAG $JFLAG $BFLAG -X5.0 -Y5.0 -K > ./$PSFILE grdimage -O $RFLAG $JFLAG $GRDFILE -C$CPTFILE -K >> ./$PSFILE pscoast -O $RFLAG $JFLAG -Dc -W5 -N1 -I1 -K >> ./$PSFILEpsscale -O -C$CPTFILE -D5.0/-1.5/8/0.5h -L -K >> ./$PSFILEpstext -O $RFLAG $JFLAG -N << EOF >> ./$PSFILE180 110 24 0.0 1 6 Jan 1979 Rainfall (CMAP)EOF
comm
ands
data.xyz grd
psbasemap
pscoastline
psscale
pstext (put title)
xyz2grd grdimage
redirects(see Lesson2)
Read until EOF appear
Omit headerOmit trailer
41
YOUR SITE HERE
LOGOPRism project
global.gmt (5/6)global.gmt (5/6)
############################################################# Comments for Options#############################################################-R: Option specifies region of interest#-J: Option for mapping projection (JX: projection type)#-B: Option for frame#-O: Option for "Omit Header" i.e. following GMT command#-K: Option for "Omit Trailer" i.e. GMT commands follow on##--psbasemap##draw basemap#-X: Shift plot origin x-direction [cm]#-Y: Shift plot origin y-direction [cm]##--xyz2grd##convert ASCII-xyz-data into binary-grid-data##--gridimage##conert binary-grid-data into colored mosaic image#
X
Y
42
YOUR SITE HERE
LOGOPRism project
global.gmt (6/6)global.gmt (6/6)
#--pscoast##draw coastline#-D: Resolution#-W: Boldness#-N: National Border#-I: Rivers##--psscale##draw color bar#-D: Place to put#-L: Give Equal size rectangular##--pstext##write text between EOFs#-N: Do not clip#{horizontal[degree]/vertical[degree]/Point/Angle/Font?/Place?/Text}
43
YOUR SITE HERE
LOGOPRism project
CPT fileCPT file
0 255 0 0 1 255 255 01 255 255 0 5 0 255 05 0 255 0 10 0 255 25510 0 255 255 30 0 0 255F 0 0 128B 128 0 0N 255 255 255
Start value End value
Color (RGB)<TAB> spacingDo not use <SPACE> spacing !!
0 1 5 10 30
44
YOUR SITE HERE
LOGOPRism project
Exercise(Midterm report)
Exercise(Midterm report)
Draw a figure (1/3)– Fill ocean with blue color
– Change title of the figure
– Put unit [mm/day] to the figure Draw a figure (2/3)
– Fill land with black color
– Change title of the figure
– Put unit [mm/day] to the figure Draw a figure (3/3)
– Zoom into Iran (horizontal:60deg; Vertical: 30deg)
– Make your original CPT
– Change title of the figure
– Put unit [mm/day] to the figure
45
LOGOPRism project
Lesson 4Lesson 4Processing grid data 1
<HTOOL>
46
YOUR SITE HERE
LOGOPRism project
Today’s goalToday’s goal
Let’s analyze global precipitation using CMAP dataset– Global, 2.5ox2.5o
– 1979-2002, monthly See the difference of global
distribution of rainfall in El Nino year and La Nina year– El Nino: 1986(autumn)-1987(winter)– La Nina: 1988(spring)-1989(spring)
Draw anomaly (difference) of annual global precipitation between 1987 and 1988
Whole 1987
Whole 1988
47
YOUR SITE HERE
LOGOPRism project
CMAP dataCMAP data
Binary grid data is stored in:/home/hanasaki/pub/unix_semi/dat_bin
72 grids
144 grids
MonthlyPrecipitation data
CMAP020319870100.cmap
CMAP, ver0203,1987,Jan,monthly data
72 x 144 x 4 = 41472 byte
48
YOUR SITE HERE
LOGOPRism project
Ascii / Binary Ascii / Binary
ASCII Binary
Can we read? Yes No
Machine dependency
No Yes
Access speed Slow Fast
Data size 1 byte/char 4byte/rec
Example 1
1.0 3 byte 4 byte
Example 2
1.000000 8 byte 4 byte
49
YOUR SITE HERE
LOGOPRism project
ExampleExample
1987-Jan
1987-Feb
1987-Mar
1987-Dec
Summation of 12 months
Divide by 12
Annual Average of 1987
12 months
add
add
add
% createcmap CMAP020319870000.cmap% addcmap CMAP020319870000.cmap CMAP 020319870100.cmap CMAP020319870000.cmap% addcmap CMAP020319870000.cmap CMAP 020319870200.cmap CMAP020319870000.cmap…% addcmap CMAP020319870000.cmap CMAP 020319871200.cmap CMAP020319870000.cmap% divcmap CMAP020319870000.cmap 12 CMAP020319870000.cmap
50
YOUR SITE HERE
LOGOPRism project
ExampleExample
El Nino year(1987 average)
La Nina year(1988 average)
Difference
Draw
sub
% mon2yearcmap CMAP 0203 1987 1987 .cmap% mon2yearcmap CMAP 0203 1988 1988 .cmap% subcmap CMAP020319870000.cmap CMAP020319880000.cmap dif.cmap% cmap2ps dif.cmap dif.cpt dif.ps Anomaly 1987-1988
51
YOUR SITE HERE
LOGOPRism project
HTOOL (1)HTOOL (1)
Copy files from /home/hanasaki/pub/unix_semi/src
% cp /home/hanasaki/pub/unix_semi/src/* .
program Arguments Function
createcmap OUT Create binary file
addcmapsubcmapmulcmapdivcmapprocmapratcmap
IN1,IN2,OUTIN1,IN2,OUTIN1,num,OUTIN1,num,OUTIN1,IN2,OUTIN1,IN2,OUT
Add binary filesSubtract binary filesMultiply binary file by valueDivide binary file by valueMultiply binary files (product)Divide binary files (ratio)
asc2cmapcmap2ascxyz2cmapcmap2xyzzuracmap
IN,OUTIN,OUTIN,OUTIN,OUTIN,OUT
Convert text data to binaryConvert binary data to textConvert (lon lat data) to binaryConvert binary to (lon lat data)Shift data by 180o in binary file
cmap2ps IN,CPT,PS,title1,title2,opt Convert data file to figure
mon2yearcmap ORG RUN syear eyear suf Convert monthly data to yearly data
IN: input binary fileOUT: output binary fileCPT: cpt filePS: ps image filenum, miss: real numbertitle1, title2: charactersyear, eyear: 4 digit integerORG: project name RUN: run name
52
YOUR SITE HERE
LOGOPRism project
ExerciseExercise
Create JJA (June-July-August) and DJF (December-January-February) precipitation for 1987 and 1988 and create anomaly figure. (Create your original CPT file for your analysis)
Draw normalized anomaly between 1987 and 1988. (Create your original CPT file for your analysis)
53
LOGOPRism project
Lesson 5Lesson 5Reading/Writing binary files
<FORTRAN IO>
54
YOUR SITE HERE
LOGOPRism project
Today’s goalToday’s goal
Master FORTRAN input/output– Change binary and ascii data
FORTRAN 77– In this seminar, we use FORTRAN 77. However,
you can use whatever programming language you like (C, FORTRAN90, C++, C#, Java, Ruby…)
– FORTRAN 77 is “out of date” programming language, however, it is still widely used in science community.
– Especially important programs in our laboratory CCSR/NIES AGCM, MATSIRO, TRIP are written in FORTRAN 77
Reference– Fortran 90 プログラミング 冨田博之 培風館– Numerical recipes in Fortran 77
55
YOUR SITE HERE
LOGOPRism project
FORTRAN 77FORTRAN 77
program helloccc write(*,100) ‘Hello, World’ write(*,*) ‘Very very long sentences such as this $ must be divided in two lines’ 100 format(a11)c end
Start from the 7th column
If line continues, put some character in the 6th column
If line is comment, put c or * in the 1st column
Line number is put in 2nd to 5th column (Usually, no need to write line num)
PRism project
The first 6 columns have special meaning. Maximum: 72 columns
56
YOUR SITE HERE
LOGOPRism project
cmap2asc.f (1/4)cmap2asc.f (1/4)
program cmap2asccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc cto Convert binary (144x72) to asciicon 30th/May/2002cby nhanasakicat IIS, UTcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc implicit nonec integer nx integer ny parameter (nx=144) parameter (ny=72)c integer i,j real rdat(nx,ny) character*128 cifname character*128 cofnamec integer iargc
Declare code type
Do not use implicit variable type
Define integer variable “nx”
Put “144” to the variable “nx” (fixed)
Define real number array “rdat”
Define character variable “cofname”
nx ny i j cifname cofname iargc
144 72
rdat
57
YOUR SITE HERE
LOGOPRism project
cmap2asc.f (2/4)cmap2asc.f (2/4)
cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Get file namecccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc if (iargc().ne.2) then write(6,*) 'Usage: cmap2asc Infile Outfile' stop end ifc call getarg(1,cifname) call getarg(2,cofname)
<In plain English>If the number of arguments is not equal to 2, then write “Usage: cmap2asc Infile Outfile” to the screen (format is default) and stop the programOtherwise, just continue process.
Put the first argument string to variable “cifname” andPut the second argument string to variable cofname
58
YOUR SITE HERE
LOGOPRism project
cmap2asc.f (3/4)cmap2asc.f (3/4)
cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Read Binary Datacccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc open(15, file=cifname, access='direct', recl=nx*4) do j=1, ny read(15,rec=j) (rdat(i,j),i=1,nx) end do close(15)
<In plain English>[Line 1] Allocate file number 15 and open file “cifname”. The file is written in binary (direct access) and the record length is nx *4 (=144*4=576) byte.[Line 2] Change j from 1 to “ny” (or iterate “ny” times)[Line 3] Read the j th record of the file number 15. Put data to array rdat.[Line 4] Process finished, go to the next iteration (or add 1 to j)[Line 5] Close file number 15
Record length
j th
59
YOUR SITE HERE
LOGOPRism project
cmap2asc.f (4/4)cmap2asc.f (4/4)
cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Write Ascii Datacccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc open(16, file=cofname, access='sequential') do j=1, ny write(16,*) (rdat(i,j), i=1,nx) end do close(16)c end
<In plain English>[Line 1] Allocate file number 16 and open file “cofname”. The file is written in ascii (sequential access)[Line 2] Change j from 1 to “ny” (or iterate “ny” times)[Line 3] Write array rdat(i,j) to the file number 16. i is changed from 1 to nx. The format is default.[Line 4] Process finished, go to the next iteration (or add 1 to j)[Line 5] Close file number 16
Record length
j th
60
YOUR SITE HERE
LOGOPRism project
ExcerciseExcercise
Read a FORTRAN source code addcmap.f and translate into plain English or Japanese.(/home/hanasaki/bin/addcmap.f)
Write a program asc2cmap.f that has the opposite function of cmap2asc.f. (After finish editing, type% make asc2cmapon the terminal)
61
LOGOPRism project
Unix Seminar Lesson 6Unix Seminar Lesson 6Grid data processing 2
62
YOUR SITE HERE
LOGOPRism project
Today’s goalToday’s goal
How much annual precipitation we have in Tokyo? [E138.75-N36.25]*
How much is the annual average precipitation of the whole world?
How large area in the world has more annual precipitation than Tokyo?
How many people in the world live in the place less than 500 mm/year precipitation?
* Actually, the geographic location for the lon/lat is Takasaki City in Gunma prefecture. Because of large grid size of CMAP, This is the nearest grid from Tokyo!
63
YOUR SITE HERE
LOGOPRism project
MaterialMaterial
Mean Annual rainfall (1979-2001)– rain.cmap
– Unit: mm/year
– Global, 2.5x2.5
Land area– landarea.cmap (land only)
gridarea.cmap (whole globe)
– Unit: km2
– Global, 2.5x2.5
(See /home/hanasaki/pub/unix_semi/lesson6)
64
YOUR SITE HERE
LOGOPRism project
MaterialMaterial
Landmask– landmask.cmap
– Unit: none
– Global, 2.5x2.5
Polulation– pop.cmap
– Unit: capita
– Global, 2.5x2.5
– Source: CIESIN
(See /home/hanasaki/pub/unix_semi/lesson6)
65
YOUR SITE HERE
LOGOPRism project
ExampleExample
How much annual precipitation we have in Tokyo? [E138.75-N36.25]*
How much is the annual average precipitation of the whole world?
Global annual precipitation
138.75
36.25
Global annual precipitation
1. Read the file2. Sum up data for every grids3. Divide by the number of grids
1. Read the file2. Pick up data for the grid
% pointcmap rain.cmap 138.75 36.25% avecmap rain.cmap 0 [Actually, it is not correct]
66
YOUR SITE HERE
LOGOPRism project
ExampleExample
Mask
Precipitation less than 500 mm/year
Sum up
1. How large area in the world has more annual precipitation than Tokyo?
2. How many people in the world live in the place less than 500 mm/year precipitation?
% maskcmap lndara.cmap rain.cmap gt 1530 temp.cmap% sumcmap temp.cmap 0% maskcmap pop.cmap rain.cmap gt 500 temp.cmap% sumcmap temp.cmap 0
67
YOUR SITE HERE
LOGOPRism project
Htools (2)Htools (2)
Program Arguments Function
avecmapmaxcmapmincmapsumcmap
IN,missIN,missIN,missIN,miss
Calc average of binary fileFind max value in binary file Find min value in binary fileCalc summation of binary file
Pointcmap IN,lon,lat Display data at (lon,lat)
findcmaprplccmap maskcmapmaskrplccmap
IN,eq,num,OUTIN,eq,num,num,OUTIN,MASK,eq,num,OUTIN,MASK,eq,num,num,OUT
Find dataReplace dataMask dataMask data and replace
1 2 3 45 1 1 89 1 1 12
13141516
1 2 3 45 6 7 89 101112
13141516
0 0 0 00 1 1 00 1 1 00 0 0 0
0 0 0 00 6 7 00 1011 00 0 0 0
1 2 3 45 6 7 89 101112
13141516
1 2 3 45 6 7 89 101112
13141516
1 2 3 45 6 7 89 101112
13141516
Mask file
0 0 0 00 0 0 09 101112
13141516
Findif (data > 8)then pick up
0 0 0 00 0 0 01 1 1 11 1 1 1
Replaceif (data > 8)replace to 1
Maskif (mask = 1)then pickup
Mask Replaceif (mask = 1)replace to 1
68
YOUR SITE HERE
LOGOPRism project
ExerciseExercise
Calculate correctly the mean annual precipitation of the world– Firstly, think about strategy, how to process data.
– Secondly, combine Htools. (See also Lesson 4) Complete the table below
Precipitation Population Area
0-50
50-100
100-500
500-1000
1000-2000
2000-
Total
69
LOGOPRism project
Lesson 7Lesson 7Fortran and Makefile
70
YOUR SITE HERE
LOGOPRism project
Today’s goalToday’s goal
Learn some important techniques for effective handling of temporal varying files
Understand what is “Makefile”(/home/hanasaki/pub/unix_semi/src/Makefile)
71
YOUR SITE HERE
LOGOPRism project
mon2yearcmap.fmon2yearcmap.f
program mon2yearcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc implicit nonec parameter integer nx, ny parameter (nx=144, ny=72)c variable integer ix, iy integer iyear, isyear, ieyear, imon integer ndaymon !! Number of days in a month integer ndayyear !! Number of days in a year real rmondat(nx,ny) !! Monthly data array real ranudat(nx,ny) !! Annual data arrayc function integer iargc !! Default function integer igetday !! My functionc files character*128 corg, crun, csuf, ctmp character*128 cifname, cofnamecccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Get Argumentscccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc if(iargc().ne.5)then write(*,*) 'mon2year <org><run><isyear><ieyear><suf>' stop end ifc call getarg(1,corg) call getarg(2,crun) call getarg(3,ctmp) read(ctmp,*) isyear call getarg(4,ctmp) read(ctmp,*) ieyear call getarg(5,csuf)
x 31x 28x 31…x 31
Comments. (not standard grammar but valid almost every compiler)
Argument must be character.To get real/integer arguments,write like this.
sum up
divide by 365
72
YOUR SITE HERE
LOGOPRism project
mon2yearcmap.fmon2yearcmap.f do iyear=isyear,ieyearc clear variable ndayyear=0c clear array do iy=1,ny do ix=1,nx ranudat(ix,iy)=0.0 end do end doc do imon=1,12 write(*,*) iyear,imon ndaymon=igetday(iyear,imon) ndayyear=ndayyear+ndaymonc read data write(cifname,101) corg,crun,iyear,imon,0,csuf call read_binmat(rmondat,nx,ny,15,cifname)c calclulation do iy=1,ny do ix=1,nx ranudat(ix,iy)=ranudat(ix,iy)+rmondat(ix,iy)*real(ndaymon) end do end do end doc Get average write(*,*) 'Total num of days:', ndayyear do iy=1,ny do ix=1,nx ranudat(ix,iy)=ranudat(ix,iy)/ndayyear end do end doc Write results write(cofname,101) corg,crun,iyear,0,0,csuf call wrte_binmat(ranudat,nx,ny,16,cofname) end do 101 format(a4,a4,i4.4,i2.2,i2.2,a5)c end
Refreshing arrays/variables
Calling a function from the outside of program
Calling a function from the outside of program
Create file name like:AAAABBBByyyymmdd.cmap
Set file name:AAAABBBByyyymmdd.cmap
Write to display.
Ref
resh
ing
arr
ays/
vari
able
s is
extr
emel
y i
mp
ort
ant
for
accu
mu
lati
ve
pro
ces
s
73
YOUR SITE HERE
LOGOPRism project
Subroutine/FunctionSubroutine/Function
igetday
read_binmat
wrte_binmat
mon2yearcmap
igetday
read_binmat
wrte_binmat
day2moncmap
Commonsubroutine
commonfunction
Differentmain program
subroutine read_binmat(rdat,nx,ny,infile,cfname)cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc open(infile,file=cfname,access='DIRECT',status='old',recl=4*nx) do j = 1,ny read (infile,rec=j)(rdat(i,j),i=1,nx) end do close (infile)c end
call read_binmat(rarray,360,180,15,temp.bin)1) Main programcalls subroutine
2) Arguments aretransferred tosubroutine
/home/hanasaki/unix_semi/src/read_binmat.f
74
YOUR SITE HERE
LOGOPRism project
MakefileMakefile
############################################################# Macro############################################################RM = /bin/rm # path to command rmFC = f77 # path to fortran compilarFCFLAGS = -g -c –C # option for fortran compilarFL = f77 # path to fortran linkerFLFLAGS = # path to fotran linkerOBJS = addcmap subcmap mulcmap divcmap procmap ratcmap \
mon2yearcmap############################################################# Suffix rule############################################################.f.o:
$(FC) $(FCFLAGS) $<############################################################# Dependency############################################################TARGET2 = mon2yearcmapTARGET2.SUB = read_binmat.o wrte_binmat.o \
igetday.o############################################################# Compilation############################################################$(TARGET2) : $(TARGET2).o $(TARGET2.SUB)
$(FL) -o $@ [email protected] $(TARGET2.SUB)clean:
$(RM) -f *.o core *~ *traceall: $(OBJS)
<TAB>
<TAB>
<TAB>
<TAB>
Define variablesSimilar to shell script
Define variables
Set Dependency (A:B)A is dependent on BIf B is newer than A thenexecute the next line
Suffix rule.f is related with fortran
75
YOUR SITE HERE
LOGOPRism project
binary/asciibinary/ascii
1 2 3 45
sample.txt
1 2 3 45
1 byte
4 byte for 1 record
1 <s> 2 <l> 3 <s> 4 5
sample.bin
<s>: space<l>: linefeed
real data(2,2) c open(15, file=‘sample.txt’, access=‘sequential’) do j=1,2 read(15,*) (data(i,j), i=1,2) end do close(15)
real data(2,2) c open(16, file=‘sample.bin’, access=‘direct’,recl=8) do j=1,2 write(16,rec=j) (data(i,j), i=1,2) end do close(16)
8 byte for 1 (unit) record length
4 byte
Ascii file
Binary file
1 2 3 45
Memory data(i,j)
1 2
1
2
76
YOUR SITE HERE
LOGOPRism project
ExerciseExercise
Code a program that gives the number of days of any month.(Put file name “nofday,f”)
% program 1995 331% program 1995 228% program 1996 229% program 2000 229% program 1900 228
77
LOGOPRism project
Lesson 8Lesson 8Shell script
78
YOUR SITE HERE
LOGOPRism project
Today’s goalToday’s goal
To complete Exercise 2 of Lesson 6, we need to type 42 commands on the terminal. How can we reduce this troublesome procedure?
79
YOUR SITE HERE
LOGOPRism project
Example (1)Example (1)
Write Jan to Dec on the display and if the month is your birth month, write “Happy birth month!” on the screen
#!/bin/sh##################################################MONS=“01 02 03 04 05 06 07 08 09 10 11 12”BIRTHMON=“08”##################################################for MON in $MONS; do if [ $MON = $BIRTHMON ]; then echo $MON “Happy birth month!” else echo $MON fidone
80
YOUR SITE HERE
LOGOPRism project
Example (2)Example (2)
#!/bin/sh##################################################YEAR=2005MONS=“01 02 03 04 05 06 07 08 09 10 11 12”BIRTHMON=“02”BIRTHDAY=“28”##################################################for MON in $MONS; do DAY=1 # refresh day EDAY=`nofday $YEAR $MON` # set the last day of the month while [$DAY –le $EDAY ]; do if [ $MON = $BIRTHMON –a $DAY = $BIRTHDAY ]; then echo $MON $DAY “Happy birth day!” else echo $MON $DAY fi DAY=`expr $DAY + 1` # day for next iteration donedone
Write 1st Jan to 31st Dec on the display and if the date is your birthday , write “Happy birthday !” on the screen
81
YOUR SITE HERE
LOGOPRism project
Exercise (3)Exercise (3)
Review the Exercise 2 of Lesson 6.Using do-loop, for-loop and if statement, reduce the code as small as possible.Step 1. find the common procedure in the code.
# 0mm/year =< Precipitation < 50mm/yearmaskcmap pop.cmap rain.cmap ge 0 temp.pop.ge0.cmapmaskcmap temp.pop.ge0.cmap rain.cmap lt 50 temp.pop.ge0.lt50.cmapsumcmap temp.pop.ge0.lt50.cmap 0# 50mm/year =< Precipitation < 100mm/yearmaskcmap pop.cmap rain.cmap ge 50 temp.pop.ge50.cmapmaskcmap temp.pop.ge50.cmap rain.cmap lt 100 temp.pop.ge50.lt100.cmapsumcmap temp.pop.ge50.lt100.cmap 0
DAT=pop# 0mm/year =< Precipitation < 50mm/yearLL=0 # lower limitUL=50 # upper limit# 50mm/year =< Precipitation < 100mm/yearLL=50 # lower limitUL=100 # upper limit##############################maskcmap ${DAT}.cmap rain.cmap ge ${LL} temp.${DAT}.ge${LL}.cmapmaskcmap temp.${DAT}.ge${LL}.cmap rain.cmap lt ${UL} temp.${DAT}.ge${LL}.lt${UL}.cmapsumcmap temp.${DAT}.ge${LL}.lt${UL}.cmap 0
# note it doesn’t work
82
YOUR SITE HERE
LOGOPRism project
Exercise (3)Exercise (3)
Review the Exercise 2 of Lesson 6.Using do-loop, for-loop and if statement, reduce the code as small as possible.Step 2. Combining do/for loop and if statement.
#!/bin/sh##################################################for DAT in pop grdara; do JOB=1 while [ $JOB –le 7 ]; do if [ $JOB -eq 1 ]; then LL=0; UL=50 elif [ $JOB -eq 2 ]; then LL=50; UL=100 fi maskcmap ${DAT}.cmap rain.cmap ge ${LL} temp.${DAT}.ge${LL}.cmap maskcmap temp.${DAT}.ge${LL}.cmap rain.cmap lt ${UL} temp.${DAT}.ge${LL}.lt${UL}.cmap sumcmap temp.${DAT}.ge${LL}.lt${UL}.cmap 0 JOB=`expr $JOB + 1` donedone
83
YOUR SITE HERE
LOGOPRism project
SummarySummary
For loop for VAR in VARLIST; do processdone
If statement If [ ]: then process1elif [ ]; then process2else process3fi
While loop while [ ]: do processdone
If file exist If [ -f FILENAME ]; then processfi
If directory exist
If [ -d DIRNAME ]; then processfi
A = B For string:[ “A” = “B” ]For integer:[ A –eq B ]
A ≠ B For string:[ “A” != “B” ]For integer:[ A –ne B ]
A > B [ A –gt B ]
A ≥ B [ A –ge B ]
A < B [ A –lt B ]
A ≤ B [ A –le B ]
And -a
Or -o
84
YOUR SITE HERE
LOGOPRism project
Exercise (Final exam)Exercise (Final exam)
Create a shell script that generates binary data files (144 x 72 x 4 byte) for whole period from the original CMAP ascii data files.ftp://ftp.ncep.noaa.gov/pub/precip/cmap/monthly/
85
LOGOPRism project
Lesson 9Lesson 9Set UNIX environment
86
YOUR SITE HERE
LOGOPRism project
Today’s goalToday’s goal
Understanding – What is etc directory in your home directory– Why “rm –f” still ask you for your permissio
n?– Why the command “lpr” knows our default pr
inter – Why we can execute “ls” even the file does n
ot exist in the current directory Understanding our lab’s UNIX enviro
nment.
87
YOUR SITE HERE
LOGOPRism project
$HOME/.cshrc$HOME/.cshrc
See ~/.cshrc– If your login shell, the shell will first check th
e file “~./cshrc”.
#############################################################to set tcsh#by nhanasaki#on 27th June 2005#at IIS,UT############################################################set DEFD=/home/`/usr/ucb/whoami`/etc#source $DEFD/setenv.allsource $DEFD/setdef.all#if ($?prompt) then source $DEFD/setenv.shell source $DEFD/setdef.shell source $DEFD/aliases.shellendif
88
YOUR SITE HERE
LOGOPRism project
$HOME/etc/.aliases$HOME/etc/.aliases
Alias is kind of “short cut”,% alias sayhello ‘echo Hello’% sayhello% which sayhello
############################################################# Replace basic command############################################################alias cd 'cd \!*;echo $cwd; sp'alias cp 'cp -i'alias ls 'ls -bF'alias mv 'mv -i'alias rm 'rm -i'alias gv 'ghostview'alias setgv 'setenv DISPLAY fountain:0.0'############################################################# To raid############################################################alias raida 'cd /export/raidA/hanasaki'alias raid1 'cd /export/raid1/hanasaki'alias raid2 'cd /export/raid2/hanasaki'
Call original “ls”% /bin/ls
89
YOUR SITE HERE
LOGOPRism project
$HOME/etc/setenv.all$HOME/etc/setenv.all
Environmental variable is a special variable used by some programs.
############################################################# BASICS############################################################setenv PRINTER floodsetenv EDITOR /usr/xpg4/bin/visetenv VISUAL /usr/xpg4/bin/visetenv PAGER /usr/local/bin/lesssetenv MANPATH /opt/SUNWspro/SC4.2/man:/usr/share/man############################################################# TEX############################################################setenv BIBINPUTS .:/home/taikan/tex/inputsokisetenv TEXFONTS /usr/local/tex/fontssetenv TEXPOOL /usr/local/texsetenv TEXFORMATS /usr/local/texsetenv TEXINPUTS .:/usr/local/tex/macros:/usr/local/rain/texinputs
Font informationused by TeX
Default value is set here
90
YOUR SITE HERE
LOGOPRism project
Why we can call various commands?% where ls% where f77
$HOME/etc/setdef.all$HOME/etc/setdef.all
set HANAP = ( /home/hanasaki/bin )set BINP= ( /usr/bin /sbin /usr/sbin /usr/ucb)set CMPP = ( /opt/SUNWspro/bin )#if ( -d /opt2/SUNWspro/bin ) then set CMPP2 = ( /opt2/SUNWspro/bin )else set CMPP2 = ( )Endif############################################################# Set path############################################################set path=( $CMPP $CMPP2 $HANAP)############################################################# Unset MACRO############################################################unset CMPP CMPP2 HANAP BINP
91
YOUR SITE HERE
LOGOPRism project
Servers Servers
CPU FortranCompiler(f77)
Attached Raid
Rainbow 400 Mhz x4 Rainbow(ver 4.2)
Raid A-E
Hydro 1280 MHz x2 Rainbow(ver 4.2)
Raid 20-29
Chaser 1593 MHz x4 Mizu(ver 6)(f90 ver 7.1)
Raid 30-39
Reservoir 750 MHz x 6 Mizu(ver 6)(f90 ver 7.1)
Raid 1-9Raid11-18
/usr/platform/`uname -i`/sbin/prtdiag
F77 ver 4.2 and ver 6 aresignificantly different in linking outside functions,subroutines! Be careful!!
92
YOUR SITE HERE
LOGOPRism project
ExerciseExercise
Set alias to “cd” to your export directory
% which raidAraidA aliased to /export/raidA/hanasaki
Make directory bin in your home directory and set path, so as to useany commands in $HOME/bin at whatever directory.
93
LOGOPRism project
After wordAfter word
94
YOUR SITE HERE
LOGOPRism project
After wordAfter word
このテキストは 2002年から 2005年の夏学期に東京大学生産技術研究所沖研究室で行われた初心者向けの UNIX講習会の補助資料として作られました。プログラミングやUNIXの知識がゼロの学生が沖研究室で研究を始めるために最低限必要な知識をまとめて紹介しています。沖研究室は地球規模の水循環に関する研究を行っています。このテキストはその際に必要な膨大な時系列グリッドデータの操作に <偏った >内容になっています。あえて内容を偏らせることで研究にすぐに役立つように工夫したつもりです。このテキストは <なんとか >英語で作りました。沖研究室に毎年やってくる熱意ある留学生に日本語のバリアをはらないためです。下手な英語で分かりにくくなっているかもしれませんが、どうぞご了承ください。またテキストの英語は文法・用法上の誤りが多々あるかもしれません。お気づきの際は、是非私までお知らせください。
2005年 7月 5日花崎 直太
95
YOUR SITE HERE
LOGOPRism project
参考文献参考文献
UNIX– 図解標準最新UNIXハンドブック(伊藤和人・秀和システム)
FORTAN– Fortran90 プログラミング(冨田博之・培風館)
Bourne Shell– 入門 UNIX シェルプログラミング( Bruce Blinn ・ Soft Bank Publishin
g ) GMT
– GMT A Map-Making Tutorial ( Paul Wessel ) VBA
– かんたんプログラミング Excel VBA シリーズ(大村あつし・技術評論社)
Solaris– Sun テキスト Solaris9 ( Paul A Watters ・ Soft Bank Publishing )
Make– Make 改訂版( Andrew Oram & Steve Talbott ・ O’Reilly )
LaTeX– LaTeX2ε 美文章作成入門(奥村晴彦・技術評論社)
96
YOUR SITE HERE
LOGOPRism project
ReferencesReferences
UNIX– ?
FORTAN– ?
Bourne Shell– Portable Shell Programming ( Bruce Blinn )
GMT– GMT A Map-Making Tutorial ( Paul Wessel )
VBA– ?
Solaris– Solaris 10: The Complete Reference ( Paul A Watters; Mcgraw-Hi
ll) Make
– Make ( Andrew Oram & Steve Talbott; O’Reilly ) LaTeX
– ?