1

Status of STAR Asian Computing Center at KISTI23 Sept. 2010

Jonghu Lee ()KISTI GSDCjlee206ATkisti.re.kr

ContentIntroduction to KISTI and GSDC1Current Status of SACC2Future Plan of SACC321. Introduction to KISTI and GSDCJan. 1962

Jan. 1980

Jan. 1991

Feb. 1991

Apr. 1993

Sep. 1999

Jan. 2001KISTI History Organized KORSTIC (Korea Scientific & Technology Information Center) Reorganized to KIET (Korea Institute for Industrial Economics and Technology) Merged with KIEI (Korea International Economics Institute) KINITI (Korea Institute of Industry and Technology Information) spun off from KIET Established SERI (System Engineering Research Institute) division under KINITI Founded KORDIC (KORea Research and Development Information Center) Acquisition Super Computing Center from ETRI (Electronics and Telecommunications Research Institute) Established KISTI (Korea Institute of Science and Technology Information)Provide Science and Technology Informationto the public to promote R&D productivity3Three Main Functions of KISTISuper Computing CenterSupercomputing Management and OperationSupercomputingHigh Speed Research NetworkNational Grid InfrastructureKnowledge Information CenterDeveloping National Portal Systems for Information ResourcesDeveloping Next Generation Technology in Information ServicesInformation Analysis CenterCore Technologies AnalysisCore Technologies Feasibility studyForeign Information Trend AnalysisInformation Analysis System Development

4Organization of KISTI

5Location

SEOULDaejeon160kmKISTI 6 Universities 20 government research institutes 10 government-invested institutes 33 private R&D labs 824 high-tech companiesLocated in the Heart of Science Valley, Daedeok InnopolisThe DAEDEOK INNOPOLIS complex consists of a cluster of firms that represents a cross-section of Korea's cutting-edge industries, including information technology, biotechnology and nanotechnology.http://www.kisti.re.kr/english6Super Computing Center at KISTIRankSiteComputer/Year VendorCoresRmaxRpeakPower1Oak Ridge National LaboratoryUnited StatesJaguar - Cray XT5-HE Opteron Six Core 2.6 GHz / 2009Cray Inc.2241621759.002331.006950.602National Supercomputing Centre in Shenzhen (NSCS)ChinaNebulae - Dawning TC3600 Blade, Intel X5650, NVidia Tesla C2050 GPU / 2010Dawning1206401271.002984.303DOE/NNSA/LANLUnited StatesRoadrunner - BladeCenter QS22/LS21 Cluster, PowerXCell 8i 3.2 Ghz / Opteron DC 1.8 GHz, Voltaire Infiniband / 2009IBM1224001042.001375.782345.504National Institute for Computational Sciences/University of TennesseeUnited StatesKraken XT5 - Cray XT5-HE Opteron Six Core 2.6 GHz / 2009Cray Inc.98928831.701028.855Forschungszentrum Juelich (FZJ)GermanyJUGENE - Blue Gene/P Solution / 2009IBM294912825.501002.702268.0015KISTI Supercomputing CenterKorea, SouthTachyonII - Sun Blade x6048, X6275, IB QDR M9 switch, Sun HPC stack Linux edition / 2009Sun Microsystems26232274.80307.441275.9616University of EdinburghUnited KingdomHECToR - Cray XT6m 12-Core 2.1 GHz / 2010Cray Inc.43660274.70366.74

1988: The 1st Super Computer in Korea Cray-2S (2GFlops)

1993: Cray C90 (16GFlops)

2001: IBM p690(655.6GFlops) NEC SX-6(160GFlope) IBM p690+ (2,655GFlops)

2009: IBM p595 (5.9TFlops) SUN B6048 (24TFlops)

7KISTI Super Computing Center Mission

Test-bed CenterValue-adding CenterResource CenterTechnology CenterHelp Korea research communities to be equipped with proper knowledge of super computingKeep securing and providing world-classsuper computing systemsMake the best use ofWhat the center has to create the new valuesValidate newly emerging concepts, ideas, toolsand systemExpanding the Horizon of Science and Technology8Global Research Networks

GLORIADGLObal RIng Network for Advanced Applications Development with 10/40Gbps Optical lambda networkingConsortium of 11 Nations: Korea, USA, China, Russia, Canada, the Netherlands and 5 Nordic CountriesSupporting Advanced Application Developments such as HEP, Astronomy, Earth System, Bio-Medical, HDTV etc.Funded by MEST (Ministry of Education, Science and Technology) of KOREA

KREONETKorea Research Environment Open NETworkNational Science & Research Network of Korea, Funded by Government since 199820Gbps Backbone, 1 ~ 20Gbps Access Networks

9GSDCSupporting to various Data Intensive ResearchesMISSION

Physics2009 ~Next ~ALICESTARBelleCDFVariousBio-informaticsEarth ScienceAstrophysics

National Representative Playerin Data Centric Research Area10Current Status

ALICE Tier-1 Test-bedALICE Tier-2KIAF (KIsti Analysis Farm)RAW DataTier-1

RAW DataSupporting Data Centric Research Communities&Promotion of Research Collaboration Computing and Storage Infrastructure Technology Development Apply Grid Technology to legacy app.

supportKorean Gov.11ActivitiesALICE Tier-1 Test-bedCompleted set-up ALICE Tier-1 test-bed this yearWill provide official service in a few yearsALICE Tier-2Site availability: 98% since Feb. 2009BelleProviding computing resources for Belle MC production (Grid)Bell to provide their data to KISTI GSDCCDFProviding computing resources under NAMCAFSupporting CDFSoft developmentLIGOSet-up LIGO cluster test-bedGBrainPlanning to cooperate with global brain research project (mainly from McGill Univ. Canada)12MembersNoNameRole1Dr. Haengjin JangHead of GSDC2Dr. Hyungwoo ParkProject Management3Mr. Jinseung YuTechnical Staff (Network)4Mr. Heejun YoonTechnical Staff (DBA)5Dr. Beokyun KimTechnical Staff (Grid)6Dr. Christophe BonnaudTechnical Staff (Admin)7Dr. Seokmyun KwonTechnical Staff8Dr. Seo-Young NohTechnical Staff9Dr. Jonghu LeeTechnical Staff (STAR contact)10Dr. Seungyun YuPlanning11Mr. Kyungyun KimTechnical Staff12Mr. Seunghee LeeTechnical Staff13Ms. Tajia HanTechnical Staff13Current Computing Resources: Cluster ServerClusterSpecMem.NodeCorekSI2kCe-aliceDellIntel Xeon E5405 2.0 GHz Quad 2 CPU16GB64848Ce01HPIntel Xeon E5420 2.5GHz Quad 2 CPU16GB16128219Ce02IBMIntel Xeon E5450 3.0GHz Quad 2 CPU16GB38304650This YearIBMIntel Xeon X5650 2.66GHz 6 Core 2 CPU24GB364321,214Current Computing Resources:StorageYearModelDisk/TapePhysical SizeUsable Size2008NetApp FAS2050Disk(SAN)48TB30TB2009NetAppFAS6080Disk(SAN, NAS)334TB200TBThis YearHitachiUSP-VDisk(SAN)960TB600TBThis Year-Tape100TB100TBTotal1442TB930TB15Some Pictures

Newly delivered storage system in June16Network Architecture

8Gb SAN S/W.

10GbE S/W

SANU600TBFCFCController #1Controller #2

PFS Servers10GbE S/WGLORIADFCController 1,2SANU160TB2009: 8Gb FC: 1GbEthernet: 10GbEthernet

1G/10GbE S/W FCController 1,2NASU40TBNAS

NASWN Servers

1G Grid Servers 10G Grid Servers1G Grid Servers .PublicPrivate2010IBRIX Fusion Cluster File System172. Current Status SACC

Collaboration

Provide ServiceProvideComputing ResourcesSTAR Asian Users(China, India, etc.)KISTI GSDC will contribute STAR experiments.Particularly Asian user community18Network Architecture1F 2S/WDJ-F10OME6500OME6500OSRDJ_7609Cisco76091F FDF(3,4)Foundry 2402CF2F 1GFDF(3,4)FDF(3,4)VLAN 124IP : 134.75.238.1 / 26VLAN 233IP : 150.183.233.1 / 24G3/18G2/5G8/21G0/26T2/3G2/111GStarproject to BNLKISTI 2F LAB 3F FDF(3,4)Foundry SuperXGloriad-KR10G TrunkVlan 123, Vlan124VLAN 123IP : 134.75.123.1 / 2410G1G LightpathVlan12410G TrunkVlan 12310GF-1010G10G10G10G1G1G19Daejeon Seattle Performance Test(UDP)[root@seattle ~]# iperf -s -u -i 1-------------------------------------------------Server listening on UDP port 5001Receiving 1470 byte datagramsUDP buffer size: 107 KByte (default)-------------------------------------------------[3] local 134.75.205.21 port 5001 connected with 134.75.204.20 port 41490[3] 0.0- 1.0 sec 233 MBytes 1.96 Gbits/sec 0.006 ms 0/166343 (0%)[3] 1.0- 2.0 sec 233 MBytes 1.96 Gbits/sec 0.005 ms 0/166533 (0%)[3] 2.0- 3.0 sec 234 MBytes 1.96 Gbits/sec 0.005 ms 0/166643 (0%)[3] 3.0- 4.0 sec 234 MBytes 1.96 Gbits/sec 0.005 ms 0/166659 (0%)[3] 4.0- 5.0 sec 234 MBytes 1.96 Gbits/sec 0.007 ms 0/166652 (0%)[3] 5.0- 6.0 sec 234 MBytes 1.96 Gbits/sec 0.005 ms 0/166664 (0%)[3] 6.0- 7.0 sec 233 MBytes 1.96 Gbits/sec 0.006 ms 0/166363 (0%)[3] 7.0- 8.0 sec 234 MBytes 1.96 Gbits/sec 0.005 ms 0/166644 (0%)[3] 8.0- 9.0 sec 234 MBytes 1.96 Gbits/sec 0.006 ms 0/166663 (0%)[3] 0.0-10.0 sec 2.28 GBytes 1.96 Gbits/sec 0.008 ms 0/1665755 (0%)[root@localhost ~]# iperf -c 134.75.205.21 -u -b 1.8g -i 1 -t 10------------------------------------------------------------Client connecting to 134.75.205.21, UDP port 5001Sending 1470 byte datagramsUDP buffer size: 64.0 KByte (default)------------------------------------------------------------[3] local 134.75.204.20 port 41490 connected with 134.75.205.21 port 5001[ID] Interval Transfer Bandwidth[3] 0.0- 1.0 sec 233 MBytes 1.96 Gbits/sec[ID] Interval Transfer Bandwidth[3] 1.0- 2.0 sec 233 MBytes 1.96 Gbits/sec[ID] Interval Transfer Bandwidth[3] 2.0- 3.0 sec 234 MBytes 1.96 Gbits/sec[ID] Interval Transfer Bandwidth[3] 3.0- 4.0 sec 234 MBytes 1.96 Gbits/sec[ID] Interval Transfer Bandwidth[3] 4.0- 5.0 sec 234 MBytes 1.96 Gbits/sec[D] Interval Transfer Bandwidth[3] 5.0- 6.0 sec 234 MBytes 1.96 Gbits/sec[ID] Interval Transfer Bandwidth[3] 6.0- 7.0 sec 233 MBytes 1.96 Gbits/sec[ID] Interval Transfer Bandwidth[3] 7.0- 8.0 sec 234 MBytes 1.96 Gbits/sec[ID] Interval Transfer Bandwidth[3] 8.0- 9.0 sec 234 MBytes 1.96 Gbits/sec[ID] Interval Transfer Bandwidth[3] 0.0-10.0 sec 2.28 GBytes 1.96 Gbits/sec[3] Sent 1665755 datagrams[3] Server Report:[ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams[3] 0.0-10.0 sec 2.28 GBytes 1.96 Gbits/sec 0.008 ms 0/1665755 (0%)20Daejeon Seattle Performance Test(TCP)[root@seattle ~]# iperf -c 134.75.204.20 -i 1 -w 30m -t 60------------------------------------------------------------Client connecting to 134.75.204.20, TCP port 5001TCP window size: 60.0 MByte (WARNING: requested 30.0 MByte)------------------------------------------------------------[ 3] local 134.75.205.21 port 46831 connected with 134.75.204.20 port 5001[ 3] 0.0- 1.0 sec 52.3 MBytes 439 Mbits/sec[ 3] 1.0- 2.0 sec 0.00 Bytes 0.00 bits/sec[ 3] 2.0- 3.0 sec 0.00 Bytes 0.00 bits/sec[ 3] 3.0- 4.0 sec 0.00 Bytes 0.00 bits/sec[ 3] 4.0- 5.0 sec 0.00 Bytes 0.00 bits/sec[ 3] 5.0- 6.0 sec 20.9 MBytes 176 Mbits/sec

[ 3] 54.0-55.0 sec 267 MBytes 2.24 Gbits/sec[ 3] 55.0-56.0 sec 238 MBytes 1.99 Gbits/sec[ 3] 56.0-57.0 sec 270 MBytes 2.26 Gbits/sec[ 3] 57.0-58.0 sec 242 MBytes 2.03 Gbits/sec[ 3] 58.0-59.0 sec 239 MBytes 2.01 Gbits/sec[ 3] 59.0-60.0 sec 270 MBytes 2.27 Gbits/sec[ 3] 0.0-60.1 sec 8.84 GBytes 1.26 Gbits/sec

21Current ResourcesCPUsIntel Xeon E5450 300cores 960kSI2K (1core = 3.2kSI2K)shared queueStoragesDisk: 100TBTape: 100TBNetwork10G GLORIAD: ActiveSoftwareSL10i on the Scientific Linux 5.5 64bits22100TB Disks

23Resource Plan201020112012CPU (kSI2K)9602,0803,100StorageDisk (TB)100200300Tape (TB)100200300Network~10G (GLORID) ~10G (GLORID)~10G (GLORID)243. Future Plan of SACCBefore 2010

Jan. 2011

Summer 2011 Installation of additional CPUs (+430 cores) & Tests MOU between KISTI GSDC and STAR Collaboration Start user service for Asian users Collaborations between KISTI GSDC and Asian institutes RUN11 online data transfer from BNL to KISTI GSDC Installation of additional storages (100TB disk + 100TB tape)25

Our VisionGlobal Science Data PipelineAsian Hubfor Global Data centric research26Do You Have Any Questions????27