dCache at Tier3Joe UrbanskiUniversity of Chicago

US ATLAS Tier3/Tier2 Meeting, BloomingtonJune 20, 2007

What is dCache?From dCache.org:

"A system for storing and retrieving huge amounts of data, distributed among a large number of heterogenous server nodes, under a single virtual filesystem tree

Features & AdvantagesWhat can dCache do?Capable of combining hundreds of commodity disk servers to get a huge petabyte scale data storeAllows several copies of a single file for distributed data accessHas internal load balancing using cost metrics and transfers between the site's poolsHas automatic file replication on high load ("hotspot detection")

What does dCache look like?The single virtual filesystem is provided by pnfs (Pretty Normal File System).Mounting and unmounting through mount and umount and /etc/fstab, much like traditional nfs.pnfs is POSIX-like:can use: ls, mkdir, findcannot use: cp, md5sum

ClientsHow do I access files in dCache?

dCap: dCache's native method, uses dccp. Easiest for local transfers.

gridFTP: via globus-url-copy with a valid X.509 proxy

SRM: via srmcp, srmls with a valid X.509 proxy

ArchitectureWhat about the backend?Admin nodes: Provide basic admin services. One or more of these.Door nodes: Provide I/O access via SRM or GridFTP. One or more, may reside on admin node on small installs.pnfs node: Provides the unified namespace. Only one per install.Pool nodes: Provides storage. Can be installed alongside any other type of node.

UC Tier3

UC Tier3 (cont'd)3 Admin nodes:uct3-edge1: gridFTP, dCapuct3-edge2: pnfsuct3-edge3: admin, SRM

25 Pool nodes:Currently: 22 compute nodes x 1.9TB + 3 admin nodes x 1.9TB = 47TB

Installing dCacheThe VDT InstallerGreatly simplifies and quickens the install processAutomatically configures which services to run, and on what nodes to run them.Installs needed rpms.Configures dCache and its postgresql databases.Latest vdt version is v1.1.8Not to be confused with the dCache version, latest is v1.7.0.

Running the VDT InstallerDownload the latest vdt tarball, untar and cd into the install directory.Run node-config.pl to generate site-info.def file

config-node.pl[root@uct3-edge2 install] ./config-node.plHow many admin nodes (non-pool and non-door nodes) do you have? 2The recommended services for node 1 are: lmDomain poolManager adminDoor httpDomain utilityDomain gplazmaService infoProvider srm The recommended services for node 2 are: pnfsManager dirDomain

Enter the FQDN for the node 1: uct3-edge3.uchicago.eduWhich services do you wish to run on node uct2-dc3.uchicago.edu (Enter for defaults)? Enter the FQDN for the node 2: uct3-edge2.uchicago.eduWhich services do you wish to run on node uct2-dc2.uchicago.edu (Enter for defaults)?

How many door nodes do you have? 1Enter the FQDN of door number 1: uct3-edge1.uchicago.edu

node-config.pl (cont'd)Enter the private network that the pools are in.If this does not apply, just press enter to skip: 10.2.0.0Enter the number of dcap doors to run on each door node [default 1]: 1Enter a pool FQDN name(Press Enter when all are done): uct3-c001.mwt2.orgEnter the first storage location (Press Enter when all are done)): /dcacheEnter another storage location (Press Enter when all are done)): --SNIP--Enter another pool FQDN name(Press Enter when all are done): uct3-edge3.mwt2.orgEnter the first storage location (Press Enter when all are done)): /dcacheEnter another storage location (Press Enter when all are done)): Enter another pool FQDN name(Press Enter when all are done): Created site-info.def file.[root@uct2-mgt install]

Running the VDT Installer(cont'd)Copy vdt tarball, and site-info.def file to all nodes.Run './install.sh -d' for a dryrun. (This will be very verbose). If successful, run the actual install. Start dCache services in the following order:pnfs node core servicesother admin nodes core servicesall dcache pool services

Verifying the installCheck the status webpage:http://uct3-edge3.uchicago.edu:2288/cellInfo

Verifying the install (cont'd)Test the doors:dCap: use dccp

gridFTP: use globus-url-copy

SRM: use srmcpglobus-url-copy -dbg \file:////tmp/test.file \gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/test.file dccp -d999 /tmp/test.file \ /pnfs/uchicago.edu/data/ddm1/test.filesrmcp -debug file:////tmp/test.file \srm://uct3-edge3.uchicago.edu:8443/pnfs/uchicago.edu/data/ddm1/test.file

TroubleshootingAlways check the status page!

Troubleshooting (cont'd)Check the logs: Most dCache cells: /var/log/*Domain.log. Each cell, or service, will generate a log with the appropriate nameSRM: /opt/d-cache/libexec/apache-tomcat-5.5.20/logs/catalina.outpnfs: /var/log/pnfsd.log, /var/log/pmountd.log, /var/log/dbserver.logdCache, The Book:http://www.dcache.org/manuals/Book/Ask for help:dCache.org support: support@dcache.orgOSG-storage: osg-storage@opensciencegrid.orgOSG Storage Activities Meeting: Every Thursday https://twiki.grid.iu.edu/twiki/bin/view/Storage/MeetingMinutes

UC Tier3 Pool Usage

Wormholespnfs provides a way to distribute configuration information to all directories in the pnfs filesystem via 'wormholes'.Accessed via the '.(config)()' subdirectory, which acts like a symlink to /pnfs/fs/admin/etc/config/By default, reading files like this is disabled, so you'll need to enable access to these files.Without being able to read that file, pnfs won't be able to find the dcap door, and dccp's won't work!WARNING: enabling access to these files empties the file!

Wormholes (cont'd)[root@uct3-edge2 dcache-upgrade-v1.1.4]# cd \ /pnfs/fs/admin/etc/config/dCache[root@uct3-edge2 dCache]# cat dcache.conf uct2-dc1.mwt2.org:22125[root@uct3-edge2 dCache] touch ".(fset)(dcache.conf)(io)(on)"[root@uct3-edge2 dCache] echo "uct3-edge1.mwt2.org:22125" > \ /pnfs/fs/admin/etc/config/dCache/dcache.conf

Authorization with gPLAZMAGrid-aware PLuggable AuthoriZation ManagementWorks in a manner similar to PAMFour available methods:kpwd: 'legacy method'. flat file maps DN's to a local username, then username to uid, gid, and rootpathgrid-mapfile: uses a grid-mapfile, then a second file, storage-authzdb, to map username to uid, gid, and rootpathgplazmalite-vorole-mapping: concatenate DN + Role, then provide uid, gid, and rootpath via storage-authzdb.saml-vo-mapping: uses GUMS to map to username, may provide uid, gid, and rootpath or via storage-authzdb.

The Admin InterfacedCache provides a shell-like interface accessed via ssh. [root@uct3-edge3 config]# ssh -c blowfish -p 22223 -1 admin@uct3-edge3 dCache Admin (VII) (user=admin)(local) admin > cd uct3-edge2_1(uct3-edge2_1) admin > pnfs register(uct3-edge2_1) admin > ..(local) admin > cd uct3-edge3_1(uct3-edge3_1) admin > pnfs register(uct3-edge3_1) admin > ..

Autovacuuming PostgresPfns database files could potentially reach a very large size and fill up your filesystem. To turn it on, uncomment all the entries in the AUTOVACUUM PARAMETERS section and change 'autovacuum = off' to 'on' in /var/lib/pgsql/data/postgresql.conf, then restart postgresql.

Useful URL'sdCache homepagehttp://www.dcache.org/VDT Installer homepagehttp://vdt.cs.wisc.edu/software/dcache/dcache/VDT Installer HOWTOhttp://vdt.cs.wisc.edu/extras/InstallingDcacheForOSG.htmlOSG Storage Activities Meetinghttps://twiki.grid.iu.edu/twiki/bin/view/Storage/MeetingMinutesOSG dCache overviewhttp://vdt.cs.wisc.edu/software/dcache/dcache/USATLAS dCache at BNLhttp://vdt.cs.wisc.edu/extras/InstallingDcacheForOSG.html

http://twiki.mwt2.org/bin/view/Admins/DCacheUpgradeTov114