XRootD Release 4And Beyond

GSI SeminarStanford University/SLAC

July15, 2015

Andrew Hanushevsky, SLAC

http://xrootd.org

July 15, 2014 2GSI Seminar

User Visible Changes INew client library: libXrdCl.so First available in 3.3.0 now is default

Old deprecated client library is libXrdClient.so

Copy command: xrdcopy xrdcopy renamed to xrdcp Xrdcopy is now a symlink to xrdcp Original xrdcp renamed toxrdcp-old

All have same command line interface!

July 15, 2014 3GSI Seminar

User Visible Changes IINew meta command: xrdfs Logical replacement to xrd command

Different command line interface More user friendly and intuitive

When installing Release 4 RPM You must de-install previous XRootD

release They are incompatible

July 15, 2014 4GSI Seminar

Root SupportNew client is the default in ROOT Since 5.34.14 (Dec 14, 2013)

Via TNetXNGFile plug-in Setting envar XNet.UseOldClient to 1

Loads TNetXFile instead (old client)Root 6 does not support new client Due to conflicts with PROOF Expected to be resolved in release

6.1

July 15, 2014 5GSI Seminar

Admin Visible ChangesIPv6 supportPublic Private NetworkingDisk Caching ProxyHTTP Plug-inRemote DebuggingOptional Client Plug-insMiscellaneous

July 15, 2014 6GSI Seminar

IPv6 SupportThe new client supports IPv6 It prefers IPv6 whenever possible

This corresponds to HEPIX request Connects using mapped IPv4 addr if

need beThe old client is deprecated It will not support IPv6

July 15, 2014 7GSI Seminar

IPv6 ConfigurationFrankly none needed Clients detect available network

stacks May force it to use various IP modes

Via configuration file or envars Server detects interfaces and DNS

settings It’s important that DNS is setup

correctly May force server to use IPv4 only

Via command line option (-I v4)

July 15, 2014 8GSI Seminar

IPv6 Considerations IWhile servers and redirectors can accept both IPv6 and IPv4 clients. . . IPv6 clients are assumed dual-stack

May be redirected to either IPv6 or IPv4 node

Client error recovery will resolve this if unworkable

This works if all the servers are dual-stack

Generally the preferred migration path Since redirection is via hostname…

DNS entry must have no un-routable entries

July 15, 2014 9GSI Seminar

IPv6 Considerations IIClient dual-stack assumption will be lifted in release 4.1 (3Q14) Clients will be redirected to compatible

nodes If none, the client receives an error

Largely driven by public-private networks

And interactions with IPv6/4

July 15, 2014 10GSI Seminar

Public-Private Network SupportRedirectors are now network cognizant Servers inform redirectors of usable

i/f’sClients always compatibly redirected Private to private and public to public

Subject to configured network topology Via new xrd.network directive option Applies to servers and redirectors

July 15, 2014 11GSI Seminar

Public-Private Net Topologiesxrd.network routes type [use if1[,if2]] type: local | common | split

local (default) No address differentiation (i.e. pre R4 mode)

common Private incoming -> private (preferred) or public Public incoming -> only public

split Incoming address must match outgoing address

If use unspecified addresses come from DNS!

July 15, 2014 12GSI Seminar

Private Addresses & DNSPrivate addresses should never be in DNS Unless

It’s a site local DNS server or The address is zone registered (i.e. only locally

available)

This is the assumption used by XRootD To avoid DNS reverse translation

timeoutsHence, the use option might be needed If a server connects using a private

address

July 15, 2014 13GSI Seminar

Public-Private ConsiderationsThe available server interfaces Must be uniform within a cluster Must be compatible with configured

topologyRestrictions relaxed in R 4.1 Redirectors will match clients & server

i/f’s May lead to inaccessible files if no match

exists Still working through external access

issues May require a separate redirector for

external access Due to IPv6/4 and public-private network

interactions

July 15, 2014 14GSI Seminar

Networking Is Complex!There are many combinations now Client and server capabilities must now

matchThere are 4 basic combinationsPublic Private

IPv6IPv4But things are not that simple!

Dual stack clients add another 4 combinations

July 15, 2014 15GSI Seminar

Disk Caching ProxyNew proxy server mode Configured via pss.cachelib directiveCaches whole files or file segments Mode is configurableCached content available for future access Until LRU purged (configurable)Many use cases to increase access speed

July 15, 2014 16GSI Seminar

Typical Disk Caching Proxy Uses

LocalClients

CachingProxy

Remote XRootDClusters

Speed upRemote Access

CachingProxy

Speed upHD AccessSSD

XRootDCluster FS

CachingProxy

Speed upRandom Access

XRootDServer HDFS

July 15, 2014 17GSI Seminar

HTTP Plug-inBasic http, https, WebDav access Suitable for browsers, curl, wget, &

davix Provides another mode of well-known

access http is neither low latency nor high-

performance Google & Microsoft have proposed improvements

• Changes submitted to W3C and IETF as http2• Improvements are considered incremental

• They only address the most vexing problems

Configured via xrd.protocol directive And specialized http.xxx directives

July 15, 2014 18GSI Seminar

XRootD Multi-Protocol SupportAlways supported multiple protocols Improved architecture makes it much

easier New protocols can leverage XRootD

features Security, monitoring, file system plug-ins,

etc

XRootDServerXRootDBridge

LoadableProtocol

Clients Memory BasedProtocol Conversion

July 15, 2014 19GSI Seminar

Remote Debugging via DigFSXRootD provided pseudo file system Provides restricted selectable R/O

access to Configuration file Log files Core files /proc/self (Linux only)

Has authentication & authorization options

Including access control restrictions View is standardized regardless of

location Configured via xrootd.diglib directive

July 15, 2014 20GSI Seminar

core/cmsdcore/xrootd

The DigFS View

/=/

logs/cmsdlogs/xrootdproc/cmsdproc/xrootd

confconf/etc (site specific)Virtual

exported path

July 15, 2014 21GSI Seminar

DigFS AuthorizationDigFS consults authorization file Created by the site and specified in

config file xrootd.diglib * authfileall 

[-]conf[-]core[-]logs[-]proc

allow

gsihostkrb5pwdsss unix

g=grouph=hostn=nameo=orgr=role

++ÖÖÖÖÖ

Ö ÖÖ

ÖÖÖ

July 15, 2014 22GSI Seminar

XRootD Client Relationships

xrdcopy

FTS3

CopyProcess xrdfs

XrdClFilesystem

PyXRootD

Dirac

root

Gaudi

Athena

CMSSW

PROOF

XrdClFile

XrdClPostMaster

XRootD - coreXRootD - add onExternal Package

XRootD – plug-ins

Plug-ins replace original implementation All calls may be replaced

All layers above benefit with any code changes

July 15, 2014 23GSI Seminar

Client Plug-insPlug-ins are loaded at run-time Client looks for plug-in configuration

files 1st Locally:

~/.xrootd/client.plugins.d/ 2nd Globally:

/etc/xrootd/client.plugins.d/ Both locations can be over-ridden via

envar XRD_PLUGINCONFDIR

Plug-ins are strictly version checked Allows for independent development

July 15, 2014 24GSI Seminar

Miscellaneous IReadv proxy pass-through Automatic & greatly improves

performanceManual log file rotation (e.g. logrotate) Via extended –k command line optionHigh precision log file timestamps New –z command line option

Log timestamp appears in microsecond format

July 15, 2014 25GSI Seminar

Miscellaneous IISpecial stat() plug-in for odd file systems Configured via the oss.statlib directive

Includes plug-in for GPFS with tape backend

Cancellable third party copy Used by transfer toolsFast directory listings Stat info can now be included w/ dir

entry xrdfs uses this to speed up long listings

July 15, 2014 26GSI Seminar

Miscellaneous IIINew query config options Use the xrdfs command to display

Query cms - shows cmsd status Query role - shows server’s cluster

role Query sitename - shows the site’s name Query version - shows server’s version

Cluster node blacklisting Via cms.blacklist directive & blacklist

file Useful in federated environments

July 15, 2014 27GSI Seminar

Miscellaneous IVNew monitoring information User login record now also includes

Name of the client’s executable Contents of client’s XRD_MONINFO envar

Useful to tie external information to actual data usage

E.g. the Panda jobid to cross-reference I/O usage

July 15, 2014 28GSI Seminar

Looking Beyond Release 4Cross Protocol RedirectionMeta-linksI/O Throttling Plug-in

July 15, 2014 29GSI Seminar

Cross-Protocol Redirections IXRootD protocol is capable of redirecting to a protocol other than xroot On file open the server may tell the

client that it’s more efficient to try something else

E.g. read the file locally from disk Open(xroot://host/filename) -> redirect

file://filename

July 15, 2014 30GSI Seminar

Cross-Protocol Redirections IINew client already capable of processing out-of protocol redirections Server needs some development to do soRoot’s TFile framework needs updating to handle a change in protocols Changes already in development

Target for root 6.1 or 6.2

July 15, 2014 31GSI Seminar

Meta-Link Files IXML file that describes one or more data files available for access Meta-link file identified by dot suffix

metalink (v3) or meta4 (v4 & incompatible w/ v3)

File is read and parsed by the client Client picks one based on certain criteria

E.G. priority, location, etc If access fails, client can pick another one

July 15, 2014 32GSI Seminar

Metalink Files IIXRootD client will support meta-link files But the road is not straight-forward

Incompatible meta-link formats, which one? Need to avoid encumbering installation

I.e. pre-reqs for sites that don’t care about meta-links

Targeting availability in 4Q14

July 15, 2014 33GSI Seminar

I/O Throttling Plug-inAllows site to limit I/O access to disk Useful for throttling external access

Used in federated environments via proxy server

Configured via xrootd.fslib directive And specific throttling directives

Plug-in is in code review phase Already used by CMS in productionTargeting 3Q14

July 15, 2014 34GSI Seminar

AcknowledgementsCurrent Software Contributors ATLAS: Doug Benjamin, Patrick McGuigan, Ilija Vukotic CERN: Lukasz Janyst, Andreas Peters, Justin Salmon Fermi: Tony Johnson Root: Gerri Ganis, Bertrand Bellenot SLAC: Andrew Hanushevsky, Wilko Kroeger, Daniel Wang, Wei

Yang UCSD: Matevz Tadel UNL: Brian Bockelman WLCG: Mattias Ellert, Fabrizio Furano, David Smith

US Department of Energy Contract DE-AC02-76SF00515 with Stanford University