AFS and Solaris - Roma Tre Universityafscon09/docs/feiler.pdfFrom Sun admin sudent guide (SA 225...
Transcript of AFS and Solaris - Roma Tre Universityafscon09/docs/feiler.pdfFrom Sun admin sudent guide (SA 225...
1Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Solaris
Why to use Solaris10+є (e.g. for AFS file server)
September 2009
Mathias FeilerUniversität Hohenheim
2Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Speed up
For faster walk through
You may want to
follow the (approx. 35) tagged slices only
3Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Why to use Solaris 10+є ?● Solaris on Sparc is rather solid
– We don't use to boot each year– HW and software provided by same company
● Development can stay more focused in limited HW options
● Iimho. the most innovative U*.*X at present – Zfs (the only contemporary disk file system) – Zones (a very effective way of virtualization)– SMF (faster boot , better normalized management) – Some more I don't use (too much)
● FMA, Dtrace, Role Based Access(RBAC), ...
5Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Known Issues● Service dependency is hell. Isn't it ?
– Not solved by RCorder numbers:/etc/rc?.d/SNNNNxxx● Works in serial and thus rather slow● No real knowledge of dependencies – just “sooner or later”
● Is it possible to unify management somewhat?● w/o fiddling the link names and rcfiles
● What if a service dies in middle of day?● Restart service? Reboot system?
– Initd cares halfhearted , but rc does not at all● What to do with dependent tasks ?
– Guess why AFSBOS was introduced● Can't a machine come up faster?
6Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
What is SMF● Service Management Facility
– Part of “Predictive SelfHealing”● “Fault Management Architecture” (FMA) is an other one.
– A method to describe, handle and monitor a service– Much more powerful than inittab, runlevel and
the /etc/rc?.d mechanism– LiteSQL database aided
7Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
What is SMF doing● Outperforms / replaces runlevel & /etc/rc?.d● Populates database from file (profile & manifest) ● Keep track of maintained services (Nanny)● Provides helpful assistance for management ● Works in parallel as far as possible
Think of a booting machine as defective since expected functionality is not given.
A healing mechanism (smf) detects the issue and tries to repair it honouring dependencies.
8Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
SMF first touch● /etc/rc?.d is alarming empty
● Did the installation really succeed ?● No /etc/rc3.d/S89sshd but its running anyway
– Who started it – and how – and why ?
● /etc/inittab is incredibly melt down● Where had all the runlevel been gone to?
● I kill a daemon but something strikes back● It is restarted immediately
● How can I stop this dirty service I don't like ? ● except of shut down the machine
9Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
My first expirience with SMF● Remote (SSH) tuning of /etc/ssh/sshd_config
> # Take care ! Don't saw the limb you are sitting on! > ps ef | grep "[Rr]oot.* 1 .*sshd" | cut c915
123> kill 123> /usr/lib/ssh/sshd ; tail /var/adm/messages
... sshd 2345 ... error: Bind to port 22 on :: failed: ...> # Hell, it won't die – try harder> kill 9 123
kill: (123) No such process> # ??! Ok, obviously meanwhile it died > /usr/lib/ssh/sshd ; tail /var/adm/messages
... sshd 2356 ... error: Bind to port 22 on :: failed: ...> # ? ... ??? ... Something is bloody strange .. ??? > ps ef | grep "[Rr]oot.* 1 .*sshd"
Root 2340 1 0 13:45:16 ? 0:00 /usr/lib/ssh/sshd
● HEY! Some one is doing my job w/o asking me – and even faster
10Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
SMF components● svc.startd – The SMF core started by initd● svc.configd – databaseinterface & management● LiteSQL – Database to handle all the knowledge● Manifest – A full service description (XML)
● Also contains default properties and values, methods,... ● Method file – similar to the old RCfile
● Program (or scripts) to start / restart / stop services● Use of contractfs
● A kernel fs for maintaining process”families” ● Administrative commands (see below)
11Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
SMF terminology● FMRI
● The URI of SMF & FMA (see below)● Service
● A big polymorphic object SMF is working with ● Milestone
● Used in a similar way like the old runlevel : “nameservice:default” , “devices:default” , “singleuser:default” , “multiuser:default” , “multiuserserver:default”
● Properties● Initially created on behave of manifest● Stored in DB, can hold all sort of adjustments
●
12Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
How SMF comes in?
Kernel initd
svc.startd(1M) svc.confd
svc:/network/login:rloginsvc:network/ftp
delegated restartersvc:/network/inetd
svc:/system/cron
inittab LiteSQLSMF
configuration repository
svc:/application/print/server
...
...
Contractfs
delegated restartersvc:/network/openafs/server:bos
13Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
FMRI the naming scheme● Fault Management Resource Identifier ● URI used by “Predictive Selfhealing”
● Which SMF is part of● Describes all sort of system objects
● services (svc) , devices(dev) , packages(pkg), memory(mem) , diagnostic engine(fmd), ....
● In general: fmrischeme://[authority]/path● Example of a SMFFMRI:
svc:/network/ssh:defaultusually parts of FMRI can be omitted e.g.:network/ssh
14Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
What is a service in terms of SMF?From Sun admin sudent guide (SA225S10) :
... “A service is an entity which provides a known list of capabilities to other local and remote services. Services are represented as instance nodes which are children of service nodes ... One service might have many instances. ...Both...can have properties...“
In my words: A abstract bunch of all the parts you need to describe + maintain + yield function.
Other, more formal, definition are available on the net
15Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
SMFadministrativ commands> svcs
– Monitoring of SMF ● Dependence , dependencies , problems , further hints
> svcadm● Switch on/off service , set tgtmilestone (confer runlevel)
> svccfg● Database and file centric management , properties,
manifest,...
> svcprop● Print properties e.g. used by scripts to learn options
> inetadm● Management of the delegated restareter “inetd”
16Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
> svcs > svcs aSTATE STIME FMRIlegacy_run Sep_14 lrc:/etc/rcS_d/S29wrsmcfglegacy_run Sep_14 lrc:/etc/rc2_d/S10lu...disabled Sep_14 svc:/network/iscsi_initiator:defaultdisabled Sep_14 svc:/network/smtp:sendmail...online Sep_14 svc:/milestone/multiuserserver:defaultonline Sep_14 svc:/network/openafs/client:default...offline 21:21:03 svc:/application/print/rfc1179:default
17Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
> svcadm + svcs > svcadm enable svc:/application/print/rfc1179:default
> svcs xvsvc:/application/print/server:default (LP print server) State: disabled since 14 September 2009 11:13:34 CESTReason: Disabled by an administrator. See: http://sun.com/msg/SMF800005 See: man M /usr/share/man s 1M lpschedImpact: 1 dependent service is not running: svc:/application/print/rfc1179:default
> svcs l svc:/application/print/rfc1179:defaultfmri svc:/application/print/rfc1179:defaultname BSD print protocol adapterenabled truestate offlinenext_state nonestate_time 24 September 2009 21:21:03 CESTrestarter svc:/network/inetd:defaultdependency require_all/refresh svc:/application/print/server (disabled)
> svcadm disable svc:/application/print/rfc1179:default
> svcs x
18Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
> svcs dependencies
> # the D shows what services or milestone bank on this on > svcs D svc:/application/print/server:defaultSTATE STIME FMRIdisabled Sep_14 svc:/application/print/ipplistener:defaultonline Sep_14 svc:/milestone/multiuser:defaultoffline 21:21:03 svc:/application/print/rfc1179:default
> # the d shows what is needed to run this service > svcs d svc:/application/print/server:defaultSTATE STIME FMRIonline Sep_14 svc:/system/filesystem/usr:defaultonline Sep_14 svc:/system/identity:domainonline Sep_14 svc:/system/filesystem/local:defaultonline Sep_14 svc:/system/systemlog:default
19Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
> svcprop>svcprop p config svc:/network/openafs/client:defaultconfig/ExtraAfsdOpts astring ""config/LegacyCfg astring ""config/Notice astring V0.1/Feilerconfig/SizeModel integer 3config/afsdb boolean trueconfig/backuptree boolean trueconfig/blocks astring ""config/cachedir astring ""config/confdir astring ""config/dynroot boolean falseconfig/fakestat boolean trueconfig/fakestatall boolean falseconfig/memcache boolean falseconfig/mountdir astring ""config/rootvol astring root.afs.readonlyconfig/ThisCell astring unihohenheim.de....
> svcprop p config/ThisCell openafs/clientunihohenheim.de
20Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Modify properties – Set a singe value > svccfg -s openafs/client setprop config/SizeModel = 3
– Delete a propertyvalue > svccfg -s openafs/client delpropval config/LegacyCfg \*
– Delete a property> svccfg -s openafs/client delprop config/LegacyCfg
– Stdout of methode shall make it to the log file > svccfg -s openafs/client set -v
– Force changes to become effective > svcadm -v refresh openafs/client
21Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
SMF:Manifest Example 1<!DOCTYPE service_bundle SYSTEM "./service_bundle.dtd.1" ><!
ident "@(#)openafszpool.xml 0.1 08/04/19" Copyright 04.2008 M. Feiler, University of Hohenheim Germany
All rights reserved. Use is subject to license terms.This Sevice is introduced for export the zpools used by the zoned afs fileserver.
><!The bundel name usually is <packet>.<servicename> ><service_bundle type="manifest" name="openafs:zpool"><service name="system/openafs/zpool" type="service" version="1"><create_default_instance enabled="false"/><! multiuser complies the archaic runlevel 2 ><dependency name="multiuser" type="service" grouping="require_all" restart_on="none"> <service_fmri value="svc:/milestone/multiuser"/></dependency>
22Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
SMF:Manifest Example 2<dependent name="zone_multiuserserver" grouping="optional_all" restart_on="none"> <service_fmri value="svc:/system/zones"/></dependent><exec_method type="method" name="start" exec="/lib/svc/method/openafszpool.sh %m" timeout_seconds="30"/><exec_method type="method" name="stop" exec="/lib/svc/method/openafszpool.sh %m" timeout_seconds="300"/><exec_method type="method" name="refresh" exec="/lib/svc/method/openafszpool.sh %m" timeout_seconds="300"/>
<! ptoperty types: boolean,astring,count (e.g. timeout),integer ><property_group name="config" type="framework"> <! The grepsearchpattern to find Afszpools > <propval name="zpoolname" type="astring" value="afspool"/></property_group>
23Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
SMF:Manifest Example 3<! Tell the SMF/startd how to treat us ><property_group name="startd" type="framework"> <propval name="duration" type="astring" value="transient"/> <propval name="ignore_error" type="astring" value="core,signal"/></property_group><property_group name="general" type="framework"> <propval name="action_authorization" type="astring" value="solaris.smf.manage.afs"/></property_group><template> <common_name> <loctext xml:lang="C">OpenAFS zpool </loctext> </common_name> <documentation> <doc_link name="Homepage" uri="http://www.google.de"/> </documentation></template></service></service_bundle>
24Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
You might think now..● Why for heaven's sake a “registry” for Solaris ?
How else to keep track of and manage dependencies?● Why on earth XML ?
It is oblong yes, but it has also DTD's1) which helps a lot ● Cant it be made more easy, native , evident ?
SMF is just a part of a hole structure design. You surely want to have a (and keep) a unified way to talk with it.
● How to switch off the beast ? Recreate the RCfiles, refill inittab , disable most of the smf
services , recompile.... – stop, no, don't do it !
1) DTD = document type definition = what is where allowed/required
25Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Why not to switch off SMF?● Automated fault detection and repair
– usually w/o boring You.– Even dependent services are considered.
● All get logged but does not flood screen ● boot works in parallel most of the way● Switching on/off a service persistently is easy
svcadm disable ssh ; svcadm enable ssh● It is aware of Solaris RBAC.
– SMFtasks can be delegated● Configuration roll back possible (DBsnapshot)● ....
27Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
What is ZFS● Zeta File system (not “Zetta”=10²¹ or Zebi=270
● New file system implementation of Sun– Full dynamical file system – Reminds to MMU in terms of RAM
● Like file system + volume managaer– e.g UFS + SVM or VxFS + VxVM ...– However much better than that
28Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
ZFS big picture
1t 1t2.3t
2 striped disksMirrored with a partition
2t
0.3t
USB
ZfsZfs1 ZfsZfs2ZfsZfs3
ZfsZfs a ZfsZfs b
2 Zpools
7 ZFSdatasets(~Filesystems)
4 Block storagedevices
Operating system / Application
ZfsZfs XZfsZfs c
29Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Layer overview
Taken from I don't know where on the Internet seams to be property of Sun microsystems
30Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Problems of common file systems● Data corruption is always possible
– Journalling helps much , but by far not all ● Often does not cover payload data blocks● Focus poor problems: power outage , system crash
– Reread is important (disk firmware?) but takes time– check sums are usually somewhat week
● Said to detect typically only 2 1 of 2 failure ⁸ ⁸● Covers data on disk but not data on the run
● Fsck – Assumes faults to occur on system start only – Usually not designed to work while in service– On larger FS it can take ages while not in service
31Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Problems of common FS / .*VM● No much optimization possible
– Volume manager does not know data structure● Trouble with write optimization and error detection● Suitable compression is rather difficult
– File system does not know about mirror ● Trouble with error correction using mirror ● (Brain based) human intervention required
● inconvenient & time consuming administration get PV's , build PVG's holding PE's of each PV create LV's by LE's using of PE's of different PVG's, get FS on LVPartition, configure vfstab, tread boot parti. – did I miss something of the nightmare?
32Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Some ZFS design goals● Administration – easy and evident
● No config file at all , only 2 commands ● Pooled storage
● File system “knows” of its multiple disks● Hierarchical space distribution + inherited parameter● Treat disk space dynamically, almost like RAM
● End to end data integrity ● Reasonable check sum for every block/file ● Automatically access mirror disk + self healing
● Transactions only● No fsck at all , scrubbing is possible while on line● Get better performance , less downtime
33Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
ZFS and the future● The only 128 Bit file system I know
– Max file system size 2¹² ( 2 by Posix API limit) ⁸ ⁶⁴● UFS can do 2 2 ³ Byte ( = 16 TiB8ZiB )⁴⁴ ⁷
● Easy Migration of ZFS – To different machines – To different architecture – NFS ready/compliant– ISCSI ready/prepared
34Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
ZFS data migration● ZFS open eggs either way ¹ ⁾
● ZFS interprets either endian (e.g. like TIFF)● Writes in host byte order ● Thus you can cross mount w/o problems for ZFS
● ZFS stores all management info along with data● Label , device path , mount point , NFSshare,... ● One central point to manage ZFS (in ZFS) ● ZFS converges even on change of disk device file
– e.g. when moved to a different controler / system
● Diskmig: zfs export , <alteration> , zfs import 1) Jonatah Swift: Guliver's travel : Likiput vs. Blefuscu
35Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
ZFS and storage● ZFS accepts (loves cheep) blockdevices
● disk , partition , usbstick , .... and files● no partitioning needed when on hole disk ● You can set mirroring posthumous
● Dynamically spend datasets (ZFS) when needed– Think in terms of work flow and logical grouping
● Stop thinking in terms of bytes and partition– Much like AFS volumes
● No fragmentation, no garbage , no waste.● Error handling works coordinated on all layers
– Self healing is possible easily
36Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Common Volume/FS Modell
*xVM
FS
Application
*xVM
FS
Application
*xVM
FS
Application
1. Read fails 2a. Wrong meta data :FS driver got troubled !Perform system crash ?
2a. Wrong user data :App got troubled !Calculation reliable ? Elect George W. Bush?
37Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
ZFS and self healing
ZFS mirror
Application
ZFS mirror
Application
ZFS mirror
Application
1. Read fails 2. ZFS reads mirror due to false detection by CRC check sum test
3. App gets good dataand
ZFS fixes bad blocks
38Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
ZFS uses copy on write● Basically well known from AFS● ZFS introduces the “Uberblocks”
● 128 in cyclic use, 4 copies each, CRC and SN ensured● Recall
– Payload is written to free blocks on disk– Associated meta data is written to free blocks too
● Pointing to the newly written Payload– Now changes (transaction) get committed
● Writing new version of Uberblocks pointing at the new tree – The original data tree is still there.– Occasionally used by snapshots – Clean up if neither used nor referenced
39Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
ZFS and memory● Per default ZFS takes all RAM but 1GB
– Occasionally 1GB of RAM could be insufficient● Virtual machines or zones (see prstat Z' )● Applications keen on memory● Cache of other file systems
● Perhaps limit ZFS' s ARC– ARC = Adaptive Replacement Cache
● See : Solarisinternals Wiki– Such a limit can
● reduces swapping● increase speed of other file systems
● Example: / etc/ system: set zfs:zfs_arc_max = 0x780000000
40Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
ZFS integrity● Transactions
● Make use of “copy on write” + “uberblock” ● Implies no need for fsck● Scrub can check & repair on line (using transactions)
● CRC check sum ● All data● End to end
– Not only while on disk bur also while on way ● Never trust the cat to keep the cream
– Check sum is not held by the summed data itself– Check sum is held within the referring structure
● Raidz ● Raid w/o “write hole” ability ... by transactions
41Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
ZFS got also...● Almost all works on line
● Compression on/of , Block size adjustment , stripe size adjustment , Storage modification , FSresizing,...
● Modern I/O optimization ● Pipelined I/O , priority ● multiuser prefetch – something cool for streaming people
● Snapshot and cloning● NT and NFSv4 ACLs ● Encryption (Solaris Security Framework)● NFS and CIFS support● Booting from ZFS is ok now (on either endian)
42Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
ZFS conclusion● Easy
● Just does what admin expects to● Powerful
● Incorporates most modern concepts ● Save and secure
● Detects and repairs silent data corruptions● Fast
● Dynamic striping, less Disk overhead , ...● Open
● No company can wash away (like Linux)● For free
43Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
ZFS for short● No limits ahead (128 bit)● No endian issue ● Robust (self healing )● Quite fast ● Transactions● CRC end to end ● No fsck ● On line compression● On line tuning /configuation
● Eazy administration● Integrated volume manager● Dynamic storage distribution● Scores of ZFS in zero time● No vfstab / dfstab● Quota & Reservation● Snapshot & cloning● NTACLs● No solid locking (SAN)
45Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Introduction : Solaris Zones● There is no modern Solaris w/o a main zone
– The “global” zone is always there● The Solaris isolation primitive
– Application execution environment– Isolated process hierarchy (processtree)
● A Solaris way of virtualization (partitioning)– Changeroot environment– Got its roots in BSDJail (a decade ago)
● Advanced and improved quite a lot since then
46Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Solaris Zone flavours ● Four main flavour available
– Global zone ● System management
– Sparse zone● shares as much as possible
– Branded zone (whole root zone / brandz)● shares as less as possible● Can hold/run other OS as well● Logically not really to be deemed special or unique
– Container ● also adds resource management to the zone concept
47Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Solaris Global Zone● The primary zone of the host system● formerly: ' the system '● Hardware boots into the global zone● There is exact one global zone per phy.Host ● Always called “global” ● System wide administration● Management of all none global zones
– eg. configure,install, boot, shut down, ...
48Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Solaris Sparse Zone● Share most of the host OS installation
● Thus sticks to the host OS and OS level ● Limited configuration ability
● Got only a few bits in private● /etc , /var , /tmp and such – the image stays small● IPaddress
● Share most of the kernel ● Only one dispatcher, ● Only one memory management, ● Only one driverset● Spot on : This means only one AFSKernel module !
● Needs minimum of system power to run
49Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Solaris Zones – Virtualization ● Imho leanest virtualization possible
– „application virtualization“ or „Light weight virtual machine“● Virtual Page Management & swapd● Scheduler● Helper
specialhardware
OS1 OS2
hardware
OsA OsB
VMware
hardware
Solaris
Zone1 Zone2
App
2
Help
er
App 1 App 2
Helper
Har
dwar
evir
tual
isie
rung
OS
Vir
tual
izat
ion
App
licat
ion
virt
ualiz
atio
n
App
1
Help
er
App
2
Help
er
App
1
Help
er
50Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Solaris Zone – physical view● 23 Configuration files within the host ● A dedicated “root” file system (file set)
– Prepared with some important files – Most files blend in from the Hostinstallation (RO)
● Solaris resource management framework● A global process for management
zoneadmd
● Some (710) system processes within the zone zsched /sbin/init /lib/svc/bin/svc.startd /lib/svc/bin/svc.configd/usr/lib/utmpd /usr/lib/saf/ttymon/usr/lib/saf/sac /usr/sbin/syslogd
see 'ps efZ' from within the globals zone
51Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Sparse Zones : Pro & Contra● Wanted
– Modern most economical way of virtualization ● Avoid superfluous redundancy = minimum overhead
– Easy to replicate ● To a second instance ● To a second host (with same OS)
● Traded in– closely tied to the host system
● All zones got virtually the same patch level● All zones share most of the binaries & libraries
– Patching might take significant longer. – Defining a zone is somewhat strange – ask Google
52Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Why do I use Solaris● Because it
– Performs well enough in terms of my spots
– Tend to solve my problems a reasonable way
– Does not bring in much of secondary problems
● e.g. no metadbpartitions for mirroring
– Make me feel comfortable
54Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Solaris
What is the point in using all of that
Solaris stuff with
AFS file server
Since I talked on that last year in GrazI don't go into that too deep now
55Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Known AFS charakteristics● Afs favour 'many small ' over 'one huge'
– design goal (see e.g. 'vos move' ) – method of load distribution
● SysID , DirtyFlag , ... (meta data)– logically belongs to the payload (vice partition)
● e.g. a SysID should tag a set of data not a host– however it is stored on a server directory
● Data and meta data can accidentally end up on different hosts
● Or can be left behind
56Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
AFS client and Solaris● Afs cache cannot use native ZFS
– Use a “zvol” (rawdataset) instead and it works just fine.
(how ever this is where ufs and vfstab steps in again)
● Sparse zones share the kernel with the host– Tokens held in Kernel are virtually available on all
zones.– Root personally may differ over zones – Thus take care to use pagshell when root
57Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
AFS and SMF● Since I got a correct AFSmanifests and suitable
AFSmethods it works like a charm for both, the server and client.
● Configuration is stored within SMF properties● Optionally cacheinfo and ThisCell is written out using
those propertyvalues ● afsd get options according to properties
● The logically same is true for the server side● However some more is needed also
58Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
AFS file server + ZFS● Compression : speed up + save storage
● Compression ratio is 1.1 ... 1.3 for user volumes
● No waste on fsck (and no salvage in my case) ● No fiddling with device files or vfstab
(in order to mount partitions always and on invariant place)
● The hierarchical concept of zpools/datasets allows to have the interpreter stored along with associated payload. (e.g. Openafs / OSD)
59Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
AFS fileserver + Zones + ZFS● Example 1:
Cut one 5TiB server into 10 virtual:= 10 * (500 GiB + OpenAFS server installation) Always keep at least one instance totally empty
● Now You can ... while on service:
● Suppress all but AFS ports on the server IP address ● Update afs version of a zone (with vols moved aside)● Convert format by vos move (e.g. common to osd) ● Test a construct on a real live server● Starting can be designed to be parallel or unfair● Give real server training space to novices ● Add vice partition by using a further zone
60Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
AFSfs + Zones + ZFS from SAN● Example 2: like example 1 but also
– Storage is on SAN as handled by x Zpools.– Use two machines (twins), zones are installed on both
concurrently – Interpreter+config (e.g. sysid, NetInfo) is stored
together with data (same zpool but different zfs) ● Now You can :
● Do some load distribution / balancing ● Recover from a host hardware crash much faster
– No truble with IP, SysID, VLDB● ....
61Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Yes I got it running● Yes , I got that running.
● 4 * Hardware = 2 * twins● 20 * 2 Zones ● 20 * Zpool● 2 * Netapp filer + 1* InfotrendEurostor raid ● ZFS mirroring anyway
● Works for more than a year now – Got one case due to a system disk outage
● Trouble in spit of using “Solaris Disk Suite” ● Affected zones started on the twin – all works fine
● 3 of 4 Server now relay on ZFS exclusively ● Test are encouraging ● The last on should follow soon
62Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Some known issues● ZFSNetapp issue
● Solved by using old VTOC (avoid EFILabel) ● Shut down time out problem
● Solved by tuning zonemethod (Sun learns, but slowly) ● AFS client cache & ZFS
● Solved by using zvol● Export of wildroot zpools ( zpool-root != / )
● Solved by introduce a further service ( manifest example)→● No autotakeover or autostart right now
● Solved by human resources ● Which host serves a particular zone
● Patched zonemethod to write host name into zones
63Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Other dark corners● ARPcache time out issue on fail over
● Just flush the arptable when starting/stoping a zone
● No inherent solid locking of Zpools ● Badly needed for robust automated startup and failover
● SoftwareInstallation (e.g. Update) wants to have zones running – but is working w/o anyway
● No real problem up to now
64Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
What could come up next● Auto / scheduled replication
– Check whether customer would like it● e.g. for web pages ● Maybe it could cool some hotspots
– Present conception● Spent a file server zone ● Initially replicate designated volumes to this zone
– smaller volumes are preferred (short lock times) ● Run the following program within the zone
– Check all hosted volumes for date(update)>date(release)● Release those volumes
– Sleep a few minutes – Repeat from start
65Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
abstract request
Virtualization lead to think a slightly different way ...and thus gain me to
ask again for unique vice partition IDs stored on the partition
instead of have server IDs laying around on visited OS instances
66Feiler@unihohenheim.de Why do I use Solaris for OpenAFS
Thank You
Thank You for Yourkind attention
Mathias Feiler
feiler@unihohenheim.de