Tips for Administering Complex Distributed Perforce EnvironmentsMichael MirmanPerforce Administrator

2

Overview

System Overview Shelved changes in Distributed Environment Using broker as a router: Routing challenges Some administration tasks• Expand capabilities of the –df option• Removing users• Regular database defragmentation• Controlled fail-over

3

System Overview

• ~1,800 users worldwide• ~15M archive files• ~90 hosts• Database ~0.6Tb on

each server• Single production

system

Commit server

Edge servers

perforce:1666P4broker (router)

perforce:1666P4broker (router)

perforce:1666P4broker (router)

perforce:1666P4broker (router)

4

System Overview (cont.)

Local Data CenterWorkspace servers

perforce:1666P4broker (router)perforce:1666

P4broker (router)

perforce:1666P4broker (router)

perforce:1666P4broker (router)

Commit server

Archive on NFS disk

Edge servers

Remote Data Center

Archive on a local disk

Archive on a local disk

WkspXXWkspXXStandbyWkspXXUtility

EdgeXXEdgeXXStandby

5

Shelved Changes in Distributed Environment

Users want transparency using shelves Limitation in Distributed Environment to the

originated server:p4 changes –s shelved //some/path/...

Work around:1. Promote all shelves2. Broker redirection for shelved changes query

6

Promoting Shelved Changes in Distributed Environment Edge server shares the archive with the commit server:• EdgeServer#lbr.replication=shared• EdgeServer#startup.1 = pull -i 1 -J /path/to/journal

Edge server does not share the archive with the commit server:• EdgeServer#lbr.replication=readonly• EdgeServer#startup.1 = pull -i 1 -J /path/to/journal• EdgeServer#startup.2 = pull -u -i 1• EdgeServer#startup.3 = pull -u -i 1• EdgeServer#dm.shelve.promote=1

7

Broker Redirection for Shelved Changes in Distributed Environment p4 changes –s shelved //some/path/... works fine when

routed to the commit servertarget = commitserver:1666command: ^changes${ flags = -s; args = shelved; action = redirect; destination target;}

It would be nicer if we didn’t have to reroute this command

8

Routing caveats

Any command has client context

p4 clients –a –e clientname

p4broker Commit server

9

Routing caveats (cont.)

Regular unloading unused clients to conserve db space Occasional rebalancing edge servers requires removal and/or

rebuild of the routing tables Users without loaded clients don’t get into the routing table Problems for the user to come back• -> p4 reload -c myclient• Client 'myclient' is restricted to use on server 'WkspXX', not on server

'WkspYY'. Workaround: p4 -Zroute=WkspXX reload -c myclient

10

Creating a New Client in Distributed Environment Scenario 1

mkdir newdirecho P4CLIENT=myclient > $P4CONFIGp4 client# edit client spec, save; exit

Usually works fine When it doesn’t:• Current user does not exist in the routing table (default routing)

11

Creating a New Client in Distributed Environment (cont.) Scenario 2

cd /myfriend/directoryp4 client myclient# edit client spec (possibly including Root); save; exit

May have bad consequences if:• /myfriend/directory is a valid workspace, defining routing• This routing is not what current user has as the default• Future new clients may go to a different edge server• Not all clients shown in P4V; users must have multiple sessions• Seeing pending changes between clients is difficult or impossible

12

Routing caveats - Solution

Maintain “anchor” clients, which never get unloaded• Client .mmirman.anchor 2016/02/03 root /tmp/fake 'Anchor client to

ensure proper routing. Please don't delete. ' Rebuild routing tables with anchor clients Downside: user can see this client and considers it noise What we would love to see: ability to manage routing table

without rebuilding from scratch

Some Administration Tasks

14

Using –df option

Task: Allow usage of –df in “p4 change”, “p4 shelve”, “p4 client”, to the owners of corresponding items, as well as admins

Broker configuration: command: ^change$ { action = filter; flags = -d; execute = "/path/to/change.wrap"; }

15

Extensions for the –df option

Deleting client• Delete shelves or not?• We don’t like the -Fs option (save shelves): it creates clutter• Shelf cannot be deleted if its client does not exist anymore• Our implementation is a broker script with -Ds option

Deleting stream• p4 stream –df• Delete all clients, including unloaded• Allow only admins to do this• Still under discussion (big implications from client –df)

16

Removing user

Someone left company a reclaim Perforce license Remove user from Perforce groups Remove user spec ... Wait!• Remove user’s clients (including unloaded and locked)

- Remove shelved changes (shelve -d or -df); may require reverting files not resolved in other users’ clients

- Revert open files (p4 revert -k)- Disassociate changes from jobs- Remove pending changes (change -d or -df)

17

Regular database defragmentation

Have a regular schedule to maintain good performance

Example:• Rebuild all standbys at the beginning of the month• Controlled fail-over every second Sunday• Rebuild standbys again later in the month

18

Rebuild Standby servers

Task: Regularly rebuild standby servers to make them available for controlled fail-overs-> rebuild_scheduleSchedule to rebuild servers:Today (16) we rebuild these servers: Edge01Standby CommitStandbyDay 1: Edge01Standby CommitStandbyDay 2: Wksp00xxStandby Edge02StandbyDay 3: Wksp01xxStandby Wksp00yyStandbyDay 4: Wksp02xxStandby Wksp03xxStandbyDay 5: Wksp04xxStandby Wksp00xxUtilityDay 6: Wksp00yyUtility Wksp01xxUtilityDay 7: Wksp02xxUtility Wksp03xxUtilityDay 8: Wksp04xxUtilityDay 16: Edge01Standby CommitStandbyDay 17: Wksp00xxStandby Edge02StandbyDay 18: Wksp01xxStandby Wksp00yyStandbyDay 19: Wksp02xxStandby Wksp03xxStandbyDay 20: Wksp04xxStandby Wksp00xxUtilityDay 21: Wksp00yyUtility Wksp01xxUtilityDay 22: Wksp02xxUtility Wksp03xxUtilityDay 23: Wksp04xxUtility

19

Controlled Fail Over in Distributed Environment Shut down replication to the server to be failed over (p4 monitor

terminate) Shut down server and its standby (p4 admin stop) Change server.id in the root directories Remount the archive (r/o on the standby; r/w on the new server) Remove the state file (it will be automatically recreated later) Bring up the new server Bring up the new standby Restart other replications (if any) of the server

Q & AThank you!

Michael.Mirman@mathworks.com