Connections Lotusphere Worst Practices 2013
-
Upload
bill-buchan -
Category
Documents
-
view
3.245 -
download
2
Transcript of Connections Lotusphere Worst Practices 2013
![Page 1: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/1.jpg)
© 2013 IBM Corporation
BP108 Worst Practices…. Back from the depths of despair
Bill Buchan / HADSLPaul Mooney / Bluewave Technology
Monday, 4 February 13
![Page 2: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/2.jpg)
STAND UP
2
Monday, 4 February 13
![Page 3: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/3.jpg)
I
3
Monday, 4 February 13
![Page 4: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/4.jpg)
State your name....
4
Monday, 4 February 13
![Page 5: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/5.jpg)
Now your real name....
5
Monday, 4 February 13
![Page 6: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/6.jpg)
Pledge solemnly to a deity, non-deity...or possibly spaghetti
6
Monday, 4 February 13
![Page 7: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/7.jpg)
To fill out an evaluation for this session
7
Monday, 4 February 13
![Page 8: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/8.jpg)
And to fill it out in full
8
Monday, 4 February 13
![Page 9: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/9.jpg)
And to buy a beer for the person to my left
9
Monday, 4 February 13
![Page 10: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/10.jpg)
Even though I never really liked that person
10
Monday, 4 February 13
![Page 11: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/11.jpg)
After that incident.. that time
11
Monday, 4 February 13
![Page 12: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/12.jpg)
But it’s best we don’t talk about it anymore...
12
Monday, 4 February 13
![Page 13: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/13.jpg)
Because it gets kinda uncomfortable
13
Monday, 4 February 13
![Page 14: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/14.jpg)
SO SAY WE ALL!
14
Monday, 4 February 13
![Page 15: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/15.jpg)
Paul Mooney
§Geek–Lotus software since R2–Symantec Authorised Consultant–Google Certified Deployment Specialist
§Speaker, Author, Blogger, jogger, biker–www.pmooney.net
§Bluewave Technology–26 staff–Operate globally
15
Monday, 4 February 13
![Page 16: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/16.jpg)
Bill Buchan
§HE’S BACK BABY YEAH!!!
§Geek–cc:Mail–Enterprise level domino consultant since 1995–Dual PCLP in v3, v4, v5, v6, v7, v8 and v8.5
§Speaker, Blogger, Biker–http://www.billbuchan.com
§hadsl–IBM BP ISV focused on federated identity
management–http://www.hadsl.com
16
Monday, 4 February 13
![Page 17: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/17.jpg)
17
● This slide presentation may contain the following copyrighted, trademarked, and/or restricted terms:
● IBM® Lotus® Domino®, IBM® Lotus® Notes®, IBM Lotus Symphony®, LotusScript®
● Microsoft® Windows®, Microsoft Excel®, Microsoft Office®
● Linux®, Java®, Adobe® Acrobat®, Adobe Flash®● Your mileage may vary● This is “Technology Light”
● Consider it a rest!● Fill out the evaluations● @IF(enjoy;"buy us beer";"buy us beer")● Try to never be a story in this presentation● Today is “destroy all end users” day● No.. really it is
Let’s get Legal!!
Monday, 4 February 13
![Page 18: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/18.jpg)
18
What is this session about?
§Mistakes are made by everyone–How do you deal with them?–Blame?–Ignore?–Denial?
§Large or small–Enterprise or SMB
§Know your environment–Especially if you inherit it
§Prevention beats the hell out of the cure–As you will see
Monday, 4 February 13
![Page 19: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/19.jpg)
19
Agenda
§For each case study, we shall–Look at the errors–Diagnose the problem–Determine the problem–How was it resolved–What lessons can be learned
§We have 10 case studies. All new–And 2 of our personal old favourites
§We cover both infrastructure and development…
§All true–Seriously, you can’t make these up....
Monday, 4 February 13
![Page 20: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/20.jpg)
20
Case Studies
Paul
Archiving “issues”Follow the white rabbitDoes it blend?I have a cunning plan(Un)Happy New Year
Bill
Designer Disaster!Router RoutedWhere’s JohnBYOD hellServer moves to Hell
Monday, 4 February 13
![Page 21: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/21.jpg)
21
Story 1 - Designer Disaster
Monday, 4 February 13
![Page 22: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/22.jpg)
22
The Story
§Large multinational Site–Over 40k users–Over 30+ sites
§Monday morning, the helpdesk exploded–Hundreds and hundreds of calls
‘I’m not getting new mail!’
§Happening across all clusters
§Happening on mobile devices
Monday, 4 February 13
![Page 23: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/23.jpg)
23
The Investigation
§Checked servers
§Checked mail routing
§Checked replication
§Checked target mailfiles–ACL - okay–Template - okay
–Checked Database Properties...
Monday, 4 February 13
![Page 24: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/24.jpg)
24
The Cause
§The ‘Don’t Maintain Unread Marks’ flag was set
§But how? This problem affected hundreds of users!
§We then checked the template.. And found that the flag had been set there
§Designer task ran each night
§Mailfile inherits from template - including this flag
Monday, 4 February 13
![Page 25: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/25.jpg)
25
The Resolution
§We tracked down the developer who touched the template–And shot him
§We unset the flag on the template–Made sure it had replicated around
§Re-applied the template to the affected user mail databases
Monday, 4 February 13
![Page 26: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/26.jpg)
26
Lessons Learned
§Unread marks are stored in a database table–Has worked quite well since 6.0.2 / 6.5.1
§Uncontrolled changes to templates can quickly cause large scale issues
§Designer task does NOT have to run every night–Running it on demand gives you control–Paul totally disagrees with this point......
§Little things can easily cause large scale issues
§Ensure that all members of all affected teams understand how to prevent this issue–Use it as a blunt weapon to ensure change control processes are adhered to
Monday, 4 February 13
![Page 27: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/27.jpg)
27
Story 2 - Archiving issues...
Monday, 4 February 13
![Page 28: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/28.jpg)
28
The Story
§Large multinational site–Over 40k users–Over 40 countries
§One region’s mail servers low on disk space–4GB left of 1TB data drive
§Apparently aggressive archiving apparently in place–Data moved on schedule• Older, large size archive server
–Mail over 90 days moved using server-server archiving
Monday, 4 February 13
![Page 29: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/29.jpg)
29
The Investigation
§Check mail server settings
§Program documents –Compact -a
§Check archive server–Cannot connect–Apparently firewall restricts admin client subnet access to archive?
§Check logs on mail servers–They are having issues too
§No RDP access
§No ICMP response (apparently firewall)
Monday, 4 February 13
![Page 30: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/30.jpg)
30
The Cause
§The archive server was down–For 8 weeks
§Ran out of disk space–Attempted restore of entire archive directory accidentally on same server
§Nobody noticed–Nobody can access archive server from admin subnet client IPs anyway
Monday, 4 February 13
![Page 31: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/31.jpg)
31
The Resolution
§*very carefully*
§Disconnect archive server from network
§Replace directory and key system databases
§Bring up and check consistency
§Add to network
§Test archiving
§But...there’s more
Monday, 4 February 13
![Page 32: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/32.jpg)
32
The Resolution
§VPN’d in on Friday evening
§No RDP access to box
§So.. request access
§Support gave me access
§By adding my account to GlobalDomainAdmin group
Monday, 4 February 13
![Page 33: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/33.jpg)
33
Lessons Learned
§All issues here caused by laziness
§Check your servers are up, daily–Monitor your servers?
§Have a restoration process for data
§Don’t hand admin rights out to people “as needed”–I don’t care how much you like them!
§Be “PIRK”Y–Purge Interval Replication Control–See adminblast deck
Monday, 4 February 13
![Page 34: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/34.jpg)
34
Story 3 - Router Rooted
Monday, 4 February 13
![Page 35: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/35.jpg)
35
The Story
§Large multinational–30k + users–50+ sites
§A large mailserver crashed–Thousands of users affected–Auto-restart enabled - restarted the server–Took 40 minutes
§ It crashed again–It restarted it again
§ it crashed again–Auto-restart decided it had enough–Manually Restarted
§ It crashed again
§NSD’s indicated that Router was crashing
Monday, 4 February 13
![Page 36: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/36.jpg)
36
The Investigation
§Console log - no issue
§Log files - no issue
§Transaction log - no issue
§NSD analysis concluded that the router task was crashing–Whilst running LotusScript?
–LotusScript ???
§We noticed a new agent...
Monday, 4 February 13
![Page 37: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/37.jpg)
37
The Cause
§Someone had created a new ‘Before Mail Delivery Agent’ in the mail template–Designer task enabled–All users got the new agent
§Was it tested?–ummm... Yes?
§A ‘Before Mail Delivery’ agent is ran by Router when it delivers the mail to the user mailbox
–Very handy hook point for some automated processes–Documentation states that this agent has GOT to be quick
§This agent tried to open a remote database and log the message–Thousands of mail messages meant that the router task could not keep up–Crashed the server
Monday, 4 February 13
![Page 38: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/38.jpg)
38
The Resolution
§We re-educated the developer–With a bat
§ Increased change control around the template. Again.
§Removed the ‘before mail delivery’ agent
§Refreshed all user templates
Monday, 4 February 13
![Page 39: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/39.jpg)
39
Lessons Learned
§Change Control
§Before Mail Delivery agents have to be fast–Try not to open remote databases on each message being delivered–Over a 400ms wide area network–With 64kb/s bandwidth
§Testing–No. Real Testing. Large Scale Testing.–Use ‘Agent Profiling’ to give you an idea of the total time it’ll take to run
Monday, 4 February 13
![Page 40: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/40.jpg)
40
Story 4 -Follow the white rabbit
Monday, 4 February 13
![Page 41: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/41.jpg)
41
The Story
§Global customer–5k staff globally
§Fast moving company–Acquisitions / temporary projects• Built servers as needed
§Mail issues–Delivery time taking hours–Some mail never delivered• No NDRs
§We were asked to investigate
Monday, 4 February 13
![Page 42: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/42.jpg)
42
The Investigation
§Ask for copy of names.nsf
§Check–Connection documents–Configuration documents–Domain documents–NNN
§Noticed different domain entries–Adjacent domain docs–Non Adjacent domain docs
§Asked to vpn to site to investigate domain–It got interesting/emotional
Monday, 4 February 13
![Page 43: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/43.jpg)
43
The Cause
Domain 1
Domain 2
Domain 3
Monday, 4 February 13
![Page 44: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/44.jpg)
44
The Cause
Domain 1
Domain 2
Domain 3
Domain 4
Domain 5
Domain 6
Domain 7
Domain 8
Domain 9
Domain 10
Domain 11
12
13
14
15 16
1718
19 20
Monday, 4 February 13
![Page 45: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/45.jpg)
45
The Cause
§At some stage...–Someone designed separate domains for projects• Separate servers
§Agents used to add documents to primary nab
§This became a “standard” without question
§Nobody knew who did it first
§Routing hell - some domains linked through 8 hops–Some not linked at all
Monday, 4 February 13
![Page 46: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/46.jpg)
46
The Resolution
§Designed a primary domain–Began a consolidation process
§Cleaned up routing to HUB/Spoke where possible–Some servers could not do this–Ended up with four hub domains/servers until consolidation complete
§Demo’d the directory catalog.....
§Explained mail routing architecture
Monday, 4 February 13
![Page 47: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/47.jpg)
47
Lessons Learned
§ If there is a standard, and it is not traceable back to an “owner”–Question it?–Validate it?
§Enable the “delayed mail” feature in configuration document on servers
Monday, 4 February 13
![Page 48: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/48.jpg)
48
Story 5 - ‘Where’s John?’
Monday, 4 February 13
![Page 49: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/49.jpg)
49
The Story
§A multinational 30k + user site–70+ sites
§Lots of critical line-of business Notes applications–Been running for years
§Overnight application processing fails–No monitoring–No-one notices for a few days–Ambiguous help desk calls logged
§More instances fail–No-one notices
§Finally the business explodes
Monday, 4 February 13
![Page 50: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/50.jpg)
50
The Investigation
§We picked one application–No application logs–No way of validating critical processing had been performed–History of large numbers of document writes• But not recently
–Checked its agents - they looked fine
–Checked the server logs to see when they should run• Tried to confirm from server console when the agents last ran
§Checked the username associated with the agent...
Monday, 4 February 13
![Page 51: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/51.jpg)
51
The Cause
§One developer, responsible for all these applications, left at the end of his contract–He’d been added to the terminations group–All the agents he’d signed had failed to run
Monday, 4 February 13
![Page 52: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/52.jpg)
52
The Resolution
§Create a ‘Template Signing’ ID for your organisation–Have the Administrators keep control of it
§Have the administrators sign all templates going into production with this ID–No exceptions
§ If it fails, its their fault.
Monday, 4 February 13
![Page 53: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/53.jpg)
53
Lessons Learned
§Domino Applications run for years–I’ve seen ones in production for 10+ years–They need to be monitored• Scheduled agents have to run!• Use DDM ‘agent failed’ monitor - and check the results!
§Release control isn’t just for the SOX Audit–its for life–And it’ll save yours
Monday, 4 February 13
![Page 54: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/54.jpg)
54
Story 6 - Does it blend
Monday, 4 February 13
![Page 55: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/55.jpg)
55
The Story
§Mid size site–1.5k users in one region
§Recently upgraded to ND8.x on Citrix
§Full fat version
§Ongoing issues with personal and recent contacts–Everyone had everyone’s recent contacts–Some people have other people’s saved contacts–Others had no issues
§Management going berserk
Monday, 4 February 13
![Page 56: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/56.jpg)
56
The Investigation
§ Investigate a typical user setup
§Check location of home directories–Majority of users using legacy “shared network drive” for data
§Purge the recent contacts from personal address books–They come back almost instantly
§Bang head on wall–No effect
§Bang head on desk–No effect
§Bang head against deployment team–Some effect
Monday, 4 February 13
![Page 57: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/57.jpg)
57
The Cause
§There were two issues
§Contacts being shared...–All users were setup using a default copy of the client databases (NOT templates)• names.nsf, bookmark.nsf etc
–These were placed in network home folder (e.g. h:\notes\data)• TERRIBLE IDEA
–Some of the users were blackberry users• Legacy setup of blackberry for contact sharing• Users’ personal directories being replicated to BES server• BES used them as sources for contacts
–As one user replicated their personal directory with BES server• All other replicas (i.e. other personal directories) replicated too
Monday, 4 February 13
![Page 58: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/58.jpg)
58
The Cause
§ Issue Two–Recent contacts appearing everywhere
§Recent contacts are stored in recent contact view in personal directory
§That data is ALSO stored..–<Notes data directory>\workspace\.metadata\.plugins\com.ibm.notes.dip–files called DIP*.SER
§Citrix deployment was completed incorrectly–The plugin directory was being shared by all users• Being written to by all users
Monday, 4 February 13
![Page 59: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/59.jpg)
59
The Resolution
§ Issue 1–Remove the personal address books from the BES server–Setup mail policy to sync contacts with mail file–Remove personal directory property for each user in the BES administrator• Will then default to mail file contacts
–Start project to change replica id for all personal directories
§ Issue 2–Promote RTFM on Citrix deployment–Fix Citrix deployment
Monday, 4 February 13
![Page 60: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/60.jpg)
60
Lessons Learned
§ “If it works, leave it alone”–Not always the best way–e.g BES using replicated personal directories - very old school
§Citrix is a great tool–8.5.3 supports Citrix well–But you need to:• Understand Citrix• Understand the Notes client• Read the manual
Monday, 4 February 13
![Page 61: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/61.jpg)
61
Story 7 - BYOD Hell
Monday, 4 February 13
![Page 62: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/62.jpg)
62
The Story
§A single user, with an iShiny device
§One morning, the phone was dead.
§She had lost everything–Family pictures, contacts, text messages–No Backup
Monday, 4 February 13
![Page 63: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/63.jpg)
63
The Investigation
§We looked back over the user history
–She used to be an employee of BigCo–Left a number of months ago–Had the BigCo MDM profile and mail/PIM data pushed to her iShiny device
§We looked at the BigCo Mobile Device Management strategy
Monday, 4 February 13
![Page 64: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/64.jpg)
64
The Cause
§BigCo had a rather brutal and primitive MDM
§They assumed control of the users own iShiny device
§The user couldn’t pull their own data off their own phone–Because she wasn’t connected to the enterprise network
§When the user left, they nuked the device
Monday, 4 February 13
![Page 65: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/65.jpg)
65
The Resolution
§Shoot the administrator
§We advised BigCo that they should invest in a better MDM architecture
§We also advised them to at least warn their users that their own phones and iPads were rendered useless by their MDM architecture
Monday, 4 February 13
![Page 66: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/66.jpg)
66
Lessons Learned
§ ‘Bring Your Own Device’ means–The Users own the device–BigCo pushes mail to that device
§BigCo wants to secure mail on that device
§But there are better ways than just nuking the phone–For example, Traveler allows you just to nuke the Traveler data–Other systems create encrypted areas on the device which can be remotely nuked
Monday, 4 February 13
![Page 67: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/67.jpg)
67
Story 8 - “I have a cunning plan”
Monday, 4 February 13
![Page 68: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/68.jpg)
68
The Story
§Small subsidiary of a large corporate company
§Two Domino mail servers
§One (Monday) morning–All mail files corrupted–All documents marked as Rep/Save conflicts
§No databases outside of mail directories corrupted
§FTI’s corrupted
Monday, 4 February 13
![Page 69: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/69.jpg)
69
The Investigation
§Check the Domino servers–Program documents–Log files–Agents–Backup software–AV software
§All good, with exception to corruption errors
§Retrace logs to last startup–Thousands of locking errors
§Ask a few questions....
Monday, 4 February 13
![Page 70: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/70.jpg)
70
The Cause
§The Administrator was asked to make both servers available for mail access (iNotes)–Only had 1 public IP address available–Mail files were not replicas
§GENIUS IDEA–Directory links!
§Administrator decided to map a drive from server A to mail directory on Server B–And map a drive from Server B to mail directory on Server A
§Administrator created directory links on each server to the additional mail directory
Monday, 4 February 13
![Page 71: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/71.jpg)
71
The Cause
§Last restart–Server A had started first, and mapped drive to Server B’s mail directory–Server B was trying to access mail files, locking errors occurring• Corruption
§Then....–Administrator noticed and did restarts in other order–Server B started first, mapped drive to Server A’s mail directory–Server A was trying to access mail files, locking errors occurring
–leading to...
Monday, 4 February 13
![Page 72: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/72.jpg)
72
The Resolution
§Stop servers
§Unmap mappings
§Delete .dir directory link files
§Get backup tape
§Restore
§Punish Administrator–Enthusiastically
Monday, 4 February 13
![Page 73: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/73.jpg)
73
Lessons Learned
§Domino is an application/database server
§Needs ownership of its data
§File locking is result of it not owning data–Always causes issues–Backups, AV software
§Look for the easy solution to the initial problem–Saying no?–Replicating mail files to central server?–Reverse Proxy?
§Domino doesn't always have to be the solution
Monday, 4 February 13
![Page 74: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/74.jpg)
74
Story 9 - Server Moves to Hell
Monday, 4 February 13
![Page 75: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/75.jpg)
75
The Story
§Massive corporate (we mean that...)
–Moving server images from one physical machine to another• Copy data across WAN - setup identical server
§Server gets brought up and 15 minutes later–Mail routing stopped working–Replication stopped working–Traveler stopped working–Agents stop–HTTP stops–Console log goes crazy
§Servers still running
Monday, 4 February 13
![Page 76: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/76.jpg)
76
The Investigation
§We opened up the directory and saw...
Monday, 4 February 13
![Page 77: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/77.jpg)
77
The Cause
§A server migration had corrupted the transaction logs on a single server
§They started the server with no server id
§This transaction log corruption had resulted in the directory design being corrupted to look a bit different.
–
Monday, 4 February 13
![Page 78: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/78.jpg)
78
The real cause
§An administrator accidently replaced the domino directory design–with the document library
§ It replicated–There were no survivors....
§Most servers kept mail routing for a while
§Some services - such as traveler - failed–The views they relied on were missing
–
Monday, 4 February 13
![Page 79: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/79.jpg)
79
The Resolution
§Replace design on the directory–Rebuild all indexes–Restart the server
§SALT ON THE WOUND–Even AFTER discovering... –Didn’t do server restarts–Problems continued
Monday, 4 February 13
![Page 80: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/80.jpg)
80
Lessons Learned
§Limit your risk–Designated master servers for design changes
§Accidental design replaces can happen–Replication requires access–Remove the access!
§Honesty–Really - you WILL get found out...–Own up - you will sleep easier
§Detailed disaster recovery processes
Monday, 4 February 13
![Page 81: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/81.jpg)
81
Story 10 - (un)Happy New Year!
Monday, 4 February 13
![Page 82: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/82.jpg)
82
The Story
§Large site–Regional administration–6000 users in “my” region
§First support call of this year–2nd January
§Replication stopped working for application hub server–Nothing replicating
Monday, 4 February 13
![Page 83: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/83.jpg)
83
The Investigation
§Replication task– working
§Cluster replication–working
§Network connectivity–working
§Console–nothing obvious reported
§Log file–Unusual dates listed...
§Replication history–Entry dates for 1st January 2020
§Spoke to developer
Monday, 4 February 13
![Page 84: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/84.jpg)
84
The Cause
§Developer working over holiday season
§Request from senior executive–Automated email to all users to be sent at midnight –Wishing them joyous tidings for the new year
§Developer was enthusiastic–wrote an agent
§Developer couldn’t test–Did not have admin rights to test servers OS
§BUT!–He did have RDP access to production servers• And nobody was online one night
§Brought down domino–Reset time to Dec 31st 11:58, 2012 and waited• It worked
§Then...–Tried every year to end of 2019
Monday, 4 February 13
![Page 85: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/85.jpg)
85
The Cause
§Brought server up each time
§Server replicated
§Applications updated Replication history–Ending in Jan 01, 2020
§Time reset to 2012–Applications wouldn’t replicate
Monday, 4 February 13
![Page 86: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/86.jpg)
86
The Resolution
§Clear every replication history
§Rebuild view indexes
§Slow repair–Will haunt you
§Educate developer–with prejudice
Monday, 4 February 13
![Page 87: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/87.jpg)
87
Lessons Learned
§Changing OS dates is bad–For any application server
§Replication relies on replication history–Date/Time stamp based marker for last successful push of data
Monday, 4 February 13
![Page 88: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/88.jpg)
88
11 Hell’s agent!
§The Story–A critical application sits on all servers• 3GB Database / 65,000 documents• Replicates from three global hub clusters to all spokes hourly
–All server communication grinds to a halt–No Mail routing/replication–Application grows to 28GB• Masses of replication conflicts
§The Investigation–Check application for design changes–Check replication history and schedule–Check server tasks• Sniff the bandwidth
§Gotcha!–New scheduled agent
Monday, 4 February 13
![Page 89: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/89.jpg)
89
Hell’s agent!
§The cause–Developer wanted to modify all documents–Built an all documents view–Wrote an agent to modify a field–Agent set as scheduled “every hour”–Set agent to run on ….
–ALL SERVERS
–Ran on Hub first… –Hub replicated with all spokes on 1-hour replication schedule–Then ran on all servers–Then continued to run and replicate for the weekend–4.8 Million documents per hour!
Monday, 4 February 13
![Page 90: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/90.jpg)
90
Hell’s agent!
§Lessons Learned–Developers must never change design on production systems • Even basic agents
–Have separate development domain/UAT/Production domains• Developers should NOT have designer access on UAT/Production domains
§Domino is very powerful, and WILL do whatever you tell it to do – no matter how stupid..
§Never leave new code unsupervised
Monday, 4 February 13
![Page 91: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/91.jpg)
91
12 Oh, is that important?
§The Story–Big site–Over 90 servers – 65K users–One Friday, all replication and routing stops–Starts on HUB, and quickly affects all servers
§The Investigation–Check the source of the error–Logs, console, WAN/LAN links–Is server performance the problem?
§Gotcha!–Checked Server consoles… and the Admin4.nsf
Monday, 4 February 13
![Page 92: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/92.jpg)
92
Oh, is that important?
Monday, 4 February 13
![Page 93: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/93.jpg)
93
Oh, is that important?
Monday, 4 February 13
![Page 94: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/94.jpg)
94
Oh, is that important?
Monday, 4 February 13
![Page 95: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/95.jpg)
95
Oh, is that important?
Monday, 4 February 13
![Page 96: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/96.jpg)
96
Oh, is that important?
It Replicates …..
Monday, 4 February 13
![Page 97: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/97.jpg)
97
Oh, is that important?
Monday, 4 February 13
![Page 98: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/98.jpg)
98
Oh, is that important?
§The Cause–Junior Administrator deleted LocalDomainServers group using Adminp on the HUB server–This replicated to all servers–All server-server access lost–Admins attempted to stop spread by disabling replication of the names.nsf file• Forgot admin4.nsf!
§Resolution–Flush out Adminp requests–Manually add entries for LocalDomainServers back to all ACLS and directory documents (3
days to recover)
Monday, 4 February 13
![Page 99: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/99.jpg)
99
Oh, is that important?
§Lessons learned–Limit Administrator access to the Domino directory–Education, Education, Education!–Response procedures for Disaster recovery–Have a test environment that simulates your live one…
Monday, 4 February 13
![Page 100: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/100.jpg)
100
Shorties
§Archive Transaction logging relies on the backup routine clearing the transaction log–No backup, no cleanup. And the transaction log fills up
§Don’t let your platform anti-virus scan the Domino Directory–It doesn’t half kill performance
§Domino requires fast disk subsystems–0-2ms is fast. 360ms is slow
§Don't keep “encrypted” text in clear text
§Please don’t open the mail template and set the Owner Name...
§Reader/Author fields are multi-value canonicalised names–You can try, but NOTHING ELSE will work–We say this every year. And we see it every year
§Don’t switch on ‘Anonymous Access’ on your server security document..
§Don’t just have a replication stub as your mail template...
Monday, 4 February 13
![Page 101: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/101.jpg)
Shorties 2
§Can you define ‘Return on Investment’ for this?
101
Monday, 4 February 13
![Page 102: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/102.jpg)
102
Summary
§Everyone makes mistakes
§Put systems in place to prevent the obvious ones
§ Its how you deal with them that makes you professional–No Blame Culture–Admit soon, Admit well…
§Major system disasters–Sometimes cant be prevented–Are usually the combination of many small errors
§Learn from these mistakes!
Monday, 4 February 13
![Page 103: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/103.jpg)
103
Thank you
§Paul Mooney ([email protected])
§Bluewave
§pmooney.net / bluewavegroup.eu
§Bill Buchan ([email protected])
§hadsl
§billbuchan.com / hadsl.com
Monday, 4 February 13
![Page 104: Connections Lotusphere Worst Practices 2013](https://reader035.fdocuments.net/reader035/viewer/2022062712/55d50589bb61eb3c338b45cc/html5/thumbnails/104.jpg)
104
Legal Disclaimer
© IBM Corporation 2009. All Rights Reserved.
The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.
References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.
IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.
IJava and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
Monday, 4 February 13