Admin Tech Clash: Discussing Best (and Worst) Administration Practices from

hundreds of customers

Presented by: Christoph Adler

Ben Menesi

Speakers

Christoph Adler

Senior Consultant at panagenda 15 years of IBM (Lotus) solutions

experience Since 2007:

IBM Notes Client Management ICS Infrastructure Analysis and

Optimization Good to know:

Lives in Germany, hence the funny accent

Frequent traveler and speaker at international conferences

No energy drinks – squirrel alert! Beer or wine? Beer!

Head of Product at Ytria IBM Notes Administrator and Developer

wearing both hats Working with Ytria since 2011

C API Development Server side Administration Enterprise IT Security

Good to know: Born Hungarian, raised in Hungary &

Switzerland, living in Canada for the past 5 years

Frequent traveler and speaker at international conferences

Whiskey or Vodka? Beer with Whiskey!

Ben Menesi

Introduction

Your collaboration infrastructure contains a TON of valuable data

In order to extract this data you need to knowWHERE to look

Visualization helps to see trends and patterns andunderstand relationships

IBM can help with IDD (IBM Domino Doublecheck)

We (Ytria / panagenda) can help with our softwaresolutions, services and expertise BUT

Scout‘s honor: We won‘t talk about ourproducts in this session!

Agenda

User Activity

Server-side Logging

Client-side Logs & Performance

Usage Activity

Usage Activity

Why should we care?

Understanding usage data allows us to find unused / heavily used applications

In the light of this data we can calculate resourcerequirements and optimize performance

We can profile our high impact users andprocesses and optimize licensing

Usage Activity

What data source should we use?

Activity Trends

DB User Activity

Log & Catalog

3rd Party Solutions

Valuable high level reports

Not very customizable

Lack of customization leads to inaccuracies

Very detailed logs Really crucial info

buried inside one text field

Unless through 3rd

party tools: NONO

Great amount of detail

Only available per database

To be discussed later

Amazing products out there

Some installed on server side

Some client side To be discussed later

Activity Trends

How does it work??

Log.nsf Catalog.nsf Names.nsf Statistics

Activity Trend Collector

Activity.nsf

Activity Trends

Activity data – advantages DB Usage (active / inactive), space growth, etc… User details (active / inactive), connection times, dbs and

network used etc...) Server tasks (HTTP, IMAP, LDAP, Replication etc...) Some visual reports

Activity data – disadvantages What you get is what you get – no digging deeper Raw data isn’t always used properly

Think about a DB marked “Active” with only servers and agents using it

Doesn’t properly handle space used percentages

Log & Catalog

Activity data – advantages It’s only two databases to analyze NOT hundreds or thousands You’ll get the full picture instead of only application usage

Think about usage of system databases or client overhead

Activity data – disadvantages Only contains data of the last 7 days

Actually: There is an undocumented parameter for the Log= notes.ini line:

LOG=logfilename, log_option, cant_touch_this, days, size, [activitydays]

Data structure (Later)

Log & Catalog

Understanding Usage log documents in log.nsf Important: Logging works on a per User, per Session bases! These documents contain a TON of raw data

Log & Catalog

What is an activity event?

Log & Catalog

What is an activity event?

Transactions SentFrmServer SentToServer Body

Database Reads Writes Transactions

Bytes Read Bytes Written DB Open Time

UserName Pathname Reads Writes

Log & Catalog

How to get valuable data out of log.nsf

• Create additional views and select the Activity form• Rearrange columns to make more sense (e.g. User \ db)

Log & Catalog

Details: Database usage document

• Note there is a 1400 activity entry maximum per database (FIFO)• There is also a 64K size limit for the user activity• More details in IBM Technote #1086245

Log & Catalog

Database activity: catalog.nsf

• related information, but different focus

• Full text index details• Replication information• ACL overview

Note:Domino does not distinguishbetween user, server or maintenance tasks activity at this level

Log & Catalog

Example visualizations: Domino Environment Overview

Log & Catalog

Example visualizations: Network Bandwith Demand

Log & Catalog

Example visualizations: User Impact

Log & Catalog

Example visualizations: Database Demand Characteristics

Log & Catalog

Example visualizations: Network demand

Log & Catalog

Real customer example: 15 European Locations 8000 Users 1200 Domino Apps Logistics

„We want to mobilize the 20 most heavily used applications that sales users throughout Europe access in read only mode. How can we find them?“

Log & Catalog

Domino Data Collect session logs from all servers log.nsf Collect nsf inventory from all servers catalog.nsf Collect person information from all users names.nsf

Non Domino Data Merge organizational info to person info

Analyze and Visualize

Log & Catalog

Applications used from Sales in read only mode

Log & Catalog

Applications used from Sales in read only mode

Log & Catalog

Applications used from Sales in read only mode

Log & Catalog

Applications used from Sales in read only mode

DB User Activity

Recorded User Activity User Activity can be a *VERY* rich source of data (incl. HTTP!) Tip: No_Force_Activity_Logging=0 (Default value) allows on ALL

dbs Tip: Set confidential to prevent users with lower than Designer

access

DB User Activity

Prior to ODS 48 Entry size allocated 44 bytes, object size 61600 bytes (max 1400)

ODS 48+ (undocumented improvement) Entry size 92 bytes. Object size 128800 bytes (same max 1400)

Curious: only two additional counters added, yet ODS48+ entry size is over 200% the size of ODS48 – why?

Date & Time

Reads Writes User

Date & Time

Reads Writes Updates Deletes User

DB User Activity

Does Notes only log what it displays? Nope

Read more: http://techlab.ytria.com/5448/lotus-notes-articles/interpret-user-activity-traces-portrait-global-usage

Reads Adds Updates Deletes

Data

NonData

Σ

Data

NonData

Σ

Data

NonData

Σ

Data

NonData

Σ

DB User Activity

Limitations Only available per database

Free tools available to facilitate working with this data across multiple databases: http://www.agecom.com.au/useractivity

Or implement via W32_NSFDbGetUserActivity Makes no difference between servers & users

Filter user names using free or paid tools Only maintains 1400 entries

Isn’t this the same for the log file activity documents? Wait, this doesn’t need to be an issue

DB User Activity

Customer Example: calculating usage patterns from usage activity

Entry Date

Entry Date#1400

#1

5 days

Entry Date

Entry Date

120 days

#1400

#1

Database A Database B

3rd Party Solutions

panagenda iDNA, ytria databaseEZ…

3rd Party products for usage analysis IDD (IBM Domino Doublecheck) free offer

http://goo.gl/maL0fo panagenda iDNA

https://www.panagenda.com/products/idna Ytria databaseEZ & consoleEZ

http://ytria.com/databaseezhttp://ytria.com/consoleEZ

Tableau (for visualization) http://www.tableau.com

3rd Party Solutions

Server-side Logs

Mining logs

Domino Server

The Admin

Event lifecycle

1.• Events take place

2. • Events are logged

3.• Some events captured

4• Few events addressed

5• Tons of events missed, angry users on the phone

Dissecting Log.nsf

Server tasks can log events to multiple types of documents! But no duplicate logs: different information! Example: Server 1 replicator opens session to Server 2 as per

Connection Document settings

Misc.

Replication

Dissecting Log.nsf

Miscellaneous events: what do we see? Some high level info (Server, Start & Finish time) and UI shows one text type item: Date + Time & Description

What is an event?

Time Status Code Severity Type

Addin Name Target Server Target DB Target User

Dissecting Log.nsf

Miscellaneous events – event attributes logged in multi-value Text items

EventListTime &

Description

EventSeverity

EventStatus

EventType

Event 1

Severity

HEX Status

Type

Time & Description

Event 2

Severity

HEX Status

Type

Time & Description

Event 3

Severity

HEX Status

Type

Dissecting Log.nsf

Can we get more data? We can use formulas and the list of Severities, Events & Status

Codes to get a better idea of each Misc. log document

Total # of events / document [Eventlist item]

– @Elements(Eventlist)

Total # of unique events [EventStatus item]

– @Elements(@Unique(EventStatus))

Dissecting Log.nsf

Highest severity / document [EventSeverity item]

– _sev:=@If(@Sort(@Unique(@Text(EventSeverity));[Ascending])[1]="0";@Sort(@Unique(@Text(EventSeverity));[Ascending])[2];@Sort(@Unique(@Text(EventSeverity));[Ascending])[1]);

– With the _sev variable, use icons to display (from DDM resources)

– @If(_sev="0";"iconNormal";_sev="1";"iconFatal";_sev="2";"iconFailure";_sev="3";"iconHigh";_sev="4";"iconLow";_sev="5";"iconNormal";"")+".gif“

– What you’ll get:

Dissecting Log.nsf

A view displaying the highest severity events encountered, # of events, unique events and all levels of severities per document:

More info here: http://www.slideshare.net/BenedekMenesi/ibm-connected-2015

Client-side Logs & Performance

ClientClocking

NOTE: All of this should only be done on “ONE” client – slows down client, creates huge log files!

Use the following to see and measure client performance CLIENT_CLOCK=1 CONSOLE_LOG_ENABLED=1 Debug_Outfile=c:\debug.txt Debug_Console=1 DEBUG_TCP_ALL=1 (use only if needed)

Fileserver or local disk Performance debugging is virtually impossible If on >= 8.5 try adding DEBUGGINGWCTENABLED=16386 in Notes

ClientClocking

Client_Clock=1 leads to a debug file of the following format:

ClientClocking

OpenNTF – NRPC Parser

Link: http://www.openntf.org/internal/home.nsf/project.xsp?action=openDocument&name=Notes%20RPC%20Parser

Developed by: Jamie Magee, MartinScott Consulting LLC, US Andrew Magerman, Magerman Collaborative Software GmbH,

Switzerland

ClientClocking

OpenNTF – NRPC Parser

Cache.ndk

We've all been “advised to”/guilty of deleting the cache.ndk in an effort to cure Notes performance problems/symptoms

• Some companies even delete cache.ndk on a scheduled basis

Cache.ndk

HOWEVER, let us explain the impact of that on Notes performance ... Increases traffic between client and server by 4,000% (forty times

more traffic, yes)(10,000+% if the data directory is on a network drive)

The more databases end users use, the worse it gets

There is always a root cause – deleting cache.ndk only fixes symptoms

Let's look at some detailed traffic analysis for just ONE application (the mail file) of ONE user ...

Cache.ndk

100 users =

150 MB vs. 3,7 MB

1000 users =

1.5 GB vs. 37 MB

Excluding Fileservers!

After deleting cache.ndk Keeping / NOT deleting cache.ndk

ODS

On Disk Structure

ODS = On Disk Structure ODS 16 = Notes 2 ODS 17 = Notes 3 ODS 20 = Notes 4 (or Templates) ODS 41 = Notes 5 ODS 43 = Notes 6 & 7 ODS 48 = Notes 8 ODS 51 = Notes 8.5 & 9 ODS 52 = Notes 9.0.1

The difference between ODS 43 and 52 = up to 80% LESS FILE I/OThis helps with ANY client AND server hardware

On Disk Structure

CAUTION:The ODS is not automatically updated when upgrading clients

Except for names.nsf, bookmark.nsf, and desktop*.dsk/ndk – IF –you upgrade properly (think ini:Setup/TemplateSetup=)

ODS 51 requires Notes >= 8.5 and CREATE_R85_DATABASES=1 in notes.ini for NEW db's

ODS 52 requires Notes >= 9.0.1 and CREATE_R9_DATABASES=1 in notes.ini for NEW db's

On Disk Structure

Good News for Notes > 8.5 Use NSF_UpdateODS=1 with CREATE_R85/9_DATABASES=1 for a

one-time upgrade of most local databases NOTE: USE WITH EXTREME CARE IF YOUR DATA DIRECTORIES

ARE ON A NETWORK DRIVE! NOTE: End users can't access databases during compact

For databases on server: no more downtime required, use: Compact [dbpath] –Replica to create replica in the

background (new in domino 9) More info: https://www-

01.ibm.com/support/docview.wss?uid=swg27039379&aid=1

On Disk Structure

Impact examples Startup time of a Notes 8.5.3 client with 3 ODS 20 apps in Notes

data = 10 seconds After ODS upgrade: 2 seconds Reduced File I/O of your disks/SAN/NAS after ODS 41 to 52 upgrade

by 60% Removing 70% of all old files in Data directories on SAN/NAS = 45%

less managed storage (backup)

The bad news: There is no way to tell what is “out there” 3rd party tools or random inspections are highly recommended News for Notes > 8.5

On Disk Structure

IBM helps!

Encryption/Compression

Slow clients

Port settings are included in the notes.ini Type / Active / Encryption / Compression / Timeout

Bad idea Remove LAN0 / COMx / DisabledPorts

Good ideaUse TCPIP according to the client type PORTS = TCPIP TCPIP = TCP,0,15,0,,12320, Desktops TCPIP = TCP,0,15,0,,45088, Laptops TCPIP = TCP,0,15,0,,12288, Citrix / VDI

Riverbed !?Don‘t forget your Servers!

Q & A

Thank you