Nagios in a Multi-Platform Environment

lvl@omnitec.net

Leland V. Lammert, PhD
Chief Scientist
Omnitec Corporation

The Problem

Different OSs can require

Different type of connection

Different installation procedure

Nagios checks or an agent require a connection to the remote machine
(i.e. from the Nagios server to the remote machine)

No issue inside the firewall

There are issues for machines outside the firewall

The Solution SSH

SSH forward connections are a good solution for monitoring inside the firewall

An ssh reverse tunnel is a good solution for monitoring outside the firewall

Active checks can then be used on all systems

No Agents

No Complex Installation

A shell is always available for troubleshooting

Today's Outline

Secure Shell

Connection Process

Credentials

Setting up a Reverse Tunnel

Using checks

Gui's - RDP or X

Troubleshooting

Secure Shell

SSH History

SSH [Secure Shell] is a data exchange protocol that uses an encrypted connection between two network devices

SSH replaced Telnet and other insecure remote shells which send information, most notably passwords, in plaintext

SSH encryption provides confidentiality and integrity of data over an insecure network

Connection Types

Normal [Forward]

Tunnel [Reverse]

SSH Advantages

Secure [ssh2]

Host Identity Verified

User Authorization

Secure Data Transmission

Ubiquitous ('Nix, Windows [Cygwin], OSX)

Reverse Tunnels require no firewall changes

Allows Active Checks

No agent to install or configure

Simplified testing, GUI still available [X or RDP]

SSH Details

Authentication is verified with SSH keys

[Forward] ssh connections work for hosts behind the firewall

[Reverse or Tunnel] ssh connections work for hosts anywhere else

The tunnel passes traffic TO the remote machine FROM the Nagios Server (reverse operation)

SSH Connections can be used to standardize connections to ALL machines!

Forward connections

Requires

Static IP

Open port

A static IP may not available at external locations

An open port for an incoming connection is a BIG security problem

Even when an agent is used [NSCA], there is no access to the remote machine for testing and troubleshooting

Reverse Connections - Tunnels

A Tunnel is packet encapsulation using a network protocol

The data payload protocol is then encapsulated in a delivery protocol.

L2TP (layer 2) Tunneling Protocol)

SSH

Socks

Reverse Tunnels - an ssh connection from client to server, where data is transmitted from server to client (i.e. reverse)

Connection Process

Startup

Start Session

Exchange Server key

Generate Session key

Authentication

Exchange Client key

If ACK, continue

SSH Startup

SSH Authentication

SSH uses Asymetric Encryption

Public/Private Keypair

A Keypair verifies identity for:

Machine

User

Currently keys are 2048 bits in length, usual RSA

Each Key has two parts:

Public and Private

AKA the keypair

Keys are created with ssh-keygen

Credentials

Host and User Credentials

Host Key

Verifies that the host/server at the other end of the connection is the same one seen previously

User Keys

Verifies that the user is authorized for the connection

Why not Passwords?

Requires manual entry

Cannot be automated without storing as plaintext

Much less secure than key

Keys on Unix/Linux

Keys:
/home//.ssh

Private key:id_rsa

Public key:id_rsa.pub

Keys on Windows

Typical Cygwin install location
C:\Program Files\Cygwin

Home directories:
/home//.ssh

Keys: ~/.ssh

Private key:id_rsa

Public key:id_rsa.pub

Keys on OSX

Keys:
/Users//.ssh

Private key:id_rsa

Public key:id_rsa.pub

Setting up a Reverse Tunnel

Basic ssh command

sshBase command

-fRun in background

-XEnable X Forwarding ('Nix)

-nPrevent reading from STDIN

-NNo remote command

-RReverse Operation

ssh -f -X -n -N -R

Ports

REMOTE:127.0.0.1:LOCAL*

REMOTEPort on remote machine [from]

127.0.0.1localhost

LOCALPort on local machine [to]

* Remember, this is being done at the external machine
so the Remote port here is actually on the Nagios server!

Authentication

$USER_NAME@$REMOTE_HOST

USER_NAMEUser ID on remote machine

REMOTE _HOSTHostname or IP of
remote hostt

-p LOGIN_PORT

LOGIN_PORTPort for login on remote machine

-i IDENTITY_FILE

IDENTITY_FILEPrivate key file to use

Three Steps to Create a Tunnel

1. Remote to NagiosConnect remote machine to Nagios server and create Tunnel

2. Make it permanent

cron

autossh

launchctl/launchd

3. Nagios to RemoteSetup access from Nagios user or process to remote machine

Step 1. Remote to NagiosConnect remote machine to Nagios server
and create Tunnel

Step 1 'Nix

Create keypair on remote system and copy to Nagios Server

ssh-keygen

cat ~/.ssh/id_rsa.pub

ssh

vim ~/.ssh/authorized_keys

G o

:x

test

Step 1 Windows

Install Cygwin (cygwin.com/install.html)

Install in c:\program files\Cygwin

Add autossh, rsync, bash, perl

Create keypair on remote system and copy to Nagios Server [same as 'Nix]

Step 1 OSX

Create keypair on remote system and copy to Nagios Server [same as 'Nix]

Step 2.
Make it permanent

Step 2 'Nix cron

Connection variables

REMOTE_HOST="nagios._______.com"
USER_NAME="________"
REMOTE_PORT="____"
LOCAL_PORT="____" *
LOGIN_PORT="____" *
IDENTITY_KEY="/home/nagios/.ssh/id_rsa"

* NOTE: The ssh port should be changed for security, do NOT use the standard port 22.

Step 2 'Nix cron

Command to create the link

COMMAND="ssh -f -n -N -R \ $REMOTE_PORT:127.0.0.1:$LOCAL_PORT \ $USER_NAME@$REMOTE_HOST \ -p$LOGIN_PORT -i $IDENTITY_KEY"

Step 2 'Nix cron

Running? If not, start it
pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND

Working? Login to Nagios and check from the other side
ssh -i$IDENTITY_KEY -p$LOGIN_PORT $USER_NAME@$REMOTE_HOST netstat -an | egrep "tcp.*127.0.0.1:$REMOTE_PORT.*LISTEN" > /dev/null 2>&1

If not working, kill and restart.
if [ $? -ne 0 ] ; then
pkill -f -x "$COMMAND"
$COMMAND
fi

Step 2 Windows autossh

Install servicecygrunsrv -I AutoSSH -f "nagios_link" -p /usr/bin/autossh -a " \
-M : -N -R :127.0.0.1: @"

Use services.msc to set restart options and credentials:

cyg_server

Local Administrator

Step 2 OSX

Install autossh with homebrew

Command/usr/local/bin/autossh -M : -N -R -p 2206 -g 4000:127.0.0.1:22 @ -tt

LaunchctlCan install AutoSSH as System Service

Autostart at boot

Step2 OSX launchctl

Autossh file template

ID=username
HOST=hostname

if [ "X$SSH_AUTH_SOCK" = "X" ]; then
eval `ssh-agent -s`
ssh-add $HOME/.ssh/id_rsa
fi

#AUTOSSH_POLL=600
#AUTOSSH_PORT=20000
#AUTOSSH_GATETIME=30
#AUTOSSH_LOGFILE=$HOST.log
#AUTOSSH_DEBUG=yes
#AUTOSSH_PATH=/usr/local/bin/ssh
export AUTOSSH_POLL AUTOSSH_LOGFILE AUTOSSH_DEBUG AUTOSSH_PATH AUTOSSH_GATETIME AUTOSSH_PORTautossh -2 -fN -M 20000 -R 2200:localhost:22 ${ID}@${HOST}

Step 3 Nagios to Remote
Setup access from Nagios user or process
to remote machine

Step 3 - 'Nix

Copy Nagios user public key to remote

Note: Nagios may use different UID than the one for testingcat ~/.ssh/id_rsa.pub

ssh

vim .ssh/authorized_keys

G o

:x

Step 3 Windows

Same as 'Nix

Step 3 OSX

Same as 'Nix

What was just created?

Take note at the current time there are working ssh connections to:

Any 'Nix

Any Windows*

OSX

These connections can be used for active checks, as well as troubleshooting and setting up an RDP or X session if needed.

* There sometimes are authentication issues with Windows Domain servers that must be handled uniquely.

Using Checks

Normal [forward] checks

Local checksdefine service{
use local-service
host_name Nagios
service_description Root Partition
check_commandcheck_local_disk!20%!10%!/
}

Checks via ssh connectiondefine host{
use openbsd-server
host_name mx1
alias mail_mx1_server
address 206.197.251.200
icon_image envelope.gif
statusmap_image envelope.gif
check_command check_smtp
}

Reverse Check

Same as other checks, except command format includes additional datadefine service{
use generic-service
host_name hanley
service_description SSH Check Proc
normal_check_interval 15
retry_check_interval 5
notifications_enabled 1
check_command check_by_ssh_reverse!-p2210!proc
}

Additional data required in command

ssh Private key

ssh Port

Host address

Command Comparison

Local
command_name check_local_disk
command_line $USER1$/check_disk -w $ARG1$ -c $ARG2$ -p }

Forward
command_name check_by_ssh
command_line /usr/bin/ssh -i /.ssh/id_rsa \
nagios@$HOSTADDRESS$ $ARG1$ 2>/dev/null

Reverse
command_name check_by_ssh_reverse
command_line /usr/bin/ssh $ARG1$ -i /.ssh/id_rsa \
nagios@$HOSTADDRESS$ $ARG2$ 2>/dev/null

Building Plugins

Create test machine
VMs work great!

Linux

Windows

OSX

Compile set of plugins for that platform

Maintain platform plugin set

Gui's - RDP or X

X apps run natively

Be sure to include -X in tunnel setup

Connect to remote with ssh -X

Launch app

Possibly explicit invocation options, a la Firefox:

firefox no-remote

Start firefox, but run on the remote machine, forwarding the display to the Nagios server.

RDP on demand

On remote machine

#
# Reverse ssh link for RDP tunnel
#
ssh -f -n -N -R 3389:127.0.0.1:3389 @ -p

Tunnel 3389 on remote machine to 3389 on Nagios ServerCheck for open port

netstat -an | grep 3389

Use one connection at a time for sanity!

Troubleshooting

Common Problems

Host Key Verification Failed

The Host Key on the sending side is not the same as seen last time

Possible cause - the remote machine has been rebuilt

Troubleshooting checks

$64K Secret tmux

Very step-by-step

Paremeters often not required

Useful aliases

Hostkey verification failed

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for [storm]:2206 has changed,
and the key for the corresponding IP address [10.0.0.1]:2206
is unknown. This could either mean that DNS SPOOFING is happening or the IP address for the host and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
6b:da:2e:50:a9:ea:b0:b1:3d:c1:b8:4a:a3:a5:56:87.
Please contact your system administrator.
Add correct host key in /home/lvl/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/lvl/.ssh/known_hosts:5
You can use following command to remove all keys for this IP:
ssh-keygen -R storm -f /home/lvl/.ssh/known_hosts
Host key verification failed.

tmux

tmux is a terminal multiplexer

A terminal multiplexer permits switching between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal.

Essential for troubleshooting checks!

tmux session

Useful Aliases

alias nagc='cd /etc/nagios'

alias nagh='cd /var/lib/nagios'

alias naghup='ps ax|grep /sbin/nagios |grep -v grep |awk "{print \$1}" | xargs kill -HUP'

alias nagq='netstat -an | grep 127 | grep LISTEN | grep -v 25 | sort --field-separator='\'':'\'' --key=2.1'

alias nagv='/usr/sbin/nagios -v /etc/nagios/nagios.cfg'

Resources

SSH and Authentication
http://en.wikipedia.org/wiki/Secure_Shellhttp://tools.ietf.org/html/rfc4252

SSH Cookbook Tunnels
http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Tunnels

Thank you!

Leland Lammert
www.omnitec.net
lvl@omnitec.net

Click to edit the outline text format

Second Outline Level

Third Outline Level

Fourth Outline Level

Fifth Outline Level

Sixth Outline Level

Seventh Outline Level

Eighth Outline Level

Ninth Outline Level

Click to edit the title text format

Click to edit the title text format

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level