Xinwen Fu

Linux Logging Mechanisms

91.580.203 Computer & Network Forensics

2CS@UML

Outline Log files

What need to be logged Logging policies Finding log files

Syslog: the system event logger

3CS@UML

Who logs data? The accounting system The kernel Various utilities

All produce data that need to be logged Most of the data has a limited useful lifetime,

and needs to be summarized, compressed, archived and eventually thrown away

4CS@UML

Logging policies

1. Throw away all data immediately2. Reset log files at periodic intervals3. Rotate log files, keeping data for a fixed

amount of time4. Compress and archive to tape or other

permanent media

5CS@UML

Which policy to choose Depends on:

how much disk space you have how security-conscious you are

Whatever scheme you select, regular maintenance of log files should be automated using cron

6CS@UML

1. Throwing away log files Not recommend

Security problems (accounting data and log files provide important evidence of break-ins)

Helpful for alerting you to hardware and software problems

In general, keep one or two months In a real world, it may take one or two weeks

for SA to realize that site has been compromised by a hacker and need to review the logs

7CS@UML

2. Reset log files at periodic intervals Most sites store each day’s log info on

disk, sometimes in a compressed format These daily files are kept for a specific

period of time and then deleted One common way to implement this policy

is called “rotation”

8CS@UML

3. Rotating log files Keep backup files that are one day old,

two days old, and so on. logfile, logfile.1 , logfile.2, … logfile.6 Linux: /etc/logrotate.conf

Specify the frequency with which the files are reused

Each day rename the files to push older data toward the end of the chain

9CS@UML

#! /bin/shcd /var/logmv logfile.2 logfile.3mv logfile.1 logfile.2mv logfile logfile.1cat /dev/null > logfile

Some daemons keep their log files open all the time, this script can’t be used with them. To install a new log file, you must either signal the daemon, or kill and restart it.

In Unix-like operating systems, /dev/null or the null device is a special file that discards all data written to it, and provides no data to any process that reads from it. In Unix programmer jargon, it may also be called the bit bucket or black hole.

Script to archive 4 days files

10CS@UML

4. Archiving log files Some sites must archive all accounting

data and log files as a matter of policy, to provide data for a potential audit

Log files should be first rotated on disk, then written to tape or other permanent media

11CS@UML

Finding log files To locate log files, read the system startup

scripts : /etc/rc* or /etc/init.d/* If logging is turned on when daemons are run Where messages are sent

Some programs handle logging via syslog (syslogd or rsyslogd) Check /etc/syslog.conf (or rsyslog.conf on

Fedora Core 9) to find out where this data goes

12CS@UML

Finding log files (default configuration) Different operating systems put log files in

different places: /var/log/* /var/cron/log /usr/adm /var/adm …

On Linux, all the log files are in /var/log directory

13CS@UML

Outline Log files Syslog: the system event logger

how syslog works its configuration file debugging syslog the software that uses syslog

14CS@UML

What is syslog A comprehensive logging system, used to

manage information generated by the kernel and system utilities

Allow messages to be sorted by their sources and importance, and routed to a variety of destinations: Log files, users’ terminals, or even other

machines

15CS@UML

Syslog: three parts1. Syslogd: daemon that does the actual

logging Configuration file: /etc/syslog.conf

2. API: openlog, syslog, closelog Library routines that programs use to send

data to syslogd

3. logger User-level command for submitting log entries

16CS@UML

Using syslog library routineswrite log entries to a special file

/dev/log

syslogd /etc/syslog.conf

reads

consults

dispatches

Logfiles

Users’sterminals

Other machines

syslog-aware programs

/dev/klog

http://www.calpoly.edu/cgi-bin/man-cgi?syslogd

Most system logging daemons listen on one or more Unix sockets, the most typical being /dev/log; /dev/klog is kernel log socket

17CS@UML

Configuring syslogd The configuration file /etc/syslog.conf

controls syslogd’s behavior It is a text file with simple format, blank

lines and lines beginning with ‘#’ are ignored (comment). selector <TAB> action for example

mail.info /var/log/maillog

18CS@UML

Configuration file - selector Identifies

Program ‘facility’ that is sending a log message Messages’s severity level eg. mail.info

Syntax facility.level Facility names and severity levels must be

chosen from a list of defined values

19CS@UML

Configuration file - Facility Names

FACILITY PROGRAMS THAT USE ITkern the kerneluser User process, default if not specifiedmail The mail systemdaemon System daemonsauth Security and authorization related

commandslpr the BSD line printer spooling systemnews The Usenet news system

20CS@UML

Configuration file - Facility names (Cont.)

FACILITY PROGRAMS THAT USE ITuucp Reserved for UUCPcron the cron daemonmark Timestamps generated at regular intervalslocal0-7 Eight flavors of local messagesyslog syslog internal messagesauthpriv Private or system authorization messagesftp the ftp daemon, ftpd* All facilities except “mark”

UUCP stands for Unix to Unix CoPy.

21CS@UML

Configuration file - Facility names (Cont.) Facility - Mark: Timestamps can be used to log time

at regular intervals (by default, every 20 minutes), so you can figure out that your machine crashed between 3:00 and 3:20 am, not just “sometime last night”. This can be a big help if debugging problems occur on a regular basis

Start at command line: syslogd –m 1 Use syslog.conf

Start syslog daemon: syslogd Add the line to syslog.conf: mark.* /var/log/messages

22CS@UML

Configuration file - severity level

LEVEL APPROXIMATE MEANINGemerg (panic) Panic situationalert Urgent situationcrit Critical conditionerr Other error conditionswarning Warning messagesnotice Unusual things that may need

investigationinfo Informational messagesdebug For debugging

severe

not severe

23CS@UML

Configuration file - selector Levels indicate the minimum importance that a

message must have in order to be logged mail.warning - would match all the messages

from the mail system, at the minimum level of warning

Level of ‘none’ will exclude the listed facilities regardless of what other selectors on the same line may say. *.info;mail.none action

All the facilities, except mail, at the minimum level info will subject to action

24CS@UML

Configuration file – selector (Cont.) Can include multiple facilities separated with ‘,’

commas e.g., daemon,auth,mail.info action

Multiple selectors can be combined with ‘;’ e.g. daemon.level1;mail.level2 action Selectors are ‘|’ -- ORed together, a message matching

any selector will be subject to the action Can contain

* - meaning all none - meaning nothing

25CS@UML

Configuration file - action(Tells what to do with a message)

ACTION MEANING

filename Write message to a file on the local machine

@hostname Forward messages to the syslogd on hostname

@ipaddress Forward messages to the host at IP address

user1, user2,… Write messages to users’ screens if they are logged in

* Write messages to all users logged in

26CS@UML

Configuration file - action (Cont.) If a filename action used, the filename must be

absolute path. The file must exist since syslogd will not create it e.g. /var/log/messages

If a hostname is used, it must be resolved via a translation mechanism such as DNS or NIS

While multiple facilities and levels are allowed in a selector, multiple actions are not allowed.

27CS@UML

Config file examples (1)# Small network or stand-alone syslog.conf file# emergencies: tell everyone who is logged on*.emerg *

# important messages*.warning;daemon,auth.info /var/adm/messages

# printer errorslpr.debug /var/adm/lpd-errs

28CS@UML

# network client, typically forwards serious messages to # a central logging machine# emergencies: tell everyone who is logged on*.emerg;user.none *

#important messages, forward to central logger*.warning;lpr,local1.none @netloghostdaemon,auth.info @netloghost

# local stuff to central logger toolocal0,local2,local7.debug @netloghost

# card syslogs to local1 - to boulderlocal1.debug @ialab.cs.uml.edu

# printer errors, keep them locallpr.debug /var/adm/lpd-errs

# sudo logs to local2 - keep a copy herelocal2.info /var/adm/sudolog

Config file examples (2)

29CS@UML

Sample syslog output1. Mar 27 09:10:02 tcb-ia-lab-inst sshd[4100]: Accepted

password for cis418 from ::ffff:216.254.235.105 port 61940 ssh2

2. Mar 27 18:10:00 tcb-ia-lab-inst sshd[9332]: Failed password for root from ::ffff:216.254.235.105 port 62817 ssh2

3. Mar 27 18:10:08 tcb-ia-lab-inst sshd[9332]: Accepted password for root from ::ffff:216.254.235.105 port 62817 ssh2

4. Mar 27 20:08:27 tcb-ia-lab-inst sshd[10629]: Accepted password for root from ::ffff:10.0.0.111 port 42172 ssh2

5. Mar 27 20:09:48 tcb-ia-lab-inst sshd[10649]: Failed password for root from ::ffff:10.0.0.111 port 48233 ssh2

30CS@UML

Syslogd A hangup signal (HUP, signal 1) cause

syslogd to close its log files, reread its configuration file, and start logging again

If you modify the syslog.conf file, you must HUP syslogd to make your changes take effect ps -ef | grep syslogd Kill -1 pid-of-syslogd

31CS@UML

Software that uses syslogPROGRAM FACILITY LEVELS DESCRIPTIONamd auth err-info NFS automounterdate auth notice Display and set dateftpd daemon err-debug ftp daemongated daemon alert-info Routing daemongopher daemon err Internet info serverhalt/reboot auth crit Shutdown programslogin/rlogind auth crit-info Login programslpd lpr err-info BSD line printer daemon

32CS@UML

Software that uses syslog PROGRAM FACILITY LEVELS DESCRIPTIONnamed daemon err-info Name sever (DNS)passwd auth err Password setting

programssendmail mail debug-alert Mail transport systemrwho daemon err-notice romote who daemonsu auth crit, notice substitute UID prog.sudo local2 notice, alert Limited su programsyslogd syslog, mark err-info internet errors,

timestamps

33CS@UML

Syslog 's functions Liberate programmers from the tedious

mechanics of writing log files Put SA in control of logging

Before syslog, SA had no control over what information was kept or where it was stored

Can centralize the logging for a network system

34CS@UML

Debugging syslog -- logger Useful for submitting log entries from

shell scripts

Can also use it to test changes in syslogd’s configuration file. For example..

35CS@UML

Add line to syslog.conf:local5.info /var/log/test.log

verify it is working, runlogger -p local5.info “test messages”

a line containing “test messages” should be written to /tmp/test.log

If this doesn’t happen:forgot to create the test.log file

or forgot to send syslogd a hangup signal

36CS@UML

Remote logging On a central logging server: 10.0.0.192

syslogd -r On a local server: 10.0.0.45

authpriv.*;auth.* @10.0.0.192 Question: where are those events written?

37CS@UML

Process Accounting accton is used to turn on or turn off process

accounting lastcomm tracks commands each user uses

touch /var/log/pacct /sbin/accton /var/log/pacct lastcomm -f /var/log/pacct

ac prints out statistics about users' connection times in hours based on the logins and logouts in the current /var/log/wtmp file ac -p -d

sa summarizes accounting information from previously executed commands, software I/O operation times, and CPU times, as recorded in the accounting record file /var/log/pacct sa /var/log/pacct

38CS@UML

Process Accounting (Cont.) last goes through the /var/log/wtmp file and prints

out information about users' connection times

lastb is the same as last, except that by default it shows a log of the file /var/log/btmp, which contains all the bad login attempts.

39CS@UML

Using syslog in programs openlog( ident, logopt, facility);

Messages logged with the options specified by logopt begin with the identification string ident.

syslog( priority, messge, parameters…); Send message to syslogd, which logs it at the

sepecified priority level close( );

40CS@UML

/ * c program: syslog using openlog and closelog */

#include <syslog.h>main ( ) {

openlog ( “SA-BOOK”, LOG_PID, LOG_USER);syslog ( LOG_WARNING, “Testing …. “);closelog ( );

}

On the host, this code produce the following log entry:Apr 4 15:21:57 tcb-ia-lab-inst SA-BOOK[7762]: Testing ...

41CS@UML

Summary On linux, check following files:

/etc/syslog.conf : syslog configuration file /etc/logrotate.conf : logging policy, rotate /etc/logrotate.d/* /var/log/* : log files

try following commands to find out more... man logrotate man syslogd

42CS@UML

References1. Chris Prosise, Kevin Mandia, Matt Pepe, Incident Response and

Computer Forensics, Second Edition (Paperback), ISBN: 007222696X

2. Brian Hatch, Preventing Syslog Denial of Service attacks, http://www.hackinglinuxexposed.com/articles/20030220.html

3. Albert M.C. Tam, Enabling Process Accounting on Linux HOWTO, 02/09/2001, http://www.faqs.org/docs/Linux-mini/Process-Accounting.html

4. Keith Gilbertson, Process Accounting, 12/01/2002, http://www.linuxjournal.com/article/6144

43CS@UML

Notes Change host name

/etc/hosts # add the host to the end of 127.0.0.1

/etc/sysconfig/network

44CS@UML

#! /bin/shcd /var/logmv logfile.2.Z logfile.3.Zmv logfile.1.Z logfile.2.Zmv logfile logfile.1cat /dev/null > logfilekill -signal pidcompress logfile.1

signal - appropriate signal for the program writing the log filepid - process id