Logs

March 2003 Log Files 1

Log Files


ObjectivesWhen you have completed this module you will

be able to do the following:• Describe statistics collected by NetCache• Describe the contents of the logs• Compare log rotation and log pushing• Manage log files• Describe log file compression


NetCache Log Files Overview• Introduction to

NetCache Logs– Available Logs– Understanding Log

files– Messages Log

• Configuring Logs– Enabling and

formatting– Customizing logs

• Managing Logs– Routine admin– File switching

techniques– Log pushing– Log rotation

• Log Content


Available Log Files

Data > Log Files


Understanding Log Files

• All logs enabled by default• Predefined or custom formats• Extended Log File Format• Options for viewing, copying, deleting,

and forwarding log files• Log switching

– Rotation (default)– Pushing


Log Setup

Setup > Content Director > General > LoggingContent Director

Setup > GRM Server> Logging

GRM Server

Setup > ICAP> ICAP 1.0ICAP

Setup > HTTP> LoggingWeb Access

Setup > Streaming Media > LoggingStreaming Details

Setup > Streaming Media > LoggingStreaming Access

Setup > NNTP> LoggingNNTP Access

no configuration optionsMessages


Log Formats

• Two Formats– Predefined– Custom

• Logs (except Messages Log) adhere to W3C Extended Log File Format (www.w3c.org)

• File headers may be customized

W3C Extended Log File Format1

Format An extended log file contains a sequence of lines containing ASCII characters terminated by either the sequence line feed (LF) or carriage-return, line feed (CRLF). Log file generators should follow the line termination convention for the platform on which they are executed. Analyzers should accept either form. Each line may contain either a directive or an entry.

Entries consist of a sequence of fields relating to a single HTTP transaction. Fields are separated by white space, the use of tab characters for this purpose is encouraged. If a field is unused in a particular entry dash "-" marks the omitted field. Directives record information about the logging process itself.

1 Baker, Phillip; M, and Behlendorf, Brian, “Extended Log File Format:W3C Working Draft WD-logfile-960323,” available at http://www.w3.org/TR/WD-logfile


Customizing Log Files

• Log file headers may be customized• HTTP headers may be logged• Custom log files created by adding or

deleting headers in predefined log format


Managing Log Files

• Enable or disable logs• Maximum log size is 1,999 MB• Log Pushing• Log Rotation


Log Management Options


Log Switching• Log Pushing• Log Rotation (default)• Selected technique applies to all enabled logs• Max log file size = 1999 MBytes• Messages log

– Maximum of 10 files– Always starts at midnight Sunday– Files saved for nine weeks– Oldest file is deleted


Log Pushing• Web Access, NNTP and streaming logs can be

pushed to a FTP server at scheduled intervals• Naming

log_name.cache_hostname.timestamp (yyyy.mm.dd.hh.mm)

• Logs deleted after push• Messages log cannot be pushed• If the push fails:

– retry every 5 minutes– Autosupport message every hour– Once problem fixed – 10 files pushed every five

minutes until all are pushed

Log pushing Log pushing is one of two log management techniques that are available. With log pushing set, the Web access log, NNTP access log, and Streaming logs are pushed, using FTP, to one or two locations that you specify. (NetCache pushes only the log files that are enabled to the location you specify.) Log pushing enables you to save your logs.

Syntax Web access, NNTP access, and streaming log files pushed to an FTP server are named using the following convention:

log_name.hostname.timestamp

Requirements for log pushing The destination server must be running an FTP server and must allow the NetCache Appliance to rename files in the directory you specify as the log repository. To accomplish this, it is best to set up a user account for the NetCache Appliance logging on the FTP server and push the logs there.


What happens if a log push is not successful If a regularly scheduled log push is not successful, the NetCache Appliance attempts to push the logs to the FTP server every 5 minutes. The NetCache Appliance continues to attempt to push the logs every 5 minutes until it succeeds. If the log push continues to be unsuccessful, the NetCache Appliance sends an autosupport message once each hour saying that the log push failed.


Push Compressed Logs

• CLIconfig.system.logs.push_compressed [on|off]– Works only when logs.technique is push

Setup > System > Logging


Log Rotation

• Default technique

• Active log and 9 previous logs

• Rotated logs are stored on the NetCache until rotated out

• View with the NetCache Manager

• Naming - log_name.number

• Messages logs not rotated

Log rotation If you choose the log rotation mechanism for log handling, logs remain on the NetCache Appliance until they are rotated out. The NetCache Appliance keeps on disk the current log file and up to nine previous log files. Rotated logs remain in the NetCache Appliance logs directory and are viewable from the NetCache Manager utility.

When the NetCache Appliance rotates the logs, it renames the active log files by appending the host name and a timestamp to the name of the log file. The NetCache Appliance also deletes the oldest log file.

Changing from log rotation to log pushing When you change the log file handling mechanism from log rotation to log pushing, the NetCache Appliance schedules all the old logs to be pushed to the FTP server. If you do not want the old logs to be pushed to the FTP server, delete the logs before committing the configuration change.


Data > System Status > General

System Status These options enable the user to view the general status of this NetCache appliance. The information is current as of the time you open the page. However, you can set various refresh rates to update these statistics. Click “Refresh Now” to automatically update the statistics.

Refer to Online Help for specific configuration information.


Data > System Configuration

System Configuration This page describes the current status of the NetCache Appliance hardware configuration.


nclog Command

usage:nclog -d <filename> Delete the named lognclog -l List saved logsnclog -s Switch log files nownclog -t [-lines] <filename> View tail of the

named lognclog -ta [-lines] <protocol> View tail of the

active log of the named protocol

nclog -v <filename> View the named lognclog -va <protocol> View the active log

of the named protocol

nclog Command This command may be used to manage log files from the command line. Refer to the command line help for additional information.


Saved LogsNetCache> nclog -l

Saved logs:

/logs/messages 2000.08.31.21.23/logs/auditlog 1970.01.01.00.00/logs/http_log.0 2000.08.31.20.34/logs/nntp_log.0 2000.08.31.20.34/logs/streaming_log.0 2000.08.31.20.34/logs/streaming_details_log.0 2000.08.31.20.34

* /logs/nntp_log.1 2000.08.31.20.34* /logs/streaming_details_log.1 2000.08.31.20.34* /logs/streaming_log.1 2000.08.31.20.34

* /logs/http_log.1 2000.08.31.20.48

* Denotes current log


Setup > System > Logging

Logging The options on this page are used to specify how the NetCache appliance manages log files.



Setup > System > Logging cont’d


Messages Log

• Monitors NetCache status• Managed differently• Always enabled• Not customizable• Not rotated or pushed• Naming

– Active log file = messages– Saved log file = messages.0 through messages.9


Messages Log

Messages log The Messages Log records all NetCache messages, including reboots of the NetCache Appliance, device failures, and errors. It also reports status messages about normal NetCache operation.

Refer to Online Help for specific configuration.


Messages Log ExampleThu Aug 10 16:51:43 GMT [rc:notice]: netcache licensed

Thu Aug 10 16:51:43 GMT [rc:notice]: nntp licensed

Thu Aug 10 16:51:43 GMT [rc:notice]: mms licensed

Thu Aug 10 16:51:43 GMT [rc:notice]: rtsp licensed

Thu Aug 10 16:51:45 GMT [rc:info]: Listening for HTTP requests on 0.0.0.0:3128

Thu Aug 10 16:51:45 GMT [rc:info]: Listening for ICP requests on 0.0.0.0:3130

Thu Aug 10 16:51:46 GMT [rc:info]: Listening for ADMIN requests on 0.0.0.0:3132

Thu Aug 10 16:51:47 GMT [init_jvm:warning]: Java disabled: Missing /etc/java/classes.zip

Thu Aug 10 16:51:47 GMT [dyn_dev_qual_admin:info]: Firmware is up-to-date on all disk drives


Setup > HTTP > Logging

Log Format The Log Format option is used to define a log format style for the Web access log.



Data > Log Files > Web Access

Web access log The Web Access Log records all information about cache hits and misses. This log is only for web caching.



Web Access Log Example903286.34 0.010 10.32.69.226 HIT/200 11659 GET

http://www.cnn.com/virtual/2000/style/main.css - - "text/css" -971903286.51 1.910 10.32.69.226 MISS_PRIVATE_CCTRL/200 945 GET

http://www.cnn.com/virtual/editions/europe/2000/roof/change.pop/frameset.exclude.html - DIRECT/207.25.71.30 "text/html" -

971903286.62 0.100 10.32.69.226 MISS_PRIVATE_CCTRL/200 790 GET http://www.cnn.com/virtual/editions/europe/2000/roof/change.pop/top.exclude.html - DIRECT/207.25.71.30 "text/html" -

971903286.63 0.120 10.32.69.226 MISS/200 1284 GET http://www.cnn.com/virtual/editions/europe/2000/roof/change.pop/ban.editions.gif - DIRECT/207.25.71.30 "image/gif" -

971903286.64 0.010 10.32.69.226 MISS/200 499 GET http://www.cnn.com/SPECIALS/multimedia/images/9901/dot.gif -DIRECT/207.25.71.30 "image/gif" -

971903286.79 0.170 10.32.69.226 HIT/200 11659 GET http://www.cnn.com/virtual/2000/style/main.css - - "text/css" -

971903287.11 0.590 10.32.69.226 MISS_PRIVATE_CCTRL/200 18818 GEThttp://www.cnn.com/virtual/editions/europe/2000/roof/change.pop/content.html - DIRECT/207.25.71.23 "text/html" -


Log File Exercises

• View and Download Log Files• Configure Log File Pushing


Log Files Exercises

• 15 minutes in length• Utilizing Breakout rooms• Instructor will visit all rooms• Broadcast announcement 5 minutes

prior to regroup• Stay focussed, start telnet, start GUI• Share microphones, or no one else can

be heard


NetCache Log Files Exercises

Objectives When you have completed this module you will be able to do the following:

• View and manage log files

• Push log files

Exercise Overview The purpose of this activity is for you to perform the procedures to manage NetCache log files. During these exercises, you will be able to go through each step in the process, and you will have an opportunity to verify that each step was successfully completed.

Time Estimate

Required Hardware, Software, and Tools

Hardware

• Workstation

• NetCache Appliance

Software

• NetCache 5.4 or later

• Netscape Navigator 4.7

• Internet Explorer 5.5


Viewing and Downloading Log Files You can view up to the last lines of any log file in the View Log Files page of the NetCache Manager utility. You can also view the entire log file in ASCII format by downloading it to a remote workstation.

Perform the following steps to view a log file:

1. Open the NetCache Manager.

2. Click Data > Log Files > Messages.

3. View the last 45 lines (default) of the Messages Log.

Perform the following steps to save a log file:

4. Click Data > Log Files > Messages.

5. Select “Copy the file to your computer” and submit.

6. You are prompted for the location where you want the file saved.

7. Select the path and enter a file name.

8. Click Save.

9. On your workstation, view the entire file with any ASCII editor

Practice viewing logs from the command line: 10. In order to have something to look for in the web access log, go to a browser

that is using your NetCache as proxy, type in a request and see the page return.

Example: http://pdc/acls/blue.htm

11. Use telnet to access the NetCache command line.

12. View the active web access log: netcache>nclog –ta http

13. Right click the blue title bar of the telnet command window and select Edit > Find

14 In the Find window, in the Find what text box, type: http://pdc/acls/blue.htm and click the Find Next button.


15. Observe the web access log entry for your request.

Pushing Log Files You can configure the NetCache to push each active log (Web access log, NNTP access log, Streaming log, and Streaming details log) to an FTP or Web server so you can save your logs.

Set up NetCache log pushing as follows:

1. Open NetCache Manager.

2. Select Setup > System > Logging.

3. You can specify a number between 1 and 1,999 megabytes in the Maximum Log File Size, in Megabytes field. For this exercise choose 1 megabyte.

4. Select Rotate the Logs.

5. Scroll to the bottom and select Switch Log Files Now.

Push the logs

6. Scroll to How to Switch Log File and select Push the log file to the following URL.

7. Enter URL format the location where you want the log files pushed:

ftp://user_name:[email protected]_name.domain

ftp://user_name:password@[ftpserver IP Address]

Example:

ftp://anonymous:x@pdc/class/logs/user[1 - 8]

8. Establish a schedule that will switch the files every minute.

9. Commit the changes.

10. Ensure that the logs were pushed to the directory you specified.

11. Restore your saved configuration.

End of Log Files Exercises

Logs

Documents

Transcript of Logs