Splunk 6.3.1 Forwarding

7/25/2019 Splunk 6.3.1 Forwarding

1/159

SplunkEnterprise 6.3.1

Forwarding Data

Generated: 11/23/2015 7:49 pm

Copyright (c) 2015 Splunk Inc. All Rights Reserved


2/159

Table of Contents

Introduction to forwarding..................................................................................1

About forwardingand receiving...................................................................1Types of forwarders....................................................................................2The universal forwarder..............................................................................6

Plan your deployment.........................................................................................9System requirements..................................................................................9Forwarder deployment topologies.............................................................10Compatibility between forwarders and indexers.......................................15

Deploy a universal forwarder............................................................................16Universal forwarder deployment overview................................................16Enable a receiver......................................................................................18Install the universal forwarder software.....................................................21Consolidate datafrom multiple machines.................................................21Migrate from a light forwarder...................................................................23

Install the universal forwarder software..........................................................25Deploy a Windows universal forwarder via the installer GUI....................25Deploy a Windows universal forwarder via the command line..................36Remotely deploya Windows universal forwarder with a staticconfiguration.............................................................................................50

Deploy a *nix universal forwarder manually..............................................52Remotely deploya *nix universal forwarder with a static configuration....57Migrate a Windows light forwarder............................................................64Migrate a *nix light forwarder....................................................................66

Configure the universal forwarder...................................................................68Configure the universal forwarder.............................................................68Configure forwarders with outputs.conf.....................................................70Configure data collection on forwarders with inputs.conf..........................78Supported CLI commands........................................................................80

Deploy heavy and light forwarders..................................................................82Enable forwarding on a Splunk Enterprise instance.................................82Enable a receiver......................................................................................83Deploy a heavy forwarder.........................................................................86Deploy a light forwarder............................................................................89Heavy and light forwarder capabilities......................................................92

i


3/159

Table of Contents

Upgrade forwarders................................................. ..........................................95

Upgrade the Windows universal forwarder...............................................95Upgrade the universal forwarder for *nix systems....................................98Upgrade heavy and light forwarders.......................................................101

Perform advanced configuration....................................................................102Set up load balancing..............................................................................102Configure a forwarder to use a SOCKS proxy........................................106Configure an intermediate forwarder.......................................................110Protect against loss of in-flight data........................................................113Route andfilter data................................................................................119Make a universalforwarder part of a system image...............................134Forward data to third-party systems........................................................137Configure forwarders with outputs.conf...................................................142

Troubleshoot forwarding................................................................................151Troubleshoot forwarder/receiver connection...........................................151

Heavy and light forwarders.............................................................................154Heavy and light forwarder capabilities....................................................154

ii


4/159

Introduction to forwarding

About forwarding and receiving

You can forward data from one Splunk Enterprise instance to another SplunkEnterprise instance or even to a non-Splunk system. The Splunk Enterpriseinstance that performs theforwardingis typically a smaller footprint version ofSplunk Enterprise, called aforwarder.

A Splunk Enterprise instance thatreceivesdata from one or more forwarders iscalled areceiver. The receiver is usually a Splunk Enterpriseindexer, but canalso be another forwarder.

Sample forwarding layout

This diagram shows three forwarders sending data to a single receiver (anindexer), which then indexes the data and makes it available for searching:

Forwarders represent a much more robust solution for data forwarding than rawnetwork feeds, with their capabilities for:

Tagging of metadata (source, source type, and host)Configurable buffering

1


5/159

Data compressionSSL securityUse of any available network ports

The forwarding and receiving capability makes possible all sorts of interesting

Splunk Enterprise topologies to handle functions like data consolidation,loadbalancing, anddata routing.

Learn more about forwarding and receiving

To learn more about the fundamentals of Splunk Enterprise distributeddeployment, see the Distributed Deployment Manual.

For more information on the types of deployment topologies that you cancreate with forwarders, see Forwarder deployment topologies" in thismanual.

To learn about what intermediate forwarding is, see "Intermediateforwarding."

To learn about the different types of forwarders available, see "Types offorwarders."

Types of forwarders

There are three types of forwarders:

Theuniversal forwarderis a streamlined, dedicated version of SplunkEnterprise that contains only the essential components needed to forwarddata to receivers.

Aheavy forwarderis a full Splunk Enterprise instance, with somefeatures disabled to achieve a smaller footprint.

Alight forwarderis also a full Splunk Enterprise instance, with mostfeatures disabled to achieve as small a footprint as possible. The lightforwarder has been deprecated as of Splunk Enterprise version 6.0. Theuniversal forwarder supersedes the light forwarder for nearly all purposesand represents the best tool for forwarding data to indexers.

For a list of all deprecated features, see "Deprecated features" in the ReleaseNotes.

2


6/159

The universal forwarder

The universal forwarder can gather data from a variety of inputs and forward thedata to a Splunk Enterprise server for indexing and searching. It can also forwarddata to another forwarder as an intermediate step before sending the data

onwards to an indexer.

The sole purpose of the universal forwarder is to forward data. Unlike a fullSplunk Enterprise instance, you cannot use the universal forwarder to index orsearch data. To achieve higher performance and a lighter footprint, it has severallimitations:

The universal forwarder has no searching, indexing, or alerting capability.The universal forwarder does notparsedata. You can not use it to routedata based on its contents.

Unlike full Splunk Enterprise, the universal forwarder does not include abundled version of Python.

The universal forwarder is a separately downloadable piece of software. Unlikethe heavy and light forwarders, you do not enable it from a full Splunk Enterpriseinstance.

To learn how to download, install, and deploy a universal forwarder, see "Installthe universal forwarder software."

For more detail on universal forwarder capabilities, see "The universal

forwarder".

Heavy and light forwarders

While the universal forwarder is the preferred way to forward data, you mightneed to use heavy or light forwarders as well. Unlike the universal forwarder,both heavy and light forwarders are full Splunk Enterprise instances with certainfeatures disabled. Heavy and light forwarders differ in capability and thecorresponding size of their resource footprints.

Aheavy forwarder(sometimes referred to as a "regular forwarder") has asmaller footprint than an indexer but retains most of the capability, except that itcannot perform distributed searches. Much of its default functionality, such asSplunk Web, can be disabled, if necessary, to reduce the size of its footprint. Aheavy forwarder parses data before forwarding it and can route data based oncriteria such as source or type of event.

3


7/159

One key advantage of the heavy forwarder is that it can index data locally, aswell as forward data to another Splunk Enterprise instance. You must activatethis feature. See "Configure forwarders with outputs.conf"in this manual fordetails.

Alight forwarderhas a smaller footprint with much more limited functionality. Itforwards only unparsed data. Starting with version 4.2, it has been supersededby the universal forwarder, which provides very similar functionality in a smallerfootprint. The light forwarder continues to be available mainly to meet legacyneeds. We recommend that you always use the universal forwarder to forwardunparsed data. When you install a universal forwarder, the installer lets youmigrate checkpoint settings from any (version 4.0 or greater) light forwarder thatresides on the same machine. See "The universal forwarder" for a more detailedcomparison of the universal and light forwarders.

For detailed information on the capabilities of heavy and light forwarders, see

"Heavy and light forwarder capabilities."

To learn how to enable and deploy a heavy or light forwarder, see "Deploy aheavy or light forwarder."

Forwarder comparison

This table summarizes the similarities and differences among the three types offorwarders:

Features andcapabilities

Universalforwarder

Light forwarder Heavy forwarder

Type of SplunkEnterprise instance

Dedicatedexecutable

Full SplunkEnterprise, withmost featuresdisabled

Full SplunkEnterprise, withsome featuresdisabled

Footprint (memory,CPU load)

Smallest SmallMedium-to-large(depending onenabled features)

Bundles Python? No Yes Yes

Handles datainputs?

All types (butscripted inputsmight requirePythoninstallation)

All types All types

4


8/159


9/159

With raw data,the forwarder sends the data stream as raw TCP. it does notconvert the data into the Splunk communications format. The forwarder collectsthe data and forwards it on. This is particularly useful for sending data to anon-Splunk system.

With unparsed data,a universal forwarder performs minimal processing. It doesnot examine the data stream, but it does tag the stream with metadata to identifysource, source type, and host. It also divides the data stream into 64-kilobyteblocks and performs some rudimentary timestamping on the stream, for use bythe receiving indexer in case the events themselves have no discernibletimestamps. The universal forwarder does not identify, examine, or tag individualevents.

With parsed data,a heavy forwarder breaks the data into individual events,which it tags and then forwards to a Splunk Enterprise indexer. It can alsoexamine the events. Because the data has been parsed, the forwarder can

perform conditional routing based on event data, such as field values.

The parsed and unparsed formats are both referred to ascookeddata, todistinguish them from raw data. By default, forwarders send cooked data inthe universal forwarder's case, unparsed data, and in the heavy forwarder's case,parsed data. To send raw data instead, set the sendCookedData=falseattribute/value pair in outputs.conf.

Forwarders and indexes

Forwarders forward and route data on an index-by-index basis. By default, theyforward all external data, as well as data for the_auditinternal index. In somecases, they also forward data for the_internalinternal index. You can changethis behavior as necessary. For details, see "Filter data by target index".

The universal forwarder

Theuniversal forwarderis a separate Splunk Enterprise executable whose solepurpose is to send data from a host or other forwarder to a Splunk Enterprise

indexer.The universal forwarder replaces the Splunk Enterprise light forwarder.Instances of full Splunk Enterprise and the universal forwarder can co-exist onthe same system.

For information on deploying the universal forwarder, see "Universal forwarderdeployment overview".

6


10/159

How universal forwarder compares to full Splunk Enterprise

The universal forwarder only forwards data. Unlike a full Splunk Enterpriseinstance, it cannot index or search data. To achieve higher performance and alighter footprint, it has several limitations:

The universal forwarder has no searching, indexing, or alerting capability.The universal forwarder does notparsedata, except in certain cases.The universal forwarder does not output data via syslog.Unlike full Splunk Enterprise, the universal forwarder does not include abundled version of Python.

Scripted inputs and Python

Full Splunk Enterprise comes bundled with Python. The universal forwarder doesnot. Therefore, if you use scripted inputs with Python and you want to use thosescripts with the universal forwarder, you must first install your own version ofPython. If you have been using calls specific to Splunk Python libraries, youcannot with the universal forwarder, because those libraries exist only in fullSplunk Enterprise. You may use other scripting languages for scripted inputs withthe universal forwarder if the target host supports them (for example, PowerShellon Windows Server.)

How universal forwarder compares to the light forwarder

The universal forwarder includes only the essential components needed to

forward data to other Splunk Enterprise instances. Thelight forwarder, bycontrast, is a full Splunk Enterprise instance, with certain features disabled toachieve a smaller resource footprint. In all respects, the universal forwarderrepresents a better tool for forwarding data to indexers.

When you install the universal forwarder, you can migrate from an existing lightforwarder that runs version 4.0 or greater. See "Migrate from a light forwarder"for details.

Compared to the light forwarder, the universal forwarder provides a betterperforming solution to forwarding. These are the main performance differencesbetween the universal forwarder and the light forwarder:

The universal forwarder puts less load on the CPU, uses less memory,and has a smaller disk footprint.

The universal forwarder has a default data transfer rate of 256Kbps.

7


11/159

The universal forwarder cannot be converted to a full Splunk Enterpriseinstance.

Note:The light forwarder has been deprecated in Splunk Enterprise version 6.0.For a list of all deprecated features, see "Deprecated features" in the Release

Notes.

Read on!

For information on deploying the universal forwarder, see the topics that directlyfollow this one.

For information on third-party Windows binaries that the Windows version of theSplunk Enterprise universal forwarder ships with, read "Information on Windowsthird-party binaries distributed with Splunk Enterprise" in theInstallationManual.

For information about running the universal forwarder in Windows Safe Mode,read "Splunk Enterprise Architecture and Processes" in theInstallationManual.

8


12/159

Plan your deployment

System requirements

This topic discusses the system requirements for using a universal forwarder andany deployment considerations you should make when you deploy it.

Platform and hardware requirements =

The universal forwarder supports the same operating systems that full SplunkEnterprise supports. See the list in theInstallationmanual.

The hardware requirements for universal forwarders are as follows:

Recommended Dual-core 1.5GHz+ processor, 1GB+ RAM

Minimum 1.0Ghz processor, 512MB RAM

Licensing requirements

The universal forwarder ships with a pre-installed license. See "Types of SplunkEnterprise licenses" in theAdminmanual for details.

Other requirements

Sun SPARC systems

If you plan to install a universal forwarder on a Sun SPARC system that runsSolaris, confirm that you have patch level SUNW_1.22.7or later of the C library(libc.so.1). If you do not, the universal forwarder cannot run because it needsthis version of the library.

User rights

You must have admin or equivalent rights on the machine where you're installing

the universal forwarder.

Forwarders and indexer clusters

When using forwarders to send data to peer nodes in an indexer cluster, youdeploy and configure them a bit differently from the description in this topic. To

9


13/159


14/159


15/159


16/159


17/159


18/159


19/159


20/159


21/159


22/159


23/159


24/159


25/159


26/159


27/159


28/159


29/159


30/159


31/159


32/159


33/159


34/159


35/159


36/159


37/159

Once the installation is complete, the universal forwarder automatically starts.SplunkForwarderis the name of the universal forwarder service. You shouldconfirm that it is running.

Considerations for enabling data inputs in the installer

If you enable data inputs in the "Enable Inputs" dialog box when installing theuniversal forwarder, the installer saves the configuration that enables those

inputs into the Splunk Add-on for Windows that comes with the installer. Thisconfiguration includes index definitions.

This means that the indexer that this forwarder sends data to must already havethose indexes defined. The indexes are:

perfmonfor Performance Monitoring inputs.windowsfor generic Windows inputs.wineventlogfor Windows Event Log inputs.

By default, indexers do not have these indexes defined. To address that, eitherdefine the indexes before performing a universal forwarder installation, or installthe Splunk Add-on for Windows onto the indexer. This is a Splunk best practice.

Install the universal forwarder in "low-privilege" mode

When you specify a domain user and choose not to give that user localadministrator rights, the forwarder installs and runs in "low-privilege" mode.

There are some caveats to doing so:

You do not have administrative access to any resources on either theserver or the domain when you run the universal forwarder in low-privilegemode.

You might need to add the domain user to additional domain groups inorder to access remote resources. Additionally, you might need to add theuser to local groups to access local resources that only privileged users

34


38/159


39/159


40/159


41/159


42/159

Note:These steps are high-level procedures only. For step-by-step instructions,read "Prepare your Windows network for a Splunk Enterprise installation as anetwork or domain user" in theInstallationManual.

Install the universal forwarder

You install the universal forwarder from the command line by invokingmsiexec.exe, the Microsoft installer program.

For 32-bit platforms, use splunkuniversalforwarder--x86-release.msi :

msiexec.exe /i splunkuniversalforwarder--x86-release.msi

[]... [/quiet]

For 64-bit platforms, use splunkuniversalforwarder--x64-release.msi :


[]... [/quiet]

The value of varies according to the particular release; for example,splunkuniversalforwarder-4.2-86454-x64-release.msi .

Important:We do not recommend that you run the 32-bit version of the universalforwarder on a 64-bit platform.

Command line flags allow you to configure your forwarder at installation time.Using command line flags, you can specify a number of settings, including:

The user the universal forwarder runs as. (Be sure the user you specifyhas the appropriate permissions to access the content you want toforward.)

Whether or not the forwarder runs in "low-privilege" mode - as a user whodoes not have local administrative access.

The receiving Splunk Enterprise instance that the universal forwarder willsend data to.

A deployment server for updating the configuration.

The Windows event logs to index.Whether the universal forwarder should start automatically when theinstallation is completed.

The following sections list the flags available and provide a few examples ofvarious configurations.

39


43/159


44/159

System user. See"Choose theWindows userSplunk should runas".

RECEIVING_INDEXER="" Use this flag tospecify thereceiving indexerto which theuniversalforwarder willforward data.Enter the name(hostname or IPaddress) and

receiving portofthe receiver. Thisflag accepts only asingle receiver. Tospecify multiplereceivers (toimplement loadbalancing), youmust insteadconfigure thissetting through theCLI oroutputs.conf.

For information onsetting up areceiver, see"Enable areceiver".Note:This flag isoptional, but if you

don't specify it andalso don't specifyDEPLOYMENT_SERVER,the universalforwarder will beunable to function,

n/a

41


45/159


46/159


47/159


48/159

setRECEIVING_INDEXER

for them to haveany effect.

CLONEPREP=1|0

Deletes anyinstance-specificdata in preparationfor creating aclone of amachine. Thisinvokes the splunkclone-prep

command from theCLI.

0 (do not preparecloning.)

SET_ADMIN_USER=1|0 Specifies whether

or not the user youspecify is anadministrator. Ifyou set this flag to0, it allows theuniversalforwarder to run in"low-privilege"mode - as a userwithout

administratorprivileges on thelocal machine.This mode isavailable forcustomers that donot have the abilityto run programs asan administratoron servers. Read"Run the universal

forwarder inlow-privilegemode" later in thistopic for additionalinformation andcaveats.

1 (Install the univ

a user with adminprivileges. The unruns in normal m"low-privilege" mo

45


49/159


50/159


51/159

msiexec.exe /i splunkuniversalforwarder_x64.msi /l*v

install_splunkforwarder-6.1-201357-x64-release.msi.log

LOGON_USERNAME=adtest1\lowpriv-testuser LOGON_PASSWORD=win1@splunk

AGREETOLICENSE=Yes SET_ADMIN_USER=0 /quiet

Test the deployment

Test your configured universal forwarder on a single machine, to make sure itfunctions correctly, before deploying the universal forwarder across yourenvironment. Confirm that the universal forwarder is getting the desired inputsand sending the right outputs to the indexer.

If you migrated from an existing forwarder, make sure that the universalforwarder is forwarding data from where the old forwarder left off. If it isn't, youprobably need to modify or add data inputs, so that they conform to those on theold forwarder.

Important:Migration does not automatically copy any configuration files; youmust set those up yourself. The usual way to do this is to copy the files, includinginputs.conf, from the old forwarder to the universal forwarder. Compare theinputs.conffiles on the universal forwarder and the old forwarder to ensure thatthe universal forwarder has all the inputs that you want to maintain.

If you migrated from an existing forwarder, you can delete that old instance onceyour universal forwarder has been thoroughly tested and you're comfortable withthe results.

Perform additional configuration

You can update your universal forwarder's configuration, post-installation, bydirectly editing its configuration files, such as inputs.confand outputs.conf.You can also update the configuration using the CLI. See "Configure theuniversal forwarder"for information.

Note:When you use the CLI, you might need to authenticate into the forwarderto complete commands. The default credentials for a universal forwarder are:

Username:adminPassword:changeme

For information on distributing configuration changes across multiple universalforwarders, see "About deployment server" in theUpdating Splunk EnterpriseInstancesmanual.

48


52/159


53/159


54/159

Required installation flags

Besides specifying /quietmode, you must include, at a minimum, thesecommandline flags:

AGREETOLICENSE=YesRECEIVING_INDEXER=""

At least one data input flag, such as WINEVENTLOG_APP_ENABLE=1. You canadd as many data input flags as you need.

See "Deploy a Windows universal forwarder via the command line"for a list of allavailable command line flags.

Example installation

This example sets the universal forwarder to run as Local System user, getinputs from Windows security and system event logs, send data to indexer1, andlaunch automatically:

msiexec.exe /i splunkuniversalforwarder_x86.msi

RECEIVING_INDEXER="indexer1:9997" WINEVENTLOG_SEC_ENABLE=1

WINEVENTLOG_SYS_ENABLE=1 AGREETOLICENSE=Yes /quiet

Deploy with a secure configuration

To deploy a secure configuration, you can specify an SSL certifcate. Use theseinstallation flags:

CERTFILE=ROOTCACERTFILE=CERTPASSWORD=

For more information, see this list of supported commandline flags.

Test the deployment

Test your configured universal forwarder on a single machine, to make sure itfunctions correctly, before deploying the universal forwarder across yourenvironment. Confirm that the universal forwarder is getting the desired inputsand sending the right outputs to the indexer.

51


55/159

Deploy a *nix universal forwarder manually

This topic describes how to install the universal forwarder software on a *nixhost, such as Linux or Solaris. It assumes that you plan to install directly onto thehost, rather than use a deployment tool. This type of deployment best suits theseneeds:

Small deployments.Proof-of-concept test deployments.System image or virtual machine for eventual cloning.

Before following the procedures in this topic, see "Universal forwarderdeployment overview".

Steps to deployment

Once you have downloaded the universal forwarder and have planned yourdeployment, perform these steps:

1. Install the universal forwarder.

2. Configure(and optionally migrate) the universal forwarder.

3. Test the deployment.

4. Perform any additional configuration.

5. Deploy the universal forwarder across your environment.

Install the universal forwarder

The universal forwarder installation package is available for download fromsplunk.com.

You can install the universal forwarder on a *nix host with a package or a tar file.To install the universal forwarder on any of the supported *nix operating systems,

see the installation topic for installing a full Splunk Enterprise instance in theInstallation Manual:

Install on LinuxInstall on SolarisInstall on Mac OS

52


56/159


57/159


58/159


59/159


60/159

Troubleshoot your deployment

The universal forwarder forwards some internal logs to the receiving indexer.These are:

$SPLUNK_HOME/var/log/splunk/splunkd.log

$SPLUNK_HOME/var/log/splunk/metrics.log

$SPLUNK_HOME/var/log/splunk/license_audit.log

The logs can be searched on the indexer for errors (index=_internalhost=).

If the universal forwarder is malfunctioning such that it cannot forward the logs,use a text editor or the greputility to examine them on the universal forwarderitself.

Remotely deploy a *nix universal forwarder with astatic configuration

One of the main ways to deploy multiple universal forwardersremotely is throughscripting. You can also use deployment management tools such as yum andPuppet. This topic focuses on script deployment.

For information on how to install and configure a single universal forwarder, see

"Deploy a nix universal forwarder manually". That topic explains how to installonto a wide variety of *nix platforms from a package or a tar file and how toconfigure (and optionally migrate) using the CLI.

Steps to deployment

Once you have downloaded the universal forwarder and have planned yourdeployment, as described in "Universal forwarder deployment overview", performthese steps:

1.Install and configure the universal forwarder on a test machine, as described in"Deploy a nix universal forwarder manually".

2.Test and tune the configuration.

3.Create a script wrapper for the installation and configuration commands.

57


61/159

4.Run the script on representative target machines to verify that it works with allrequired shells.

5.Execute the script against the desired set of hosts.

6.Review log files on the forwarder to confirm that it has connected to thereceiving indexer.

Create and execute the script

Once you've validated your installation and configuration process by testing afully configured universal forwarder, you're ready to incorporate the process intoa script.

Script requirements

You need to place the installation package or tar file in a network locationaccessible by the target machines. You can set this up so that the script pushesthe file over to each target host, or you can place the file in a generallyaccessible location, such as an NFS mount.

The script is responsible for error reporting. Logging to Splunk either directly orvia a flat file is recommended.

Sample script

Here's a sample script you can use as a starting point. Note that this is only anexample of the type of script you could create for your deployment. Thecomments in the script provide some guidance on how to modify it for yourneeds; however, the script will likely require further modification, beyond thatindicated by the comments.

Among other things, the script:

Deploys the forwarder's tar file to a list of hosts specified in a file that theHOST_FILEvariable points to. You will need to provide this file, in theformat specified in the script comments.

Specifies the location on each destination host where the tar file will getunpacked.

Specifies a Splunk Enterprise instance to serve as adeployment serverthat can subsequently manage and update the forwarders. This is an

58


62/159

optional configuration step.

Starts the forwarder executable on each host.

The script is well commented; be sure to study it carefully before modifying it for

your environment.

Here's the sample deployment script:

#!/bin/sh

# This script provides an example of how to deploy the universal

forwarder

# to many remote hosts via ssh and common Unix commands.

#

# Note that this script will only work unattended if you have SSH host

keys# setup & unlocked.

# To learn more about this subject, do a web search for "openssh key

management".

# ----------- Adjust the variables below -----------

# Populate this file with a list of hosts that this script should

install to,

# with one host per line. You may use hostnames or IP addresses, as

# applicable. You can also specify a user to login as, for example,

"foo@host".

## Example file contents:

# server1

# server2.foo.lan

# you@server3

# 10.2.3.4

HOSTS_FILE="/path/to/splunk.install.list"

# This is the path to the tar file that you wish to push out. You may

# wish to make this a symlink to a versioned tar file, so as to minimize

# updates to this script in the future.

SPLUNK_FILE="/path/to/splunk-latest.tar.gz"

# This is where the tar file will be stored on the remote host during

# installation. The file will be removed after installation. You

normally will

# not need to set this variable, as $NEW_PARENT will be used by default.

#

59


63/159

# SCRATCH_DIR="/home/your_dir/temp"

# The location in which to unpack the new tar file on the destination

# host. This can be the same parent dir as for your existing

# installation (if any). This directory will be created at runtime, if

it does

# not exist.

NEW_PARENT="/opt"

# After installation, the forwarder will become a deployment client of

this

# host. Specify the host and management (not web) port of the

deployment server

# that will be managing these forwarder instances. If you do not wish

to use

# a deployment server, you may leave this unset.

#

# DEPLOY_SERV="splunkDeployMaster:8089"

# A directory on the current host in which the output of each

installation

# attempt will be logged. This directory need not exist, but the user

running

# the script must be able to create it. The output will be stored as

# $LOG_DIR/. If installation on a host fails,

a

# corresponding file will also be created, as

# $LOG_DIR/.failed.

LOG_DIR="/tmp/splunkua.install"

# For conversion from normal Splunk Enterprise installs to the universal

forwarder:

# After installation, records of progress in indexing files (monitor)

# and filesystem change events (fschange) can be imported from an

existing

# Splunk Enterprise (non-forwarder) installation. Specify the path to

that installation here.

# If there is no prior Splunk Enterprise instance, you may leave this

variable empty ("").

#

# NOTE: THIS SCRIPT WILL STOP THE SPLUNK ENTERPRISE INSTANCE SPECIFIED

HERE.

## OLD_SPLUNK="/opt/splunk"

# If you use a non-standard SSH port on the remote hosts, you must set

this.

# SSH_PORT=1234

# You must remove this line, or the script will refuse to run. This is

60


64/159


65/159


66/159

# end of remote script.

#

#

exec 5>&1 # save stdout.

exec 6>&2 # save stderr.

echo "In 5 seconds, will copy install file and run the following script

on each"

echo "remote host:"

echo

echo "===================="

echo "$REMOTE_SCRIPT"

echo "===================="

echo

echo "Press Ctrl-C to cancel..."

test -z "$MORE_FASTER" && sleep 5

echo "Starting."

# main loop. install on each host.

for DST in `cat "$HOSTS_FILE"`; do

if [ -z "$DST" ]; then

continue;

fi

LOG="$LOG_DIR/$DST"

FAILLOG="${LOG}.failed"

echo "Installing on host $DST, logging to $LOG."

# redirect stdout/stderr to logfile.

exec 1> "$LOG"

exec 2> "$LOG"

if ! ssh $SSH_PORT_ARG "$DST" \

"if [ ! -d \"$NEW_PARENT\" ]; then mkdir -p \"$NEW_PARENT\"; fi";

then

touch "$FAILLOG"

# restore stdout/stderr.

exec 1>&5

exec 2>&6

continue

fi

# copy tar file to remote host. if ! scp $SCP_PORT_ARG "$SPLUNK_FILE" "${DST}:${DEST_FILE}"; then

touch "$FAILLOG"


exec 1>&5

exec 2>&6

continue

fi

63


67/159

# run script on remote host and log appropriately.

if ! ssh $SSH_PORT_ARG "$DST" "$REMOTE_SCRIPT"; then

touch "$FAILLOG" # remote script failed.

else

test -e "$FAILLOG" && rm -f "$FAILLOG" # cleanup any past attempt

log. fi


exec 1>&5

exec 2>&6

if [ -e "$FAILLOG" ]; then

echo " --> FAILED


68/159


69/159


70/159

Important:Make sure you install the universal forwarder into a different directoryfrom the existing light forwarder. Since the default install directory for theuniversal forwarder is /opt/splunkforwarderand the default install directory forfull Splunk Enterprise (including the light forwarder) is /opt/splunk, you'll be safeif you just stick with the defaults.

3.In the universal forwarder's installation directory, (the new $SPLUNK_HOME),create a file named old_splunk.seed; in other words:$SPLUNK_HOME/old_splunk.seed. This file must contain a single line, consisting ofthe path of theoldforwarder's $SPLUNK_HOMEdirectory. For example:/opt/splunk.

4.Start the universal forwarder:

$SPLUNK_HOME/bin/splunk start

The universal forwarder will migrate the checkpoint files from the forwarderspecified in the $SPLUNK_HOME/old_splunk.seedfile. Migration only occurs thefirst time you run the startcommand. You can leave the old_splunk.seedinplace; it only gets examined the first time you start the forwarder after installing it.

5.Perform any additional configuration of the universal forwarder, as described in"Deploy a nix universal forwarder manually." Since the migration process onlycopies checkpoint files, you will probably want to manually copy over the oldforwarder's inputs.confconfiguration file (or at least examine it, to determinewhat data inputs it was monitoring).

Once the universal forwarder is up and running (and after you've tested to ensuremigration worked correctly), you can uninstall the old forwarder.

67


71/159


72/159


73/159

Configure forwarders with outputs.conf

The outputs.conf file defines howforwarders send data to receivers. You canspecify some output configurations at installation time (Windows universalforwarders only) or through Splunk Web (heavy/light forwarders only) or the CLI,but most advanced configuration settings require that you directly editoutputs.conf. The topics describing various topologies, such as load balancingand data routing, provide detailed examples on configuring outputs.conftosupport those topologies.

Important:Although outputs.confis a critical file for configuring forwarders, itspecifically addresses theoutputsfrom the forwarder. To specify theinputsto aforwarder, you must separately configure the inputs, as you would for any SplunkEnterprise instance. For details on configuring inputs, see "Add data andconfigure inputs" in the Getting Data In manual.

Types of outputs.conf files

A single forwarder can have multiple outputs.conffiles (for instance, one locatedin an apps directory and another in /system/local). No matter how manyoutputs.conffiles the forwarder has and where they reside, the forwardercombines all their settings, using the rules of location precedence, as describedin "Configuration file precedence". Your installation will contain both default andcustom outputs.conffiles.

Default versions

Splunk Enterprise ships with these default versions of outputs.conf:

On the universal forwarder: The universal forwarder has two defaultoutputs.conffiles, one in $SPLUNK_HOME/etc/system/defaultand theother in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default . Thedefault version in the SplunkUniversalForwarderapp has precedenceover the version under /etc/system/default.

On heavy and light forwarders: These have a single defaultoutputs.conffile, located in $SPLUNK_HOME/etc/system/default.

Important:Do not touch default versions of any configuration files, for reasonsexplained in "About configuration files".

70


74/159


75/159


76/159

as well as forward the data to receiving indexers in the target groups. If setto "false" (the default), the forwarder just forwards data but does not indexit. This attribute is only available for heavy forwarders; universal and lightforwarders cannot index data.

Default target groups

To set default groups for automatic forwarding, include the defaultGroupattributeat the global level, in your [tcpout]stanza:

[tcpout]

defaultGroup= , , ...

The defaultGroup specifies one or more target groups, defined later intcpout:stanzas. The forwarder will send all events to the

specified groups.

If you donotwant to forward data automatically, don't set the defaultGroupattribute. (Prior to 4.2, you were required to set the defaultGroupto some value.This is no longer necessary.)

For some examples of using the defaultGroupattribute, see "Route and filterdata".

Target group stanza

The target group identifies a set of receivers. It also specifies how the forwardersends data to those receivers. You can define multiple target groups.

Here's the basic pattern for the target group stanza:

[tcpout:]

server=, , ...

=

=

...

To specify a receiving server in a target group, use the format:, where is the receiving server'sreceiving port. For example, myhost.Splunk.com:9997. You can specify multiplereceivers and the forwarder will load balance among them.

73


77/159

See "Define typical deployment topologies", later in this topic, for information onhow to use the target group stanza to define several deployment topologies.

Single-server stanza

You can define a specific configuration for an individual receiving indexer.However, the receiver must also be a member of a target group.

When you define an attribute at the single-server level, it takes precedence overany definition at the target group or global level.

Here is the syntax for defining a single-server stanza:

[tcpout-server://:]

=

= ...

Example

The following outputs.confexample contains three stanzas for sending tcpout toSplunk Enterprise receivers:

Global settings. In this example, there is one setting, to specify adefaultGroup.

Settings for a single target group consisting of two receivers. Here, we are

specifying a load-balanced target group consisting of two receivers.

Settings for one receiver within the target group. In this stanza, you canspecify any settings specific to the mysplunk_indexer1receiver.

[tcpout]

defaultGroup=my_indexers

[tcpout:my_indexers]

server=mysplunk_indexer1:9997, mysplunk_indexer2:9996

[tcpout-server://mysplunk_indexer1:9997]

Define typical deployment topologies

This section shows how you can configure a forwarder to support several typicaldeployment topologies. See the other topics in the "Forward data" section of thisbook for information on configuring forwarders for other topologies.

74


78/159


79/159

The forwarder will send full data streams to both cloned_group1andcloned_group2.The data will be load-balanced within each group, rotatingamong receivers every 30 seconds (the default frequency).

Note:For syslog and other output types, you must explicitly specify routing as

described here: "Route and filter data".

Commonly used attributes

The outputs.conffile provides a large number of configuration options that offerconsiderable control and flexibility in forwarding. Of the attributes available,several are of particular interest:

Attribute DefaultWhere

configuredValue

defaultGroup n/aglobalstanza

A comma-separated list of one ormore target groups. Forwardersends all events to all specifiedtarget groups. Don't set thisattribute if you don't want eventsautomatically forwarded to a targetgroup.

indexAndForward falseglobalstanza

If set to "true", the forwarder willindex all data locally, in addition toforwarding the data to a receiving

indexer.

Important:This attribute is onlyavailable for heavy forwarders. Auniversal forwarder cannot indexlocally.

server n/atarget groupstanza

Required. Specifies the server(s)that will function as receivers for theforwarder. This must be set to avalue using the format

:,where is the receivingserver's receiving port.

disabled false any stanzalevel

Specifies whether the stanza isdisabled. If set to "true", it isequivalent to the stanza not being

76


80/159


81/159

run-time interval = dnsResolutionInterval + (number of receivers in

server attribute - 1) * 30

The run-time interval is extended by 30 seconds for each additional receiverspecified in the serverattribute; that is, for each additional receiver across whichthe forwarder is load balancing. The dnsResolutionIntervalattribute defaults to300 seconds.

For example, if you leave the attribute at the default setting of 300 seconds andthe forwarder is load-balancing across 20 indexers, DNS resolution will occurevery 14 minutes:

(300 + ((20 - 1) * 30)) = 870 seconds = 14 minutes

If you change dnsResolutionIntervalto 600 seconds, and keep the number ofload-balanced indexers at 20, DNS resolution will occur every 19.5 minutes:

(600 + ((20 - 1) * 30)) = 1170 seconds = 19.5 minutes

Configure data collection on forwarders withinputs.conf

This topic discusses how to configure data inputs on a universal forwarder byediting the inputs.conf configuration file.

Universal forwarders can collect any type of data that a full Splunk Enterpriseinstance can. If you install the Windows universal forwarder, you can collectWindows Event Logs, performance metrics, Registry changes, and any otherWindows data that a full instance can gather.

Universal forwarders can have apps and add-ons installed, and those apps andadd-ons can collect data. The one difference is that a universal forwarder cannotdisplay any data, as there is no Web interface to do so. There also is no interface

to edit configuration files, so unless you install an app or add-on that has aconfigured inputs.conffile, you must configure that file yourself.

In nearly all cases, you must edit inputs.confin the$SPLUNK_HOME/etc/system/localdirectory. If you have an app installed and wantto make changes to its input configuration, edit

78


82/159


83/159


84/159

As described above, it's theobjectthat determines whether a command is validin the universal forwarder. For example, the above list includes the monitorobject. Therefore, the add monitorand edit monitorcommand/objectcombinations are both valid. For more information on the monitorobject, see"Use the CLI to monitor files and directories" in the Getting Data In manual.

For more details on using the CLI in general, see the "Administer SplunkEnterprise with the CLI" chapter in the Admin manual. In particular, the topic "CLIadmin commands" provides details on CLI syntax, including a list of allcommands supported by full Splunk Enterprise and the objects they can actupon.

81


85/159

Deploy heavy and light forwarders

Enable forwarding on a Splunk Enterprise instance

This topic lists the key steps involved in setting up heavy and light forwarders onfull Splunk Enterprise instances, with links to more detailed topics. You mustinstall a full Splunk Enterprise instance before enabling and configuring a heavyor light forwarder.

Note:This topic assumes that your receivers are indexers. However, in somescenarios, discussed elsewhere, a forwarder also serves as receiver. The set-upis basically much the same for any kind of receiver.

If you want to forward data across a proxy, see "Configure a forwarder to use aSOCKS proxyin this manual.

Set up forwarding and receiving: heavy or light forwarders

Note:The light forwarder has been deprecated in Splunk Enterprise version 6.0.For a list of all deprecated features, see the topic "Deprecated features" in theRelease Notes.

1.Install the full Splunk Enterprise instances that will serve as forwarders andreceivers. See theInstallation Manualfor details.

2.Use Splunk Web or the CLI to enable receiving on the instances designated asreceivers. See "Enable a receiver" in this manual.

3.Use Splunk Web or the CLI to enable forwarding on the instances designatedas forwarders. See Deploy a heavy forwarder" or "Deploy a light forwarder" inthis manual.

4.Specify data inputs for the forwarders in the usual manner. See "What SplunkEnterprise can index" in theGetting Data Inmanual.

5.Specify the forwarders' output configurations - the receiver(s) that they shouldsend data to. You can do so through Splunk Web, the CLI, or by editing theoutputs.conffile. You get the greatest flexibility by editing outputs.conf. Fordetails, see "Deploy a heavy or light forwarder", as well as the other topics in thissection, including "Configure forwarders with outputs.conf."

82


86/159


87/159

will receive data on port 9997. By convention, receivers listen on port 9997, butyou can specify any unused port. You can use a tool like netstatto determinewhat ports are available on your system. Make sure the port you select is not inuse by splunkweb or splunkd.

6.ClickSave.You must restart the instance to complete the process.

Set up receiving with Splunk CLI

To enable receiving, run the CLI command:

splunk enable listen -auth :

For , substitute the port you want the receiver to listen on (the receivingport). For example, if you enter "9997," the receiver will receive data on port

9997. By convention, receivers listen on port 9997, but you can specify anyunused port. You can use a tool like netstatto determine what ports areavailable on your system. Make sure the port you select is not in use bysplunkweb or splunkd.

The splunk enable listencommand creates a [splunktcp]stanza ininputs.conf. For example, if you set the port to "9997", it creates the stanza[splunktcp://9997].

Set up receiving with the configuration file

You can enable receiving on your Splunk Enterprise instance by configuringinputs.confin $SPLUNK_HOME/etc/system/local. To configure a universalforwarder as an intermediate forwarder (a forwarder that functions also as areceiver), use this method.

To enable receiving, add a [splunktcp]stanza that specifies the receiving port.In this example, the receiving port is 9997:

[splunktcp://9997]

disabled = 0

For further details, refer to the inputs.conf spec file.

Note:The forms [splunktcp://9997]and [splunktcp://:9997](one colon ortwo) are semantically equivalent. Use either one.

84


88/159


89/159

Deploy a heavy forwarder

To enable forwarding and receiving, you configure both areceiverand aforwarder. The receiver is the Splunk Enterprise instance receiving the data; theforwarder sends data to the receiver.

You must first set up the receiver, as described in "Enable a receiver". You canthen set up forwarders to send data to that receiver.

Setting up aheavyforwarder is a two step process:

1.Install a full Splunk Enterprise instance.

2.Enable forwarding on the instance.

The sections that follow describe these steps in detail.

Important:This topic describes deployment and configuration issues specific toheavy forwarders. For information on how to deploy auniversal forwarder, see"Universal forwarder deployment overview".

Install a full Splunk Enterprise instance

To deploy a heavy forwarder, you must first install a full Splunk Enterpriseinstance. For detailed information about installing Splunk Enterprise, includingsystem requirements and licensing issues, see the Installation manual.

Once the instance has been installed, you can enable forwarder functionality onit.

Set up forwarding

You can use Splunk Web or the CLI as a quick way to enable forwarding in aSplunk Enterprise instance.

You can also enable, as well as configure, forwarding by creating an

outputs.conffile on the Splunk Enterprise instance. Although setting upforwarders with outputs.confrequires a bit more initial knowledge, there areobvious advantages to performing all forwarder configurations in a singlelocation. Most advanced configuration options are available only throughoutputs.conf. In addition, if you will be enabling and configuring a number offorwarders, you can easily accomplish this by editing a single outputs.conffile

86


90/159


91/159


92/159

Important:Before doing an upgrade, consider whether you really need to. Inmany cases, there's no compelling reason to upgrade a forwarder. Forwardersare always compatible with later version indexers, so you do not need to upgradethem just because you've upgraded the indexers they're sending data to.

Back up your files first

Before you perform the upgrade, we strongly recommend that you back up all ofyour files. Most importantly, back up your Splunk Enterprise configuration files.For information on backing up configurations, read "Back up configurationinformation" in the Admin manual.

If you're upgrading a heavy forwarder that's indexing data locally, you also needto back up the indexed data. For information on backing up data, read "Back upindexed data" in theManaging Indexers and Clusters of Indexersmanual.

You cannot downgrade to a previous version; if you need to revert to an olderforwarder release, reinstall the instance.

Deploy a light forwarder


To enable forwarding and receiving, you configure both areceiverand aforwarder. The receiver is the Splunk Enterprise instance receiving the data; theforwarder sends data to the receiver.

You must first set up the receiver. You can then set up forwarders to send data tothat receiver.

Setting up alightforwarder is a two step process:

1.Install a full Splunk Enterprise instance.

2.Enable forwarding on the instance.

The sections that follow describe these steps in detail.

Important:This topic describes deployment and configuration issues specific tolight forwarders. For information on how to deploy auniversal forwarder, see

89


93/159

"Universal forwarder deployment overview".

Install a full Splunk Enterprise instance

To deploy a light forwarder, you must first install a full Splunk Enterprise

instance. For detailed information about installing Splunk Enterprise, includingsystem requirements and licensing issues, see the Installation manual.

Once the instance has been installed, you can enable light forwarder functionalityon it.

Note:When you install a Splunk Enterprise instance to be used as a lightforwarder, select the forwarder license. For more information, see "Types ofSplunk licenses".

Set up forwarding

You can use the CLI as a quick way to enable forwarding.

You can also enable, as well as configure, forwarding by creating anoutputs.conffile on the Splunk Enterprise instance. Although setting upforwarders with outputs.confrequires a bit more initial knowledge, there areobvious advantages to performing all forwarder configurations in a singlelocation. Most advanced configuration options are available only throughoutputs.conf. In addition, if you will be enabling and configuring a number offorwarders, you can easily accomplish this by editing a single outputs.conffile

and making a copy for each forwarder. See the topic "Configure forwarders withoutputs.conf"for more information.

Set up light forwarding with the CLI

With the CLI, setting up forwarding is a two step process. First you enableforwarding on the instance. Then you start forwarding to a specified receiver.

To access the CLI, first navigate to $SPLUNK_HOME/bin/.

To enable the light forwarder mode,enter:

splunk enable app SplunkLightForwarder -auth :

To disable the light forwarder mode,enter:

90


94/159

splunk disable app SplunkLightForwarder -auth :

By disabling forwarding, this command reverts the forwarder to a full SplunkEnterprise instance.

Important:After invoking either of these commands, restart the forwarder.

Start forwarding activity from the CLI

To access the CLI, first navigate to $SPLUNK_HOME/bin/.

To start forwarding activity,specify the receiver with the splunk addforward-servercommand:

splunk add forward-server : -auth :

To end forwarding activity,enter:

splunk remove forward-server : -auth :

Note:Although this command ends forwarding activity, the instance remainsconfigured as a forwarder. To revert the instance to a full Splunk Enterpriseinstance, use the disablecommand, as described earlier in this topic.

Important:After invoking either of these commands, restart the forwarder.

Upgrade a forwarder

To upgrade a forwarder to a new version, just upgrade the instance in the usualfashion. For details, read the upgrade section of the Installation manual.

Important:Before doing an upgrade, consider whether you really need to. Inmany cases, there's no compelling reason to upgrade a forwarder. Forwardersare always compatible with later version indexers, so you do not need to upgradethem just because you've upgraded the indexers they're sending data to.

Back up your files first

Before you perform the upgrade, we strongly recommend that you back up all ofyour files. Most importantly, back up your configuration files. For information onbacking up configurations, read "Back up configuration information" in the Admin

91


95/159

manual.

Heavy and light forwarder capabilities

Certain capabilities are disabled in heavy and light forwarders. This sectiondescribes forwarder capabilities in detail.


Heavy forwarder details

The heavy forwarder has all Splunk Enterprise functions and modules enabled by

default, with the exception of the distributed search module. The file$SPLUNK_HOME/etc/apps/SplunkForwarder/default/default-mode.conf includesthis stanza:

[pipeline:distributedSearch]

disabled = true

For a detailed view of the exact configuration, see the configuration files for theSplunkForwarder application in$SPLUNK_HOME/etc/apps/SplunkForwarder/default .

Light forwarder details

Most features of Splunk Enterprise are disabled in the light forwarder.Specifically, the light forwarder:

Disables event signing and checking whether the disk is full($SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/default-mode.conf ).

Limits internal data inputs to splunkdand metrics logs only($SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/inputs.conf ).

Disables all indexing($SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/indexes.conf ).

Does not use transforms.confand does not fully parse incoming data,but the CHARSET, CHECK_FOR_HEADER, NO_BINARY_CHECK,PREFIX_SOURCETYPE,and sourcetypeproperties from props.confare used.

Disables the Splunk Web interface($SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/web.conf ).

92


96/159

Limits throughput to 256KBps($SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/limits.conf ).

Disables the following modules in$SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/default-mode.conf :

[pipeline:indexerPipe]

disabled_processors= indexandforward, diskusage,

signing,tcp-output-generic-processor, syslog-output-generic-processor,

http-output-generic-processor, stream-output-processor

[pipeline:distributedDeployment]

disabled = true

[pipeline:distributedSearch]

disabled = true

[pipeline:fifo]

disabled = true

[pipeline:merging]

disabled = true

[pipeline:typing]

disabled = true

[pipeline:udp]

disabled = true

[pipeline:tcp]

disabled = true

[pipeline:syslogfifo]

disabled = true

[pipeline:syslogudp]

disabled = true

[pipeline:parsing]

disabled_processors=utf8, linebreaker, header, sendOut

[pipeline:scheduler]

disabled_processors = LiveSplunks

These modules include the deployment server (not the deployment client),distributed search, named pipes/FIFOs, direct input from network ports, and thescheduler.

93


97/159

The defaults for the light forwarder can be tuned to meet your needs byoverriding the settings in$SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/default-mode.conf ona case-by-case basis.

Purge old indexes

When you convert an indexer instance to a light forwarder, among other things,you disable indexing. In addition, you no longer have access to any datapreviously indexed on that instance. However, the data still exists.

If you want to purge that data from your system, you must first disable theSplunkLightForwarder app, then run the CLI cleancommand, and then renablethe app. For information on the cleancommand, see "Remove indexed data fromSplunk" in theManaging Indexers and Clusters of Indexersmanual.

Considerations for forwarding structured data

Note:When you forward structured data (data with source types that use theINDEXED_EXTRACTIONSfeature) you must perform any parsing, extraction, orfiltering changes on the forwarder, not the indexer. See Forward data extractedfrom header files" in the Getting Data In manual.

94


98/159

Upgrade forwarders

Upgrade the Windows universal forwarder

This topic describes the procedure for upgrading your Windows universalforwarder from version 5.0.x, 6.0.x, 6.1.x, or 6.2.x to 6.3.

When you upgrade a universal forwarder, the installer performs an upgrade withno configuration changes. If you need to change any configuration settings onyour forwarders, you can do so after the upgrade. A deployment server canassist in the configuration update process.

This topic describes three upgrade scenarios:

Upgrade a single forwarder with the GUI installerUpgrade a single forwarder with the command line installerPerform a remote upgrade of a group of forwarders

For deployments of any size, you will most likely want to use this last scenario.

Before you upgrade

Be sure to read this section before performing an upgrade. Also, read "How toupgrade Splunk Enterprise" in the Installation Manual for up-to-date information

and potential issues you might encounter when upgrading.

Confirm that an upgrade is necessary

Before performing an upgrade, consider whether you really need to. In mostcases, there is no compelling reason to upgrade a forwarder. Forwarders arealways compatible with later version indexers, so you do not need to upgradethem just because you have upgraded the indexers they're sending data to.

No platform architecture changes

Due to how the universal forwarder installer is configured, you cannot upgrade a32-bit version of the universal forwarder with the 64-bit universal forwarderinstaller. If you are in this situation, the follow these instructions:

1.Back up your configurations, including any apps or add-ons (in%SPLUNK_HOME%\etc\apps). Also back up the checkpoint files located

95


99/159

in %SPLUNK_HOME%\var\lib\modinputs\

2.Uninstall the existing 32-bit forwarder.

3. Installthe 64-bit forwarder.

4.Restore your apps, configurations and checkpoints by copying them to theappropriate directories:

%SPLUNK_HOME%\etc\system\localfor configuration files.%SPLUNK_HOME%\etc\appsfor apps and add-ons.%SPLUNK_HOME%\var\lib\modinputsfor checkpoint files.

Back your files up

Before you perform the upgrade, we strongly recommend that you back up your

configuration files. For information on backing up configurations, read "Back upconfiguration information" in theAdminmanual.

Splunk Enterprise does not provide a means of downgrading to a previousversion; if you need to revert to an older forwarder release, just uninstall thecurrent version and reinstall the older release.

Upgrade using the GUI installer

You can upgrade a single forwarder with the GUI installer:

1.Download the new MSI file from the universal forwarder download page.

2.Double-click the MSI file. The installer displays the "Accept license agreement"panel.

3.Accept the license agreement and click "Install." The installer then upgradesthe forwarder while retaining the existing configuration.

Note:You do not need to stop the forwarder before upgrading. The installer doesthis automatically as part of the upgrade process.

4.The forwarder will start automatically when you complete the installation.

The installer puts a log of upgrade changes in the %TEMP%directory. It also reportsany errors in the Application Event Log.

96


100/159

Upgrade using the command line

You can upgrade a single forwarder by running the command line installer. Toupgrade a group of forwarders, you can load the command line installer into adeployment tool, as described below.

Here are the steps for using the command line installer to upgrade a singleforwarder:

1.Download the new MSI file from the Splunk universal forwarder downloadpage.

2.Install the universal forwarder from the command line by invoking msiexec.exe.

For 32-bit platforms, usesplunkuniversalforwarder--x86-release.msi :


[AGREETOLICENSE=Yes /quiet]

For 64-bit platforms, usesplunkuniversalforwarder--x64-release.msi :


[AGREETOLICENSE=Yes /quiet]

The value of varies according to the particular release; for example,splunkuniversalforwarder-5.0-142438-x64-release.msi .

Note:You cannot make configuration changes during an upgrade. The installerignores any command line flags that you specify except for"AGREETOLICENSE".

3.The forwarder starts automatically when you complete the installation.

The installer puts a log of upgrade changes in the %TEMP%directory. It also reportsany errors in the Application Event Log.

Perform a remote upgrade

To upgrade a group of forwarders across your environment:

97


101/159

1.Load the universal forwarder MSI into your deployment tool. Specify thecommand like as follows:

msiexec.exe /i splunkuniversalforwarder-.msi AGREETOLICENSE=Yes

/quiet

See the previous section, "Upgrade using the command line", for details on theMSI command.

2.Execute deployment with your deployment tool.

3.Verify that the universal forwarders function properly.

You might want to test the upgrade locally on one machine before performing aremote upgrade across all your forwarders.

Upgrade the universal forwarder for *nix systems

This topic describes the procedure for upgrading your universal forwarder fromversion 5.0.x, 6.0.x, 6.1.x, or 6.2.x to 6.3.

This topic describes two upgrade scenarios:

Upgrade a single forwarder manuallyPerform a remote upgrade of a group of forwarders

For deployments of any size, you will most likely want to use this secondscenario.

Before you upgrade

Be sure to read this section before performing an upgrade. Also, read "How toupgrade Splunk Enterprise" in the Installation Manual for up-to-date informationand potential issues you might encounter when upgrading.

Confirm that an upgrade is necessary

Before doing an upgrade, consider whether you really need to. In most cases,there's no compelling reason to upgrade a forwarder. Forwarders are alwayscompatible with later version indexers, so you do not need to upgrade them justbecause you've upgraded the indexers they're sending data to.

98


102/159

Back your files up

Before you perform the upgrade, back up your configuration files. For informationon backing up configurations, read "Back up configuration information" in theAdminmanual.

Splunk Enterprise does not provide a means of downgrading to a previousversion; if you need to revert to an older forwarder release, just reinstall it.

How upgrading works

After performing the installation of the new version, configuration changes do notoccur until you start the universal forwarder. You can run the migration previewutility at that time to see what will change before the files are updated. If youchoose to view the changes before proceeding, the forwarder writes theproposed changes to$SPLUNK_HOME/var/log/splunk/migration.log.

Upgrade a single forwarder

1.Execute the stopcommand:

$SPLUNK_HOME/bin/splunk stop

Important:Make sure no other processes can start the forwarder automatically

(such as Solaris SMF).

2.Install the universal forwarder package over the existing deployment:

If you use a .tar file, expand it into the same directory with the sameownership as the existing universal forwarder instance. This overwritesand replaces matching files but does not remove unique files.

If you use a package manager, such as an RPM, type in rpm -U.rpmfrom a shell prompt.

If you use a .dmg file (on MacOS), double-click it and follow theinstructions. Be sure to specify the same installation directory as your

existing installation.

If you use init scripts, be sure to include the following so the End-UserLicense Agreement (EULA) gets accepted:

./splunk start --accept-license

99


103/159


104/159


105/159

Perform advanced configuration

Set up load balancing

Withload balancing, a forwarder distributes data across several receivingSplunk Enterprise instances. Each receiver gets a portion of the total data, andtogether the receivers hold all the data. To access the full set of forwarded data,you need to set up distributed searching across all the receivers. For informationon distributed search, see "About distributed search" in the Distributed Searchmanual.

Load balancing enables horizontal scaling for improved performance. In addition,its automatic switchover capability ensures resiliency in the face of machineoutages. If a machine goes down, the forwarder simply begins sending data tothe next available receiver.

Load balancing can also be of use when getting data from network devices likerouters. To handle syslog and other data generated across port 514, a singleheavy forwarder can monitor port 514 and distribute the incoming data acrossseveral indexers.

Note:When implementing load balancing between forwarders and receivers, youmust use the forwarder's inherent capability.Do not use an external loadbalancer.The use of external load balancers between forwarders and receivers

will not work properly.

How load balancing works

Forwarders perform "automatic load balancing". The forwarder routes data todifferent indexers based on a specified time interval. For example, assume youhave a load-balanced group consisting of three indexers: A, B, and C. At somespecified interval, such as every 30 seconds, the forwarder switches the datastream to another indexer in the group, selected at random. So, the forwardermight switch from indexer B to indexer A to indexer C, and so on. If one indexeris down, the forwarder immediately switches to another.

To expand on this a bit, there is a data stream for each of the inputs that theforwarder is configured to monitor. The forwarder determines if it is safe for adata stream to switch to another indexer. Then, at the specified interval, itswitches the data stream to the newly selected indexer. If it cannot switch thedata stream to the new indexer safely, it keeps the connection to the previous

102


106/159


107/159

The main advantage of a static list is that it allows you to specify a different portfor each receiver. This is useful if you need to perform load balancing acrossmultiple receivers running on a single host. Each receiver can listen on aseparate port.

Static list target

To use a static list for the target, you simply specify each of the receivers in thetarget group's [tcpout]stanza in the forwarder's outputs.conffile. In thisexample, the target group consists of three receivers, specified by IP addressand receiver port number:

[tcpout: my_LB_indexers]

server=10.10.10.1:9997,10.10.10.2:9996,10.10.10.3:9995

The universal forwarder will load balance between the three receivers listed. Ifone receiver goes down, the forwarder automatically switches to another one onthe list.

DNS list target

To use a DNS list, edit your forwarder's outputs.conffile to specify a single hostin the target group's [tcpout]stanza. For example:

[tcpout:my_LB_indexers]

server=splunkreceiver.mycompany.com:9997

In your DNS server, create a DNS A record for each host's IP address,referencing the server name you specified in outputs.conf.For example:

splunkreceiver.mycompany.com A 10.10.10.1



The forwarder will use the DNS list to load balance, sending data in intervals,switching among the receivers specified. If a receiver is not available, theforwarder skips it and sends data to another one on the list.

If you have a topology with many forwarders, the DNS list method allows you toupdate the set of receivers by making changes in just a single location, withouttouching the forwarders' outputs.conffiles.

104


108/159

Configure load balancing for horizontal scaling

To configure load balancing, first determine your needs, particularly yourhorizontal scaling and failover requirements. Then develop a topology based onthose needs, possibly including multiple forwarders, as well as receivers and a

search head to search across the receivers.

Assuming a topology of three universal forwarders and three receivers, set upDNS-based load balancing with these steps:

1.Install and enable a set of three Splunk Enterprise instances as receivers. Thisexample uses a DNS list to designate the receivers, so they must all listen on thesame port. For example, if the port is 9997, enable each receiver by going to its$SPLUNK_HOME/bin/location and using this CLI command:

./splunk enable listen 9997 -auth :

2.Install the set of universal forwarders, as described here.

3.Set up a DNS list with an A record for each receiver's IP address:




4.Create a single outputs.conffile for use by all the forwarders. This one

specifies the DNS server name used in the DNS list and the port the receiversare listening on:

[tcpout]

defaultGroup=my_LB_indexers

[tcpout:my_LB_indexers]

disabled=false

autoLBFrequency=40

server=splunkreceiver.mycompany.com:9997

This outputs.conffile uses the autoLBFrequencyattribute to set a load-balancefrequency of 40 seconds. Every 40 seconds, the forwarders will switch to anotherreceiver. The default frequency, which rarely needs changing, is 30 seconds.

5.Distribute the outputs.conffile to all the forwarders. You can use thedeployment serverto handle the distribution.

105


109/159

The steps are similar if you're using a static list instead of DNS.

Specify load balancing from the CLI

You can also use the CLI to specify load balancing. You do this when you start

forwarding activity to a set of receivers, using this syntax:

./splunk add forward-server : -method autobalance

where :is the host and receiver port of the receiver.

This example creates a load-balanced group of four receivers:

./splunk add forward-server indexer1:9997 -method autobalance




Configure a forwarder to use a SOCKS proxy

This topic discusses how to configure a forwarder with a Socket Secure version 5(SOCKS5) proxy server as a target with the intent of forwarding data to anindexer beyond the proxy server.

By default, a Splunk Enterprise forwarder requires a direct network connection toany receiving indexers. If a firewall blocks connectivity between the forwarderand the indexer, the forwarder cannot send data to the indexer.

Starting with version 6.3 of Splunk Enterprise, you can configure a forwarder touse a SOCKS5 proxy host to send data to an indexer. You can do this byspecifying attributes in a stanza in the outputs.confconfiguration file on theforwarder. After you configure and restart the forwarder, it connects to theSOCKS5 proxy host, and optionally authenticates to the server on demand if youprovide credentials. The proxy host establishes a connection to the indexer and

the forwarder begins sending data through the proxy connection.

Any type of Splunk Enterprise forwarder can send data through a SOCKS5 proxyhost.

This implementation of the SOCKS5 client complies with the Internet Engineering

106


110/159

Task Force (IETF) Request for Comments (RFC) Memo #1928. For informationon this memo, see "Network Working Group: Request for Comments: 1928"(http://www.ietf.org/rfc/rfc1928.txt) on the IETF website.

Configure a SOCKS5 proxy connection with configuration files

To configure a SOCKS5 proxy connection, edit stanzas in outputs.confandspecify certain attributes to enable the proxy. For a list of valid proxy attributes,see "Proxy configuration values." You cannot configure proxy servers in SplunkWeb.

1.Make a copy of $SPLUNK_HOME/etc/system/default/outputs.conf and place itinto $SPLUNK_HOME/etc/system/local.

2.Open $SPLUNK_HOME/etc/system/local/outputs.conf for editing.

3.Define forwarding servers or output groups in outputs.confby creating[tcpout]or [tcpout-server]stanzas. See "Configure forwarders withoutputs.conf."

4.In the stanza for connections that should have SOCKS5 proxy support, addattributes for SOCKS that fit your proxy configuration. You must specify at leastthe socksServerattribute to enable proxy support.

5.Save the file and close it.

6.Restart the forwarder.

7.On the receiving indexer, user the Search and Reporting app to confirm thatthe indexer received the data.

Proxy configuration values

Use the following attributes to configure SOCKS5 on the forwarder:

Attribute Description Default

socksServer Tells the forwarder the host name or IP addressand port of the SOCKS5 proxy it should connect tofor forwarding data.

You can specify one of host:portor IPaddress:port. You must specify both the host

N/A

107


111/159

name or the IP address and the port. You mustspecify this attribute to enable SOCKS5 support.

socksUsername

(Optional) Tells the forwarder to use this usernameto authenticate to the SOCKS5 proxy host if it

demands authentication during the connectionphase.

N/A

socksPassword

(Optional) Tells the forwarder to provide thispassword when authenticating into a SOCKS5proxy host that demands authentication during theconnection phase.

The forwarder obfuscates this password when itloads the configuration that is associated with thestanza. However, there are some securityconsiderations. See "Security considerations".

N/A

socksResolveDNS

(Optional) Tells the forwarder whether or not itshould use DNS to resolve the host names ofindexers in the output group before passing thatinformation on to the SOCKS5 proxy host.

When you set this attribute to true, the forwardersends the name of the indexers to the SOCKS5proxy host as is, and the SOCKS5 proxy host mustthen resolve the indexer host names through DNS.Set to trueif, for example, the forwarder and the

proxy server are on different networks served bydifferent DNS servers.

When you set it to false, the forwarder attempts toresolve the indexer host names through DNS itself,and if it is successful, sends the resolved IPaddresses of the indexers to the SOCKS5 proxyhost.

This attribute only applies if you specify host

names for indexers in the[tcpout]

or[tcpout-server]stanzas. If you specify IPaddresses, DNS resolution does not happen.

false

108


112/159

Examples of SOCKS5 support

Here are some examples of outputs.confstanzas with SOCKS5 proxy supportenabled:

This example establishes a connection to a SOCKS5 proxy host that forwardsthe data to indexers beyond the host:

[tcpout]

defaultGroup = proxy_indexers

[tcpout:proxy_indexers]

server = indexer1.slapstick.com:9997, indexer2.slapstick.com:9997

socksServer = prx.slapstick.com:1080

This example uses credentials to authenticate into the proxy host beforeattempting to send data, and tells the proxy host to resolve DNS to determine theindexers to connect for sending data:

[tcpout]

defaultGroup = socksCredentials

[tcpout:socksCredentials]

server = indexer3.slapstick.com:9997

socksServer = prx.slapstick.com:1081

socksUsername = proxysrv

socksPassword = letmein

socksResolveDNS = true

Security considerations

Note the following caveats when using this feature:

SOCKS5 proxy support only exists between the forwarder and the indexerinclusive. There is no support for the usage of SOCKS with any otherSplunk Enterprise features, apps, or add-ons.

The SOCKS5 protocol sends authentication credentials in clear text. Dueto this implementation, these credentials are vulnerable to aman-in-the-middle attacker. This means that an attacker can secretly relay

and possibly change communication between the SOCKS client and theSOCKS proxy host. This is a caveat of the SOCKS protocol, not theimplementation of this feature in Splunk Enterprise.

For the most secure results, use the SOCKS attributes only on forwarderswhich are inside networks that a SOCKS proxy host protects. Deploying aforwarder in an unprotected environment can result in the interception of

109


113/159

SOCKS credentials by a third par

Splunk 6.3.1 Forwarding

Documents

Transcript of Splunk 6.3.1 Forwarding