How to Build Your First DevOps Lab – Part 4: Configuring Splunk

Splunk Overview

Splunk is a widely used tool for monitoring, analyzing, and searching large amounts of machine data. With vast capabilities, there is no wonder why it has become so popular amongst many companies. You can gather various types of data from remote endpoints to ingest into your Splunk instance. Splunk mainly relies on its universal forwarder to send data to the Splunk servers, though, not all systems can install a universal forwarder. This is where syslog comes in handy with Splunk. Syslog servers help centralize data from devices that can’t install a universal forwarder, such as a router, or firewall device. Then the syslog server can send all of its aggregated logs to the Splunk server. Splunk has many uses, and we’ll see how to set it up in the coming sections.

Configuring Splunk on the Ubuntu Master Node

To keep the environment as simple as possible, the Splunk server will be our Ubuntu Master node. Typically, you would not have Splunk on a server with other computational processes as Splunk is quite resource hungry as it is. However, for a small setup like this, there shouldn’t be much, if any, contention. In this lab build, Splunk is not integrated with any of the other products, although it could be. In this section, we’ll be covering how to setup Splunk on the Ubuntu Master VM, followed by the Universal Forwarder setup. See below on how to install Splunk on the Ubuntu Master VM:

1. Navigate to and either create a Splunk account or sign in with an existing one.

a. There is a developer license that can be obtained for Splunk if you need to build out a bigger environment that ingests more data. The steps for obtaining one can be found here.

b. Click here for more information on the different types of Splunk licenses.

2. After logging in, select .deb from the Linux tab

3. Click download from command line and copy the contents of the popup.

4. Sign into the Ubuntu Master VM and open a terminal window.

5. Install Splunk

a. Change Directory to Downloads

cd ~\Downloads

b. Download Splunk package

sudo <copied_contents>

c. Install the downloaded file

sudo dpkg -i <downloaded_filename>

d. Start Splunk and accept the license agreement; Also input a username and password that will be used to login to Splunk

sudo /opt/splunk/bin/splunk start --accept-license --answer-yes

6. Login to the Splunk instance at http://[IP]:8000 replacing IP with your Ubuntu Master VM’s IP

7. Navigate to Settings > Receiving and Forwarding

8. Add receiving rule for 9997

Configuring the Windows Node

Now that the Splunk server is setup, we need to actually get data into Splunk. There are a few ways to do this, but to get continuous real time data that is actually useful, we will setup a Universal Forwarder on the Windows VM. Another way you could get data into Splunk quickly is from a file that you import manually. The universal forwarder installation is quite simple, especially if done from the command line as we will be doing. To install the Universal Forwarder, follow these steps: