Implement splunk to record snmp traps on CentOS6

Splunk free doesn’t allow alerts to be set. So I’m moving on from this solution…
[updated: check_eventdb made me think that it’s possible, but you just have to figure out a way to programmatically query the splunk database]

Here’s an easy way to handle SNMP traps, because SNMPTT+nagios/icinga is annoying.

Okay… okay… so maybe I’m lazy. Maybe after spending hours trying to get snmptt to work with nagios/centreon (let me break my arm patting myself on the back, yea… I wrote most of that) and reading the equally lovely page on the icinga wiki, I’m just sick of dealing with trying to deal with a sensitive system with too many moving parts.

What better way to do it than to use something that just binds to a port or a file and eats data. Splunk.

In case you didn’t know, it can be used for a lot more than just snmp trap handling, but that’s for another day… but…

This is for a small installation:
I have very little stuff to manage (to be honest), so I am not concerned with the implications of installing icinga (including all the modules) and splunk on the same server, but you may be [I don’t have a 500 server wide farm]. If you’re interested, review the capacity planning doc.
Note that the free license for Splunk only allows indexing of 500MBs a day (per splunk instance of course). I’ll be working on a better way to handle logs that’s free shortly (with a focus on security).

Download Splunk
In order to download splunk you must sign up for a splunk account. I will proceed as if you have completed the account creation process and logged on to the site.
1) Access: http://www.splunk.com/download
2) Download the linux RPM package for i386 (as of August 27th, 2012) x84_64 (as of August 27th, 2012).
3) Copy the RPM file to your server. I’ll be installing it on the same server where we had installed icinga.

Install and configure the splunk startup script:

cd
rpm -i splunk-*.i386.rpm #installs splunk to /opt/splunk
curl -k https://gist.github.com/mbrownnycnyc/3488819/raw/6ad378f086b9380963e6e0314a05a82ce168a29f/splunk > /etc/init.d/splunk # I will check for allowed port in iptables later
chmod +x /etc/init.d/splunk
chkconfig splunk on

Switch to free license type:

echo "[license]" >> /opt/splunk/etc/system/local/server.conf
echo "active_group = Free" >> /opt/splunk/etc/system/local/server.conf

Set up apache for a reverse proxy and authentication:
I’m going to write this as though you’ve forced HTTPS in part 1.

echo "root_endpoint =/splunk" >> /opt/splunk/etc/system/local/web.conf
service splunk restart

Configure authentication:
This doesn’t work with Auth right now.

echo "" >> /etc/httpd/conf/httpd.conf
echo "<Location /splunk>" >> /etc/httpd/conf/httpd.conf
echo "ProxyPass http://127.0.0.1:8000/splunk" >> /etc/httpd/conf/httpd.conf
echo "ProxyPassReverse http://127.0.0.1:8000/splunk" >> /etc/httpd/conf/httpd.conf
echo "SetEnvIf X-Url-Scheme https HTTPS=1" >> /etc/httpd/conf/httpd.conf
echo "#AuthName \"splunkweb Access\"" >> /etc/httpd/conf/httpd.conf
echo "#AuthType Basic" >> /etc/httpd/conf/httpd.conf
echo "#AuthUserFile /opt/splunk/htpasswd.users" >> /etc/httpd/conf/httpd.conf
echo "#Require valid-user" >> /etc/httpd/conf/httpd.conf
echo "</Location>" >> /etc/httpd/conf/httpd.conf
htpasswd -c /opt/splunk/htpasswd.users splunker #to set the password for a user named splunker
service httpd restart

Configuring the SNMP trap daemon:
1) Having already installed the net-snmp package during step 1, we can proceed with the configuration of snmptrapd.

echo 'OPTIONS="-p /var/run/snmptrapd.pid -m +ALL -Lf /var/log/snmp-traps"' >> /etc/sysconfig/snmptrapd

2) Authorize your community string to send traps, you can also specify a source subnet:

echo "authCommunity log public" >> /etc/snmp/snmptrapd.conf #public is the authorized community string

3) Already having specified the -m argument, download all MIBs containing traps to /usr/share/snmp/mibs
4) Enable the daemon to start and start it:

chkconfig snmptrapd on
service snmptrapd start

5) Send a test trap and review the log (this is necessary to create the splunk source type):

snmptrap -v2c -c public localhost 1 1 && head -n 5 /var/log/snmp-traps # you should see a trap from 127.0.0.1 for sysuptime

Configure Splunk to monitor the snmptrapd log file
1) Access the splunk web interface at https://SERVER/splunk
2) Enter the username splunker and password you configured earlier with htpasswd (only if authentication is enabled)
3) Accept the free license disclaimer.
4) On the Home app/Welcome page click Add data.
5) Click “From files and directories” under “Choose a Data Source”
6) With “Preview data before indexing” selected, click Browser Files.
7) Navigate /var/log/snmp-traps, select and continue.
8) With “Start a new sourcetype” selected, click continue.
9) Splunk should auto-detect to break on timestamp, but click “adjust timestamp and event break settings.”
10) You can make further adjustments, but we will accept the default Auto (break on timestamp) by clicking Continue.
11) Name your new sourcetype “snmp_traps” and click Save source type.
12) Click “Create input”
13) On the “Add new” page, leave all the defaults then click Save.
14) You may be forwarded to an invalid page, just make sure the beginning of the URL reads https://SERVER/splunk/&#8230;
15) Your file source will not be listed under Manager/Inputs as a File. You can break down a lot of fields, etc, but it’s not really necessary as this can be done with searches.

Creating notifications on new snmp traps received
Splunk free does not provide alerting mechanism. I’m moving on to another solution.

Catching non-MIBed out traps
You can do this by utilizing snmptt. Have snmptrapd forward the snmptrap to snmptt then have snmptt parse the trap and create a new trap to send to snmptrapd. I will cover this later.

With reference:

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: