Implement splunk to record snmp traps on CentOS6
[updated: check_eventdb made me think that it’s possible, but you just have to figure out a way to programmatically query the splunk database]
Here’s an easy way to handle SNMP traps, because SNMPTT+nagios/icinga is annoying.
Okay… okay… so maybe I’m lazy. Maybe after spending hours trying to get snmptt to work with nagios/centreon (let me break my arm patting myself on the back, yea… I wrote most of that) and reading the equally lovely page on the icinga wiki, I’m just sick of dealing with trying to deal with a sensitive system with too many moving parts.
In case you didn’t know, it can be used for a lot more than just snmp trap handling, but that’s for another day… but…
This is for a small installation:
I have very little stuff to manage (to be honest), so I am not concerned with the implications of installing icinga (including all the modules) and splunk on the same server, but you may be [I don’t have a 500 server wide farm]. If you’re interested, review the capacity planning doc.
Note that the free license for Splunk only allows indexing of 500MBs a day (per splunk instance of course). I’ll be working on a better way to handle logs that’s free shortly (with a focus on security).
In order to download splunk you must sign up for a splunk account. I will proceed as if you have completed the account creation process and logged on to the site.
1) Access: http://www.splunk.com/download
2) Download the linux RPM package for i386 (as of August 27th, 2012) x84_64 (as of August 27th, 2012).
3) Copy the RPM file to your server. I’ll be installing it on the same server where we had installed icinga.
Install and configure the splunk startup script:
cd rpm -i splunk-*.i386.rpm #installs splunk to /opt/splunk curl -k https://gist.github.com/mbrownnycnyc/3488819/raw/6ad378f086b9380963e6e0314a05a82ce168a29f/splunk > /etc/init.d/splunk # I will check for allowed port in iptables later chmod +x /etc/init.d/splunk chkconfig splunk on
Switch to free license type:
echo "[license]" >> /opt/splunk/etc/system/local/server.conf echo "active_group = Free" >> /opt/splunk/etc/system/local/server.conf
Set up apache for a reverse proxy and authentication:
I’m going to write this as though you’ve forced HTTPS in part 1.
echo "root_endpoint =/splunk" >> /opt/splunk/etc/system/local/web.conf service splunk restart
This doesn’t work with Auth right now.
echo "" >> /etc/httpd/conf/httpd.conf echo "<Location /splunk>" >> /etc/httpd/conf/httpd.conf echo "ProxyPass http://127.0.0.1:8000/splunk" >> /etc/httpd/conf/httpd.conf echo "ProxyPassReverse http://127.0.0.1:8000/splunk" >> /etc/httpd/conf/httpd.conf echo "SetEnvIf X-Url-Scheme https HTTPS=1" >> /etc/httpd/conf/httpd.conf echo "#AuthName \"splunkweb Access\"" >> /etc/httpd/conf/httpd.conf echo "#AuthType Basic" >> /etc/httpd/conf/httpd.conf echo "#AuthUserFile /opt/splunk/htpasswd.users" >> /etc/httpd/conf/httpd.conf echo "#Require valid-user" >> /etc/httpd/conf/httpd.conf echo "</Location>" >> /etc/httpd/conf/httpd.conf htpasswd -c /opt/splunk/htpasswd.users splunker #to set the password for a user named splunker service httpd restart
Configuring the SNMP trap daemon:
1) Having already installed the net-snmp package during step 1, we can proceed with the configuration of snmptrapd.
echo 'OPTIONS="-p /var/run/snmptrapd.pid -m +ALL -Lf /var/log/snmp-traps"' >> /etc/sysconfig/snmptrapd
2) Authorize your community string to send traps, you can also specify a source subnet:
echo "authCommunity log public" >> /etc/snmp/snmptrapd.conf #public is the authorized community string
3) Already having specified the -m argument, download all MIBs containing traps to /usr/share/snmp/mibs
4) Enable the daemon to start and start it:
chkconfig snmptrapd on service snmptrapd start
5) Send a test trap and review the log (this is necessary to create the splunk source type):
snmptrap -v2c -c public localhost 1 1 && head -n 5 /var/log/snmp-traps # you should see a trap from 127.0.0.1 for sysuptime
Configure Splunk to monitor the snmptrapd log file
1) Access the splunk web interface at https://SERVER/splunk
2) Enter the username splunker and password you configured earlier with htpasswd (only if authentication is enabled)
3) Accept the free license disclaimer.
4) On the Home app/Welcome page click Add data.
5) Click “From files and directories” under “Choose a Data Source”
6) With “Preview data before indexing” selected, click Browser Files.
7) Navigate /var/log/snmp-traps, select and continue.
8) With “Start a new sourcetype” selected, click continue.
9) Splunk should auto-detect to break on timestamp, but click “adjust timestamp and event break settings.”
10) You can make further adjustments, but we will accept the default Auto (break on timestamp) by clicking Continue.
11) Name your new sourcetype “snmp_traps” and click Save source type.
12) Click “Create input”
13) On the “Add new” page, leave all the defaults then click Save.
14) You may be forwarded to an invalid page, just make sure the beginning of the URL reads https://SERVER/splunk/…
15) Your file source will not be listed under Manager/Inputs as a File. You can break down a lot of fields, etc, but it’s not really necessary as this can be done with searches.
Creating notifications on new snmp traps received
Splunk free does not provide alerting mechanism. I’m moving on to another solution.
Catching non-MIBed out traps
You can do this by utilizing snmptt. Have snmptrapd forward the snmptrap to snmptt then have snmptt parse the trap and create a new trap to send to snmptrapd. I will cover this later.