Part 4: Sending snmp traps to icinga


Happy Gilmore convinces an snmp trap to become an icinga alert.

Tested twice.
Good to go.


I am shamelessly going to rely heavily on this dude’s blog post. In fact, if there was such a thing as plagiarism on the open source web, this article “by” me will likely end up being it. If anything, I’ll be translating that article so that it looks pretty and changing some minor things to my preference. It’s just a warning… but what’s it matter? At 150 hits a day, this blog is really for my reference only. Anyway… Thanks Askar! :)

I “hated on it” badly, but it seems that without implementing a full blown logging mechanism (which primarily completes a different challenge), it seems that using snmptrapd> snmptt> nagios/icinga command pipe is really the best solution for snmp traps.

I’m writing this as if you’ve followed Parts 1, 2, and 3 of the reliability monitoring technology solution.

Configure iptables:

vim /etc/sysconfig/iptables
#-A INPUT -p udp -m udp --dport 162 -j ACCEPT
service iptables restart

Download, install and configure snmptt and dependencies:

cd
yum -y install perl-Time-HiRes net-snmp-perl
groupadd snmptt
adduser -g snmptt snmptt
wget http://sourceforge.net/projects/snmptt/files/latest/download?source=files
tar zxvf snmptt*.tgz
cd snmptt*
cp ./snmptt ./snmpttconvert ./snmpttconvertmib ./snmptthandler /usr/sbin/
cp ./snmptt.ini /etc/snmp/
cp ./snmptt-init.d /etc/init.d/snmptt
chkconfig snmptt on
chown snmptt:snmptt /etc/snmp/snmptt.ini
mkdir /var/spool/snmptt
chown snmptt:snmptt /var/spool/snmptt
cd
sed s/mode\ =\ standalone/mode\ =\ daemon/ -i /etc/snmp/snmptt.ini
sed s/syslog_enable\ =\ 1/syslog_enable\ =\ 0/ -i /etc/snmp/snmptt.ini
sed s/net_snmp_perl_enable\ =\ 0/net_snmp_perl_enable\ =\ 1/ -i /etc/snmp/snmptt.ini
sed s/log_enable\ =\ 1/log_enable\ =\ 0/ -i /etc/snmp/snmptt.ini
sed s/syslog_enable\ =\ 1/syslog_enable\ =\ 0/ -i /etc/snmp/snmptt.ini
sed s/translate_log_trap_oid\ =\ 0/translate_log_trap_oid\ =\ 1/ -i /etc/snmp/snmptt.ini
sed s/#mibs_environment\ =\ ALL/mibs_environment\ =\ ALL/ -i /etc/snmp/snmptt.ini

Download submit_check_result:

mkdir -p /usr/local/nagios/libexec/eventhandlers/
curl http://nagios.svn.sourceforge.net/viewvc/nagios/nagioscore/trunk/contrib/eventhandlers/submit_check_result?revision=2122 > /usr/local/nagios/libexec/eventhandlers/submit_check_result
sed s@/usr/local/nagios/var/rw/nagios.cmd@/var/icinga/rw/icinga.cmd@ -i /usr/local/nagios/libexec/eventhandlers/submit_check_result
chmod +x /usr/local/nagios/libexec/eventhandlers/submit_check_result

Create snmptt definition files from MIBs
1) Copy all your MIBs to the /usr/share/snmp/mibs directory.
2) Create the snmptt conf files from the trap mibs:

echo '#!/bin/bash' > /usr/bin/snmptt_mib_import.sh
echo "cd /usr/share/snmp/mibs" >> /usr/bin/snmptt_mib_import.sh
echo "rm -rf /etc/snmp/snmptt.conf.bak" >> /usr/bin/snmptt_mib_import.sh
echo "mv /etc/snmp/snmptt.conf /etc/snmp/snmptt.conf.bak" >> /usr/bin/snmptt_mib_import.sh
echo "#import MIBs, changing trap severity from Normal to critical for all traps" >> /usr/bin/snmptt_mib_import.sh
echo "find . -type f | cut -d '/' -f 2 | xargs -I {} snmpttconvertmib --severity=Critical --in={} --out=/etc/snmp/snmptt.conf --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result \$r \"snmp_traps\" 2 \"\$O: \$+*\"'" >> /usr/bin/snmptt_mib_import.sh
echo "#change certain trap's severity from Critical to Normal" >> /usr/bin/snmptt_mib_import.sh
echo "#  Changing linkUp trap to be Normal:" >> /usr/bin/snmptt_mib_import.sh
echo 'sed "/^EVENT.*linkUp.*Critical$/ s/Critical/Normal/" -i /etc/snmp/snmptt.conf' >> /usr/bin/snmptt_mib_import.sh
echo "#  Changing linkUp trap to send an OK, the string 'A linkUp trap' is located in the EXEC line of the snmptt definition:" >> /usr/bin/snmptt_mib_import.sh
echo 'sed "/^EXEC.*A\ linkUp\ trap.*/ s/\ 2/\ 0/" -i /etc/snmp/snmptt.conf' >> /usr/bin/snmptt_mib_import.sh
chmod +x /usr/bin/snmptt_mib_import.sh
bash /usr/bin/snmptt_mib_import.sh

3) Run the script every time you add any more MIBs.
4) In the above script, I have opted to change the snmptt.conf file so that the severity of all the traps is changed from the default Normal to Critical. You may want to change the severity for specific instances.

Following the line “change certain trap’s severity from Critical to Normal,” I have used sed to change linkUp from Critical to a Normal severity. The available severity strings are: ‘Minor’, ‘Major’, ‘Normal’, ‘Critical’, ‘Warning’. You can use this sed line, replacing ‘lineUp’ with the trap you wish to affect, to affect other trap definitions.

Following the line “Changing linkUp trap to send an OK,” I have used sed to change the severity of the nagios alert for linkUp from Critical to a OK. The available severity integers are: 0 (OK), 1 (WARNING), 2 (CRITICAL), 3 (UNKNOWN). You can use this sed line, replacing ‘A\ linkUp\ trap’ (note the escape character ‘\’ before the space characters) with some content of the EXEC line you wish to affect, to affect other trap definitions.

Create catch all snmptt trap definition (optional):

echo '#' >> /etc/snmp/snmptt.conf
echo '#' >> /etc/snmp/snmptt.conf
echo '#' >> /etc/snmp/snmptt.conf
echo '#' >> /etc/snmp/snmptt.conf
echo 'EVENT CatchAll .1.* "snmptt catchall" Critical' >> /etc/snmp/snmptt.conf
echo 'FORMAT $D' >> /etc/snmp/snmptt.conf
echo 'EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result "$r" "snmp_traps" 2 "$O: $1 $2 $3 $4 $5 $5 $6 $7 $8 $9 $10 $11 $12 $13 $14 $15"' >> /etc/snmp/snmptt.conf
echo 'SDESC' >> /etc/snmp/snmptt.conf
echo 'This is the catch all snmptt MIB definition.  This means that this trap does not have a MIB definition in snmptt.conf on the server.' >> /etc/snmp/snmptt.conf
echo 'EDESC' >> /etc/snmp/snmptt.conf

Configure snmptrapd:

echo 'OPTIONS="-t -c /etc/snmp/snmptrapd.conf -On -Lsd -p /var/run/snmptrapd.pid -m ALL"' >> /etc/sysconfig/snmptrapd
echo '# -t to stop logging traps to syslog' >> /etc/sysconfig/snmptrapd
echo '# -Lf /var/log/snmptrapd.log to logging to a file' >> /etc/sysconfig/snmptrapd
echo "authCommunity log,execute,net public" >> /etc/snmp/snmptrapd.conf #public is the authorized community string
echo "traphandle default /usr/bin/snmptthandler" >> /etc/snmp/snmptrapd.conf

Add and start both snmptrapd and snmptt

chkconfig snmptt on
chkconfig snmptrapd on
service snmptt start
service snmptrapd start

Creating a passive service to receive traps:
Unfortunately, is_volatile is not part of the schema included in LConf version 1.2; which stops users from being able to set the is_volatile property of a service within the icinga-web LConf module.

In order to work around this constraint, we will have to add another cfg_dir to icinga.cfg and then populate cfg_files with service definitions with the hosts that will send our traps.

Add the cfg_dir:

mkdir /etc/icinga/snmptrap_cfg
chown icinga:apache /etc/icinga/snmptrap_cfg
chmod 775 /etc/icinga/snmptrap_cfg
echo "cfg_dir=/etc/icinga/snmptrap_cfg" >> /etc/icinga/icinga.cfg

Create the cfg_file:

curl -k https://gist.github.com/mbrownnycnyc/3503677/raw/9965139383ed4014d47d25d0a32cab16f8d36173/generic-trap.cfg > /etc/icinga/snmptrap_cfg/snmptrap_template.cfg
curl -k https://gist.github.com/mbrownnycnyc/3540126/raw/1e637d701707f13f4af53b13955719b50011ea8d/ex-service.cfg > ~/ex_service.cfg

~/ex_service.cfg contains the very short service definition that utilizes the generic-trap template contained in snmptrap_template.cfg.

Remember, each service is tied to a host, and, in the case of this service, we want the host to be one that is sending us snmptraps.

In order to link the service to a host, copy ~/ex_service.cfg to /etc/icinga/snmptrap_cfg/snmptrap_service.cfg and create your service definitions within it. An example can be reviewed at this gist.

You can do this easily using sed:

sed s/SERVERNAME/defined_hostA/ ~/ex_service.cfg > /etc/icinga/snmptrap_cfg/traps_services.cfg
echo "" >> /etc/icinga/snmptrap_cfg/traps_services.cfg
sed s/SERVERNAME/defined_hostb/ ~/ex_service.cfg >> /etc/icinga/snmptrap_cfg/traps_services.cfg
echo "" >> /etc/icinga/snmptrap_cfg/traps_services.cfg

Creating a service that alerts when traps are received from applications that reside on a host:

Certain applications may have the ability to send traps which are defined in a MIB. For example, anti-virus, backup, SIEM, and IDS/IPS software may have this ability. The way to configure a service to receive specific traps is to generate your own snmptt.conf file from the MIB in question, designating the arguments specified in the following example, replacing “backup_exec” with the desired unique service_description:

snmpttconvertmib --severity=Critical --in=/usr/share/snmp/test_mib/bkupexec.mib --out=/etc/snmp/backupexc-snmptt.conf --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $r "backup_exec" 2 "$O: $+*"'

Alternately, you can manually isolate the definitions out of an already generated snmptt.conf file and change the generated fields manually. This is more applicable when tying a service into a meta-type service, such as tying a ‘VPN down’ trap sent from a VPN end-point to send CRITICAL to another service that represents the connectivity to an application or service.

It can get quite complex, but you can have multiple checks affect the status of a single service, specifically when not using icinga to perform the checks and update the status (as is the case with a passive service).

Considering duplicate traps
There is a setting within snmptt.ini that allows you to set a second threshold where duplicate traps will be thrown away.

To stop duplicate trap received within a 10 second from being processed, set the following:

duplicate_trap_window=10

Debugging the processes:
The flow for traps is as follows:

trap generation -> snmptrapd host UDP port 162 -> snmptrapd (snmptthandler 'traphandler' configured in /etc/snmp/snmptrapd.conf) -> snmptthandler populates file in spool_directory (/var/spool/snmptt/) -> snmptt (recognize EVENT and EXEC as configured in /etc/snmp/snmptt.conf) -> /usr/local/nagios/libexec/eventhandlers/submit_check_result (prepend datetime append host and service info to a PROCESS_SERVICE_CHECK_RESULT) -> /var/icinga/rw/icinga.cmd (accept PROCESS_SERVICE_CHECK_RESULT) -> icinga (affects service status)

To “tap into” this flow use the following:
snmptrapd host UDP port 162:

tcpdump -n -i 1 -v port 162 -c 100

snmptrapd:

tail -f /var/log/messages #since snmptrapd is configured with OPTIONS in /etc/sysconfig/snmptrapd to -Lsd (log to syslog under LOG_DAEMON)

snmptthandler:

vim /etc/snmp/snmptrapd.conf
#change traphandler to:
# /usr/bin/snmptthandler --debug=2 --debugfile=/var/log/snmptthandler.log
touch /var/log/snmptthandler.log
chown snmptt:snmptt /var/log/snmptthandler.log
service snmptrapd restart
tail -f /var/log/snmptthandler.log

snmptt:

sed s/log_enable\ =\ 0/log_enable\ =\ 1/ -i /etc/snmp/snmptt.ini
mkdir /var/log/snmptt/
touch /var/log/snmptt/snmptt.log
chown snmptt:snmptt /var/log/snmptt/* /var/log/snmptt/
tail -f /var/log/snmptt/snmptt.log

or

sed s@OPTIONS="--daemon"@OPTIONS="--daemon --debug=1 --debugfile=/var/log/snmptt/snmptt_debug.log"@ -i /etc/init.d/snmptt
service snmptt restart

submit_check_result:

#you can add some debugging lines to the script

icinga.cmd:

#since icinga is written to bind to the icinga.cmd named pipe/FIFO, you must kill the icinga process then you can tail the icinga.cmd
service icinga stop
tail -f /var/icinga/rw/icinga.cmd

icinga:

tail -f /var/log/icinga/icinga.log | grep PASSIVE

Testing:
Send a test trap from any host that’s configured in /etc/icinga/snmptrap_cfg/traps_services.cfg.

Advertisements
  1. Colin England
    June 18, 2013 at 5:56 am

    Hi,

    I am trying to start the service snmptt after the “Configure snmptrapd” part but am unable to get it to start as it is coming up with the following error message:

    BEGIN failed–compilation aborted at /usr/sbin/snmptt line 4026.

    • June 18, 2013 at 10:48 am

      That’s interesting. Are you installing on centos? Can you find a package to download with a package manager like yum or apt-get? The package managers will resolve dependencies for you. Otherwise, try to find the dependencies.

  2. Jonathan
    September 19, 2013 at 10:00 am

    Hello friend. Thanks for this guide. After much searching a lot and I get to this document. I think there is a lot of information and your input is really very useful.

    My problem is:
    After performing all the steps, now icinga not start and I get this error

    Processing object config file '/etc/icinga/snmptrap_cfg/traps_services.cfg'...
    Error: Unexpected token or statement in file '/etc/icinga/snmptrap_cfg/traps_services.cfg' on line 1.
       Error processing object config files!
    

    Could you help?

    Thank you!

    Jonathan from Argentina

    • September 20, 2013 at 11:44 am

      Please allow me some more time to respond.

  3. Jonathan
    September 23, 2013 at 9:07 am

    Hey! course! will be right here!

    thanks

    • September 23, 2013 at 9:32 am

      Woah, major problems with the source for the cfg_files:

      The post has been corrected. Please re-read and complete the “Create the cfg_file” section. The source files are new.

      • Jonathan
        September 24, 2013 at 8:09 am

        Hello again. Thank you very much for your previous update.

        It seems that now all goes well, although the traps are not reaching icinga.
        I can doubt my host settings, but I’ve also done this test with this result:

        [root@localhost mibs]# service icinga stop
        Stopping icinga: Stopping icinga done.
        [root@localhost mibs]# tail -f /var/icinga/rw/icinga.cmd
        tail: cannot open `/var/icinga/rw/icinga.cmd' for reading: No such file or directory
        

        I also see that the folder “var” does not exist.

        [root@localhost nagios]# ls /usr/local/nagios/
        libexec
        

        You can be here my mistake?

        thanks

    • September 24, 2013 at 8:34 am

      Hello,

      The command_file (see /etc/icinga/icinga.cfg and definition of “command_file” in icinga docs) is only present when the `icinga` process is running. This is because it is generated by the `icinga` process to provide an interface to itself from outside processes (it if a “fifo file”).

      The `/usr/local/nagios/var` directory doesn’t exist anywhere on my system either. I specifically use `sed` in the above config (under “Download submit_check_result”) to change references to this to the proper location.

      My strongest suggestion would be to start over. It is a fairly complex system to troubleshoot, but you can also refer to “Debugging the processes” to cover each step in the flow of traps. Please run through making sure that each step in the flow is covered:

      1) trap generation by external host
      2) arrives to snmptrapd host UDP port 162
      3) where snmptrapd is bound (snmptthandler ‘traphandler’ configured in /etc/snmp/snmptrapd.conf
      4) snmptthandler populates file in spool_directory (/var/spool/snmptt/)
      5) snmptt (recognize EVENT and EXEC as configured in /etc/snmp/snmptt.conf)
      6) /usr/local/nagios/libexec/eventhandlers/submit_check_result (prepend datetime append host and service info to a PROCESS_SERVICE_CHECK_RESULT)
      7) /var/icinga/rw/icinga.cmd (accept PROCESS_SERVICE_CHECK_RESULT)
      8) icinga (affects service status)

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: