Part 1: Implement icinga, icinga-web and icinga-reports on CentOS6


PLEASE READ THIS BEFORE CONTINUING!


Afrika Bambaataa.

To communicate with the community of developers and users, join #icinga on freenode and check out monitoring-portal.org.

To do: SElinux stuff
Updated February 4th, 2013: (not totally tested) to include some recommended inclusions for icinga 1.7 (noted as `#thanks Luke!`).
Please make sure you are using the latest versions of the cronks as webapps. The versions I `wget` are NOT the latest, as noted in the #comments.


For a system that’s fairly easy to administer and use, the installation procedure for icinga is confusingly split up across many web pages and guides.

The copy-pasta friendly version of Installing icinga with IDOutils + new web interface + ingraph + reporting:

I am working with the latest centos minimal install.

Some prerequisites you want:

yum -y install wget vim bind-utils ntp
chkconfig ntpd on
ntpdate pool.ntp.org
service ntpd start

#add the fedora package repo
echo [fedora_repo] >> /etc/yum.repos.d/fedora_repo.repo #allow yum access to the fedora repo
echo name=fedora_repo >> /etc/yum.repos.d/fedora_repo.repo
echo baseurl=http://download1.fedora.redhat.com/pub/epel/\$releasever/\$basearch/ >> /etc/yum.repos.d/fedora_repo.repo
echo enabled=1 >> /etc/yum.repos.d/fedora_repo.repo
echo skip_if_unavailable=1 >> /etc/yum.repos.d/fedora_repo.repo
echo gpgcheck=0 >> /etc/yum.repos.d/fedora_repo.repo

Temporarily set selinux to permissive:

setenforce permissive

Install the rpmforge repo:

cd
wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.i686.rpm
rpm -i rpmforge-release-0.5.2-2.el6.rf.i686.rpm

Install some prerequisites and icinga:

yum -y install make httpd gcc glibc glibc-common gd gd-devel openssl-devel openssl mysql-server mysql libdbi libdbi-dbd-mysql libdbi-devel libdbi-drivers libjpeg libjpeg-devel libpng libpng-devel net-snmp net-snmp-devel net-snmp-utils perl-Net-SNMP perl-Config-IniFiles nagios-plugins-perl php php-cli php-pear php-xmlrpc php-xsl php-pdo php-soap php-gd php-ldap php-mysql subversion
#yum -y install icinga icinga-gui icinga-doc icinga-idoutils
yum -y install icinga icinga-gui icinga-doc icinga-idoutils-libdbi-mysql #thanks Luke!
mkdir -p /usr/lib/nagios/plugins
# or for 64-bit: mkdir -p /usr/lib64/nagios/plugins #thanks MtK!
cd 
wget http://sourceforge.net/projects/nagiosplug/files/latest/download?source=files
tar zxvf nagios-plugins-*.tar.gz
cd nagios-plugins-*
./configure && make && make install
cd
svn checkout http://svn.centreon.com/trunk/plugins-2.x/src/
cd src
find . -iname *svn* | xargs /bin/rm -rf
grep -rl '@NAGIOS_PLUGINS@' . | xargs sed 's&@NAGIOS_PLUGINS@&/usr/local/nagios/libexec&g' -i
mkdir -p /var/icinga/centplugins/ && chown icinga:icinga /var/icinga/centplugins/ #thanks MTK
grep -rl '@CENTPLUGINS_TMP@' . | xargs sed 's&@CENTPLUGINS_TMP@&/var/icinga/centplugins&g' -i 
sed s/max_msg_size\(5000\)/max_msg_size\(30000\)/ -i ./check_centreon_snmp_cpu #http://forum.centreon.com/showthread.php/12097-Is-it-safe-to-adjust-the-max_msg_size-in
/bin/cp -rf * /usr/local/nagios/libexec
chmod +x /usr/local/nagios/libexec/*
cd
sed s@/usr/lib/nagios/plugins@/usr/local/nagios/libexec/@ -i /etc/icinga/resource.cfg
# or for 64-bit: sed s@/usr/lib64/nagios/plugins@/usr/local/nagios/libexec/@ -i /etc/icinga/resource.cfg #thanks MtK!

Secure your mysql instance:

chkconfig mysqld on
service mysqld start
/usr/bin/mysql_secure_installation
#enter for no root password
#Y to set root password
#enter a password
#Y to remove anonymous user
#Y to disallow root login remotely
#Y to remove test database and access to it
#Y to reload privilege tables now
#done

Install the icinga schema:
Note the DB password icinga_DBUSER_PASSWORD, which will be used by ido2db to access the database.

echo "CREATE DATABASE icinga;" > ~/create_icinga.sql
echo "GRANT USAGE ON icinga.* TO 'icinga'@'localhost' IDENTIFIED BY 'icinga_DBUSER_PASSWORD' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0;" >> ~/create_icinga.sql
echo "GRANT SELECT , INSERT , UPDATE , DELETE, DROP, CREATE VIEW ON icinga.* TO 'icinga'@'localhost';" >> ~/create_icinga.sql
echo "FLUSH PRIVILEGES;" >> ~/create_icinga.sql
mysql -p < ~/create_icinga.sql
#mysql -p icinga < /etc/icinga/idoutils/mysql/mysql.sql
mysql -p icinga < /usr/share/doc/icinga-idoutils-libdbi-mysql-1.7.2/db/mysql/mysql.sql #thanks Luke!

Configure ido2db to use the DB user password

sed s/db_pass=icinga/db_pass=icinga_DBUSER_PASSWORD/ -i /etc/icinga/ido2db.cfg

Install SSMTP and mailx to send Email notifications
Follow this post to install SSMTP.

yum -y install mailx

Download and install icinga-web:

cd
wget http://sourceforge.net/projects/icinga/files/icinga-web/1.7.2/icinga-web-1.7.2.tar.gz/download #Go to http://sourceforge.net/projects/icinga/files/icinga-web/ for the latest icinga-web package
tar zxvf icinga-web-*.tar.gz
cd icinga-web-*
#optionally review the defaults with ./configure --help | more
./configure && make install && make install-apache-config && make install-done
make testdeps #Test the PHP framework (all should pass, except for php5-pdo-pgsql...)

Configure icinga-web to use both the DB users passwords
icinga-web must access both the icinga-web and the icinga databases, so note the following use of both the icinga_web_DBUSER_PASSWORD and icinga_DBUSER_PASSWORD.

sed s/icinga_web:icinga_web@localhost/icinga_web:icinga_web_DBUSER_PASSWORD@localhost/ -i /usr/local/icinga-web/app/config/databases.xml
sed s/icinga:icinga@localhost/icinga:icinga_DBUSER_PASSWORD@localhost/ -i /usr/local/icinga-web/app/config/databases.xml
sed s/icinga_web:icinga_web@localhost/icinga_web:icinga_web_DBUSER_PASSWORD@localhost/ -i ./app/config/databases.xml #referred to by installer
sed s/icinga:icinga@localhost/icinga:icinga_DBUSER_PASSWORD@localhost/ -i ./app/config/databases.xml #referred to by installer

Correct the timezone setting for the icinga server
In order for icinga-web to function properly, a change in the timezone setting is needed.
1) Locate your timezone’s string notation via PHP docs.
2) Modify /etc/php.ini:

sed s@\;date.timezone\ \=@date.timezone\ \=\ \"America/New_York\"\;@ -i /etc/php.ini
/usr/local/icinga-web/bin/clearcache.sh
service httpd restart

[an alternative to this is to modify /usr/local/icinga-web/etc/conf.d/translation.xml with the setting]

Configure the icinga-web database:
Note the DB password icinga_web_DBUSER_PASSWORD, which will be used by the PHP “program” icinga-web to access the database.

echo "CREATE DATABASE icinga_web;" > ~/create_icinga_web.sql
echo "GRANT USAGE ON *.* TO 'icinga_web'@'localhost' IDENTIFIED BY 'icinga_web_DBUSER_PASSWORD' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0;" >> ~/create_icinga_web.sql
echo "GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER, INDEX ON icinga_web.* TO 'icinga_web'@'localhost';" >> ~/create_icinga_web.sql
echo "FLUSH PRIVILEGES;" >> ~/create_icinga_web.sql
mysql -p < ~/create_icinga_web.sql
cd ~/icinga-web-* #yes, back here again
make db-initialize #hit y to the option to use the user found in /usr/local/icinga-web/app/config/databases.xml

Configure the ability for you to use commands in the web site to affect icinga

usermod -G icingacmd apache
sed s@/usr/local/icinga/var/rw/icinga.cmd@/var/icinga/rw/icinga.cmd@ -i /usr/local/icinga-web/etc/conf.d/access.xml
/usr/local/icinga-web/bin/clearcache.sh

Add port exception for httpd to iptables:

vim /etc/sysconfig/iptables
# add this line below the last -A INPUT
## -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

Re/start all the affected processes:

chkconfig ido2db on
chkconfig icinga on
chkconfig httpd on
service iptables restart
service ido2db restart
service icinga restart
service httpd restart

Hit it in a web browser at http://SERVER/icinga-web/.

default user is root default password is password

Install Jasper Reports Server:

#find the jasperreports installer for your OS http://sourceforge.net/projects/jasperserver/files/JasperServer/
cd
wget http://sourceforge.net/projects/jasperserver/files/JasperServer/JasperServer%204.5.0/jasperreports-server-cp-4.5.0-linux-x86-installer.run
chmod +x jasperreports-server-*-installer.run
./jasperreports-server-*-installer.run
#accept license (y)
#keep default all defaults (hit enter a bunch of times: bundled tomcat, bundled postgres, 8080, 8005, 5432... all for use with JasperServer itself)
#install sample data, yes
#install ireports, yes
#don't register at JasperForge and JasperHeartbeat
/opt/jasperreports-server-cp-*/ctlscript.sh start

Install the Icinga Reporting module:

#get the latest http://sourceforge.net/projects/icinga/files/icinga-reporting/
cd
wget http://sourceforge.net/projects/icinga/files/icinga-reporting/1.7.0/icinga-reports-1.7.0.tar.gz/download
tar zxzf icinga-reports-*.tar.gz
cd icinga-reports-*
./configure --with-jasper-server=/opt/jasperreports-server-cp-4.5.0  # looks like the config script checks for: /opt/jasperreports-server-cp-4.0.0 /opt/jasperreports-server-cp-4.1.0 and /opt/jasperreports-server-cp-4.2.1
make install-mysql-connector #so that jasperserver can access our icinga DB
/opt/jasperreports-server-cp-4.5.0/ctlscript.sh stop tomcat
/opt/jasperreports-server-cp-4.5.0/ctlscript.sh start tomcat
make install
/opt/jasperreports-server-cp-4.5.0/ctlscript.sh stop tomcat
/opt/jasperreports-server-cp-4.5.0/ctlscript.sh start tomcat

Create an init script that starts tomcat and postgres:

Configure the init script:

curl -k https://gist.github.com/mbrownnycnyc/3437554/raw/f304ac207b363e36257f41390f03daafeaec7f99/icinga-reporting > /etc/init.d/icinga-reporting
chmod +x /etc/init.d/icinga-reporting
chkconfig --add icinga-reporting
chkconfig icinga-reporting on

Use the script to check status and restart the tomcat and postgresql servers:

service icinga-reporting status
service icinga-reporting restart

Configure JasperServer to access the icinga DB
1) Add a temporarily iptables rule for access to the jasperserver:

vim /etc/sysconfig/iptables
# add this line below the last -A INPUT
## -A INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT
service iptables restart

2) Hit http://SERVER:8080/jasperserver in a web browser
3) Logon with user jasperadmin password jasperadmin
4) On the right-side tree, nagivate to root/Icinga/Data Sources, highlight IDO, click Edit
5) On the “Set Data Source Type and Properties” page, scroll down and change the password to the icinga_DBUSER_PASSWORD that was set in the section Install the icinga schema then click the Test Connection button at the bottom.
6) After the “Connection passed” message appears at the top of the page, click the blue Submit button at the bottom.
7) Remove the iptables rule for access to the jasperserver. Remember to restart iptables.
8) Hit https://SERVER/, access Reporting on the left side pane, and check out some Reports in the tree, and notice that you can also schedule reports.

Some management stuffs:

  • If you to change a config file, like /usr/local/icinga-web/app/config/databases.xml, you must run /usr/local/icinga-web/bin/clearcache.sh after in order to flush the config.
  • make icinga-reset-password will reset any web user’s password.
  • The logs for icinga-web, which is a PHP app written using the agavi framework, are located according to /usr/local/icinga-web/app/config/logging.xml

Some extra stuff:
Force HTTPS:
Refer to another HTTPS setup.

Install mod_ssl:

yum -y install mod_ssl

Add port exception for httpd:

vim /etc/sysconfig/iptables
# add this line below the last -A INPUT
## -A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
service iptables restart

Modify httpd config to re-write all requests to HTTPS:

echo "RewriteEngine On" >> /etc/httpd/conf/httpd.conf
echo "RewriteCond %{HTTPS} off" >> /etc/httpd/conf/httpd.conf
echo "RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}" >> /etc/httpd/conf/httpd.conf
service httpd restart

Serve icinga-web from the root of the server:

sed s@#DocumentRoot\ \"/var/www/html\"@DocumentRoot\ \"/usr/local/icinga-web/pub\"@ -i /etc/httpd/conf.d/ssl.conf
service httpd restart

SELinux:
The first thing we should do is take care of the booleans. If the booleans don’t allow something, file context changes don’t matter.
The booleans are described on a CentOS wiki page.

setsebool -P httpd_builtin_scripting on
setsebool -P httpd_can_network_connect_db on

For icinga-web:
To make the changes at run time:

chcon -Rvt httpd_sys_rw_content_t /usr/local/icinga-web/log/
chcon -Rvt httpd_sys_rw_content_t /usr/local/icinga-web/app/cache

To make the changes survive a SElinux file system relabel:

yum -y install policycoreutils-python
semanage fcontext -a -t httpd_sys_rw_content_t "/usr/local/icinga-web/log(/.*)?"
semanage fcontext -a -t httpd_sys_rw_content_t "/usr/local/icinga-web/app/cache(/.*)?"

Notes:

  1. This guide installs the centreon plugins from the project’s SVN repo. You will have to build them into the check command cfg in order to use them.
  2. You can optimize ido2db by following some guidelines in the icinga wiki.

Continue onto Implement Lconf with icinga and icinga-web on CentOS 6 with SElinux in order to easily manage icinga configuration files.

with reference:

  1. July 26, 2012 at 6:43 pm

    Concerning the selinux modifications, how did you determine the paths that required context changes?

    I’m currently installing using someone else’s walkthrough on CentOS 6 and i don’t have any icinga-related directories in /usr/local.

    Thanks.

    • July 26, 2012 at 6:58 pm

      I’m away right now, but if you use the site search for selinux primer you will find the page that describes a method to determine which directories to tag. Off the bat, I would say, look at the httpd conf and see which files are being served… They must be tagged/associated with the same domain where httpd lives. It’s not the end of the world to start over with this tutorial… I reinstalled centos and rolled Parts 1, 2 and 3 out to the machine at leaat three times in the last two days. Parts 2&3 are also pretty much done.

  2. Jo
    August 27, 2012 at 3:36 am

    cant login. username “root” and password “password” is not working.. :(

  3. September 14, 2012 at 9:16 am

    Great set of instructions! I followed all of the steps and still can’t resolve the following ido2db error:
    ido2db: Error: Could not connect to mysql database: 1045: Access denied for user ‘icinga’@’localhost’
    I’ve checked that /usr/local/icinga-web/app/config/databases.xml matches ido2db.cfg.
    Any help would be greatly appreciated.

    • September 14, 2012 at 9:43 am

      Thanks for the compliment. I’m glad you find this write up helpful!

      Clearly this means that ido2db is not receiving the proper credentials from /etc/icinga/ido2db.cfg. Did you set credentials within the section “Install the icinga schema”?

      Remember these conversations:

      icinga > ido2db <> DB
      icinga-web <> DB
      icinga-web <> icinga command pipe
      

      ido2db (“icinga data out to database”, from ndo2db “nagios…”) provides a pathway for icinga to change data in a database. This is why you configure a DB user and password under Install the icinga schema section, and then configure /etc/icigna/ido2db.cfg to use this user and password.

      The /usr/local/icinga-web/app/config/databases.xml is actually a configuration file used by the agavi framework based web application icinga-web.

  4. September 14, 2012 at 9:39 am

    Figured out the ido2db problem. I needed to restart ido2db! Now, all is good. Thanks again for the instructions!

    • September 14, 2012 at 9:57 am

      Glad you figured out the problem. I’ve also implemented Lconf (which, once it’s set up is pretty awesome), as well as InGraph, which is actually a kick ass graphing solution. I’m likely going to also look at log management with EventDB/graylog2/octopussy/kabina shortly, which will help you guys manage your Apache logs.

      In complement, I will also be documenting using sagan to review logs, as well as a full roll out of snort, sguil, and squert.

      All these solutions should be very helpful for you guys. Good luck!

  5. Greattutorial
    September 21, 2012 at 7:14 am

    Nice blog !

  6. Brian
    September 28, 2012 at 8:00 pm

    This is a great blog. I am having a small issue, and being that I AM a noob to all of this, any help would be appreciated. I just pased the “Configure JasperServer to access the icinga DB
    ” portion and when I try to navigate to my URL, I get a 404 “Requested Resource is not available”. I am not quite sure where to look. I have restarted services, double checked my firewall rules.

    • September 28, 2012 at 11:12 pm

      Thanks for the compliment!

      Are you sure you are accessing the port for which the java process is bound? Is SElinux set to permissive?

  7. Luke
    February 1, 2013 at 6:53 pm

    Some changes with Icinga 1.7

    Install some prerequisites and icinga:
    18: yum -y install icinga icinga-gui icinga-doc icinga-idoutils-libdbi-mysql

    Install the icinga schema:
    6: mysql -p icinga < /usr/share/doc/icinga-idoutils-libdbi-mysql-1.7.2/db/mysql/mysql.sql

    • February 1, 2013 at 8:06 pm

      Thanks very much Luke. I’ll update the original article citing this comment.

    • February 4, 2013 at 3:36 pm

      Updated! Thanks!

  8. Chris
    February 4, 2013 at 1:32 pm

    This guide is MARVELOUS for a 100% n00b like me. I very much appreciate the effort you’ve put into this, it’s incredible.

    Once I had completed the icinga-web part, I loaded it up in a browser and was greeted by this message: “Request Failed. The server encountered an error: internal exception, check your logs!”

    I opened the java console on the page and found some errors that linked to these lines:
    SQLSTATE[42S02]: Base table or view not found: 1146 Table ‘icinga.icinga_hosts’ doesn’t exist.

    and

    SQLSTATE[42S02]: Base table or view not found: 1146 Table ‘icinga.icinga_hoststatus’ doesn’t exist

    So it seems that the tables icinga would like to get at don’t exist. Here comes the n00b part… how do I create them? Could I have skipped a section of copy-pasta? Looking back it seems not, but… much of what you’ve created it pure magic to me lol

    Any advice you could offer would be greatly appreciated.

    • February 4, 2013 at 3:38 pm

      Hello Chris,

      I have to admit that this guide was written was icinga 1.6. The previous commenter noted some changes that must have taken place for icinga 1.7.

      If you went ahead with this, please search this page for the string “#thanks Luke!” to see the two lines that must be changed.

      They are very much related to the DB!

      My strong suggestion is to start over :(

      Thanks for your patience,

      Matt

      • Chris
        February 4, 2013 at 4:19 pm

        Thanks Matt! There’s a reason a I keep a CentOS VM template around :) I’ll just blast this one and run through it again. Your copy-pasta is so easy that redoing it really isn’t a big deal at all :) I’ll let you know what sorts of results I get.

      • Chris
        February 4, 2013 at 7:13 pm

        It worked!

        I did get some errors at at “Configure the icinga-web database”, but the app appears to be operational nevertheless.

        Here are where the errors appeared:

        [root@localhost icinga-web-1.7.2]# mysql -p db-create:

        Error during schema creation – are you sure the table doesn’t already exist? If so, use the update command.

        SQLSTATE[HY000]: General error: 1005 Can’t create table ‘icinga_web.#sql-6e4_15’ (errno: 121). Failing Query: “ALTER TABLE cronk ADD CONSTRAINT cronk_cronk_user_id_nsm_user_user_id FOREIGN KEY (cronk_user_id) REFERENCES nsm_user(user_id)”. Failing Query: ALTER TABLE cronk ADD CONSTRAINT cronk_cronk_user_id_nsm_user_user_id FOREIGN KEY (cronk_user_id) REFERENCES nsm_user(user_id)
        WARNING: Initial data import may have failed due too many references.

      • February 4, 2013 at 7:45 pm

        You said you installed on a totally bare system? Meaning there were no tables in the MySQL db? If so, I’d like to look into this error a bit more Ecuador it could indicate a problem. I’ll look into it a bit more tomorrow.

  9. Chris
    February 4, 2013 at 7:49 pm

    Yes indeed–just CentOS 6.3 with nano installed, openssh enabled, and an IP set on 1 eth adapter.

    Incidentally, do you know of a plugin that facilitates auto-discovery of hosts/devices?

    • February 4, 2013 at 8:17 pm

      Hmm… sorry actually I got confused with my other project OSSIM, and it’s autodiscover feature. Nagios and icinga work together with a lot of the same tools. So take a look at the nagiosexchange. There are also some other tools available. I do suggest you check out lconf though. What kind of stuff would you want to discover? Just a host definition? Services?

  10. Ferdinand
    February 7, 2013 at 12:40 pm

    Hi,

    The implementation of icinga-reports with icinga-web works well? … Because I try it on red hat 6, that suppose is the same as CentOS 6. The thing is that all works great even jasperreports but not from the new web interface of icinga … I tried to use it but show me an error, something like cannot connect to the server and cannot run any report fron there.

    thanks

    • February 7, 2013 at 2:25 pm

      Hello Ferdinand,

      Please make sure you are:

      #get[ting] the latest http://sourceforge.net/projects/icinga/files/icinga-reporting/
      

      There could be incompatibility problems with the latest icinga (db) and the icinga-reports that I originally included in the write up.

      See: http://sourceforge.net/projects/icinga/files/icinga-reporting/1.8.1/.

      Let me know if that solves your issues.

      Thanks,

      Matt

      • Ferdinand
        February 7, 2013 at 2:55 pm

        Casually I used the latest sources icinga-report 1.8.1 and jasperreports 5.0.0. Then I tried icinga-report 1.8.1 with jasperreports and also didn’t work. Now I will try with the previous version of both. I will let you know if it works.
        Thanks

  11. Federico
    March 1, 2013 at 3:34 pm

    I cannot connect jasper for reports with database… i am using jasper 1.8.1, could you help me?
    Regards,

    • March 1, 2013 at 3:54 pm

      Does the following file exist?

      /opt/jasperreports-server-cp-4.5.0/ctlscript.sh
      

      Also, please run the following and reply with the output:

      find / -iname ctlscript.sh
      netstat -nlp | grep java
      

      ….
      Sorry, I just re-read your comment… Do you mean you can access the JasperReporting web UI, but you can not add the icinga DB entries needed?

      Thanks,

      Matt

  12. Federico
    March 4, 2013 at 1:51 pm

    Thanks! it is working….but when i am starting the service icinga…i have this log:

    # service icinga restart && tail -f /var/log/icinga/
    archives/   gui/        icinga.log  
    [root@ICINGA ~]# service icinga restart && tail -f /var/log/icinga/icinga.log 
    Running configuration check...OK
    Stopping icinga: Stopping icinga done.
    Starting icinga: Starting icinga done.
    [1362422980] Successfully shutdown... (PID=8914)
    [1362422980] idomod: Shutdown complete.
    [1362422980] Event broker module '/usr/lib64/icinga/idomod.so' deinitialized successfully.
    [1362422980] Icinga 1.7.2 starting... (PID=8957)
    [1362422980] Local time is Mon Mar 04 15:49:40 ART 2013
    [1362422980] LOG VERSION: 2.0
    [1362422980] idomod: IDOMOD 1.7.2 (08-27-2012) Copyright(c) 2005-2008 Ethan Galstad, Copyright(c) 2009-2012 Icinga Development Team (https://www.icinga.org)
    [1362422980] idomod: Successfully connected to data sink.  0 queued items to flush.
    [1362422980] Event broker module 'IDOMOD' version '1.7.2' from '/usr/lib64/icinga/idomod.so' initialized successfully.
    [1362422980] Finished daemonizing... (New PID=8961)
    [1362422980] Event loop started...
    [1362422982] error executing command '/usr/lib64/nagios/plugins/check_load': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
    [1362423015] error executing command '/usr/lib64/nagios/plugins/check_users': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
    

    Could you help me please?

    • March 4, 2013 at 2:05 pm

      The concerning lines are:

      [1362422982] error executing command '/usr/lib64/nagios/plugins/check_load': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
      19
      [1362423015] error executing command '/usr/lib64/nagios/plugins/check_users': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
      

      Can you run the following and post the output:

      ls -al /usr/lib64/nagios/plugins/check_load
      ls -al /usr/lib64/nagios/plugins/check_users
      grep '\$USER1\$' /etc/icinga/resource.cfg
      

      Thanks,

      Matt

  13. Pemontto
    March 28, 2013 at 12:05 pm

    Excellent tutorial, thanks so much.

    Followed this guide and I was having a bit of trouble sending commands through from icinga-web, it would say they succeeded but no result in Icinga, sending commands from the ClassicUI worked fine.

    Turned out to be Icinga not set to look in the correct location. I added a step

    sed s@command_file=/var/spool/icinga/cmd/icinga.cmd@command_file=/var/icinga/rw/icinga.cmd@ -i /etc/icinga/icinga.cfg
    
    • March 28, 2013 at 12:11 pm

      Thanks for the compliments!

      Not sure how I missed including this on this page. It is included on the Part 4: Sending snmp traps to icinga page. I’ll definitively add it to this page to avoid confusion!

      Thanks for the input!

  14. Walt
    April 17, 2013 at 4:16 pm

    This is a great tutorial, and everything appears to be running successfully, website, services, etc. But I must be missing something because localhost (and its services) show down and unknown in both the classic web site and the icinga-web site. I expected the localhost site to come right up… Any suggestions?

    • April 17, 2013 at 4:56 pm

      Thanks for the complement. Good point. You must configure the snmpd.conf to allow polling of snmp values.

      # grep -v "^#\|^$" /etc/snmp/snmpd.conf
      #note that 192.168.100.10 is the local ip
      #'commstring' is the community id
      com2sec readonly 192.168.100.10/32    commstring
      group   readonlygroup   v1      readonly
      group   readonlygroup   v2c     readonly
      view    systemview    included   .1.3.6.1
      access  readonlyGroup ""      any       noauth    exact  systemview none none
      syslocation New York
      syscontact Matt
      dontLogTCPWrappersConnects yes
      
  15. sachin
    May 19, 2013 at 1:14 am

    hi mbrownnyc,
    i made all icinga-web setup, when i try to open icinga-web in the browser it says 404 not found nginx 1.4.x
    help me out …..i have gone any where wrong……
    thnx in advance……..

    • May 19, 2013 at 2:03 am

      Sounds like you are running nginx and not apache. Make sense?

  16. sachin
    May 19, 2013 at 8:01 am

    yes ur right……help me out from this……

    • May 19, 2013 at 10:56 am

      You’ll have to find out more about nginx. I am not familiar. Good luck!

  17. MtK
    August 16, 2013 at 8:28 am

    great tutorial.

    one question and one correction:
    1. this directory is never used:

    mkdir -p /usr/lib/nagios/plugins
    

    2. at this point `/var/icinga` doesn’t exisist, so this is ether the wrong directory, or use `mkdir -p`:

    mkdir /var/icinga/centplugins/ && chown icinga:icinga /var/icinga/centplugins/
    
    • August 16, 2013 at 10:48 am

      Thanks for the complements and the corrections! I didn’t notice those and have updated the original article noting “#thanks MTK!” in line. Thanks!

      • MtK
        August 16, 2013 at 10:54 am

        Thanks :)

        but also /var/icinga/centplugins/ is empty and /var/icinga has only this directory.

        Also, facing the same issue Walt mentioned:

        Walt :
        This is a great tutorial, and everything appears to be running successfully, website, services, etc. But I must be missing something because localhost (and its services) show down and unknown in both the classic web site and the icinga-web site. I expected the localhost site to come right up… Any suggestions?

        while my snmpd.conf is (default):

        # grep -v "^#\|^$" /etc/snmp/snmpd.conf
        com2sec notConfigUser  default       public
        group   notConfigGroup v1           notConfigUser
        group   notConfigGroup v2c           notConfigUser
        view    systemview    included   .1.3.6.1.2.1.1
        view    systemview    included   .1.3.6.1.2.1.25.1.1
        access  notConfigGroup ""      any       noauth    exact  systemview none none
        syslocation Unknown (edit /etc/snmp/snmpd.conf)
        syscontact Root  (configure /etc/snmp/snmp.local.conf)
        dontLogTCPWrappersConnects yes
        

        and the error log shows (/var/log/icinga/icinga.log):

        [1376664531] error executing command '/usr/lib64/nagios/plugins/check_swap': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
        [1376664565] error executing command '/usr/lib64/nagios/plugins/check_procs': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
        [1376664656] error executing command '/usr/lib64/nagios/plugins/check_load': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
        [1376664665] error executing command '/usr/lib64/nagios/plugins/check_http': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
        [1376664677] error executing command '/usr/lib64/nagios/plugins/check_users': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
        [1376664704] The command defined for service Icinga Startup Delay does not exist
        [1376664731] error executing command '/usr/lib64/nagios/plugins/check_ping': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
        [1376664765] error executing command '/usr/lib64/nagios/plugins/check_disk': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
        [1376664784] error executing command '/usr/lib64/nagios/plugins/check_ping': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
        [1376664798] error executing command '/usr/lib64/nagios/plugins/check_ssh': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
        
  18. August 16, 2013 at 11:07 am

    What you’re describing seems to be rooted in the fact that a checkout of source from the centreon project takes place:

    cd
    svn checkout http://svn.centreon.com/trunk/plugins-2.x/src/
    

    Followed by several lines of modification to the internal scripts:

    cd src
    find . -iname *svn* | xargs /bin/rm -rf
    grep -rl '@NAGIOS_PLUGINS@' . | xargs sed 's&@NAGIOS_PLUGINS@&/usr/local/nagios/libexec&g' -i
    mkdir -p /var/icinga/centplugins/ && chown icinga:icinga /var/icinga/centplugins/ #thanks MTK
    grep -rl '@CENTPLUGINS_TMP@' . | xargs sed 's&@CENTPLUGINS_TMP@&/var/icinga/centplugins&g' -i
    sed s/max_msg_size\(5000\)/max_msg_size\(30000\)/ -i ./check_centreon_snmp_cpu #http://forum.centreon.com/showthread.php/12097-Is-it-safe-to-adjust-the-max_msg_size-in
    /bin/cp -rf * /usr/local/nagios/libexec
    chmod +x /usr/local/nagios/libexec/*
    

    Did you perform these lines as well?

    The following indicates that you do not have the scripts in the correct location::

    error executing command '/usr/lib64/nagios/plugins/check_swap': No such file or directory. Make sure that the file actually exists
    

    Try to find this one script with:

    find / -iname check_swap
    

    Does this make sense?

    • MtK
      August 16, 2013 at 11:14 am

      yes, of course I performed those lines, otherwise I wouldn’t be able to comment on the missing -p.
      anyway, those lines do nothing to /var/icinga/centplugins/ the files are copied to /usr/local/nagios/libexec

      and for the other issue:

      # find / -iname check_swap
      /usr/local/nagios/libexec/check_swap
      /usr/local/nagios/libexec/nagios-plugins-1.4.16/plugins/check_swap
      
      • August 16, 2013 at 11:27 am

        The presence of:

        /usr/local/nagios/libexec/nagios-plugins-1.4.16/plugins/check_swap
        

        …indicates that you did not perform the exact syntax, as you should have switched to your home directory with `cd` before performing the `wget` & `tar zxvf` commands as noted above.
         

        Here is some more info that may be of additional assistance:

        Did you perform:

        grep -rl '@NAGIOS_PLUGINS@' . | xargs sed 's&@NAGIOS_PLUGINS@&/usr/local/nagios/libexec&g' -i

        …within the `~/src` directory which should be the directory where the centreon-plugins svn checkout targeted?

        If so, you should have replaced all instances of the string `@NAGIOS_PLUGINS@` with `/usr/local/nagios/libexec`.
         

        Then the:

        /bin/cp -rf * /usr/local/nagios/libexec
        

        …should copy all the scripts to `/usr/local/nagios/libexec/`.
         

        `/usr/local/nagios/libexec/` should already exist due to:

        cd #<====== I do this so that we return the same directory easily!  You can make this any directory, but `cd` automatically places you in your home directory (aka `~/`)
        wget http://sourceforge.net/projects/nagiosplug/files/latest/download?source=files
        tar zxvf nagios-plugins-*.tar.gz
        cd nagios-plugins-*
        ./configure && make && make install
        

        Does this make sense?

  19. MtK
    August 16, 2013 at 11:44 am

    let’s seperate the issues so it is easier to follow.

    I rolled back into a fresh CentOS 6.4 installation and re-did this:

    cd #&lt;====== I do this so that we return the same directory easily!  You can make this any directory, but `cd` automatically places you in your home directory (aka `~/`)
    wget <a href="http://sourceforge.net/projects/nagiosplug/files/latest/download?source=files" rel="nofollow">http://sourceforge.net/projects/nagiosplug/files/latest/download?source=files</a>
    tar zxvf nagios-plugins-*.tar.gz
    cd nagios-plugins-*
    ./configure &amp;&amp; make &amp;&amp; make install
    

    which resulted in this:

    # find / -iname check_swap
    /usr/local/nagios/libexec/check_swap
    /root/nagios-plugins-1.4.16/plugins/check_swap
    

    so this looks OK, but still nothing in:

    # ls -lah /usr/lib64/nagios/plugins/
    total 92K
    drwxrwxr-x. 2 root root 4.0K Aug 16 18:40 .
    drwxr-xr-x. 3 root root 4.0K Aug 16 18:39 ..
    -rwxr-xr-x. 1 root root  38K Aug 17  2012 negate
    -rwxr-xr-x. 1 root root  36K Aug 17  2012 urlize
    -rw-r--r--. 1 root root 2.1K Aug 17  2012 utils.pm
    -rwxr-xr-x. 1 root root 2.7K Aug 17  2012 utils.sh
    
    • August 16, 2013 at 11:56 am

      The reason I conflated the issues is because they are directly related. It doesn’t appear that you’ve followed the syntax exactly. Please do this and your problems will be solved. I understand… “but we must fix what’s broken! It must be possible!” But it’s not worth either of our time. It will be much faster if you simply use a fresh image of CentOS6 and start from the beginning and use the exact syntax.

       
      But… of course… I can’t resist a good Friday troubleshooting session:

      `/usr/lib64/nagios/plugins/` should not exist. Does it exist?

      # ls -al /usr/lib64/nagios/plugins/
      ls: cannot access /usr/lib64/nagios/plugins/: No such file or directory
      
      # ls -al /usr/local/nagios/libexec | head
      total 4124
      drwxr-xr-x. 5 root root   4096 Jul 25 10:19 .
      drwxr-xr-x. 5 root root   4096 Aug 23  2012 ..
      drwxr-xr-x. 3 root root   4096 Aug 23  2012 Centreon
      -rwxr-xr-x. 1 root root   1425 Aug 23  2012 centreon.conf
      -rwxr-xr-x. 1 root root   3566 Aug 23  2012 centreon.pm
      -rwxr-xr-x. 1 root root  91250 Aug 23  2012 check_apt
      -rwxr-xr-x. 1 root root   2254 Aug 23  2012 check_breeze
      -rwxr-xr-x. 1 root root 104612 Aug 23  2012 check_by_ssh
      -rwxr-xr-x. 1 root root   3825 Aug 23  2012 check_centreon_dummy
      

      Ahh hah…

      # ls -al /usr/lib/nagios/plugins
      total 88
      drwxrwxr-x. 2 root root  4096 Aug 23  2012 .
      drwxr-xr-x. 3 root root  4096 Aug 23  2012 ..
      -rwxr-xr-x. 1 root root 36680 Jul  1  2012 negate
      -rwxr-xr-x. 1 root root 33288 Jul  1  2012 urlize
      -rw-r--r--. 1 root root  1939 Jul  1  2012 utils.pm
      -rwxr-xr-x. 1 root root   862 Jul  1  2012 utils.sh
      

      Hope this helps.

      • MtK
        August 16, 2013 at 12:15 pm

        Well, as I showed you above it does exist:

        # ls -lah /usr/lib64/nagios/plugins/
        total 92K
        drwxrwxr-x. 2 root root 4.0K Aug 16 18:40 .
        drwxr-xr-x. 3 root root 4.0K Aug 16 18:39 ..
        -rwxr-xr-x. 1 root root  38K Aug 17  2012 negate
        -rwxr-xr-x. 1 root root  36K Aug 17  2012 urlize
        -rw-r--r--. 1 root root 2.1K Aug 17  2012 utils.pm
        -rwxr-xr-x. 1 root root 2.7K Aug 17  2012 utils.sh
        

        so the issue here is that this tutorial is for 32bit while my system is 64bit.
        so line #21 on that same section should be:

        sed s@/usr/lib64/nagios/plugins@/usr/local/nagios/libexec/@ -i /etc/icinga/resource.cfg
        
  20. August 16, 2013 at 12:36 pm

    Ahh… this sounds correct!

    # arch
    i686
    

    I’ve updated the original post! Nice catch!

  21. MtK
    August 16, 2013 at 12:37 pm

    mbrownnyc :
    Ahh… this sounds correct!

    # arch
    i686
    

    now, after finishing the installation, the localhost is UP and it’s services and OK (and not UNKNOWN)

    thanks :)

    • August 16, 2013 at 12:44 pm

      Thanks for sharing! This is actually a very useful solution. I might even be able to build in a conditional to the line to determine the `arch`, but I will just note it for now!

      Apologies for the earlier confusion. Thanks!

  22. MtK
    August 16, 2013 at 12:46 pm

    mbrownnyc :
    Thanks for sharing! This is actually a very useful solution. I might even be able to build in a conditional to the line to determine the `arch`, but I will just note it for now!
    Apologies for the earlier confusion. Thanks!

    sound great!

    now we can go back to the fact that:

    # ls -l /var/icinga/centplugins/
    total 0
    

    (like I stated earlier)

    • August 16, 2013 at 12:54 pm

      This directory is used by some centreon command files as temporary space (specifically for a cache).

      # cd 
      # cd src
      # grep -lr centplugins
      ./check_centreon_snmp_value_table.pl
      ./check_centreon_snmp_packetErrors
      ./check_centreon_snmp_remote_storage
      ./check_centreon_snmp_value
      ./check_centreon_snmp_traffic
      
      # ls -l /var/icinga/centplugins/
      total 40
      -rw-r--r--. 1 icinga icinga 120 Aug 16 10:47 remote_storage_cache_hosta
      -rw-r--r--. 1 icinga icinga 130 Aug 16 12:43 remote_storage_cache_hostb
      

      So, you must:

      mkdir -p /var/icinga/centplugins/ && chown icinga:icinga /var/icinga/centplugins/
      
  23. MtK
    August 24, 2013 at 9:14 am

    just to let you know, the selinux configuration does not survive a reboot of the machine.

    Could not connect to the JasperServer (Raw: SOAP-ERROR: Parsing WSDL: Couldn't load from 'http://127.0.0.1:8080/jasperserver/services/repository?wsdl' : failed to load external entity "http://127.0.0.1:8080/jasperserver/services/repository?wsdl" ).
    Seems that reporting is unconfigured or not installed!

    if I set:

    setenforce permissive

    no error is displayed.

    • August 26, 2013 at 7:47 am

      Yes, I don’t cover this here. To be more verbose, I don’t cover it here because it is bad practice. icinga-web is documented as to not be secure enough to be served to the public, and could contain many flaws. Therefore, I have permanently set selinux to permissive on my host box… however, that’s generally not a good thing, and definitely not a thing I’d like to proliferate.

      Feel free to comment back on the solution or post it on your blog and link it :)

      • September 30, 2013 at 3:33 pm

        I found this error also occurs when you have changed the default jasperserver credentials (probably a good idea). Modify /etc/icinga-web/conf.d/module_reporting.xml with the new credentials to get Icinga-Web to be able to talk with JasperServer again.

  24. Vladan
    August 29, 2013 at 5:56 am

    Hello,
    I just installed Jasperreport srv and I have problem that when I try to generate some report from Icinga in Jasper. I don’t have anything to select in hostname tab.
    Any idea what could be the problem ?

    • August 29, 2013 at 9:03 am

      Hello Vladan,

      Thanks for following the blog by clicking “Follow” at the top.

      Did you follow the “Configure JasperServer to access the icinga DB” section?

      Thanks,

      Matt

  25. September 24, 2013 at 10:36 am

    I’m having trouble with the icinga-reporting init.d script import from github. It would appear th path is not valid any longer:

    curl -k https://raw.github.com/gist/3437554/f304ac207b363e36257f41390f03daafeaec7f99/icinga-reporting > /etc/init.d/icinga-reporting
    init.d/icinga-reporting contains the headers of the GitHub 404 page instead of usable information.
    

    Could you post the full content of the init.d script or provide an updated link?

    BTW, this guide has been most helpful. I had several issues during initial configuration, but they were all from my failing to update version numbers in the various packages as I downloaded and installed them.

    • September 24, 2013 at 10:55 am

      Thanks Matthew. I will update the link, then go through the entire blog and do the same, as this caused confusion on another post.

      Try:

      <!-- Invalid Gist ID -->
      
      • September 24, 2013 at 11:53 am

        That works perfectly.Thank you.

        As a note, I am running Jasper Server version 5.2.0 and so had to change the JasperCTLScript path in the file. Everything appears to be working well now.

      • September 24, 2013 at 12:26 pm

        Thanks for this info!

  26. Sven
    October 4, 2013 at 2:47 am

    Hi,
    the Nagios Plugins are no longer distributed via SourceForge, you can download them at: https://www.nagios-plugins.org/download/nagios-plugins-1.5.tar.gz

    if i try to do the mkdir and chown at “centplugins” the system told me, that there is no icinga user. thats right, because we must do an useradd before, or not?

    • October 17, 2013 at 1:31 pm

      Thanks Sven. I changed the order of the tutorial and it should resolve the user issue. As for the nagios plugins, it’s unfortunate, but there isn’t a “nagios-plugins-latest.tar.gz” or the like on nagios-plugins.org. Also I noticed that nagios-plugins.org latest was updated actually one day later than sourceforge. Do you work on the project and have new info that for sure it will only be distributed via nagios-plugins.org?

      Thanks for the info nonetheless!

  27. October 4, 2013 at 12:57 pm

    Another question: How/where do you go about changing the credentials Icinga uses to connect to JasperServer? Right now I have to leave the default jasperadmin credentials or Icinga-Reporting is not able to connect.

    • October 4, 2013 at 1:48 pm

      Nevermind, I figured it out. There are two module_reporting.xml locations:

      /etc/icinga-web/conf.d/module_reporting.xml
      /usr/local/icinga-web/etc/conf.d/module_reporting.xml

      I’d changed the credentials in the /etc/… one, but that didn’t do the trick. Using locate I found the other and changed the credentials there, and it worked.

  28. josh
    March 18, 2014 at 11:29 pm

    Nagios Plugins are no longer hosted on SourceForge. They’re here now: http://nagios-plugins.org/download/nagios-plugins-2.0.tar.gz

  29. josh
    March 19, 2014 at 1:07 am

    With Jasper Report Server 5.5 it seems the /etc/init.d/icinga-reporting script needs to be changed a bit:

    1. change reference to 4.5 to 5.5
    2. change “postgres” to “postgresql”

  30. josh
    March 19, 2014 at 1:12 am

    I updated the /etc/init.d/icinga-service script to look like the below (updated the version of jasper server to reflect 5.5 and also updated how they now refer to “postgresql” – previously just “postgres”).

    I don’t think your step-by-step had a section that made tomcat start/stop at server boot, so I added it here.

    JASPERCTLSCRIPT="/opt/jasperreports-server-cp-5.5.0/ctlscript.sh "
    
    start()
    {
            $JASPERCTLSCRIPT start postgresql
            $JASPERCTLSCRIPT start tomcat
    
    }
    

    ….. do the same thing for the stop(), restart(), and status() sections

  31. Researcher
    October 17, 2014 at 4:06 am

    You forgot:

     # Import the resources for icinga-report to jasperserver
    
     cd /opt/jasperreports-server-cp-*
    ./js-import.sh --input-zip /root/icinga-reports-*/reports/icinga/package/js-icinga-reports.zip
    
  32. sk
    October 5, 2015 at 10:13 am

    Thanks a lot for this blog,,really loved it..
    I followed your blog seems to be everything fine but i am unable to execute commands in Icinga-web GUI prompt by getting below error.
    “Logger “4294967296” has not been configured.”
    I googled it but no use..can any plz help

    I am requesting you to provide steps for icinga and icingaa-web upgrade as well..

    • October 30, 2015 at 7:34 am

      You should check on monitoring-portal.org.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: