Home > Uncategorized > Install, configure, and use ntop to monitor traffic

Install, configure, and use ntop to monitor traffic

Note that this write up is old and not for ntopng. You probably want ntopng.

If you like ntop, you might like flow-inspector, an up and coming open source project. Leveraging d3.js to render flow statistics, it is a very useful tool to quickly get flow information, utilizing a variety of visualization techniques. flow-inspector can use argus, VERMONT, and Bro IDS connection logs (and soon snmp connection stats) as flow sources.


This is a work in progress:

  • testing has not yet been done.
  • Problem installing libgts, hence graphviz (updated December 22nd)

This is in complement to a previous post about the iptables module that is a Netflow generator, ipt_NETFLOW.

The current goal of this project is to use a switch to mirror traffic to a box with a single NIC, generate Netflow data for the packets, direct them back to the same box where ntop sits.  This is the latter portion of that… installing, configuring, and using ntop to gather data on network traffic.

Holistically, the final goal of the project is to produce useful information; produce a report on source and destinations, which should help identify “abnormal” network traffic.

At our perimeter we have a transparent proxy performing some application layer filtering of HTTP, FTP, SMTP, and IMAP.  There are also firewall policies in place that do what a firewall does, provides access control up to the transport layer (IP, UDP, and TCP based access control).  With these two methods combined, I have reduced the surface of extrusions from the internal network to the outside world, getting a tighter grip on protocol usage, odd destinations, and hopefully can provide further insight during postmortem analysis of malware incidents, as well as generally monitoring the usage of the Internet.

I will start by using ntop as a solution to this challenge, but may look into a network visualizer, like visualfirewall, or a “flow perspective-izer,” like argus, if and when this solution fails to produce the information I want (I suppose live monitoring and analysis of the past are two different subjects).

Alright, enough self-affirming banter!

Building and Installing ntop
Install some prerequisites:

yum -y install lsof vim gcc make wget gcc-c++ glibc-devel glib2-devel m4 autoconf automake gdbm gdbm-devel libpcap libpcap-devel libtool rrd rrdtool-devel openssl openssl-devel zlib zlib-deveyum l

Install libGeoIP prerequisite:

cd
wget http://geolite.maxmind.com/download/geoip/api/c/GeoIP.tar.gz
tar zxvf GeoIP.tar.gz
cd GeoIP-*
./configure
make
make install

Install GraphViz for IP> Local> Network Traffic Map

#install the graphviz repo
curl http://www.graphviz.org/graphviz-rhel.repo > /etc/yum.repos.d/graphviz-rhel.repo
#install graphviz and its dependencies
yum -y install graphviz
#remove the graphviz repo
rm -f /etc/yum.repos.d/graphviz-rhel.repo

Install ntop:

Install stable (no nDPI):

cd
wget http://sourceforge.net/projects/ntop/files/latest/download?source=files
tar zxvf ntop-*.tar.gz
cd ntop-*
./autogen.sh
make
make install
make install-selinux-policy
less docs/1STRUN.tx

Install unstable (with nDPI and a nicer UI):
Follow another post and return to this post.

Initialize the ntop user database:

useradd ntop #create a user to run ntop
passwd ntop #assign that user a password
chown ntop:ntop /usr/local/share/ntop
mkdir /var/opt/ntopdb
chown ntop:ntop /var/opt/ntopdb
which ntop
su - ntop -c "/usr/local/bin/ntop -P /var/opt/ntopdb -u ntop -A"
ls /var/opt/ntopdb/

Delete ntop_pw.db if you forget the admin password and run this again.

Configure ntop to run at system startup:

ntop init script:

vim /etc/init.d/ntop

The contents of /etc/init.d/ntop should be as follows:
Note that you MUST change the ifconfig lines to match the interface you are mirroring traffic to!

#!/bin/sh
#
#
#
#
# chkconfig: 345 90 10
# description:
#init script for ntop, written by
#branded on freenode
#mbrownnyc everywhere else
#

### BEGIN INIT INFO
# Provides:          ntop
# Required-Start:    $local_fs $network
# Required-Stop:     $local_fs $network
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 3 4 5 6
# Description:       ntop pre-configured by OPTIONS in /etc/sysconfig/ntop
### END INIT INFO

# Source function library.
. /etc/rc.d/init.d/functions

exec="/usr/local/bin/ntop"
prog="ntop"
progname="ntop"
lockfile=/var/lock/subsys/ntop

[ -e /etc/sysconfig/$progname ] && . /etc/sysconfig/$progname

  rh_status() {
	echo "Checking status of $prog"
        status -p /var/run/$prog.pid -l $lockfile $progname
  }

  rh_status_q() {
        rh_status
	#>/dev/null 2>&1
  }

  start() {
        echo "Starting $progname:"
	echo "   $exec $OPTIONS"
	echo
        $exec $OPTIONS
	RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch $lockfile
        return $RETVAL
		}
  stop() {
        echo -n "Stopping $progname: "
        killproc $prog
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && rm -f $lockfile
        return $RETVAL
  }
  restart() {
        stop
        start
  }

case "$1" in
    start)
	rh_status_q && exit 0
        $1
	;;
    stop)
	rh_status_q || exit 0
	$1
	;;
    restart)
	$1
	;;
    status)
	status -p /var/run/$prog.pid -l $lockfile $progname
	;;
    *)
	echo $"Usage: $0 {start|stop|restart|status}"
	exit 1
esac
exit 0

Create a config file to pull configuration from:

vim /etc/sysconfig/ntop

Contents of /etc/sysconfig/ntop:

OPTIONS="--daemon --max-table-rows 200 --local-subnets '192.168.10.0/24' --trace-level 2 --user ntop <strong>--http-server 0 </strong>--https-server 443 --domain domain.local --use-syslog=local3 --db-file-path /var/opt/ntopdb --mapper --interface none --no-fc --ipv4"

Note that these are options that I use in my config. The following arguments should be the same according to the previous config steps: –user, –db-file-path. Read the manpage to learn all of the options.

Add the service to be managed:

chkconfig --add ntop
chkconfig --levels 345 ntop on #configure ntop to execute at the desired runlevels

Change permissions on the file and execute the init script using the service manager:

chmod +x /etc/init.d/ntop
service ntop start

You may see the following error:

**ERROR** ++++ DEMON MODE=1

After reading the manpage, it became clear that not everything logged at an ERROR level is an error. This is the case with this error, it’s just a notification that ntop is in  mode.

Locate the ports where ntop is bound

ntop serves over http and https by default. Review the manpage for more info. I have used the –https-server parameter to control the port where ntop’s https server is bound.

netstat -apn | grep ntop

Access the web UI and review the config

Hit the host in a web browser at https://%5Bhost%5D

On the menu, hover on About, and click Show Configuration to review the currently running configuration. The manpage describes each setting in detail including their default value.

Poke around in the Admin menu, authenticating with the username admin and the password you previously configured.

Configure the Netflow plugin

Go to Plugins> Netflow> View/Configure

Confirm the Netflow device name.

Make sure that the Local Collector UDP port is 2055.  This will bind ntop to collect Netflow packets at UDP 2055.

Set some whitelist entries (a comma-separated list of hosts you’d like to capture), and blacklist entry (at least [the local IP of the server]/32).

Disable debugging.  You can re-enable this later.

Note that the plugin config will be retained through cycling of the process.

Verify your Netflow probe is sending Netflow data properly

I’m going to assume that you are using a local iptables instance with the ipt_NETFLOW module to sniff packets and produce Netflow packets.

To verify the port config is correct, run the following command which reveals the runtime/current settings for ipt_NETFLOW:

sysctl -a | grep net.netflow

Configure mirroring on your switch, and test.

About these ads
  1. Aaron
    September 21, 2012 at 2:51 pm

    Thanks for the guide! Almost a year later since you wrote it, but wanted to point out an issue I found with your init script. You point to /var/run/$prog.id as the location of the pid file but following your instructions (albeit using 5.0.2) the pid is in /var/opt/ntopdb/. Other than that, this is hands down the best (and only real working) guide that The Google has led me to. Thanks again.

    • September 21, 2012 at 3:04 pm

      Thanks very much for the compliments!

      Can you confirm that you installed from source or did you use a package (like an rpm or yum)?

      I no longer have ntop in production, and have since moved on to argus. But really, it seems that ntop and argus solve two different problems (at least for me).

      I mentioned the idea of implementing nDPI as an argus client, and the argus dev reacted sort of poorly (probably reasonably, since I didn’t RTFM). That would be quite useful. argus takes a lot more effort to make usable, and doesn’t feature a GUI. It is my dream to program a front end leveraging d3.js to render argus data that’s stored in a DB.

  2. Aaron
    September 25, 2012 at 8:47 am

    It was installed using your alternate instructions for using nDPI and that svn command. I’m guessing that’s the difference.

    Beyond that, I’m actually having a couple of minor issues and I’d like to see if you’ve run across them. I used the local3 setup in /etc/rsylog.conf (CentOS 6.2) but I’m still logging to /var/log/syslog and not to the log file I’ve set up. I’ve stopped/started/restarted, etc.

    I also use the –skip-version-check option, but I’m still seeing CHKVER messages being logged.

  3. Aaron
    September 25, 2012 at 8:52 am

    Blarg. Too much working with Ubuntu. It’s logging to /var/log/messages not syslog. Sorry about the extra post.

    And to clarify, here’s my entry in /etc/rsyslog.conf:

    # NTOP
    local3.* /var/log/ntop.log

    And the flag from /etc/sysconfig/ntop:

    –use-syslog=local3

    • September 25, 2012 at 12:09 pm

      The –skip-version-check option’s description in the ntop.html might indicate something:

      By default, ntop accesses a remote file to periodically check if the most current version is running. This option disables that check.
      Please review the privacy notice at the bottom of this page for more information. By default, the recheck period is slightly more than 15 days. This can be adjusted via a constant in globals-defines.h.
      If the result of the initial check indicates that the ntop version is a new development version (that is newer than the latest published development version), the recheck is disabled.  This is because which fixes and enhancements were present/absent from the code.
      

      …but probably not. Sounds like a question for the ntop mailing group citing the minor build you’ve compiled.

      From what I can see, the following will log all local3.* to your /var/log/ntop.log files

      local3.* /var/log/ntop.log
      

      So it should be logging when –use-syslog=local3 since you will…

      Use this parameter to send log messages to the system log instead of stdout... The complex form, --use-syslog=facility will set the log facility to whatever value (e.g. local3, security) you specify.
      

      If the setting –use-syslog is not functioning in the SVN build, sounds like a job for the mailing list.

      You can test using the following:

      logger -p local3.info "test from logger at command line, to local3.info"
      

      This should send a syslog message to local3 facility with info severity.

      Let me know how it works out.

  4. Ray
    October 2, 2012 at 6:13 pm

    You mentioned you switched to ARGUS I am very interested in installing argus I know support is very limited and argus offers no GUI. However, accurate install details will be a great help.

  5. Ray
    October 2, 2012 at 6:31 pm

    I am getting error when enabling “su – ntop -c “/usr/local/bin/ntop -P /var/opt/ntopdb -u ntop -A”

    Will appreciate how to rectify.
    Error details:

    ” to”[root@localhost ntop-4.0.1]# su – ntop -c “/usr/local/bin/ntop -P /var/opt/ntopdb -u ntop -A”
    Tue Oct 2 15:28:55 2012 NOTE: Interface merge enabled by default
    Tue Oct 2 15:28:55 2012 Initializing gdbm databases
    Tue Oct 2 15:28:55 2012 **ERROR** ….open of /var/opt/ntopdb/prefsCache.db failed: File open error
    Tue Oct 2 15:28:55 2012 Possible solution: please use ‘-P ‘
    Tue Oct 2 15:28:55 2012 **FATAL_ERROR** GDBM open failed, ntop shutting down…
    Tue Oct 2 15:28:55 2012 CLEANUP[t139917259868416]: ntop caught signal 2 [state=2]
    Tue Oct 2 15:28:55 2012 ntop is now quitting…

    • October 2, 2012 at 9:11 pm

      Unfortunately, if you’ve received an error, it’s pretty likely you’ve not followed this page.

      As for this specific error, I have no idea how to solve it. I could get to googling, but I don’t have time right now. Additionally, you may refer to the ntop.org page on support.

    • Boomi
      October 15, 2012 at 7:45 am

      Hi Ray,
      Just rename the existing prefsCache.db and ntop_pw.db. After that again run the same command. Now it do not shows error.

  6. sethuper
    August 24, 2014 at 2:55 am

    I have followed step by step in this tutorial, everything is describe, how to install and configure newest version (new generation) of ntop – ntopng on Debian/Ubuntu.

    http://terminal28.com/ntopng-ntop-web-based-network-traffic-monitoring-system-linux-debian/

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 32 other followers

%d bloggers like this: