Part 5.c: Build, deploy, and use argus

Now that you have flow-inspector rendering some useful views of your data about the network flows, you will want to drill down deeply to obtain irrefutable answers about specific flows.

Why?
Currently, VERMONT is an IPFIX probe, and can output to a database. In fact, the flow-inspector project also contains a module to import flow records from an “rasqlinserted” argus database. So why use argus + VERMONT + flow-inspector?

During very quick (!) testing, I was unable to verify that the data imported to flow-inspector from an rasqlinserted argus database versus the data available directly from the VERMONT probe versus data in the argus files were the same. I might deeply test this more so that my test can be irrefutable itself. :) [updated note: this might be due to >rasqlinsert’s cache, or might be due to preprocessor.py’s flushing of it’s cache.]

But, because of this, and needing to roll out something stable quickly that would give me very robust data, I chose to double-up on my probes. VERMONT is the source of data for flow-inspector, and argus is, well, the source of data for rastream.

That way, I can use flow-inspector to give me a good overview of what’s happening, and then use the argus ra* clients to drill down to see a variety of statistics that are not yet available through flow-inspector.

Download and Install argus:

yum -y install gcc make bison libpcap libpcap-devel cyrus-sasl readline-devel ncurses-devel flex rrdtool rrdtool-perl geoip-devel geoip perl-Geo-IP 
cd
wget http://qosient.com/argus/src/argus-latest.tar.gz http://qosient.com/argus/dev/argus-clients-latest.tar.gz
tar zxvf argus-latest.tar.gz
cd argus-*
./configure
make && make install
cd
tar zxvf argus-clients-latest.tar.gz
cd argus-clients-*
./configure
make && make install

Run the argus probe:
You can review the argus.conf man page and adjust settings.

cp ./argus-*/support/Config/argus.conf /etc/argus.conf
#if you have interest in inspecting data buffers set:
# ARGUS_CAPTURE_DATA_LEN=256
argus -d -i eth0 -P 561

Write argus data to files:
`-d` daemonizes the executable internally.

cd 
mkdir /var/opt/argus
cp ./argus-clients-*/support/Config/rastream.sh /usr/local/bin/rastream.sh
chmod 770 /usr/local/bin/rastream.sh
rastream -d -S 127.0.0.1:561 -B 15s -M time 1h -w /var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh #note resolution of %T will be to the `-M time N`

init.d scripts:
I have re-written init scripts for use with enterprise linux based *nixes as gists:

Run analysis against argus data files:

argus
the argus data flow, with a few example clients

 

Primarily I’ll use racluster to aggregate files, ragraph to show some useful time-based stuff, and rasort to sort binary ra* output. All of the ra* clients reference ra syntax. The NSM wiki contains some nearly impossible to understanding queries using ra* clients, so enjoy that.

I suggest you use a scrap working directory when messing around with the data files. This avoids overwriting the source data for instance. Check out this gist which is a quick script that dumps the gunzipped datafiles to a working directory. Run it when you are within the directory with the gzipped data files.

In the following queries, `-nr *` means that the query targets all files in current directory, and it will not resolve the names of ports (as listed in /etc/services).

Global overview:
1) How many bytes sent per node (sorted asc by bytes)

racluster -m saddr bytes -nr * -w - - ip | rasort -m bytes -s saddr bytes | less

2) How many bytes received per node (sorted asc by bytes).

racluster -m daddr bytes -nr * -w - - ip | rasort -m bytes -s daddr bytes | less

3) How many bytes sent per node, per port (sorted asc by saddr).

racluster -m saddr bytes dport -nr * -w - - ip | rasort -m saddr -s saddr dport bytes  | less

4) How many bytes received per node, per port (sorted asc by daddr).

racluster -m daddr bytes dport -nr * -w - - ip | rasort -m daddr -s daddr dport bytes  | less

5) How many bytes sent per node, per transport protocol.

racluster -m saddr bytes proto -nr * -w - - ip | rasort -m bytes -s saddr proto bytes | less

6) How many bytes sent per node, per transport protocol.


Digging down to target a host:
1) How many bytes sent, per node, to node.

racluster -m saddr daddr bytes dport -nr * -w - - ip and src host 192.168.100.23 | rasort -m bytes -s saddr daddr bytes dport | less

`ip and src host` is the filtering statement. You are safe to play around with the Berkeley Packet Filter syntax, but there are some modifications noted on page 8 of the ra(1) man page in the “Filter Expression” section.

2) how many bytes sent, per node, to port of node:

racluster -m saddr daddr dport bytes -nr * -w - - ip and src host 192.168.100.23 | rasort -m bytes -s saddr daddr dport bytes | less

Let’s get graphing:
1) chart all bytes with a given saddr, with one second resolution

ragraph saddr bytes -M 1s -w test.png -n -n -nr * - src host 192.168.100.23

2) chart of bytes with a given saddr and a given daddr, with one second resolution

ragraph saddr bytes -M 1s -n -n -nr * - src host 192.168.100.23 and dst host 108.160.160.15

3) chart of bytes with a given saddr and a given daddr and dport, with one second resolution

ragraph dport bytes -M 1s -n -n -nr * - src host 192.168.100.23 and dst host 108.160.160.15

4) chart of bytes with a given src net, excluding a given dst net and icmp, with one second resolution, and a title, saving it to a file icmp100cidr24.png.

ragraph bytes -M 1s -w icmp100cidr24NOT101cidr24.png -title "ICMP from 192.168.100.0/24 (exc. 192.168.101.0/24)" -n -n -nr * - src net 192.168.100.0/24 and not dst net 192.168.101.0/24 and icmp

`icmp` is from /etc/protocols.

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: