Part 5.c: Build, deploy, and use argus
Now that you have flow-inspector rendering some useful views of your data about the network flows, you will want to drill down deeply to obtain irrefutable answers about specific flows.
Currently, VERMONT is an IPFIX probe, and can output to a database. In fact, the flow-inspector project also contains a module to import flow records from an “rasqlinserted” argus database. So why use argus + VERMONT + flow-inspector?
During very quick (!) testing, I was unable to verify that the data imported to flow-inspector from an rasqlinserted argus database versus the data available directly from the VERMONT probe versus data in the argus files were the same. I might deeply test this more so that my test can be irrefutable itself. :) [updated note: this might be due to >rasqlinsert’s cache, or might be due to preprocessor.py’s flushing of it’s cache.]
But, because of this, and needing to roll out something stable quickly that would give me very robust data, I chose to double-up on my probes. VERMONT is the source of data for flow-inspector, and argus is, well, the source of data for rastream.
That way, I can use flow-inspector to give me a good overview of what’s happening, and then use the argus ra* clients to drill down to see a variety of statistics that are not yet available through flow-inspector.
Download and Install argus:
yum -y install gcc make bison libpcap libpcap-devel cyrus-sasl readline-devel ncurses-devel flex rrdtool rrdtool-perl geoip-devel geoip perl-Geo-IP cd wget http://qosient.com/argus/src/argus-latest.tar.gz http://qosient.com/argus/dev/argus-clients-latest.tar.gz tar zxvf argus-latest.tar.gz cd argus-* ./configure make && make install cd tar zxvf argus-clients-latest.tar.gz cd argus-clients-* ./configure make && make install
Run the argus probe:
You can review the argus.conf man page and adjust settings.
cp ./argus-*/support/Config/argus.conf /etc/argus.conf #if you have interest in inspecting data buffers set: # ARGUS_CAPTURE_DATA_LEN=256 argus -d -i eth0 -P 561
Write argus data to files:
`-d` daemonizes the executable internally.
cd mkdir /var/opt/argus cp ./argus-clients-*/support/Config/rastream.sh /usr/local/bin/rastream.sh chmod 770 /usr/local/bin/rastream.sh rastream -d -S 127.0.0.1:561 -B 15s -M time 1h -w /var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh #note resolution of %T will be to the `-M time N`
I have re-written init scripts for use with enterprise linux based *nixes as gists:
Run analysis against argus data files:
Primarily I’ll use racluster to aggregate files, ragraph to show some useful time-based stuff, and rasort to sort binary ra* output. All of the ra* clients reference ra syntax. The NSM wiki contains some nearly impossible to understanding queries using ra* clients, so enjoy that.
I suggest you use a scrap working directory when messing around with the data files. This avoids overwriting the source data for instance. Check out this gist which is a quick script that dumps the gunzipped datafiles to a working directory. Run it when you are within the directory with the gzipped data files.
In the following queries, `-nr *` means that the query targets all files in current directory, and it will not resolve the names of ports (as listed in /etc/services).
1) How many bytes sent per node (sorted asc by bytes)
racluster -m saddr bytes -nr * -w - - ip | rasort -m bytes -s saddr bytes | less
2) How many bytes received per node (sorted asc by bytes).
racluster -m daddr bytes -nr * -w - - ip | rasort -m bytes -s daddr bytes | less
3) How many bytes sent per node, per port (sorted asc by saddr).
racluster -m saddr bytes dport -nr * -w - - ip | rasort -m saddr -s saddr dport bytes | less
4) How many bytes received per node, per port (sorted asc by daddr).
racluster -m daddr bytes dport -nr * -w - - ip | rasort -m daddr -s daddr dport bytes | less
5) How many bytes sent per node, per transport protocol.
racluster -m saddr bytes proto -nr * -w - - ip | rasort -m bytes -s saddr proto bytes | less
6) How many bytes sent per node, per transport protocol.
Digging down to target a host:
1) How many bytes sent, per node, to node.
racluster -m saddr daddr bytes dport -nr * -w - - ip and src host 192.168.100.23 | rasort -m bytes -s saddr daddr bytes dport | less
`ip and src host` is the filtering statement. You are safe to play around with the Berkeley Packet Filter syntax, but there are some modifications noted on page 8 of the ra(1) man page in the “Filter Expression” section.
2) how many bytes sent, per node, to port of node:
racluster -m saddr daddr dport bytes -nr * -w - - ip and src host 192.168.100.23 | rasort -m bytes -s saddr daddr dport bytes | less
Let’s get graphing:
1) chart all bytes with a given saddr, with one second resolution
ragraph saddr bytes -M 1s -w test.png -n -n -nr * - src host 192.168.100.23
2) chart of bytes with a given saddr and a given daddr, with one second resolution
ragraph saddr bytes -M 1s -n -n -nr * - src host 192.168.100.23 and dst host 220.127.116.11
3) chart of bytes with a given saddr and a given daddr and dport, with one second resolution
ragraph dport bytes -M 1s -n -n -nr * - src host 192.168.100.23 and dst host 18.104.22.168
4) chart of bytes with a given src net, excluding a given dst net and icmp, with one second resolution, and a title, saving it to a file icmp100cidr24.png.
ragraph bytes -M 1s -w icmp100cidr24NOT101cidr24.png -title "ICMP from 192.168.100.0/24 (exc. 192.168.101.0/24)" -n -n -nr * - src net 192.168.100.0/24 and not dst net 192.168.101.0/24 and icmp
`icmp` is from /etc/protocols.