flow-inspector and ntopng are very useful for this sort of thing, as it generally takes care of visualizing all the traffic stats; but you can utilize ragraph all the same as follows. As I’ve yet to implement ntopng, specifically the historic feature, I’m relying on flow-inspector.
Graph of bytes per second (like load, bps) downloads initiated internally:
ra -r * -w - -t 10:50:00-11:00:00 - src net 192.168.100.0/24 | ragraph saddr dbytes -M 1s -r -
this should indicate the most downloading-ist client:
Then you can drill further into this client
ra -r * -w - -t 10:50:00-11:00:00 - src host 192.168.100.46 | ragraph dbytes daddr -title 'dbytes per second requested by 192.168.100.46' -M 1s -r -
then you can actually see what’s going on with…
this will give you the resulting load by destination address
ra -r * -w - -t 10:50:00-11:00:00 - src host 192.168.100.46 | racluster -M daddr -r - -w - | rasort -M byte load -s saddr daddr load:15 -r - | less
and also… this will give you load of all transactions between a src host and a dst host, per second:
ra -r * -w - -t 10:50:00-11:00:00 - src host 192.168.100.46 and dst host 184.108.40.206 | rabins -M 1s -s stime ltime saddr daddr load:15 -r - | less
which can also be expressed…
ra -r * -w - -t 10:50:00-11:00:00 - src host 192.168.100.46 and dst host 220.127.116.11 | ragraph dbytes -title 'dbytes initiated by 192.168.100.46 downloaded from 18.104.22.168' -M 1s -r -
flow-inspector: open source web UI and backend processor and storage to visualize network flows with d3.js!
Update Devember 7th, 2012: I’ve started an email thread with Lothar Braun and Mario Volke; Lothar got back to me right away with some questions about usage and to explain the inner workings of flow-inspector some more.
I’ve sent him some info on argus, and if I understand his implication correctly, the process to push data from anything (including argus) to flow-inspector is very simple:
1) define fields in flow_aggr_values and flow_aggr_sums (you wish to pivot against/process data for).
2) push JSON (field:data) to a redis queue named ‘entry:queue’.
3) processor.py connects to the redis queue, processes some data and places it in to the configured DB (waiting for an entry simply called “END” to stop listening).
Wow! How did I miss this?!
flow-inspector is EXACTLY what I wanted to build for use with argus. And I kid you not, EXACTLY what I was looking for.
Thank you guys so much for releasing this open source and releasing it at all! Some times (too many times) I come across research papers and the research source is never released. This is exactly what is needed by the community.
Mario Volke is the original developer.
Now to work argus into it.
I’ve opted to use throttling at the site’s internet firewall to control traffic traversing the ssh port for which the syslog messages are sent.
Dealing with remote sites with low interconnection bandwidth (aka “slow links”), even if they are over VPNs on the unpredictable Internet, I try to do my best to allot the little bandwidth I have. I learned a baseline throughput by using iperf. Even if this testing methodology is not good since the inter-site throughput fluctuates due to the variety of traffic conditions on the Internet, it still gives me a half-way decent framework to work with.
This is the first post in a topic I’ll call “Me thinks,” which is similar to that of “Misguided Opinion,” being mostly editorial. In Me thinks I will be posting ideas that I have, even if they make no sense at all in the long run. So it’s like all gut, no thought, me thinks.
I’ve been working with argus for about two weeks. It’s quite great stuff, and I have been working on a way to optimally store, retrieve and display argus data so that it can be most useful for analysts. I had an idea that may or may not work, if argus is generating flows, and ntop is generating flows, and ntop can output what it thinks is happening (with nDPI), it would be quite useful to combine both, or simply just add what ntop has diagnosed the session to contain. This would be great!
In effort to analyze I/O specifics, I had coded a program that performs various calls to the underlying fileio system.
My final program, although I’m proud of her, is clunky, and written in .NET.
I just came across an interesting program called FileTest while researching device filter drivers. It is written in C so it is less clunky and likely more efficient than whatever I wrote. So be it!
FileTest is in complement to FileSpy, “a more advanced version of SysInternals FileMon].”