Archive

Posts Tagged ‘analysis’

How I use argus to learn about outlying bandwidth consumers

December 19, 2013 Leave a comment

flow-inspector and ntopng are very useful for this sort of thing, as it generally takes care of visualizing all the traffic stats; but you can utilize ragraph all the same as follows. As I’ve yet to implement ntopng, specifically the historic feature, I’m relying on flow-inspector.

Graph of bytes per second (like load, bps) downloads initiated internally:

ra -r * -w - -t 10:50:00-11:00:00 - src net 192.168.100.0/24 | ragraph saddr dbytes -M 1s -r -

this should indicate the most downloading-ist client:

Then you can drill further into this client

ra -r * -w - -t 10:50:00-11:00:00 - src host 192.168.100.46 | ragraph dbytes daddr -title 'dbytes per second requested by 192.168.100.46' -M 1s -r -

then you can actually see what’s going on with…
this will give you the resulting load by destination address

ra -r * -w - -t 10:50:00-11:00:00 - src host 192.168.100.46 | racluster -M daddr -r - -w - | rasort -M byte load -s saddr daddr load:15 -r - | less

and also… this will give you load of all transactions between a src host and a dst host, per second:

ra -r * -w - -t 10:50:00-11:00:00 - src host 192.168.100.46 and dst host 128.122.215.45 | rabins -M 1s -s stime ltime saddr daddr load:15 -r - | less

which can also be expressed…

ra -r * -w - -t 10:50:00-11:00:00 - src host 192.168.100.46 and dst host 128.122.215.45 | ragraph dbytes -title 'dbytes initiated by 192.168.100.46 downloaded from 128.122.215.45' -M 1s -r -
Advertisements

Anomaly detection methodology with argus data

June 21, 2013 Leave a comment

flow-inspector: open source web UI and backend processor and storage to visualize network flows with d3.js!

December 5, 2012 Leave a comment

Update Devember 7th, 2012: I’ve started an email thread with Lothar Braun and Mario Volke; Lothar got back to me right away with some questions about usage and to explain the inner workings of flow-inspector some more.

I’ve sent him some info on argus, and if I understand his implication correctly, the process to push data from anything (including argus) to flow-inspector is very simple:
1) define fields in flow_aggr_values and flow_aggr_sums (you wish to pivot against/process data for).
2) push JSON (field:data) to a redis queue named ‘entry:queue’.
3) processor.py connects to the redis queue, processes some data and places it in to the configured DB (waiting for an entry simply called “END” to stop listening).


 

Wow! How did I miss this?!

flow-inspector is EXACTLY what I wanted to build for use with argus. And I kid you not, EXACTLY what I was looking for.

Research paper published November 8, 2012.

Original commit (seven months ago).

forked as referred to by paper.

Simply incredible!

Thank you guys so much for releasing this open source and releasing it at all! Some times (too many times) I come across research papers and the research source is never released. This is exactly what is needed by the community.

Mario Volke is the original developer.

Now to work argus into it.

nxlog: Rate limiting, controlling packets per second/bandwidth throttling

November 27, 2012 Leave a comment
Not done. What is wrong with my math? Is it correct? Should I just assume that there will be more, specifically since the nxlog docs recommend limiting to 2000 messages/sec?
I’ve opted to use throttling at the site’s internet firewall to control traffic traversing the ssh port for which the syslog messages are sent.

 

Dealing with remote sites with low interconnection bandwidth (aka “slow links”), even if they are over VPNs on the unpredictable Internet, I try to do my best to allot the little bandwidth I have. I learned a baseline throughput by using iperf. Even if this testing methodology is not good since the inter-site throughput fluctuates due to the variety of traffic conditions on the Internet, it still gives me a half-way decent framework to work with.

Read more…

Me thinks: is argus + ntop with nDPI + c5 sigma the way to go?

May 31, 2012 2 comments

This is the first post in a topic I’ll call “Me thinks,” which is similar to that of “Misguided Opinion,” being mostly editorial. In Me thinks I will be posting ideas that I have, even if they make no sense at all in the long run. So it’s like all gut, no thought, me thinks.

I’ve been working with argus for about two weeks. It’s quite great stuff, and I have been working on a way to optimally store, retrieve and display argus data so that it can be most useful for analysts. I had an idea that may or may not work, if argus is generating flows, and ntop is generating flows, and ntop can output what it thinks is happening (with nDPI), it would be quite useful to combine both, or simply just add what ntop has diagnosed the session to contain. This would be great!

Read more…

Hook it up, son.

October 11, 2011 2 comments

Create and monitor I/O calls

August 30, 2011 Leave a comment

In effort to analyze I/O specifics, I had coded a program that performs various calls to the underlying fileio system.

My final program, although I’m proud of her, is clunky, and written in .NET.

I just came across an interesting program called FileTest while researching device filter drivers. It is written in C so it is less clunky and likely more efficient than whatever I wrote. So be it!

FileTest is in complement to FileSpy, “a more advanced version of SysInternals FileMon].”

%d bloggers like this: