Home > Uncategorized > nxlog: Rate limiting, controlling packets per second/bandwidth throttling

nxlog: Rate limiting, controlling packets per second/bandwidth throttling

Not done. What is wrong with my math? Is it correct? Should I just assume that there will be more, specifically since the nxlog docs recommend limiting to 2000 messages/sec?
I’ve opted to use throttling at the site’s internet firewall to control traffic traversing the ssh port for which the syslog messages are sent.

 

Dealing with remote sites with low interconnection bandwidth (aka “slow links”), even if they are over VPNs on the unpredictable Internet, I try to do my best to allot the little bandwidth I have. I learned a baseline throughput by using iperf. Even if this testing methodology is not good since the inter-site throughput fluctuates due to the variety of traffic conditions on the Internet, it still gives me a half-way decent framework to work with.

The need of syslog packet routing/relaying, may be something you consider if you have a lot of servers sending logs. Configure imudp or imtcp as in, have processors or other modules affect the syslog messages, then send them out with omudp or omtcp. You may run into the challenge of designating the proper message source, since the source may be the IP address of the relay.

Regardless, being able to limit the packet that packets are sent is very valuable and can easily be done.

Determining maximum throughput between syslog client and server:

On the syslog server:

yum -y install iperf #from the fedora repo
service iptables stop
iperf -s
service iptables start

On the sending client (Windows):
1) Download iperf from the University of Central Florida.
2) connect to the server and send some data:

iperf -c SYSLOGSERVER

In my instance, the throughput average of five tests using TCP (as above) is: 861kbps

It might actually be wise to have “something” run (maybe icinga/nagios) throughout a week or so to see patterns.

Determining maximum UDP and TCP packet size between syslog client and server:

On the syslog server run the following, increasing or decreasing the buffer size noted as 1408, util you see “Frag needed and DF set”:

ping -M do -s 1408 SENDINGSYSLOGSERVER
#note that you do not see the "Frag needed" error, but do not see echo replies.  Note the 1400+8 bytes.

Ethernet encapsulation is 42 bytes. An IPv4 header is 20-24 bytes (usually 20). An ICMP header is 8 bytes. In the above case, the maximum packet size is 1408+42-20-8 = 1422 bytes (including the Ethernet encapsulation). This is the point-to-point maxiumum payload, but doesn’t express the maximum TCP or UDP packet size.

A TCP header can be 20-60 bytes. If Ethernet encapsulation is 42 bytes, and IPv4 header is 20-24 bytes (usually 20), and the TCP header can be 20-60 bytes. The only variant is the TCP header; maximum TCP payload = 1408+42-20-20 to 1408+42-20-60; 1370-1410 bytes = 10960-11280 bits.

A UDP header is 2 bytes. If Ethernet encapsulation is 42 bytes, and IPv4 header is 20-24 bytes (usually 20), and the UDP header is 2 bytes. The only variant is the UDP header; maximum UDP payload = 1408+42-20-2 = 1428 bytes = 11424 bits.

Syslog messages maximum sizes are dynamically configured, and most of the concern should lay with the max supported size for the server. In rsyslog’s instance, there is a directive ($MaxMessageSize) which defaults to 2k. There is a detailed write up in the rsyslog docs about $MaxMessageSize. In my instance, I have configured the max message size to be 64 kilobytes; so we can’t quite expect to not have these fragmented across many TCP packets (since the max size of our TCP payload would be 1410 bytes).

How many syslog messages can we send per second?

– Maximum transmittal of TCP packets is 11280bits/packet
– Maximum transmittal of data is 861kbps = 7053312bps
== Maximum TCP packets per second = 625 (about)

– Maximum transmittal of TCP packets is 11280bits/packet
– Maximum syslog message size is 2 to 64KB = 16384 to 524288bits
== Maximum syslog messages (over TCP) per packet = 0 to 0.688
== Maximum syslog messages (over TCP) per second = 13 to 430+ (depending on size)

Configuring rate limiting in nxlog
Now that we have all these great numbers, let’s put them to use.

It appears that the recommended (aka only) way to do rate limiting is by calling the sleep() function somewhere in the processing of a message. Specifically, the nxlog docs recommend that it is performed upon receipt.

Out of my 861kbps interconnection throughput, I’d like to utilize no more than (50kbps = 51200 bits/sec) for syslog transmittal. How many messages per second would that be? 3. This is WAY too low. So what is wrong with my math?

I highly recommend implementing pm_buffer if you will use rate limiting or else messages will be dropped when the input module blocks on sleep().

Reference:

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: