Home > Uncategorized > Understanding a Vermont config file

Understanding a Vermont config file

Documentation summary:
I will be summarizing the documentation available within the project’s github wiki to bring a simple understand of a fairly complex configuration process.

For our example we are concerned with setting up Vermont to be an IPFIX generator, receiver, and transmitter to a local redis queue “entry:queue.”

Note that as of April 30th, 2013, there is no redis queue output module for vermont. I have reached out to Lothar to find out more about this possibility. Until then, since our goal is to configure vermont to be a probe to populate flow-inspector’s backend DB, I will be using Vermont to receive PCAP and output to a database. However, I will cover outputting IPFIX packets using the ipfixExporter module. See the bottom of this page for more output options, including IpfixDbWriter which I will be using for my flow-inspector instance.

Working with the modular design:
Vermont config files operate like our friends’ nsclient++, nxlog, and rsyslog config files, in a modular design.

The data flow is decided upon by the “id” of each module (declared in the module root xml node), and the “next” child node of that module root node.

Multiple “next” tags can be included, but there are special stipulations for this. Translating docs from developer speak to admin speak:
1) Each receiving module will see the same packets, not a copy.
2) So… if a module modifies the data, then it is modified. This can be confusing for the other modules.
If multiple receivers are used, then a queue can be used. This allows the modules to do work in a synchronous manner.

Configuring an IPFIX generator:
We’ll be working off a file ipfix-export.xml and I will describe simply non-obvious portions.

A lot of these are defined in code: ./src/modules/ipfix/*.h

  • The root node is the name of the config.
  • “sensorManager”: use the “checkInterval” setting to control how many frequently (in seconds) sensors are polled.
  • “observer” (Input type: none, Output type: Packet): takes input from pcap interface.
  • “packetQueue” (Input type: Packet, Output type: Packet): holds packets in queue, up to child node “maxSize”, until the “next” module is ready. If full, pauses previous module. The large maxSize, the more RAM is utilized.
  • “packetAggregator” (Input type: Packet, Output type: IpfixRecord): takes incoming packets and makes IPFIX records out of them, using the provided settings:
    • “rule”: defines scope
      • “templateId”: IPFIX Template ID.
      • “flowKey”: Parent node of ieName… that includes in aggregation? **I don’t understand the wiki’s definition: “Flow key information element – flows are aggregated according to those keys.”
      • “nonFlowKey”: Parent node of ieName… that excludes from aggregation? **I don’t understand the wiki’s definition: “Non-flow key information element – those IEs are aggregated.”
      • “ieName”: IPFIX information elements
      • “match”: matches the ieName
        • protocolIdentifier: “TCP”, “UDP”, “ICMP”, or IANA number (for IPv4 RFC791, for IPv6 RFC2460).
        • (sourceIPv4Address, destinationIPv4Address, ipNextHopIPv4Address, bgpNextHopIPv4Address, sourceIPv4Prefix, destinationIPv4Prefix, mplsTopLabelIPv4Address, exporterIPv4Address, collectorIPv4Address, postNATSourceIPv4Address, postNATDestinationIPv4Address, staIPv4Address), (sourceIPv6Address, destinationIPv6Address, ipNextHopIPv6Address, bgpNextHopIPv6Address, exporterIPv6Address, mplsTopLabelIPv6Address, destinationIPv6Prefix, sourceIPv6Prefix, collectorIPv6Address, postNATSourceIPv6Address, postNATDestinationIPv6Address) (not clear if supports ipv6): for IPv4 use CIDR notation, for IPv6 see RFC5952.
        • udpSourcePort, udpDestinationPort, tcpSourcePort, tcpDestinationPort, sourceTransportPort, destinationTransportPort, collectorTransportPort, exporterTransportPort, postNAPTSourceTransportPort, postNAPTDestinationTransportPort: an unsigned16 that represents a destination port (for UDP see RFC768, for TCP see RFC793, for SCTP see RFC2960, for NAPT see RFC3022. Note that a port range is can be defined as [start port]:[end port].
        • tcpControlBits: “URG”, “ACK”, “PSH”, “RST”, “SYN”, “FIN”. Combine in a comma-separated list.
    • “expiration”: defines scope
      • “inactiveTimeout”: timeout for inactive flows.
        • “unit”: “sec”, “msec”, “usec”.
      • “activeTimeout”: timeout for long-lasting active flows.
    • “pollInterval”: interval when flow should be passed to next module.
  • “ipfixQueue” (Input type: IpfixRecord; Output type: IpfixRecord): holds IPFIX records in queue until the next module is ready to process them.
  • “ipfixExporter” (Input type: IpfixRecord, Output type: none): module that sends IPFIX outbound to the network.
    • “observationDomainId”: IPFIX spec for observation domain as defined in RFC5153
    • “transportProtocol”: aka “exportTransportProtocol” in the IPFIX spec. Accepted values in the code: “17” and “UDP”, “132” and “SCTP”, “DTLS_OVER_UDP”, “DTLS_OVER_SCTP”, “TCP”

Configuring an IPFIX to MySQL DB receiver:
You should now understand the concept of the modular design, and by referring to the available module list and their input and output specs, should understand how to implement modules effectively.

Here is what we generally want:

packets -> NIC of Vermont probe -> take certain fields -> DB

Knowing the list of available modules, we already know how to configure Vermont to be an IPFIX receiver to MySQL DB receiver:

observer -> packetqueue -> packetaggregator -> ipfixqueue -> ipfixdbwriter

Outputting to stdout:
With the modules and some cli-fu:

observer -> packetqueue -> packetaggregator -> ipfixprinter -> pipe -> [data processing script] -> redis API or `redis-cli -x`

The challenge here is that the output of ipfixprinter to stdout is not engineered to be input to `redis-cli -x`, so a script or program to process the data is necessary.

Tuning for performance:
As evidenced in the above example, queuing becomes sort of an arbitrarily important part of your probe config design. The queues are memory mapped, so this is a consideration, low RAM = not so large queues; while saturated CPU would mean you need a larger queue. Some monitoring and tweaking would need to be done in this area.

Four facts should be gained: incoming packet flow, process RAM utilization, process CPU utilization, outgoing packet flow. This will give you very basic information on if you can tweak your queues better. Remember, unpredictably large packet flow might prove difficult to engineer for.

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: