Home > Uncategorized > nxlog: configure pattern matching for Windows Events

nxlog: configure pattern matching for Windows Events

November 8th, 2012: It isn’t possible to do this with pm_pattern, but only with if and Exec. Go to the bottom section (“Nopenopenope”) to see the correct way to block on multiple condition.

 

In this example, I am using the event source/input module im_mseventlog. You could use another input module and the process would be relatively the same.

In order to filter event logs, it is recommended that you use the process module pm_pattern; but I did not have success matching multiple conditionals. Note that pm_pattern will match your patterns in the order of most occurrences to least found in the event. This means that you could consider that each pattern may get matched, and randomly at that, so consider the list to be a logical OR. ‘Exec’ treats rules as that of a firewall, first hit equals action, and no more rules are processed.

Creating an example event with:

eventcreate /l application /so EVENT_SOURCE /t error /id 1 /d "test event"

nxlog Output:

$raw_event = THE_SERVER ERROR 1 test event
xm_xml.to_xml() = 
<Event>
	<EventTime>2012-11-08 09:38:11</EventTime>
	<EventTimeWritten>2012-11-08 09:38:11</EventTimeWritten>
	<Hostname>THE_SERVER</Hostname>
	<EventType>ERROR</EventType>
	<SeverityValue>4</SeverityValue>
	<Severity>ERROR</Severity>
	<SourceName>EVENT_SOURCE</SourceName>
	<FileName>Application</FileName>
	<EventID>1</EventID>
	<CategoryNumber>0</CategoryNumber>
	<RecordNumber>19131</RecordNumber>
	<Message>test event  </Message>
	<EventReceivedTime>2012-11-08 09:38:11</EventReceivedTime>
	<SourceModuleName>in</SourceModuleName>
	<SourceModuleType>im_mseventlog</SourceModuleType>
</Event>

The follow fails… (patterndb.xml):

<?xml version='1.0' encoding='UTF-8'?>

<!--
	This is an example patterndb.xml to go along with the nxlog write up on https://mbrownnyc.wordpress.com
	It should be stored in..\nxlog\conf\patterndb.xml
-->

<!--
	Find log messages that match our desired pattern:
	
	Which are events matching:
	Event log: DFS Replication
	Source: DFSR
	Event ID: 4202, 4208, 4302, 4304, 5004
	
	eventcreate /l application /so EVENT_SOURCE /t error /id 1 /d "test event"
	$raw_event
		THE_SERVER ERROR 1 test event
	xm_xml.to_xml()
		<Event>
			<EventTime>2012-11-08 09:38:11</EventTime>
			<EventTimeWritten>2012-11-08 09:38:11</EventTimeWritten>
			<Hostname>THE_SERVER</Hostname>
			<EventType>ERROR</EventType>
			<SeverityValue>4</SeverityValue>
			<Severity>ERROR</Severity>
			<SourceName>EVENT_SOURCE</SourceName>
			<FileName>Application</FileName>
			<EventID>1</EventID>
			<CategoryNumber>0</CategoryNumber>
			<RecordNumber>19131</RecordNumber>
			<Message>test event  </Message>
			<EventReceivedTime>2012-11-08 09:38:11</EventReceivedTime>
			<SourceModuleName>in</SourceModuleName>
			<SourceModuleType>im_mseventlog</SourceModuleType>
		</Event>
	
-->


<patterndb>
	<group>
		<name>eventlog</name>
		<id>1</id>
		
		<pattern>
			<id>1</id>
			<name>test event</name>
			
			<matchfield>
				<name>FileName</name>
				<type>exact</type>
				<value>Application</value>
			</matchfield>
			
			<matchfield>
				<name>SourceName</name>
				<type>exact</type>
				<value>EVENT_SOURCE</value>
			</matchfield>
			
			<matchfield>
				<name>EventID</name>
				<type>exact</type>
				<value>1</value>
			</matchfield>
			
			<!--
				the following exec is evaluated when this pattern matches
				you can have multiple execs, and clearly multiple <matchfields>, where <type> can have values 'exact' or 'regexp' ['regexp' can't be used against a boolean or integer]
			-->
			
			<exec>
				drop();
			</exec>
		</pattern>
		
	</group>
</patterndb>

Blocking multiple event IDs from the same Event Source
So, given the desire to filter the following events:

Event log name (aka the event log filename): DFS Replication
Event Source: DFSR
Event IDs: 4202, 4208, 4302, 4304, 5004.

The pattern XML file would look like this:

<patterndb>
	<group>
		<name>eventlog</name>
		<id>1</id>
		<pattern>
			<id>1</id>
			<name>dfs events</name>
			<matchfield>
				<name>FileName</name>
				<type>string</type>
				<value>DFS Replication</value>
			</matchfield>
			<matchfield>
				<name>SourceName</name>
				<type>string</type>
				<value>DFSR</value>
			</matchfield>
			<matchfield>
				<name>EventID</name>
				<type>rege</type>
				<value>(4202|4208|4302|4304|5004)</value>
			</matchfield>
			<!--
				the following exec is evaluated when this pattern matches
				you can have multiple execs, and clearly multiple <matchfields>
			-->
			<exec>
				drop();
			</exec>
		</pattern>
	</group>

nxlog can pull the matching data in raw, and overloads it into the capturefield objects, similar to string formatting in many programming languages (see sprintf in ruby):

<matchfield>
      <name>Message</name>
      <type>REGEXP</type>
      <value>^pam_unix\(sshd:session\): session opened for user (\S+) by \(uid=(\d+)\)</value>
      <capturedfield>
         <name>AccountName</name>
         <type>STRING</type>
      </capturedfield>
      <capturedfield>
         <name>AccountID</name>
         <type>INTEGER</type>
      </capturedfield>
   </matchfield>

The ..\nxlog\conf\nxlog.conf file would look like this (merged with the above In and Out module definitions) [correct ‘..\nxlog\conf\patterndb.xml’]:

<Extension xml>
	Module	xm_xml
</Extension>

<Input in>
	Module	im_mseventlog
	Exec	to_xml();
</Input>

<Processor pattern>
	Module	pm_pattern
	PatternFile	..\nxlog\conf\patterndb.xml
</Processor>

<Output out>
	Module	om_tcp
	Host	127.0.0.1
	Port	22222
</Output>

<Route 1>
	Path	in => pattern => out
</Route>

Notice that you CAN NOT use regexp when dealing with an element that is a INTEGER. You can test it with the following replacing the above:

<matchfield>
	<name>EventID</name>
	<type>regexp</type>
	<value>.*(\d+).*</value>
	
	<capturedfield>
		<name>eventidnumber</name>
		<type>string</type>
	</capturedfield>
</matchfield>

You will receive error:

ERROR invalid use of REGEXP match with INTEGER type EventID

Nopenopenope
It doesn’t look possible to match an INTEGER field against multiple values with pm_pattern.

To solve this problem, you can use binary operators and the Exec command in nxlog.conf within :

Exec	if $EventID == 4202 or $EventID == 4208 or $EventID == 4302 or $EventID == 4304 or $EventID == 5004 drop();

If you were to write a block of code to act upon the conditional outcome, you must escape the EOL:

<Input in>
	Module	im_mseventlog
	Exec if ($EventID == 4202 or $EventID == 4208 or $EventID == 4302 or $EventID == 4304 or $EventID == 5004) drop();\
	else\
	{\
		parse_syslog_ietf();\
		$Message = $FileName + ": " + $Message;\
		$SyslogFacility = syslog_facility_string(22);\
		$SyslogFacilityValue = syslog_facility_value("local6");\
		if ( $EventType == "INFO" ) $SyslogSeverityValue = 6;\
		if ( $EventType == "WARNING" ) $SyslogSeverityValue = 4;\
		if ( $EventType == "ERROR" ) $SyslogSeverityValue = 3;\
	}
</Input>

This example does three things:

1) Conditionally performs a drop() function (dropping the event from the Input module), if the EventID is a certain event id (see log_info(to_xml())) and ends the function. Otherwise it performs the following…

2) Changes the facility of the syslog message to local6 (as it is configured in the EventDB write up, mostly to isolate which syslog events get written to the DB using ommysql).

3) Changes the severity of the syslog message to the syslog accepted as described in the RFC 5424 to match the source severity.

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: