Part 5: Implement EventDB on CentOS6 (and nxlog on Windows)

Update December 31st, 2013:
So… it’s been a while and my perspective has changed.
I no longer believe EventDB is a good way to move, and that you’re better off using Logstash + ElasticSearch + Kibana for this same purpose. Also, a big “whoops,” I don’t think SSH is the way to go for on-the-wire encryption for this instance, as it isn’t flexible enough. I had fun making use of security by obscurity defense-in-depth strategy I cover here, but it just can’t cut it when it comes to ease of management and stability.

So, I’ll be retiring this solution after I finish writing a write up on logstash+elasticsearch+kibana.

If my analysis bores you, jump to “Implementing SSH” to get technical.


Originally, I had used the CheckEventLog library with NSClient++ to actively poll for new events in a variety of Event Logs on Windows servers. It became quite clear that this idea was fundamentally flawed from a security perspective, and optimally any event should be sent immediately when it is logged in the local logging mechanism.

The scope of this article will cover quite a large amount; it will not be focusing on security implications of log entries, but will provide a secure architecture for logging and sending logs to a collector.

Deciding on clients:
Which hosts will be sending their logs to the collector? Are they Windows boxes, linux boxes, firewalls, switches, routers? Some of these devices have their own clients to “ship” logs to a log collector (whether they be syslog or otherwise), and some do not.

Later versions of Windows include a service that can be directly configured to send logs to other Windows machines that are running event log collectors. However, since we will be implementing the collector on a CentOS box, we can not use whatever proprietary upper layer protocol used by the event collecting services; we will rely on syslog clients.

There are several “eventlog to syslog” clients for Windows, but we must first make another decision and we will circle back to select our client due to our need for capabilities.

Security should be a strong consideration:
When evaluating the security implications of implementing a system, it’s important to consider what you are trying to guarantee. I usually live by the simple and elegant concepts of the C.I.A. triad: confidentiality, integrity and availability. I consider authenticity to fall under integrity, but I guess it could conceptually be a separate point; for the purpose of this article, I’ll treat it as the same.

For us, knowing that communication between our clients and our server(s) could consist of the transmittal of log data in plain text, it is wise to focus on confidentiality, and maybe what provides us confidentiality will also provide us integrity and availability/reliability.

Confidentiality: SSH, SSL or IPsec tunnels?
When evaluating which method to use to secure communication, it is important to consider flaws in implementation and possible attacks. Clearly the goal is to use the technology with the least inherent flaws and make sure that you have configured it to have the smallest attack surface of both.

Two important considerations I have when securing communication are: How easy are man-in-the-middle attacks (violations of data integrity and data capture) and how easy is it to attack captured data (violations of confidentiality of captured data)?

You can leverage public keys with all three technologies we’re considering. When considering public key cryptography you must consider key distribution, which is a robust process to implement correctly. I will not cover key distribution here, but hopefully will talk about it a bit more in the near future.

A great place to perform research to make your decision is the Security stackexchange, which covers the variety of questions that you should be asking.

Using SSH to tunnel traffic:
I arrived on using SSH over SSL from my Windows servers for three reasons:

  • It is easy to implement with a little bit of overhead (no more than stunnel for instance)
  • It provides encryption.
  • As trust is apparent, when implemented correctly, SSH guarantees identity of each end point in a manageable way: guaranteeing integrity of data (a rogue client can’t fake data), and guaranteeing the server is the server (avoids MITM).

SSL ain’t that bad either:
It is arguable that SSL can be implemented in a much similar way, but generally (and with this implementation) a trust hierarchy is not needed to trust the endpoints identity with SSH, but may be with SSL. It’s possible to create your own completely isolated trust hierarchy for you to use with SSL, including self-sign certificate pinning. Since that removes the need to trust any third parties, then it’s on you (as it is with SSH) to guarantee the security of your key distribution and management.

I believe that it is much easier to poorly configure clients and poorly design key distribution architecture for SSL encryption solutions than SSH, and this is what makes it less secure than SSH.

Considering “eventlog to syslog” clients:
I believe the following features to be beneficial in an “eventlog to syslog” client:
1) Free.
2) Reliable delivery of syslog messages somewhere in the stack (not necessarily RFC3195 compliance, but it would be nice).
3) Encryption of messages somewhere in the stack.

Here are my problems with a variety of clients:

Clearly, I’m less concerned than you might think with native RFC3195 compliance and native transport encryption. Why? Since I’ve decided to use SSH to encapsulated the syslog messages, both TCP for reliable delivery of packets (so not syslog messages, but this is pretty good) and encryption of messages are provided for.

I think “Big Syslog” has a point that “[RFC 793 sections 3.4 and 3.7 are] the ultimate evidence that you can not build a reliable syslog infrastructure just on top of TCP (without app-layer acks).” But I suppose you can make the call. Keep in mind reliable UDP and PGM are options here.

Implementing SSH:
As Redman expresses, it’s time for some action.

Install SSH on CentOS6:
I assume you have followed the previous parts of the Reliability Monitoring guide.
Oh, you don’t need to install SSH on the server, it’s already there receiving connections on port 22.

We’ll configure our SSH clients in a few steps.

Configure ssh on the syslog server:
There are a few articles on securing ssh, but I will cover a few steps here:

useradd syslogger -s /bin/false
passwd syslogger
vim /etc/sysconfig/iptables
# add the following two lines before the line that ACCEPTs tcp 22 connections
# this will DROP packets from any IP who's connected more than four times in 60 seconds to TDP port 22
## -A INPUT -p tcp -m tcp --dport 22 -m recent --set --name ssh --rsource
## -A INPUT -p tcp -m tcp --dport 22 -m recent ! --rcheck --seconds 60 --hitcount 4 --name ssh --rsource -j ACCEPT
vim /etc/ssh/sshd_config
#add syslogger to have ssh access, note to add whatever other users must have ssh access!!!
## AllowUsers syslogger
#enable key based authentication
sed s/#RSAAuthentication/RSAAuthentication/ -i /etc/ssh/sshd_config
sed s/#PubkeyAuthentication/PubkeyAuthentication/ -i /etc/ssh/sshd_config
sed s@#AuthorizedKeysFile\ .ssh/authorized_keys@AuthorizedKeysFile\ /home/%u/.ssh/authorized_keys@ -i /etc/ssh/sshd_config
mkdir /home/syslogger/.ssh
service sshd restart
service iptables restart

Configure rsyslog server:
As comes with the distro, rsyslog is a very solid project with a great lead developer. My goal is to use a connection sourced on the log sending server to connect to the syslog server via a tcp socket. Technically, we’ll use the ssh tunnel we’ll configure in the next step to access the syslog server, then connect to the syslog daemon on syslog server via a locally source connection from the syslog server itself, since that’s where we are connected with the SSH client.

It’s generally pretty cool and sort of confusing, so here’s a diagram (of course I left out ommysql!):


I used gliffy.com to make the diagram, and greenshot to grab it.

Configure rsyslog as follow:

sed s/#$ModLoad\ imtcp/$ModLoad\ imtcp/ -i /etc/rsyslog.conf
sed s/#$InputTCPServerRun\ 514/$InputTCPServerRun\ 514/ -i /etc/rsyslog.conf
echo '#$TCPServerAddress 127.0.0.1 #this is not a real option' >> /etc/rsyslog.conf
echo '$InputTCPMaxSessions 500' >> /etc/rsyslog.conf
service rsyslog restart

Note that currently the rsyslog input module for TCP (imtcp) does not support setting the IP address to bind to. Only the rsyslog input module for UDP (imudp) supports this. Optimally, you’d stop all network sourced packets from accessing the rsyslog server if it was bound to 127.0.0.1; but we can use TCP to guarantee delivery, and use iptables/netfilter to block incoming requests to TCP/514. This strategy removes a layer of defense-in-depth. If you aren’t comfortable with this, simply use imudp instead and ‘UDPServerAddress 127.0.0.1’.

Configure your SSH clients:

I will be focusing on plink.exe, since this article is focused primarily on tunneling Windows Event Logs to the syslog server. Using ssh on *nix to create a tunnel is quite similar.

Install and configure the SSH tunnel on Windows:
The following explains the following, including a strange, derived method to use EFS to protect the private key on the client:
– create the local Windows user, sshsyslogsvc, who will run plink.exe as a service.
– create a public-private key pair using puttygen.exe with this user.
– restrict access to the private key only the sshsyslogsvc user by encrypting this file using EFS. (I may attempt to password protect the private key later as well, but this could be troublesome, and I’ll update the article with info).

I. Create public/private key pair for authentication:

On a DC, you obviously can’t log on as a local user, nor as a standard domain user. I’ve detailed the procedure at the end of this page, as it is significantly different than the normal procedure.

All parts that are related to the solution to use EFS to encrypt the private key file so that we don’t need to protect it with a passphrase, are marked with a ~. You can avoid these if you wish not to use EFS or it is banned, and just have the private key protected by file system ACLs.

1) Create a user on your log sending Windows server:

net user sshsyslogsvc sshsyslogsvc_PASSWORD /add

[sshsyslogsvc should not part of the local admin group]

2) Download puttygen.exe to c:\windows\system32\

3) Start a command prompt as this user to perform the remaining steps in this section (I. Create public/private key pair for authentication):

runas /user:sshsyslogsvc cmd.exe

4) Create an RSA key pair with 3072 bit encryption (because that’s how I roll):
– open puttygen.exe
– click Generate (and then move mouse around in the middle of the PuTTY Key Generator window)
– change comments
– do NOT add passphrase (maybe I’ll write a method to handle this later)
– make sure SSH-2 RSA is bulleted (it’s the default)
– change your bits from 1024 to 3072
– click Save private key and save to %userprofile%\ssh_priv.ppk (yes you’re sure)
– click Save public key and save to %userprofile%\ssh_priv.pub (yes you’re sure)
– close puttygen

5) ~Generate an EFS key:~
In the command prompt running as .\sshsyslogsvc run:

cipher /k /q

6) ~Protect the private key file from unauthorized access:~
In the cmd running as .\sshsyslogsvc run:

xcacls "%userprofile%\ssh_priv.ppk" /P sshsyslogsvc:OF /y

7) Encrypt the private key file using EFS:
In the cmd running as .\sshsyslogsvc run:

cipher /a /e "%userprofile%\ssh_priv.ppk"

8) Add the public key of the client’s (plink’s) key pair to the list of public keys that can be used to authenticate as the syslogger user for the openssh server (in /home/syslogger/.ssh/authorized_keys):

There’s a variety of ways to do this, but I’ll keep it as simple as possible form the command prompt.
– Log in via putty to your server open /home/syslogger/.ssh/authorized_keys.tmp for editing.

vim /home/syslogger/.ssh/authorized_keys.tmp

– On the client, run the following in the command prompt window running as sshsyslogsvc, then paste it into the /home/syslogger/.ssh/authorized_keys.tmp file on the SSH server

more "%userprofile%\ssh_priv.pub"

– On the ssh server, run the following to convert the key from puttygen generated to openssh accepted:

ssh-keygen -i -f /home/syslogger/.ssh/authorized_keys.tmp >> /home/syslogger/.ssh/authorized_keys
rm -rf /home/syslogger/.ssh/authorized_keys.tmp

– Set access flags on the authorized_keys file:

chmod 644 /home/syslogger/.ssh/authorized_keys
chown syslogger /home/syslogger/.ssh/authorized_keys

– Restart sshd:

service sshd restart

** Note that you can either repeat this process for each server, adding the new public key to the /home/syslogger/.ssh/authorized_keys, or simply use the same private key (this private key) for each client. Note that even the EFS cert can be transferred between stores.

II. Create the service to open the tunnel.

In order to utilize the service control system of Windows, the executable that is the service must be sort of aware of the service control system so that it can properly communicate process status.
If you attempt to run plink.exe directly with the service controller, the service controller will fail with “[SC] StartService FAILED 1053: The service did not respond to the start or control request in a timely fashion,” even if a plink.exe has been spawned and none of its threads are blocked.

Therefore, you must use a command host; a parent command that will execute and is aware of dynamic child process’s status, and “translates” the status of this child process to the service controller. Unfortunately, cmd.exe and cmd.exe’s start do not do this. However, the Microsoft tool srvany.exe is written exactly for this purpose. Too bad srvany.exe is no longer supported. Up steps The Non-Sucking Service Manager.

Configure the service SSH tunnel service on the client:

1) Download non-sucking service manager (NSSM) (latest build as of November 15th, 2012), extracting to c:\program files\ and plink.exe to c:\windows\system32\

2) Create the service:

Replace sshsyslogsvc_PASSWORD with the proper password for the Windows local user sshsyslogsvc. Replace SYSLOGSERVER with the proper syslog server hostname. Replace ..\sshsyslogsvc\ with the proper user profile path.

sc create ssh_to_syslogserver binpath= "c:\program files\nssm\nssm.exe" start= auto error= critical depend= tcpip/afd displayname= "plink SSH tunnel" obj= .\sshsyslogsvc password= sshsyslogsvc_PASSWORD 
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ssh_to_syslogserver\Parameters
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ssh_to_syslogserver\Parameters /v Application /t reg_sz /d "c:\windows\system32\plink.exe"
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ssh_to_syslogserver\Parameters /v AppParameters /t reg_sz /d "-ssh -2 -L 22222:SYSLOGSERVER:514 -l syslogger -i \"..\sshsyslogsvc\ssh_priv.ppk\" SYSLOGSERVER:22 -N"
#this creates a plink process bound to 127.0.0.1:22222, which then tunnels to SYSLOGSERVER:514 through SYSLOGSERVER:22 (see previous diagram)
sc failure ssh_to_syslogserver actions= restart/30000/restart/30000/restart/30000/ reset= 120
#configure the service to restart after 30 seconds, three times, then try again three times in 2 minutes, forever (network connectivity problems may cause temporary service disruption)

3) Add the ssh server’s public key to the PuTTY key store:

There is an alternate way to do this using pageant, but the easiest way is to do the following (and the only way via the command line):

runas /user:sshsyslogsvc cmd.exe
#then start:
c:\windows\system32\plink.exe -ssh -2 -L 22222:SYSLOGSERVER:514 -l syslogger -i "%userprofile%\ssh_priv.ppk" SYSLOGSERVER:22 -N
#accept the key to store it in the PuTTY key store

4) Start and test the service:
On server:

tail -f /var/log/messages

On client:

sc start ssh_to_syslogserver
telnet 127.0.0.1 2222

After connecting with telnet/ncat, type something and it should be logged to /var/log/messages on the server.

Secure the user session for sshsyslogsvc:
If you would like, you add the “Deny logon locally” to sshsyslogsvc either through secpol.msc (Local Policies\User Rights Assignment) or using ntrights.exe:

ntrights +r SeDenyInteractiveLogonRight -u sshsyslogsvc

Security caveats:
This configuration allows any locally sourced packet to be forwarded to the syslog service via the ssh tunnel bound to localhost:22222. I have not tested using Windows Vista+ firewall to block outbound traffic to port 22222, only allowing nxlog.exe.

Download, install, and configure the nxlog client:

1) Download and install the latest nxlog community edition from sourceforge (latest MSI for Windows).

2) Open ..\nxlog\conf\nxlog.conf and perform the following changes (note I’m filtering out some events and other conditionals as you wish.:

1) Verify 'define ROOT' is the correct ..\nxlog\ directory
2) Configure the Input (I am using a nested conditional to exclude events with certain Event IDs from a certain event log ("FileName")... take note of the changes to the facility and the severity, the change to facility is mandatory for this writeup):
	<Input in>
		Module	im_mseventlog
		Exec if (\
			not (\
				$FileName == "Application"\
				or\
				$FileName == "System"\
				or\
				$FileName == "Security"\
				or\
				$FileName == "DFS Replication"\
			)\
			or not (\
				$EventType == "WARNING"\
				or\
				$EventType == "ERROR"\
				or\
				$EventType == "AUDIT_FAILURE"\
			)\
			or (\
				$FileName == "DFS Replication"\
				and\
				(\
					$EventID == 4202\
					or\
					$EventID == 4208\
					or\
					$EventID == 4302\
					or\
					$EventID == 4304\
					or\
					$EventID == 5004\
				)\
			)\
		) drop();\
		else {\
			parse_syslog_ietf();\
			$Message = $FileName + " / " + $SourceName + ": " + $Message;\
			$SyslogFacility = syslog_facility_string(22);\
			$SyslogFacilityValue = syslog_facility_value("local6");\
			if ( $EventType == "INFO" ) $SyslogSeverityValue = 6;\
			if ( $EventType == "WARNING" ) $SyslogSeverityValue = 4;\
			if ( $EventType == "ERROR" ) $SyslogSeverityValue = 3;\
			if ( $EventType == "AUDIT_FAILURE" ) $SyslogSeverityValue = 3;\
		}
	</Input>

3) Configure the output module om_tcp as follows:
	<Output out>
		Module	om_tcp
		Host	127.0.0.1
		Port	22222
		Exec	to_syslog_ietf();
	</Output>
4) be happy as the module name notation indicates that the developers seem to like rsyslog.
5) be even happier that this program is extremely extensible (routing between modules? processor modules! awesome! oh yes! we'll be coming back to you later nxlog...).

3) In order to not drop messages, consider using pm_buffer, but it may be problematic in 2.0.927.

Download, install, and configure EventDB:
Finally, let’s implement EventDB.
Note that I will be using the rsyslog module ommysql instead of syslog-ng2mysql.

1) Download and decompress the latest EventDB source (2.0.4rc is the latest release as of November 5th, 2012 which was recommended over stable 2.0.2).

cd
mkdir eventdb
cd eventdb
wget https://www.netways.org/attachments/download/643/eventdb-2.0.4rc.tar.gz
tar zxvf eventdb*.tar.gz

2) Create the DB user for EventDB:
Note the DB password eventdb_DBUSER_PASSWORD, which will be used to access the database.

cd 
echo "CREATE DATABASE eventdb;" >> ~/eventdb_user.sql
echo "GRANT SELECT, INSERT, UPDATE, CREATE, DELETE ON eventdb.* to 'eventdb'@'localhost' IDENTIFIED BY 'eventdb_DBUSER_PASSWORD';" >> ~/eventdb_user.sql
echo "FLUSH PRIVILEGES;" >> ~/eventdb_user.sql
mysql -p < ~/eventdb_user.sql
rm -f ~/eventdb_user.sql

3) Import the EventDB schema into the database:

mysql -p eventdb < ~/eventdb/db/mysql/createTables.sql

4) Install the icinga-cronk to be used with icinga-web:
Note the DB password eventdb_DBUSER_PASSWORD, which will be used to access the database.

cd ~/eventdb/icinga-cronk/
sed s/dbpass\ =\ \"eventdb\"/dbpass\ =\ \"eventdb_DBUSER_PASSWORD\"/ -i db.ini
./install.sh
#Do you want to install this cronk? [hit enter]
#Location of icinga-web [hit enter]
Which db backend do you want to use? [hit enter]

5) Configure the cronk to connect to the DB:
Note the DB password eventdb_DBUSER_PASSWORD, which will be used by rsyslog’s ommysql to access the database.

sed s@mysql://eventdb:eventdb@mysql://eventdb:eventdb_DBUSER_PASSWORD@g -i /usr/local/icinga-web/app/modules/EventDB/config/databases.xml

5) Install the check_eventdb icinga plugin:

cd
install -m 755 ./eventdb/plugin/check_eventdb.py /usr/local/nagios/libexec/

6) Modify check_eventdb so that it deals with UTC times instead of localtime:

sed s/import\ time/import\ time,\ datetime/ -i /usr/local/nagios/libexec/check_eventdb.py
sed s/curTime\ =\ time.time\(\)/curTime\ =\ time.mktime\(datetime.datetime.utcnow\(\).timetuple\(\)\)/ -i /usr/local/nagios/libexec/check_eventdb.py

7) Clear icinga-web cache and restart httpd:

/usr/local/icinga-web/bin/clearcache.sh
service httpd restart

If you are logged onto icinga-web, log out and back in in order to see the EventDB cronk icon populate the column of your hosts.

Configure rsyslog to output to the DB:
We will use the ommysql output module for rsyslog to output to the eventdb.events database.

Note the DB password eventdb_DBUSER_PASSWORD, which will be used to access the database.

yum -y install rsyslog-mysql
mkdir -p /etc/rsyslog/workingdir
echo "\$ModLoad ommysql" > /etc/rsyslog.d/ommysql.conf
echo "\$WorkDirectory /etc/rsyslog/workingdir" >> /etc/rsyslog.d/ommysql.conf
echo "\$ActionQueueType LinkedList" >> /etc/rsyslog.d/ommysql.conf
echo "\$ActionQueueFileName dbq" >> /etc/rsyslog.d/ommysql.conf
echo "\$ActionResumeRetryCount -1" >> /etc/rsyslog.d/ommysql.conf
echo "\$ActionOmmysqlServerPort 3306" >> /etc/rsyslog.d/ommysql.conf
echo "\$template eventdbtemplate,\"INSERT INTO eventdb.event (host_name, host_address, type, facility, priority, program, message, created) VALUES ('%HOSTNAME%', '%fromhost-ip%', 0, %syslogfacility%, %syslogpriority%, 'rsyslog', '%msg%', '%timereported:::date-mysql%')\",SQL" >> /etc/rsyslog.d/ommysql.conf
echo "local6.* :ommysql:localhost,eventdb,eventdb,eventdb_DBUSER_PASSWORD;eventdbtemplate" >> /etc/rsyslog.d/ommysql.conf
service rsyslog restart

Polling EventDB for non-ACKed log entries:
1) Create the command definition in your icinga config file by going to the Lconf web interface.

2) I prefer to keep system type commands under the structural object I had created previously called default_conf, so right click on default_conf> Create new node as child> Create new command> Next and fill out the following information:

cn: check-eventdb
lconfCommandLine: $USER1$/check_eventdb.py --dbuser=eventdb --dbpass=eventdb_DBUSER_PASSWORD -H $HOSTNAME$ $ARG1$

3) Create a service within the ldap tree in Lconf where it is appropriate. For instance, refering to the previous Lconf write up, you will notice that I created templates for each one of my hosts. This might be too broad, as I may not be sending events from each host to EventDB, so instead, you may want to duplicate the service definition for each applicable host.

You must make a decision on which events you wish the icinga active service check to monitor. The following are some select options from check_eventdb.py so that we can easily use it to monitor syslog source messages:

--msg=MESSAGE: Message as logged by the agent (SQL Format wildcards)
--facility=FACILITY: The facilities to respect, comma separated
--priority=PRIORITY: Priority as logged by the agent
--type=LOGTYPE: The logtype (syslog,snmptrap,mail)
--program=PROGRAM: Program as logged by the agent
--warning-priorities=PRIO_WARNING: A comma seperated set of priorities which will be used for determine the warning state
--critical-priorities=PRIO_CRITICAL: A comma seperated set of priorities which will be used for determine the warning state
--label=LABEL: Label for plugin output
--url=URL: URL for EventDB link in plugin output
--maxage=MAXAGE: Maximum age of EventDB entry (eg. 1m, 2h, 3d)
--resetregexp=RESETREGEX: Regular Expression for message entry in eventdb to change each state back to OK
--perfdata=PERFDATA: Performance data from the last check (e.g. \$SERVICEPERFDATA\$)
--ip=IPADDRESS: Filter by ip address
--warning=WARNING: Number of matches to result in warning state
--critical=CRITICAL: Number of matches to result in critical state
--cventry: returns the custom variable entry for this call (needed in order to use icinga-web cronk integration)

You have a choice to reset the status of service back to OK:

  • you could use use the –resetregexp option of the script
  • you could limit the scope of the returned log entry hits age with –maxage
  • you could use the freshness function of icinga.

In my instance, I wish to have icinga change the status of the service when I see any ERROR or WARNING from the event log source “DFS Replication,” and have it no seek entries older than five minutes (since this is when my service check takes place). I configure an lconfService object with the follow property values.

Matching conditionals like this isn’t provided for in the code of check_eventdb; so I workaround using ‘%’ AND to take care of the prescribed message LIKE pre-populated string in the code, then move on to condition matching; closing the string with something that’s always true: 1.

cn: dfsr_check
lconfservicecheckcommand: check-eventdb!--maxage=5m --msg="%' AND ((message like '%ERROR%' OR message LIKE '%WARNING%') and (message LIKE '%DFS\ Replication:%')) AND '1" -w 1 -c 1
#note the use of escape characters within the string --msg.  This is because the entire argument is processed into a command line used by icinga (executed with python's popen), then the --msg string is used within the python script by an SQL API.

The SQL query looks like:

SELECT COUNT(id) as count, MAX(id) as last ,  COUNT(id) AS count_warning ,  COUNT(id) AS count_critical  from event where id > 0  AND upper(host_name) = '$HOSTNAME$'   AND message LIKE '%%' AND ((message like '%%ERROR%%' OR message LIKE '%%WARNING%%') and (message LIKE '%%DFS Replication:%%')) AND '1'   AND type = 0  AND created >= '2012-11-20 10:52:54'   AND ack = 0

Conclusion: Why EventDB?

My decision to move the “responsibility” of tracking Windows event logs events from the icinga DB to the EventDB DB is because EventDB is written to handle event messages better than Icinga. You may only want the alert to alert when certain Windows events occur, and not on all error or warning level Windows events. Additionally, although CheckEventLog.dll’s real-time event monitoring, from NSClient++, also has immediate sending of events, nxlog is also a very robust, extensible client. Also, nxlog produces syslog messages, which I will be using Sagan to filter through.
 

Configuring a service user and the service on a domain controller:

1) you must create a service domain user on the DC:

net user sshsyslogsvc sshsyslogsvc_USER_PASSWORD /add

2) you must add this service user to a dedicated security group, set it as their primary security group, and then remove them from Domain Users.

3) you must create the service.

sc create ssh_to_syslogserver binpath= "c:\program files\nssm\nssm.exe" start= auto error= critical depend= tcpip/afd displayname= "plink SSH tunnel" obj= DOMAIN\sshsyslogsvc password= sshsyslogsvc_USER_PASSWORD
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ssh_to_syslogserver\Parameters
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ssh_to_syslogserver\Parameters /v Application /t reg_sz /d "c:\windows\system32\plink.exe"
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ssh_to_syslogserver\Parameters /v AppParameters /t reg_sz /d "-ssh -2 -L 22222:SYSLOGSERVER:514 -l syslogger -i \"c:\documents and settings\sshsyslogsvc\ssh_priv.ppk\" SYSLOGSERVER:22 -N"
sc failure ssh_to_syslogserver actions= restart/30000/restart/30000/restart/30000/ reset= 120

4) you must allow the user to log on a service

ntrights +r SeServiceLogonRight -u DOMAIN\sshsyslogsvc

5) you must create the service (as below step 6), and attempt to start it:

sc start ssh_to_syslogserver

It will succeed to start, but the tunnel will not be established, since SYSLOGSERVER’s host key is not accepted into the PuTTY cache.
The SYSTEM has built the user profile, including the HKU registry key…

6) you must run the plink command to and accept the hostkey for SYSLOGSERVER:

echo y | c:\windows\system32\plink.exe -ssh -2 -L 22222:SYSLOGSERVER:514 -l syslogger SYSLOGSERVER:22 -N

then retrieve the hashed host key from the registry data:

reg query HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\SshHostKeys /v rsa2@22:SYSLOGSERVER

7) you must then determine the SID of the DOMAIN\sshsyslogsvc user and merge the hashed key into it’s HKU\ registry

sc stop ssh_to_syslogserver
reg add HKU\[SID of DOMAIN\sshsyslogsvc]\Software\SimonTatham\PuTTY\SshHostKeys
reg add HKU\[SID of DOMAIN\sshsyslogsvc]\Software\SimonTatham\PuTTY\SshHostKeys /v rsa2@22:SYSLOGSERVER /t REG_SZ /d "0x23,0xkeykeykeykeykey"

8) you must then place the private key in c:\documents and settings\sshsyslogsvc\ssh_priv.ppk

9) start the service and test

sc start ssh_to_syslogserver
telnet 127.0.0.1 22222

With reference:

  1. Alex Slatalla
    April 15, 2014 at 12:23 am

    I have been using your nxlog configuration script to bring my event logs over to Loganalyzer and it has been wonderful.

    I was just wondering if it would be possible to modify the part of the script which sets severity values to also set the windows event ID to be a syslog process ID?

    • April 15, 2014 at 6:20 am

      Happy you’re finding the post useful. I haven’t looked at this is a while, but you can refer to nxlog documentation and review $ProcessID and $EventID.

  1. October 31, 2013 at 10:16 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: