Modify raw_event to send custom logs using NXLog-CodePudding

I would like to modify the IIS logs for further transfer to the destination. Now I am parsing the IIS log with the xm_csv module, as in the template. UndefValue is disabled to not get empty.

How can I interact with parsed data from w3c_parser?

For example, I want to combine into a variable $request = '"' $cs-method ' ' $cs-uri-stem ' ' $cs-version '"'; such a value, but I get an error. When I try to write a field from w3c_parser to $raw_event, I also get an error. Any other data is added without error.

For example $raw_event = $c-ip -- error

$raw_event = $EventTime ' ' $http_host -- no error

Example error, logs and config file below

2022-03-23 16:49:56 ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.conf:59; couldn't parse statement at line 71, character 32 in C:\Program Files\nxlog\conf\nxlog.conf; syntax error, unexpected , expecting (

2022-03-23 16:49:56 ERROR module 'iis_w3c' has configuration errors, not adding to route 'uds_to_file' at C:\Program Files\nxlog\conf\nxlog.conf:84

2022-03-23 16:49:56 ERROR route uds_to_file is not functional without input modules, ignored at C:\Program Files\nxlog\conf\nxlog.conf:84

2022-03-23 16:49:56 WARNING no routes defined!

2022-03-23 16:49:56 WARNING not starting unused module iis_w3c

2022-03-23 16:49:56 WARNING not starting unused module file

2022-03-23 16:49:56 INFO nxlog-ce-3.0.2272 started

Current log format

date time s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-bytes cs-bytes time-taken

2022-03-23 08:00:01 HOST.DOMAIN 99.XX.XX.4 GET /AnalyticsService - 443

XX.XX.XX.XXX HTTP/1.1 Zabbix - - site.host.domain 200 3918 144 4

Required log format

$http_host $remote_addr $remote_user [$time_local] UNIX-TIME-$msec "$request" $status "$sent_http_content_type" $body_bytes_sent "$http_referer" "$http_user_agent" "$http_cookie" $request_time "$upstream_addr" NGINX-CACHE-$upstream_cache_status "$request_id" "$request_body"

host.domain 99.99.99.249 - [11/Mar/2022:20:09:56 0300] UNIX-TIME-1647018596.031 "GET /api/company.php?id=853747 HTTP/1.1" 200 "text/xml; charset=UTF-8" 1455 "-" "-" "20b6b325ea192383cb1244412247c5ea=3002538ef353c9daab4f742176a840; etpsid=f488b343a23d1a4a2332e089a0" 0.059 0.059 "10.10.10.111:80" NGINX-CACHE-- "d0b5ac12cf82671067aa5e6c5c" "-"

Panic Soft
#NoFreeOnExit TRUE

define ROOT C:\Program Files\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf\nxlog.d
define LOGDIR %ROOT%\data

define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data

<Extension _syslog>
Module xm_syslog
</Extension>

<Extension fileop>
Module xm_fileop
</Extension>

<Extension _charconv>
Module xm_charconv
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>

<Extension _exec>
Module xm_exec
</Extension>

#Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c_parser>
Module xm_csv
Fields $date, $time, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-bytes, $cs-bytes, $time-taken
FieldTypes string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer
Delimiter ' '
EscapeChar '"'
QuoteChar '"'
EscapeControl FALSE
# UndefValue -
</Extension>

<Extension w3c_out>
Module xm_csv
Fields $http_host, $c-ip, $cs-username, $EventTime1, $sc-status, $Unix
FieldTypes string, string, string, string, string, string
Delimiter ' '
# UndefValue -
QuoteMethod None
</Extension>

<Input iis_w3c>
Module im_file
File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log'
SavePos TRUE
<Exec>
if $raw_event =~ /^#/ drop();
else
{
w3c_parser->parse_csv();
$EventTime = parsedate($date   " "   $time);
$EventTime = $EventTime   (3 * 3600);
$EventTime1 = strftime($EventTime, '[%d/%b/%Y:%H:%M:%S]');
# $EventTime1 = '$EventTime1'   '  0003]';
$Unix = integer($EventTime);
$Unix = 'UNIX-TIME-'   $Unix;
$http_host = "site.host.domain";
# $request = '"'   $cs-method   ' '   $cs-uri-stem   ' '   $cs-version   '"';
# $request = $cs-method;
w3c_out->to_csv();
}
</Exec>
</Input>

<Output file>
Module om_file
File 'C:\inetpub\logs\LogFiles\Parser\w3c.txt'
</Output>

<Route uds_to_file>
Path iis_w3c => file
</Route>

CodePudding user response：

let's start with NXLog language in the conf files. Dashes in explicit format are not allowed - you can check: https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#lang_fields

Hence, one needs to apply curly braces to reach the goal ({}). If I understand your issue correctly, this may help with most of your issues.