Custom

Seer Box side configuration

The Custom format allows you to create a format that fits traffic logs sent by any source that is not one of the Seer Box default sources. To set it up, simply select the value Custom in the Log type field in step 3 of traffic source creation.

info

Creating a custom format is an advanced operation that requires in-depth knowledge of the identified log format. We recommend you to proceed only after consulting the source documentation.

Log format field

The Log format field allows you to build the output log format of the selected traffic source using appropriate keywords, so that it can be correctly interpreted by Seer Box.

The available keywords trace all specific fields of an HTTP transaction (request and response) and are identified by the `%' symbol used as the first character.

Default keywords

Keyword	Description
%transaction_id	HTTP transaction ID
%timestamp	Timestamp, to be defined in the Date/Time format field
%date	Date-only timestamp, to be defined in the Date/Time format field
%time	Hour-only timestamp, to be defined in the Date/Time format field
%timezone	Timezone, to be defined in the Date/Time format field
%client_address	Request source IP address
%client_port	Request source port
%server_address	Request destination IP address
%server_port	Request destination port
%q_Host_	Request URL domain
%path	Request URL uri path
%url	Request URL
%method	Request method (eg. POST, GET, ...)
%protocol	Request HTTP version
%query_string	Raw request query string
%q_Referer_	Request Referer header
%status_code	Server response code (eg. 404, 200, ...)
%request_body	Request body
%response_body	Response body
%request_bytes	Request bytes sent
%response_bytes	Response bytes received
%q_User-Agent_	Request User-Agent header
%q_Cookie_	Request Cookie header
%q_Content-Type_	Request Content-Type header
%r_Content-Type_	Request Content-Type response
%q_X-Forwarded-For_	Request X-Forwarded-For header

Custom keywords

If an HTTP request or response header is not among the default keywords, it can be defined using the following syntax:

• For a request header: %q_FIELD-NAME_ (eg: %q_Custom-Req-Field_)
• For a response header: %r_FIELD-NAME_ (eg: %q_Custom-Resp-Field_)

This keyword is processed using the name defined in the format.

Special keywords

If the log format contains unknown fields or fields that should not be processed and analyzed, it is possible to ignore their presence with the special %ignore syntax.

Examples

CLF format

Log row

66.249.75.237 - - [11/Sep/2022:06:42:21 +0200] "GET /robots.txt HTTP/1.1" 200 1969

Log format

%client_address %q_Remote-Logname_ %q_Remote-User_ [%timestamp] "%method %path %protocol" %status_code %response_bytes

NCSA format

Log row

66.249.75.237 - - [11/Sep/2022:06:42:21 +0200] "GET /robots.txt HTTP/1.1" 200 1969 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Log format

%client_address %q_Remote-Logname_ %q_Remote-User_ [%timestamp] "%method %path %protocol" %status_code %response_bytes "%q_Referer_" "%q_User-Agent_"

Non-standard format 1

In this example it is not possible to detect all log fields, so we will use the %ignore keyword to ignore them.

Log row

66.249.75.237 | 11/Sep/2022:04:51:40 +0000 | GET /api/1.0/items?pageId=789 HTTP/1.1 | 200 | 144 | - | - | http://example.com/pages/viewpage.action?pageId=789 | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36 | JSESSIONID=AE2920007DC0005E0008BE1934F5DBF; Hm_lvt_0ef9c12592473ef5637c9827ed88=1480921285,1480949630,1480985334,1481035256; Hm_lpvt_0e754a018e3ec6f9e74276d65114ef74=1481086302 | 66.249.75.237 | example.com | 10.1.2.3:80 | 0.006 | 0.006

Log format

%client_address | %timestamp | %method %path %protocol | %status_code | %response_bytes | %ignore | %ignore | %q_Referer_ | %q_User-Agent_ | %q_Cookie_ | %q_X-Forwarded-For_ | %q_Host_ | %ignore | %ignore | %ignore

Non-standard format 2

In this example it is not possible to detect all log fields, so we will use the %ignore keyword to ignore them.

Log row

13.66.139.0 - - [11/Sep/2022:13:57:26 +0100] "GET /index.php?option=com&view=category&Itemid=53 HTTP/1.1" 200 32653 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"

Log format

%client_address %ignore %ignore [%timestamp] "%method %path %protocol" %status_code %response_bytes "%ignore" "%q_User-Agent_" "%ignore"

Date/Time format field

The Date/Time format field characterizes the specific format of the keywords %timestamp, %date, %time and %timezone.

The first can be used in cases where the timestamp is entirely defined within a single field. The other three allow the date, time, and timezone to be separated into different fields.

There are 3 available options for Date/Time format field:

1. unix - UNIX Timestamp
2. timestamp - ISO/IEC 9899:2018 strftime for timestamp format representation. It also needs Timestamp and Time zone fields to be defined.
3. datetime - ISO/IEC 9899:2018 strftime for date and time format representation. It also needs Date and/or Time, and Time zone fields to be defined.

Timestamp/Date/Time fields

Timestamp, Date and Time fields define the specific format for the time stamp representation, using the following specifiers:

Specifier	Description	Example
%a	Abbreviated weekday name	Sun
%A	Full weekday name	Sunday
%b	Abbreviated month name	Mar
%B	Full month name	March
%c	Date and time representation	Sun Aug 19 02:56:02 2012
%d	Day of the month (01-31)	19
%H	Hour in 24h format (00-23)	14
%I	Hour in 12h format (01-12)	05
%j	Day of the year (001-366)	231
%m	Month as a decimal number (01-12)	08
%M	Minute (00-59)	55
%p	AM or PM designation	PM
%S	Second (00-61)	02
%U	Week number with the first Sunday as the first day of week one (00-53)	33
%w	Weekday as a decimal number with Sunday as 0 (0-6)	4
%W	Week number with the first Monday as the first day of week one (00-53)	34
%x	Date representation	08/19/12
%X	Time representation	02:50:06
%y	Year, last two digits (00-99)	01
%Y	Year	2012
%Z	Timezone name or abbreviation	CDT
%z	Timezone positive/negative integer	+0100

Examples

Timestamp

11/Sep/2022:06:42:21 +0200

Format

%d/%b/%Y:%H:%M:%S %z

Time zone field

The timezone of the HTTP transactions, formatted like "+0000" or "-0000".