In the previous installment we examined default Apache logging. Now let's pump up the default Apache combined log format in order to supercharge forensic capability! We'll utilize the "LogFormat" directive in order to define the "enhanced" log format within the /etc/apache2/apache2.conf configuration file:
LogFormat "%{[%a %D @ %I:%M:%S.}t%{msec_frac}t %{%p %Z]}t [%h (%{X-Forwarded-For}i) > %v:%p] [%H %m \"%U\" \"%q\" %I > %>s %D %O %k %L] \"%{Referer}i\" \"%{User-Agent}i\" %{USER}C %{JSESSIONID}C" enhancedA sample Apache enhanced log format entry looks a little something like this:
[Wed 07/30/14 @ 10:45:59.420 PM CDT] [10.1.1.101 (-) > 192.168.1.1:80] [HTTP/1.1 GET "/example.html""?foo=bar" 666 > 200 295 999 0 -] "http://192.168.1.1/from.html""Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0" jdoe 09162009
Jeepers Creepers! That looks like even more gibberish than last time, so let's break down the enhanced log format step by step:
The %{[%a %D @ %I:%M:%S.}t%{msec_frac}t %{%p %Z]}t component logs the time in a more intuitive format using strftime conversion specifications and embedded Apache tokens. The day of the week is now included, the time is now specified in 12-hour format with millisecond precision, and the time zone is now specified by abbreviation. Note that all production servers should be synchronized with a Network Time Protocol (NTP) server in order to ensure consistent time settings across the enterprise. In the example log entry this value is "[Wed 07/30/14 @ 10:45:59.420 PM CDT]".
The [%h (%{X-Forwarded-For}i) > %v:%p] component logs the source and destination of the request. The %h directive specifies the client IP address, and the %{X-Forwarded-For}i directive specifies the underlying client IP address for proxied requests. Note that the value of the "X-Forwarded-For" header could be spoofed by the client. In the example log entry these values are "10.1.1.1" and "-", respectively. The %v directive specifies the server name or IP address, and the %p directive specifies the server port. The server port is useful to determine whether a request was transmitted over an HTTP or SSL network connection. In the example log entry these values are "192.1681.1.1" and "80", respectively.
The [%H %m \"%U\" \"%q\" %I > %>s %D %O %k %L] component logs details regarding the request and response. The %r directive from the combined log format has been split into individual elements for easier sorting and searching functionality. The %H directive logs the request protocol. Uncommon protocols such as "HTTP/1.0" could indicate automated scanning tools or targeted attacks. . In the example log entry this value is "HTTP/1.1". The %m directive logs the request method. Uncommon protocols such as "PUT" could indicate automated scanning tools or targeted attacks. In the example log entry this value is "GET ". The %U directive logs the requested URL path. Uncommon URL paths such as "/admin.html" could indicate automated scanning tools or targeted attacks. In the example log entry this value is "/example.html". The %q directive logs the query string, which can contain a wealth of useful forensic information. Common attacks such as Cross-Site Scripting (XSS) and SQL Injection can be identified by indicative attack strings within the URL such as ' or 1=1 or <script>. In the example log entry this value is "?foo=bar". The %I directive logs the total number of bytes received from the client, including headers and the request itself. An unusually high number of bytes received could indicate certain types of attacks such as buffer overflows. In the example log entry this value is "666". The %>s directive logs the status code of the request. Uncommon status codes such as "405" (i.e., "Method Not Allowed") could indicate automated scanning tools or targeted attacks. In the example log entry this value is "200". The %D directive logs the number of microseconds taken to serve the request. Unusually long times could indicate certain types of attacks such as successful time-based SQL injection. In the example log entry this value is "295". The %0 directive logs the total number of bytes sent by the server, including headers and the response itself. An unusually high number of bytes could indicate certain types of attack such as successful SQL injection. In the example log entry this value is "999". The %k directive logs the number of keepalive requests processed on the connection. Automated scanners do not typically utilize keepalive requests. In the example log entry this value is "0". Finally the %L directive logs the request log identifier from the error log. This identifier can be used to correlate the request with errors log entries within the /var/log/apache2/error.log logfile. In the example log entry this value is "-".
The %{Referer}i directive logs the "Referer" header sent by the client. Note that the value of the "Referer" header could be spoofed by the client. In the example log entry this value is "http://10.1.1.101/".
The %{User-Agent}i directive logs the "User-Agent" header sent by the client. Note that the value of the "User-Agent" header could be spoofed by the client. In the example log entry this value is "Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0".
The %{USER}C directive logs the "USER" cookie. The application server would set this cookie to contain the username of the authenticated user after form-based authentication. Note that the value of the "USER" cookie could be spoofed by the client. However, the application server could verify the "USER" cookie value sent with each request. If the cookie value does not match the authenticated user correlated with the submitted session identifier value then a deterministic status code such as "403 Forbidden" could be returned to the client. Consequently a "200 OK" status code would confirm a valid "USER" cookie value. The identity of the authenticated user can be extremely useful during a forensic investigation. In the example log entry this value is "jdoe".
The %{SESSIONTRACK}C directive logs the "SESSIONTRACK" cookie. The application server would set this cookie to contain an eight-digit identifier in order to track requests throughout the duration of a session. Note that the value of the "SESSIONTRACK" cookie could be spoofed by the client. However, the application server could verify the "SESSIONTRACK" cookie value sent with each request. If the cookie value does not match the tracking identifier correlated with the submitted session identifier value then a deterministic status code such as "403 Forbidden" could be returned to the client. Consequently a "200 OK" status code would confirm a valid "SESSIONTRACK" cookie value. Tracking requests throughout the duration of a session can be extremely useful during a forensic investigation. However, it is important to note that the "SESSIONTRACK" tracking identifier is not the session identifier itself (e.g., "JSESSIONID") and cannot be utilized to resume a session. If the session identifier itself was logged then attackers could leverage a compromised logfile in order to hijack authenticated sessions. Therefore, session identifiers and other sensitive security tokens should never be logged. In the example log entry this value is "09162009".
As you can see, the enhanced log format captures a wealth of information that would be extremely useful during a forensic investigation. Obviously additional information means larger logfiles, but disk space is cheap and the benefits of extended logging certainly outweigh the nominal increase in resource consumption. All that's left now is to enable the enhanced log format within the /etc/apache2/sites-available/000-default.conf and /etc/apache2/sites-available/default-ssl.conf configuration files:
CustomLog ${APACHE_LOG_DIR}/access.log enhancedAnd finally we'll restart the Apache daemon in order to load the configuration changes:
root@debian $ service apache2 restart
Done and Done! Apache will now begin logging each request in the enhanced log format, providing additional information that would be extremely useful during a forensic investigation. There's no doubt about it, our pumped up log format is a lean mean forensic machine!