Brief introduction of log file formats

You can choose the format that your Web server uses to record user activity. The following formats are available:

• W3C Extended Log File Format
• Microsoft IIS Log Format
• NCSA Common Log File Format
• ODBC Logging

The W3C Extended log file format, Microsoft IIS log file format, and NCSA log file format are all ASCII text formats.

The W3C Extended and NCSA formats record logging data in four-digit year format.
The Microsoft IIS format uses a two-digit year format and is provided for backward compatibility with earlier IIS versions.

The W3C Extended log file format provides the most features of all of the log formats because you can customise the fields which are logged thereby reducing the size of log files.

W3C Extended format is a customizable ASCII format with a variety of different fields.

You can include fields important to you, while limiting log size by omitting unwanted fields.
Fields are separated by spaces.
Time is recorded as UTC (Greenwich Mean Time).

The example below shows lines from a file using the following fields: Time, Client IP Address, Method, URI Stem, HTTP Status, and HTTP Version.

#Software: Microsoft Internet Information Services 5.0  #Version: 1.0  #Date: 1998-05-02 17:42:15 
#Fields: time c-ip cs-method cs-uri-stem sc-status cs-version 
17:42:15 172.16.255.255 GET /default.htm 200 HTTP/1.0

The preceding entry indicates that:

On May 2, 1998 at 5:42 P.M. UTC


	A user with HTTP version 1.0 and the IP address of 172.16.255.255 issued an HTTP GET command for the file Default.htm.

The request was returned without error.  

The #Date: field indicates when the first log entry was made, which is when the log was created.
The #Version: field indicates that the W3C logging format 1.0 was used.

Any of the fields can be selected, but some fields may not have information available for some requests.

For fields that are selected, but for which there is no information, a dash (—) appears in the field as a placeholder.

If you have your site configured for Anonymous user authentication the user shows as a dash (—).

Microsoft IIS format is a fixed (non-customizable) ASCII format. It records more items of information than the NCSA Common format.

The Microsoft IIS format includes basic items such as:

The user's IP address
User name
Request date and time
HTTP status code
The number of bytes received.

In addition, it includes detailed items such as:

The elapsed time of the request
The number of bytes sent
The action (for example, a download carried out by a GET command) and the target file.
The items are separated by commas
The time is recorded as local time.

When you open a Microsoft IIS format file in a text editor, the entries are similar to the following examples:

192.168.114.201,—,03/20/98,7:55:20,W3SVC2,SALES1,192.168.114.201,4502,163,3223,200,0,GET,DeptLogo.gif  
172.16.255.255,anonymous,03/20/98,23:58:11,MSFTPSVC,SALES1,192.168.114.201,60,275,0,0,0,PASS,intro.htm

The preceding sample entries are interpreted as follows:

Users IP Address
User Name
Date
Time
Service and Instance
Computer Name
IP Address of Server
Time taken in Ms
Bytes Received
Bytes Sent
Service Status Code
Windows NT/2000 Status Code
Request type
Target of the operation

So as text we have:

An anonymous user with the IP address of 192.168.114.201
issued an HTTP GET command for the image file DeptLogo.gif at 7:55 A.M. on March 20, 1998
From a server named SALES1 at IP address 172.21.13.45.


	The 163-byte HTTP request had an elapsed processing time of 4502 milliseconds (4.5 seconds) to complete, and returned, without error, 3223 bytes of data to the anonymous user.

In the log file:

All fields are terminated with a comma (,).
A hyphen acts as a placeholder if there is no valid value for a certain field.

NCSA Common format is a fixed (non-customizable) ASCII format, available for Web sites but not for FTP sites.

It records basic information about user requests, such as:

Remote host name
User name
Date
Time
Request type
HTTP status code
Number of bytes received by the server.

The item are separated by spaces
Time is recorded as local time.

When you open an NCSA Common format file in a text editor, the entries are similar to the following example:

172.21.13.45 — REDMOND\fred [08/Apr/1997:17:39:04 -0800]
"GET /scripts/iisadmin/ism.dll?http/serv HTTP/1.0" 200 3401

Note: In the preceding entry, the second field (which would show the remote log name of the user) is empty and is represented by the hyphen following the IP address 172.21.13.45.

The preceding sample entry is interpreted as follows:

Remote Host Name
User Name
Date
Time & GMT offset
Request Type
Service Status Code
Bytes Sent

All fields are terminated with a space.
A hyphen acts as a placeholder if there is no valid value for a certain field.

The entry indicates that a user named Fred in the REDMOND domain, with the IP address of 172.21.13.45, issued an HTTP GET command (that is, downloaded a file) at 5:39 P.M. on April 8, 1998. The request returned, without error, 3401 bytes of data to the user named Fred.

ODBC logging format is a record of a fixed set of data fields in an ODBC-compliant database, such as Microsoft Access or Microsoft SQL Server.

Some of the items logged are:

The user's IP address
User name
Request date and time
HTTP status code
Bytes received
Bytes sent


	Action carried out (for example, a download carried out by a GET command) The target (for example, the file that was downloaded).



The time is recorded as local time
You must specify the database to be logged to.
You must setup the database table manually to receive the data.

To use ODBC logging, you must complete the following steps:

1. Create a database containing a table with the appropriate fields for the logging data.

   IIS includes a SQL template file which can be run in a SQL database to create a table that accepts log entries from IIS.

   The file is called Logtemp.sql and is available in the following location by default:

   c:\winnt\system32\inetsrv\logtemp.sql

   The following fields are required and in this format

   Field Name	SQL Server Field Type	MS Access Field Type
   ClientHost	varchar(255)	text (255)
   Username	varchar(255)	text (255)
   LogTime	datetime	datetime
   Service	varchar(255)	text (255)
   Machine	varchar(255)	text (255)
   ServerIP	varchar(50)	text (50)
   ProcessingTime	int	int
   BytesRecvd	int	int
   BytesSent	int	int
   ServiceStatus	int	int
   Win32Status	int	int
   Operation	varchar(255)	text (255)
   Target	varchar(255)	text (255)
   Parameters	varchar(255)	text (255)

   
   
2. Create an ODBC Data Source Name (DSN), which is a name that the ODBC software uses to find the database. 
    
3. Supply IIS with the name of the database and table. If a user name and password are needed for accessing the database, you must also specify these in IIS.
    
   For more information see this article:

Process accounting is a new feature in IIS 5+. It adds additional fields to the W3C Extended log file to record information about how Web sites use CPU resources on the server.

This information is used to determine if sites are using a disproportional amount of CPU resources, or to detect malfunctioning scripts or CGI processes.

Process accounting can be enabled on a per-site basis.

It does not provide details:

CPU usage of individual applications
Logs information only about out-of-process applications.
It is available only for Web sites
Is only available with the W3C Extended log file format.

The process accounting information is interspersed with other logging information in the file.

To enable process accounting use the following procedure:

1. Open the Internet Service Manager
2. Select a Web site, right click and select Properties.
3. On the Web Site property sheet, click Properties.
4. On the Extended Properties sheet, select the Process Accounting check box.
5. Select the additional fields to log.
6. Click OK.

The information gained from process accounting can be used to determine if process throttling should be enabled on a Web site.

Process throttling limits the amount of processor time a Web site can use - Process throttling can cause clients to be denied access to your web site.

When IIS logging is enabled, as it is by default, new log entries are generated whenever users are accessing the server. This causes a progressive increase in log file size or in the number of log files. You may need to balance the gathering of detailed data against the need to limit files to a manageable size and number. IIS offers two options for managing the generation of log data and the creation of new log files.

One way to manage log data is to use W3C Extended logging and customize it so that you gather only the data you need.

Another option for managing log files is to limit log size by changing the frequency of log file creation.

Log files are simply ASCII (text) files. If you find you have created many small files when you prefer one large file, you can combine the files by using the copy command.

Note: If your server runs out of disk space when IIS is attempting to add a log entry to a file, IIS logging shuts down. At the same time, in the Application log of Windows Event Viewer, an event is logged. When disk space is once again available, IIS logging resumes. This causes an additional event to be logged in the Application log of Windows Event Viewer.

Back To Tips of Web Log Analysis