Home | Sitemap | FAQ | Contact Us

Brief introduction of log file formats

You can choose the format that your Web server uses to record user activity. The following formats are available:



Sample Report
Tutorial
Compare Editions
FAQ
Localization
Testimonials
Our Customers
Newsletter
 

The W3C Extended log file format, Microsoft IIS log file format, and NCSA log file format are all ASCII text formats.

The W3C Extended and NCSA formats record logging data in four-digit year format.
The Microsoft IIS format uses a two-digit year format and is provided for backward compatibility with earlier IIS versions.

The W3C Extended log file format provides the most features of all of the log formats because you can customise the fields which are logged thereby reducing the size of log files.

W3C Extended Log File Format

W3C Extended format is a customizable ASCII format with a variety of different fields.

    You can include fields important to you, while limiting log size by omitting unwanted fields.
    Fields are separated by spaces.
    Time is recorded as UTC (Greenwich Mean Time).

The example below shows lines from a file using the following fields: Time, Client IP Address, Method, URI Stem, HTTP Status, and HTTP Version.

#Software: Microsoft Internet Information Services 5.0  #Version: 1.0  #Date: 1998-05-02 17:42:15 
#Fields: time c-ip cs-method cs-uri-stem sc-status cs-version 
17:42:15 172.16.255.255 GET /default.htm 200 HTTP/1.0

The preceding entry indicates that:

    On May 2, 1998 at 5:42 P.M. UTC
    A user with HTTP version 1.0 and the IP address of 172.16.255.255 issued an HTTP GET command for the file Default.htm.
     
    The request was returned without error.  

    The #Date: field indicates when the first log entry was made, which is when the log was created.
    The #Version: field indicates that the W3C logging format 1.0 was used.

Any of the fields can be selected, but some fields may not have information available for some requests.

For fields that are selected, but for which there is no information, a dash (—) appears in the field as a placeholder.

If you have your site configured for Anonymous user authentication the user shows as a dash (—).

Microsoft IIS Log Format

Microsoft IIS format is a fixed (non-customizable) ASCII format. It records more items of information than the NCSA Common format.

The Microsoft IIS format includes basic items such as:

    The user's IP address
    User name
    Request date and time
    HTTP status code
    The number of bytes received.

In addition, it includes detailed items such as:

    The elapsed time of the request
    The number of bytes sent
    The action (for example, a download carried out by a GET command) and the target file.
    The items are separated by commas
    The time is recorded as local time.

When you open a Microsoft IIS format file in a text editor, the entries are similar to the following examples:

192.168.114.201,—,03/20/98,7:55:20,W3SVC2,SALES1,192.168.114.201,4502,163,3223,200,0,GET,DeptLogo.gif  
172.16.255.255,anonymous,03/20/98,23:58:11,MSFTPSVC,SALES1,192.168.114.201,60,275,0,0,0,PASS,intro.htm

The preceding sample entries are interpreted as follows:

    Users IP Address
    User Name
    Date
    Time
    Service and Instance
    Computer Name
    IP Address of Server
    Time taken in Ms
    Bytes Received
    Bytes Sent
    Service Status Code
    Windows NT/2000 Status Code
    Request type
    Target of the operation

So as text we have:

    An anonymous user with the IP address of 192.168.114.201
    issued an HTTP GET command for the image file DeptLogo.gif at 7:55 A.M. on March 20, 1998
    From a server named SALES1 at IP address 172.21.13.45.
    The 163-byte HTTP request had an elapsed processing time of 4502 milliseconds (4.5 seconds) to complete, and returned, without error, 3223 bytes of data to the anonymous user.
     

In the log file:

    All fields are terminated with a comma (,).
    A hyphen acts as a placeholder if there is no valid value for a certain field.
NCSA Common Log File Format

NCSA Common format is a fixed (non-customizable) ASCII format, available for Web sites but not for FTP sites.

It records basic information about user requests, such as:

    Remote host name
    User name
    Date
    Time
    Request type
    HTTP status code
    Number of bytes received by the server.

    The item are separated by spaces
    Time is recorded as local time.

When you open an NCSA Common format file in a text editor, the entries are similar to the following example:

172.21.13.45 — REDMOND\fred [08/Apr/1997:17:39:04 -0800]
"GET /scripts/iisadmin/ism.dll?http/serv HTTP/1.0" 200 3401

Note: In the preceding entry, the second field (which would show the remote log name of the user) is empty and is represented by the hyphen following the IP address 172.21.13.45.

The preceding sample entry is interpreted as follows:

    Remote Host Name
    User Name
    Date
    Time & GMT offset
    Request Type
    Service Status Code
    Bytes Sent
    All fields are terminated with a space.
    A hyphen acts as a placeholder if there is no valid value for a certain field.

The entry indicates that a user named Fred in the REDMOND domain, with the IP address of 172.21.13.45, issued an HTTP GET command (that is, downloaded a file) at 5:39 P.M. on April 8, 1998. The request returned, without error, 3401 bytes of data to the user named Fred.

ODBC Logging

ODBC logging format is a record of a fixed set of data fields in an ODBC-compliant database, such as Microsoft Access or Microsoft SQL Server.

Some of the items logged are:

    The user's IP address
    User name
    Request date and time
    HTTP status code
    Bytes received
    Bytes sent
    Action carried out (for example, a download carried out by a GET command)
    The target (for example, the file that was downloaded).
     


    The time is recorded as local time
    You must specify the database to be logged to.
    You must setup the database table manually to receive the data.

To use ODBC logging, you must complete the following steps:

  1. Create a database containing a table with the appropriate fields for the logging data.

    IIS includes a SQL template file which can be run in a SQL database to create a table that accepts log entries from IIS.

    The file is called Logtemp.sql and is available in the following location by default:

    c:\winnt\system32\inetsrv\logtemp.sql

    The following fields are required and in this format

    Field Name
    SQL Server Field Type
    MS Access Field Type
       ClientHost
    varchar(255)
    text (255)
       Username
    varchar(255)
    text (255)
       LogTime
    datetime
    datetime
       Service
    varchar(255)
    text (255)
       Machine
    varchar(255)
    text (255)
       ServerIP
    varchar(50)
    text (50)
       ProcessingTime
    int
    int
       BytesRecvd
    int
    int
       BytesSent
    int
    int
       ServiceStatus
    int
    int
       Win32Status
    int
    int
       Operation
    varchar(255)
    text (255)
       Target
    varchar(255)
    text (255)
       Parameters
    varchar(255)
    text (255)

  2. Create an ODBC Data Source Name (DSN), which is a name that the ODBC software uses to find the database. 
     
  3. Supply IIS with the name of the database and table. If a user name and password are needed for accessing the database, you must also specify these in IIS.
     
    For more information see this article:
Process Accounting

Process accounting is a new feature in IIS 5+. It adds additional fields to the W3C Extended log file to record information about how Web sites use CPU resources on the server.

This information is used to determine if sites are using a disproportional amount of CPU resources, or to detect malfunctioning scripts or CGI processes.

Process accounting can be enabled on a per-site basis.

It does not provide details:

    CPU usage of individual applications
    Logs information only about out-of-process applications.
    It is available only for Web sites
    Is only available with the W3C Extended log file format.

The process accounting information is interspersed with other logging information in the file.

To enable process accounting use the following procedure:

  1. Open the Internet Service Manager
  2. Select a Web site, right click and select Properties.
  3. On the Web Site property sheet, click Properties.
  4. On the Extended Properties sheet, select the Process Accounting check box.
  5. Select the additional fields to log.
  6. Click OK.

The information gained from process accounting can be used to determine if process throttling should be enabled on a Web site.

Process throttling limits the amount of processor time a Web site can use - Process throttling can cause clients to be denied access to your web site.

Log File Size and the Creation of New Log Files

When IIS logging is enabled, as it is by default, new log entries are generated whenever users are accessing the server. This causes a progressive increase in log file size or in the number of log files. You may need to balance the gathering of detailed data against the need to limit files to a manageable size and number. IIS offers two options for managing the generation of log data and the creation of new log files.

One way to manage log data is to use W3C Extended logging and customize it so that you gather only the data you need.

Another option for managing log files is to limit log size by changing the frequency of log file creation.

Log files are simply ASCII (text) files. If you find you have created many small files when you prefer one large file, you can combine the files by using the copy command.

Note: If your server runs out of disk space when IIS is attempting to add a log entry to a file, IIS logging shuts down. At the same time, in the Application log of Windows Event Viewer, an event is logged. When disk space is once again available, IIS logging resumes. This causes an additional event to be logged in the Application log of Windows Event Viewer.


Back To Tips of Web Log Analysis

Home | Features | Free Trial | Buy Now! | Support
Contact us | Privacy Policy | Reseller
Copyright © 2001-2014 Nihuo Software,Inc