Website Hit Counter from Logs

[Home]   [Puzzles & Projects]    [Delphi Techniques]   [Math topics]   [Library]   [Utilities]

 

Search

Search WWW

Search DelphiForFun.org

As of October, 2016, Embarcadero is offering a free release of Delphi (Delphi 10.1 Berlin Starter Edition ).     There are a few restrictions, but it is a welcome step toward making more programmers aware of the joys of Delphi.  They do say "Offer may be withdrawn at any time", so don't delay if you want to check it out.  Please use the feedback link to let me know if the link stops working.

 

Support DFF - Shop

 If you shop at Amazon anyway,  consider using this link. 

     

We receive a few cents from each purchase.  Thanks

 


Support DFF - Donate

 If you benefit from the website,  in terms of knowledge, entertainment value, or something otherwise useful, consider making a donation via PayPal  to help defray the costs.  (No PayPal account necessary to donate via credit card.)  Transaction is secure.

Mensa Daily Puzzlers

For over 15 years Mensa Page-A-Day calendars have provided several puzzles a year for my programming pleasure.  Coding "solvers" is most fun, but many programs also allow user solving, convenient for "fill in the blanks" type.  Below are Amazon  links to the two most recent years.

Mensa 365 Puzzlers  Calendar 2017

Mensa 365 Puzzlers Calendar 2018

(Hint: If you can wait, current year calendars are usually on sale in January.)

Contact

Feedback:  Send an e-mail with your comments about this program (or anything else).

Search DelphiForFun.org only

 

 

Problem Description

A program to read website log files and count "hits" or other site events based on record content.

Background & Techniques

Our home page has a "hit counter" component which stopped working sometime in the April to June, 2009 timeframe.  It may be fixable, but in the meantime I wanted to recover the count of home page visits since my last checkpoint at the end of in March.   I save raw log data routinely to analyze the popularity of program downloads, so I have the data but needed a program to count home page visit events.

Logs are text files are produced daily which record interactions between web pages and the outside world including all the events which occur when user clicks to open a page.   The event I want to count is the one that invokes the FrontPage hit counter program FPCount.exe. 

I did try to generalize the program to make easier to apply to solving other problems.

The program  allows the user to select the folder to be searched for log files. 

Within that folder, a "mask" controls which specific files are scanned.  The rules for file masks are summarized in the sidebar at right. Typically we see masks like "*.*" (any file name containing a dot ), or as in our default mask in this case "ex*.log" since I name my log files as exyymmdd.log where   yymmdd is the year, month and day numbers of the day it contains.

Finally, the specific text to be searched within each record is a user input.  I only need one field in this case, but included provisions for  a second match filed required  which is ignored if left blank.  

In this version, only hit count totals and the highest date found  are kept, but future version might include option to save the specific matching record for further processing.   The date and count totals are kept in a text file named :HitCount.ini.

File mask rules

A file mask is a string of characters against which file names may be tested.  Here's a summary of  the file masking rules:

  1. Characters in the string other than * and ? must match the corresponding position in the file name.
  2. A ?  matches any character in that position.
  3. An * matches any number of consecutive characters until the character following the * in the mask is found or the end of the string is reached.
  4. Matches are case insensitive.

To test the operation, I have included in the download zips a few hundred records from two late June log files in.

Programmer's Notes

A new feature, at least for me, was using ShBrowseForFolder Windows API function which displays an Explorer type dialog allowing user to select the target folder.  The GetFolder function contained in a new USelectFolder unit  was adapted from  Angus Johnson's  TZip component which I downloaded several years ago from the excellent Torry.net website but never got around to implementing.

Once the program has the folder to scan, that folder name and file mask field is passed to SelectFiles procedure to fill a String list named FileList with the names of the files to scan.  The files are successively read in a loop, checking each record for matches against the match fields supplied by the user.  When a match is found, the HitCount integer is incremented and the date extracted from the log record.  Windows logs use a somewhat unusual format for date-time stamps, at least not one handled automatically by Delphi's date routines, e,g,  28/Jun/2009:00:15:42.   I ended up extracting the three character month abbreviations, converting them to a month number and then using then EncodeDate function to convert year, month, and day numbers to a TDate field, Filedate

SaveDate and SaveCount fields are saved in HitCounter.ini file and have no built-in backup/undo features, I have tried to get user verification before updating the fields; not perfect, but better  than nothing. .  

 

Running/Exploring the Program 

bullet Download source
bullet Download  executable

Suggestions for Further Explorations

bullet Future uses might require saving the file names or the records which matched for further analysis.
bullet It would be good to save a history of results so that errors could be corrected, perhaps saving a separate section in the Ini file which incorporates run date and time in the section name.  By default, the section with the most current run data would provide the starting values for the next run. 
bullet The program should work for non-log text files except for the date extraction, but i have not tested it.

 

Original Date: June 16, 2009

Modified: July 29, 2017

 

  [Feedback]   [Newsletters (subscribe/view)] [About me]
Copyright 2000-2017, Gary Darby    All rights reserved.