Finding and evaluating patterns in Web repository using database technology and data mining algorithms

Abstract

Web mining is a very hot research topic, which combines two of the active research areas: Data Mining and World Wide Web. The Web mining research relates to several research communities such as Database, Statistics, Artificial Intelligence and Visualization. Although there exists some confusion about the Web mining, the most recognized approach is to categorize Web mining into three areas: Web content mining, Web structure mining, and Web usage mining. Web content mining focuses on the discovery/retrieval of the useful information from the Web contents/data/documents, while the Web structure mining emphasizes to the discovery of how to model the underlying link structures of the Web. Sometimes the distinction between these two categories is not very clear. Web usage mining is relatively independent, but not isolated category, in which the following studies continue; General Web Usage Mining, Site Modification, Systems Improvement and Personalization. General Web Usage Mining systems aim to discover general trends and patterns from the log files by adapting data mining techniques. The objective of the Site Modification systems is to improve the design of a web site by suggesting modifications in its content and structure. The research on System Improvement focuses on using the web usage mining for improving the web traffic. Finally, personalization systems aim to understand individual trends used for personalizing the web sites. The study subject to this thesis, IYTE Web Usage Mining (WUM) System was an example of system development in the field of General Web Usage Mining with a database approach where the flexible query capability of SQL (Structured Query Language) was explored. The data mining and database techniques were applied on the access/error/user logs of the web server of Izmir Institute of Technology. The main objective was to create a site improvement tool for the web administrator by reporting the distribution of the hits received by the web server according to the time stamp, users, service and URL types and at the same time revealing the nature of the errors generated by the web server. All data cleaning and transaction identification processes were handled by the software routines coded in Java. Clean transactions were imported into IYTE Web Usage Mining (IYTE WUM) relational database. Flexible features of SQL were utilized for application of algorithm Apriori to discover most frequent pair of URL s visited, in addition to extraction of general knowledge from data.