About RAT

The Result Assessment Tool (RAT) is a software toolkit that allows researchers to conduct large-scale studies based on results from (commercial) search engines and other information retrieval systems. It consists of modules for (1) designing studies, (2) collecting data from search systems, (3) collecting judgments on the results, (4) downloading/analysing the results.

Due to the modularity, individual components can be used for a multitude of studies relating to web content, such as qualitative content analyses. Through automated scraping, web documents can be analysed quantitatively in empirical studies.

The modular web-based software can automatically record data from search engines. Studies with questions and scales can be flexibly designed, and the information objects can be evaluated by jurors on the basis of the questions.

The design of the RAT prototype has been guided by the requirement of researchers to query external information retrieval systems in general, and search engines in particular. This need derives from researchers' interest in the quality of their search results, the composition of their results (lists or pages), and potential biases in the results, to name but a few. This is a unique approach that separates RAT from software developed to aid retrieval evaluation where researchers have complete control over the systems evaluated.

RAT allows studies to be conducted under the two major evaluation paradigms, namely system-centered and user-centered evaluations.

RAT will also be useful to any other researchers aiming to use results from search engines for further analysis. While the basis of RAT lies in information retrieval evaluation, its use goes well beyond this research area. Already in the prototype phase, we found that there is a significant need for a software tool like RAT in the search engine and information retrieval communities, respectively. Some projects have already demonstrated this.