Анализ «лабораторной» парадигмы оценки систем поиска

Abstract

В отчете представлены предварительные результаты экспериментального анализа некоторых методологических аспектов «лабораторной» парадигмы оценки систем информационного поиска. Исследование оценки методом «общего котла» проводилось на основе материалов семинара РОМИП за 2003 и 2004 годы. Рассматривались следующие вопросы: Насколько эффективен метод «общего котла»? Как параметры эксперимента влияют на выводы? До какой степени можно использовать полученные таблицы релевантности для оценки других систем? Насколько на результат влияет «человеческий фактор»? Большинство опубликованных исследований схожих методологических вопросов проводились на основе данных TREC. Целью этой работы являлось не только получение новых результатов, но также и проверка уже опубликованных выводов на альтернативных данных.

This work focuses on evaluation of pooling-based methodology widely used to evaluate information retrieval systems. Number of previous works studied pooling characteristics and impact based on TREC data. In our research we are using results of first two years of Russian Information Retrieval Seminar (ROMIP) (see also http://romip.narod.ru). Four main groups of questions are considered: Is pooling effective way to reduce evaluation costs for all participants? Does it provide good approximation of set of relevant documents? How reliable are results of such experiments? Will conclusions change if some experiment parameters will be changed? E.g. if other queries will be judged. Are resulted collections and relevance tables are reusable? Can they be used to reasonable evaluate system run omitted from pool? To which extent “human factors” (such as subjectivity) can have an effect on the evaluation of retrieval results? What may change if other assessors will be judging system results? Does increasing number of assessors help to improve stability? Some of these questions were considered earlier using TREC data. We are interested to verify some of published results as well as to see if ROMIP-based dependencies are similar to TREC ones. Most of our results are in line with previous research. Therefore we mention only few specific ones here: Calculated quantitative estimations of minimal difference in scores for main metrics on ROMIP data. Shown that evaluation of system omitted from the pool is reasonably reliable for ROMIP data. Proved that random errors in assessor judgments do not have significant impact on conclusions. Demonstrated that individual assessor can make the difference in the conclusions and redundant assessment helps to improve stability.