Domain: NeuIR

Overview

Developing effective retrieval models is a central challenge in information retrieval (IR) research. Many different retrieval models have been proposed over the past decades, such as vector space models and probabilistic models.

Dataset List

Robust04 is a small news dataset. The topics are collected from TREC Robust Track 2004. Here the Robust04-Desc means that the description of the topic are used as query. The collection is consist of 0.5M documents and 250 queries. The vocabulary size is 0.6M, and the collection length is 252M. Thes...
BASELINE MAP NDCG@20 P@20 Evaluation
DRMM(LCH-IDF) 0.2750 0.4370 0.3710 Detail
NWT 0.2680 0.4130 0.3530 Detail
QL 0.2460 0.3910 0.3340 Detail
BM25 0.2410 0.3990 0.3370 Detail
MatchPyramid(COS) 0.1900 0.3300 0.1620 Detail
MatchPyramid(IND) 0.1420 0.3190 0.1180 Detail
MatchPyramid(DOT) 0.1040 0.1590 0.0920 Detail
DSSM-D 0.0780 0.1690 0.1450 Detail
CDSSM-D 0.0500 0.1130 0.0930 Detail
ARC-II 0.0420 0.0860 0.0740 Detail
ARC-I 0.0300 0.0470 0.0450 Detail
ClueWeb09B is a large Web collection, whose topics are accumulated from TREC Web Tracks 2009, 2010, and 2011. And ClueWeb09B is filtered to the set of documents with spam scores in the 60th percentile, us ing the Waterloo Fusion spam scores [1]. The collection consist of 34M documents and 150 querie...
BASELINE MAP nDCG@20 P@20 Evaluation
DRMM(LCH-IDF) 0.1130 0.2580 0.3650 Detail
NWT 0.1070 0.2360 0.3410 Detail
BM25 0.1010 0.2250 0.3260 Detail
QL 0.1000 0.2240 0.3280 Detail
MatchPyramid(COS) 0.0660 0.2220 0.2900 Detail
CDSSM-T 0.0640 0.1530 0.2140 Detail
MatchPyramid(IND) 0.0560 0.2080 0.2810 Detail
DSSM-T 0.0540 0.1320 0.1850 Detail
CDSSM-D 0.0540 0.1340 0.1770 Detail
MatchPyramid(DOT) 0.0440 0.1580 0.1550 Detail
DSSM-D 0.0390 0.0990 0.1310 Detail
ARC-II 0.0330 0.0870 0.1230 Detail
ARC-I 0.0240 0.0730 0.0890 Detail
Robust04 is a small news dataset. The topics are collected from TREC Robust Track 2004. Here the Robust04-Title means that the title of the topic are used as query. The collection is consist of 0.5M documents and 250 queries. The vocabulary size is 0.6M, and the collection length is 252M. These dat...
BASELINE MAP NDCG@20 P@20 Evaluation
DRMM(LCH-IDF) 0.2760 0.4310 0.3820 Detail
NWT 0.2740 0.4260 0.3800 Detail
BM25 0.2550 0.4180 0.3700 Detail
QL 0.2530 0.4150 0.3690 Detail
MatchPyramid(COS) 0.1890 0.3300 0.2900 Detail
MatchPyramid(IND) 0.1690 0.3190 0.2810 Detail
DSSM-D 0.0950 0.2010 0.1710 Detail
MatchPyramid(DOT) 0.0830 0.1590 0.1550 Detail
CDSSM-D 0.0670 0.1460 0.1250 Detail
ARC-II 0.0670 0.1470 0.1280 Detail
ARC-I 0.0410 0.0660 0.0650 Detail
the WordEmbedding dataset contains word embeddings used in Robust04 and Clueweb09B dataset. The word embeddings are trained on corresponding corpus with the word2vec toolkit. These data can only be used for academic research purposes....
BASELINE MAP NDCG@20 P@20 Evaluation
ClueWeb09B is a large Web collection, whose topics are accumulated from TREC Web Tracks 2009, 2010, and 2011. And ClueWeb09B is filtered to the set of documents with spam scores in the 60th percentile, us ing the Waterloo Fusion spam scores [1]. The collection consist of 34M documents and 150 querie...
BASELINE MAP NDCG@20 P@20 Evaluation
DRMM(LCH-IDF) 0.0870 0.2270 0.2940 Detail
NWT 0.0800 0.2040 0.2640 Detail
BM25 0.0800 0.1960 0.2550 Detail
QL 0.0750 0.1830 0.2340 Detail
MatchPyramid(COS) 0.0570 0.1400 0.1710 Detail
CDSSM-T 0.0550 0.1390 0.1710 Detail
CDSSM-D 0.0490 0.1250 0.1600 Detail
DSSM-T 0.0460 0.1190 0.1430 Detail
MatchPyramid(IND) 0.0430 0.1180 0.1580 Detail
DSSM-D 0.0340 0.0780 0.1030 Detail
MatchPyramid(DOT) 0.0330 0.0730 0.1020 Detail
ARC-II 0.0240 0.0560 0.0750 Detail
ARC-I 0.0170 0.0360 0.0510 Detail