Domain: Learning to Rank

Overview

Learning to rank is the application of machine learning, typically supervised, semi-supervised or reinforcement learning, in the construction of ranking models for information retrieval systems. Training data consists of lists of items with some partial order specified between items in each list. This order is typically induced by giving a numerical or ordinal score or a binary judgment (e.g. "relevant" or "not relevant") for each item. The ranking model's purpose is to rank, i.e. produce a permutation of items in new, unseen lists in a way which is "similar" to rankings in the training data in some sense. Learning to rank is a relatively new research area which has emerged in the past decade.

Dataset List

Dataset Introductions MQ2007 is a query set from Million Query track of TREC 2007. There are about 1700 queries in it with labeled documents. In MQ2007, the 5-fold cross validation strategy is adopted and the 5-fold partitions are included in the package. In each fold, there are three subsets fo...
BASELINE P@1 P@2 P@3 P@4 P@5 MAP NDCG@1 Evaluation
LambdaMART 0.4811 0.4550 0.4395 0.4255 0.4193 0.4658 0.4122 Detail
RankBoost 0.4799 0.4578 0.4440 0.4252 0.4113 0.4624 0.4126 Detail
svm_struct 0.4746 0.4495 0.4315 0.4193 0.4135 0.4644 0.4096 Detail
RankNet 0.4515 0.4303 0.4129 0.4004 0.3915 0.4500 0.3893 Detail
AdaRank 0.4480 0.4335 0.4253 0.4190 0.4073 0.4602 0.3913 Detail
ListNet 0.4456 0.4311 0.4129 0.4031 0.3935 0.4461 0.3822 Detail
BASELINE NDCG@2 NDCG@3 NDCG@4 NDCG@5 MeanNDCG Evaluation
LambdaMART 0.4085 0.4115 0.4141 0.4185 0.4985 Detail
RankBoost 0.4147 0.4185 0.4191 0.4191 0.4995 Detail
svm_struct 0.4073 0.4062 0.4084 0.4142 0.4966 Detail
RankNet 0.3895 0.3907 0.3924 0.3966 0.4821 Detail
AdaRank 0.3963 0.4021 0.4091 0.4125 0.4922 Detail
ListNet 0.3868 0.3894 0.3944 0.3974 0.4798 Detail
Dataset Introductions MQ2008 is a query set from Million Query track of TREC 2008. There are about 800 queries in it with labeled documents. In MQ2008, the 5-fold cross validation strategy is adopted and the 5-fold partitions are included in the package. In each fold, there are three subsets for l...
BASELINE P@1 P@2 P@3 P@4 P@5 MAP NDCG@1 Evaluation
LambdaMART 0.4489 0.4113 0.3843 0.3596 0.3405 0.4753 0.3741 Detail
RankBoost 0.4413 0.4094 0.3962 0.3689 0.3480 0.4758 0.3665 Detail
AdaRank 0.4374 0.4081 0.3848 0.3657 0.3431 0.4783 0.3720 Detail
svm_struct 0.4273 0.4068 0.3903 0.3695 0.3474 0.4695 0.3626 Detail
ListNet 0.4107 0.3865 0.3656 0.3511 0.3342 0.4555 0.3486 Detail
RankNet 0.4068 0.3839 0.3622 0.3466 0.3280 0.4514 0.3422 Detail
BASELINE NDCG@2 NDCG@3 NDCG@4 NDCG@5 MeanNDCG Evaluation
LambdaMART 0.4056 0.4288 0.4486 0.4689 0.4852 Detail
RankBoost 0.3919 0.4281 0.4487 0.4680 0.4820 Detail
AdaRank 0.4064 0.4289 0.4585 0.4759 0.4885 Detail
svm_struct 0.3984 0.4285 0.4508 0.4695 0.4832 Detail
ListNet 0.3808 0.4061 0.4322 0.4546 0.4682 Detail
RankNet 0.3776 0.4042 0.4281 0.4475 0.4642 Detail
Dataset Introductions MQ2007 is a query set from Million Query track of TREC 2007. There are about 1700 queries in it with labeled documents. In MQ2007, the 5-fold cross validation strategy is adopted and the 5-fold partitions are included in the package. The data format in...
BASELINE P@1 P@2 P@3 P@4 P@5 MAP NDCG@1 Evaluation
BASELINE NDCG@2 NDCG@3 NDCG@4 NDCG@5 MeanNDCG Evaluation
Dataset Introductions MQ2008 is a query set from Million Query track of TREC 2008. There are about 800 queries in it with labeled documents. In MQ2008, the 5-fold cross validation strategy is adopted and the 5-fold partitions are included in the package. The data format...
BASELINE P@1 P@2 P@3 P@4 P@5 MAP NDCG@1 Evaluation
BASELINE NDCG@2 NDCG@3 NDCG@4 NDCG@5 MeanNDCG Evaluation
Dataset Introductions MQ2007 is a query set from Million Query track of TREC 2007. There are about 1700 queries in it with labeled documents. In MQ2007, the 5-fold cross validation strategy is adopted and the 5-fold partitions are included in the package. In each fold, there are three subsets for le...
BASELINE P@1 P@2 P@3 P@4 P@5 MAP NDCG@1 Evaluation
BASELINE NDCG@2 NDCG@3 NDCG@4 NDCG@5 MeanNDCG Evaluation
Dataset Introductions MQ2008 is a query set from Million Query track of TREC 2008. There are about 800 queries in it with labeled documents. In MQ2008, the 5-fold cross validation strategy is adopted and the 5-fold partitions are included in the package. In each fold, there are three subsets for lea...
BASELINE P@1 P@2 P@3 P@4 P@5 MAP NDCG@1 Evaluation
BASELINE NDCG@2 NDCG@3 NDCG@4 NDCG@5 MeanNDCG Evaluation
Dataset Introductions MQ2007 is a query set from Million Query track of TREC 2007. There are about 1700 queries in it with labeled documents. In MQ2007, the 5-fold cross validation strategy is adopted and the 5-fold partitions are included in the package. The data format in MQ2007-list is the same a...
BASELINE P@1 P@2 P@3 P@4 P@5 MAP NDCG@1 Evaluation
BASELINE NDCG@2 NDCG@3 NDCG@4 NDCG@5 MeanNDCG Evaluation
Dataset Introductions MQ2008 is a query set from Million Query track of TREC 2008. There are about 800 queries in it with labeled documents. In MQ2008, the 5-fold cross validation strategy is adopted and the 5-fold partitions are included in the package. The data format in MQ2008-list is the same as...
BASELINE P@1 P@2 P@3 P@4 P@5 MAP NDCG@1 Evaluation
BASELINE NDCG@2 NDCG@3 NDCG@4 NDCG@5 MeanNDCG Evaluation