Rank:ndcg is predicting all values to be zero, while pairwise builds the trees correctly

#1

Hello!

I am working a ranking task (cannot share the original data, but shared a simulated set where we can see this pattern), I have 25 features and 25 positive and 25 negative examples. Positive values are increasing order of relevance such as 1,2,3 …,25 (specific to my domain), and all negative labels are set to 0.

Here the trees from rank:ndcg are all as follows

model.get_dump()

['0:leaf=0\n', '0:leaf=0\n', '0:leaf=0\n', '0:leaf=0\n', '0:leaf=0\n']

but when I use rank:pairwise on this labels, I get to see the trees, and splits. (you can run the code below to validate this behaviour - I am on xgboost '1.7.1' version)

Create synthetic Data

new_df = pd.DataFrame(np.random.random((50,25)))
new_df["label"] = [0]*25 + list(range(1, 26))
dtrain = xgb.DMatrix(
            new_df.iloc[:,:-1],
            new_df.iloc[:,-1]
)
dtrain.set_group([50])

rank:ndcg

params = {
    'objective': 'rank:ndcg',
    'max_depth':4,
    'tree_method': 'hist',
    'eval_metric': "ndcg@10"
}

model =  xgb.train(params=params, dtrain=dtrain ,num_boost_round=5,
                   evals=[(dtrain,"train")], verbose_eval=True)

rank:pairwise

params = {
    'objective': 'rank:pairwise',
    'max_depth':4,
    'tree_method': 'hist',
    'eval_metric': "ndcg@10"
}

model =  xgb.train(params=params, dtrain=dtrain ,num_boost_round=5,
                   evals=[(dtrain,"train")], verbose_eval=True)

Please let me know if there is any mistake in my implementation. From the previous forum discussion I have made sure that

  1. Label are non-negative and int64 in increasing order of relevance
  2. I also read xgb uses2^relevance in ndcg computation - so clustered the labels to 1 - 25
  3. truncated the numbers precision to max 9-digit precision

Thank you for you time!

Dealing with xgboost multiple missing values
#2

Could you please try 2.0? We have some significant updates to ltr.

#3

@jiamingy - Yes, 2.0.0 seems to fix this issue. I am able to replicate this issue till 1.7.6 May I know what fixed this? Also I still see rank:pairwise having a better ndcg than rank:ndcg (in my offline experiments)

[0]	train-ndcg@10:0.00005
[1]	train-ndcg@10:0.79768
[2]	train-ndcg@10:0.95345
[3]	train-ndcg@10:0.97790
[4]	train-ndcg@10:0.98448

rank:pairwise

[0]	train-ndcg@10:0.00221
[1]	train-ndcg@10:0.64081
[2]	train-ndcg@10:0.94269
[3]	train-ndcg@10:0.99705
[4]	train-ndcg@10:0.99778
#4

We have detailed documents about learning to rank in 2.0. You may want to take a look at what are these objectives and how they work.