I am trying to build a model trained on binary labels that has a high precision for the top k predicted instances, and don’t care too much about recall or precision more generally. I was then interested in the idea of ranking instances in my classification problem rather than looking at prediction probabilities, so thought I could play around with XGBRanker with a single query group.
Firstly, does this make sense? I noticed when using learning to rank, I’m getting much better results than with classification using pre@k as an evaluation metric
Secondly, I noticed that Learning to Rank parameters can be passed to XGBClassifier and doesn’t raise any errors, and in fact with a single query group XGBClassifier and XGBRanker seem to output the same results (see code below to reproduce in xgboost v2.0.3). Using XGBClassifier would be simpler as it then doesn’t break sklearn compatibility, but I am unsure whether this is a correct usage. If not, I would expect an error to be raised but this isn’t the case.
from sklearn.datasets import make_classification
import numpy as np
import xgboost as xgb
# Make a synthetic ranking dataset for demonstration
seed = 1994
X, y = make_classification(random_state=seed)
rng = np.random.default_rng(seed)
n_query_groups = 1
qid = rng.integers(0, n_query_groups, size=X.shape[0])
# Sort the inputs based on query index
sorted_idx = np.argsort(qid)
X = X[sorted_idx, :]
y = y[sorted_idx]
qid_sorted = qid[sorted_idx]
ranker = xgb.XGBRanker(lambdarank_num_pair_per_sample=8, objective="rank:map")
ranker.fit(X, y, qid=qid_sorted)
classif = xgb.XGBClassifier(lambdarank_num_pair_per_sample=8, objective="rank:map")
classif.fit(X, y)
rank_prediction = ranker.predict(X)
classif_prediction = classif.predict(X)
classif_prediction_proba = classif.predict_proba(X)[:, 1]
assert np.array_equal(rank_prediction, classif_prediction_proba)