Early stopping returns minimum value if the eval_metric is AUCPR

Vivekreddyd · June 11, 2019, 6:18pm

Hello Everyone,

I’m performing xgboost with eval_metric as ‘aucpr’ and early_stopping_rounds = 20.

When performing the early stopping it is only checking for the minimum value and returning if the value does not decrease, but the case should be it should pick up the maximum value instead. Can you please let me know the workaround.

Thanks,
Vivek.

PabloCanovas · June 17, 2019, 9:46am

You should add the parameter maximize = True during training.

Vivekreddyd · June 17, 2019, 6:14pm

I’m using python API XGBClassifier (https://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.sklearn) which does not has maximize parameter.

hcho3 · June 18, 2019, 9:09pm

In latest version, AUCPR is maximized in early stopping:

github.com

dmlc/xgboost/blob/bbe0dbd7ec781147327d5326700fbda8b70f2d61/python-package/xgboost/callback.py#L189


def init(env):
    """internal function"""
    bst = env.model


    if not env.evaluation_result_list:
        raise ValueError('For early stopping you need at least one set in evals.')
    if len(env.evaluation_result_list) > 1 and verbose:
        msg = ("Multiple eval metrics have been passed: "
               "'{0}' will be used for early stopping.\n\n")
        rabit.tracker_print(msg.format(env.evaluation_result_list[-1][0]))
    maximize_metrics = ('auc', 'aucpr', 'map', 'ndcg')
    maximize_at_n_metrics = ('auc@', 'aucpr@', 'map@', 'ndcg@')
    maximize_score = maximize
    metric_label = env.evaluation_result_list[-1][0]
    metric = metric_label.split('-', 1)[-1]


    if any(metric.startswith(x) for x in maximize_at_n_metrics):
        maximize_score = True


    if any(metric.split(":")[0] == x for x in maximize_metrics):
        maximize_score = True

Can you double check if you are using a recent version of XGBoost?

Vivekreddyd · July 10, 2019, 8:58pm

Thank you very much. I was using old version