A potential bug in Poisson regression: eval_metric="logloss" vs "poisson-nloglik"

john · January 16, 2022, 7:01pm

I believe there is a bug in Poisson regression and I have checked with other regression such as gamma. My understanding is the results should be the same with eval_metric=“logloss” or “poisson-nloglik”. But the outputs are quite different. An example follows:

library(“xgboost”)
set.seed(15)
x <- matrix(rnorm(100*2),100,2)
g2 <- sample(c(0,1),100,replace=TRUE)
fit1 <- xgboost(data=x, label=g2, objective = “count:poisson”, eval_metric=“logloss”, nrounds=10)
[1] train-logloss:0.604225
[2] train-logloss:0.533964
[3] train-logloss:0.477919
[4] train-logloss:0.430387
[5] train-logloss:0.390779
[6] train-logloss:0.367838
[7] train-logloss:0.348413
[8] train-logloss:0.330266
[9] train-logloss:0.315782
[10] train-logloss:0.302425
fit2 <- xgboost(data=x, label=g2, objective = “count:poisson”, eval_metric=“poisson-nloglik”, nrounds=10)
[1] train-poisson-nloglik:0.781148
[2] train-poisson-nloglik:0.745825
[3] train-poisson-nloglik:0.717435
[4] train-poisson-nloglik:0.693129
[5] train-poisson-nloglik:0.672651
[6] train-poisson-nloglik:0.660460
[7] train-poisson-nloglik:0.650108
[8] train-poisson-nloglik:0.640582
[9] train-poisson-nloglik:0.633043
[10] train-poisson-nloglik:0.625980
sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.6 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] xgboost_1.5.0.2

loaded via a namespace (and not attached):
[1] compiler_4.1.2 Matrix_1.4-0 grid_4.1.2 data.table_1.14.0
[5] jsonlite_1.7.2 lattice_0.20-41

hcho3 · January 26, 2022, 8:30am

What leads to this conclusion? In XGBoost, these two evaluation metrics have different implementations.

logloss:

github.com

dmlc/xgboost/blob/58a6723eb10e18336ada3b9ea32c2fd9a2aee1c6/src/metric/elementwise_metric.cu#L203


  }


  XGBOOST_DEVICE bst_float EvalRow(bst_float y, bst_float py) const {
    const bst_float eps = 1e-16f;
    const bst_float pneg = 1.0f - py;
    if (py < eps) {
      return -y * std::log(eps) - (1.0f - y)  * std::log(1.0f - eps);
    } else if (pneg < eps) {
      return -y * std::log(1.0f - eps) - (1.0f - y)  * std::log(eps);
    } else {
      return -y * std::log(py) - (1.0f - y) * std::log(pneg);
    }
  }


  static double GetFinal(double esum, double wsum) {
    return wsum == 0 ? esum : esum / wsum;
  }
};


struct EvalRowMPHE {
  char const *Name() const {

poisson-nloglik:

github.com

dmlc/xgboost/blob/58a6723eb10e18336ada3b9ea32c2fd9a2aee1c6/src/metric/elementwise_metric.cu#L271


};


struct EvalPoissonNegLogLik {
  const char *Name() const {
    return "poisson-nloglik";
  }


  XGBOOST_DEVICE bst_float EvalRow(bst_float y, bst_float py) const {
    const bst_float eps = 1e-16f;
    if (py < eps) py = eps;
    return common::LogGamma(y + 1.0f) + py - std::log(py) * y;
  }


  static double GetFinal(double esum, double wsum) {
    return wsum == 0 ? esum : esum / wsum;
  }
};


/**
 * Gamma deviance
 *

john · January 26, 2022, 9:06pm

Sorry, but I had a hard time to digest what you meant. What results are the same? The trees from these two different eval_metric? In my opinion, the logloss should be only used for logistic regression, not Poisson regression. Of course, those numbers still can be computed somehow from the computer code. But I don’t know what is the meaning:

fit1 <- xgboost(data=x, label=g2, objective = “count:poisson”, eval_metric=“logloss”, nrounds=10)

hcho3 · January 27, 2022, 12:04am

Yes, the trees should be the same, since you didn’t use early stopping. What’s different is the choice of eval_metric.

Good point. We can simply throw an error when the user tries to use logloss metric together with Poisson regression.