Custom objective function - explanation of "logregobj"

#1

Hi,

as it is possible to make a customized objective function such as here for binary classification:
logregobj <- function(preds, dtrain) {
labels <- getinfo(dtrain, “label”)
preds <- 1/(1 + exp(-preds))
grad <- preds - labels
hess <- preds * (1 - preds)
return(list(grad = grad, hess = hess))
}

My questions regarding this example is:

  1. What is really preds here? It looks like preds are by default logit of predicted probabilities of being in class 1, say. Is this always the case? I can’t find the function where the objective function with “preds” is explicitly used.

  2. I don’t understand the computation of grad and hessian. Using minus log likelihood og bernoulli random variables this gives a sum of y*log p + (1-y)log(1-p) (y is 0 or 1, p pred. prob.). Differentiation w.r.t. p (which is what you shoud do, right?) gives gradient (preds-labels)/(preds(1-preds).

Thank you for any replies.

1 Like
#2

OK, so I understand. The reason I got confused is that the input for the function logregobj called “pred” is really not predictions but logit prediction (should have been called logitpreds or something). Logitpreds it the output of any regression tree given input, where transformation to binary classification probabilities is done by:
pred(class = 1) = 1/(1 + exp(-logitpreds)). I guess for multiple classes it gets even more messy.

#3

@paalvj I have the same question but I still do not understand the answer you gave here. Why is the gradient simply equal to preds-labels? What is the exact mathematical form of the loss function?

#4

@paalvj Also why is the output of any regression tree given in the form of logitpreds?