How can I manually fit a single tree with a customized cost function ( from pytorch) and then update the model trees in xgboost?

#1

Hello,

I am working this paper : Multi-Label Learning to Rank through Multi-Objective
Optimization

The algorithm is to used an customized gradient “lambda” from multiple cost functions (c_1, c_2, …c_K) in the xgboost training process.
image

Within the algorithm mentioned in the paper, I need to fit the t-th tree using the data ( see equation (3) below) where the label is constructed with the new gradients “lambda”. image
image

My question here is:

When I use Pytorch to build a customized cost function as well as its gradient, how can I use xgboost package to

  • fit a single tree with the new data (i.e. the set of (feature, residual) pairs )
  • update the model by adding the newly fitted tree in xgboost package.

Thank you here for your help.

Best,
Zongjun

#2

Have you looked into custom objective?

#3

Hello Jiamingy, thanks for replying. I have looked into customized objective function. I will need to define a gradiend and a hessian as output of the customized loss function.

Here, I can get the gradient using pytorch auto-gradient functionality, but what I still need is to use the gradient to update the existing trees in the xgboost package. I need to construct a tree as a functional proxy to fit the residuals data $D_{T_\theta^t}$. Because there is a sub-optimization algorithm used to balance the gradients in each tree step construction, so I need to control how xgboost fits trees step by step.

In short, I want to know

  • how to do a one step tree fitting in Xgboost with a given data ( given the residual values from the loss after N steps of tree fitting )
  • how to append the new fitted tree to the previous N trees and then use these N+1 trees to compute the new residual .

Many thanks.