Custom tweedie regression objective


It has been a while I am investigating an error I have when trying to implement the reg:tweedie objective myself as a custom objective function in Python and I couldn’t find the solution, neither in other topics, neither in the source code of the library.

The issue I encounter is that my results are terrible with the custom implementation on my dataset but are good on the same dataset with the built-in implementation.

My custom implementation is the following (forcing rho = tweedie_variance_power at 1.5):

def custom_tweedie_grad(
    y_true: np.ndarray,
    y_pred: np.ndarray,
) -> np.ndarray:
    a = -y_true * np.exp((1.0 - 1.5) * y_pred)
    b = np.exp((2.0 - 1.5) * y_pred)
    grad = a + b
    return grad

def custom_tweedie_hessian(
    y_true: np.ndarray,
    y_pred: np.ndarray,
) -> np.ndarray:
    a = -y_true * (1.0 - 1.5) * np.exp((1.0 - 1.5) * y_pred)
    b = (2.0 - 1.5) * np.exp((2.0 - 1.5) * y_pred)
    hess = a + b
    return hess

def custom_tweedie_objective(
    y_true: np.ndarray,
    y_pred: np.ndarray,
    return (
        custom_tweedie_grad(y_true, y_pred),
        custom_tweedie_hessian(y_true, y_pred),

I then create a regressor with this custom objective function and call the fit method like this:

self.xgb_model = xgb.XGBRegressor(
x_train =["train"]["X"]
y_train =["train"]["y"], y_train)

The results I obtain on my dataset are the following (using a custom evaluation method):

{'test_mse': 7578884.044259707,
 'test_r_squared': -0.12819942553491792,
 'train_mse': 6836209.780847064,
 'train_r_squared': -0.13481724642997617}

When learning another regressor with the exact same parameters, dataset and evaluation method but using the built-in tweedie objective, the results are the following:

{'test_mse': 4243417.3363827625,
 'test_r_squared': 0.3683211178249963,
 'train_mse': 2994838.0419956953,
 'train_r_squared': 0.5028540712950178}

Does someone know what could be wrong with my implementation ? It is supposed to be the same as the one in the source code as my implementation is based on it.

Thank you very much for your help :slight_smile:

1 Like

I can reproduce the issue. I checked the formulas and cross-checked with scikit-learns implementation for Tweedie loss and they are the same.