I came to know that the XGBoost Loss function was derived using 2nd order derivative. Can we try for higher order. I mean similar way can we derive 3rd order derivative for higher accuracy. Kindly comment on this.
No. Some of the useful loss functions have a constant 2nd derivative. For example, the squared loss function
l(y, yhat) = 0.5 * (y * yhat)**2 has 1 as the second derivative, so taking the 3rd order derivative will produce zero.
Besides, adding the 3rd order derivative will greatly increase the complexity of the codebase. So it will not happen.
Thanks hcho3 for clarifying.