XGBoost GPU Floats vs Doubles

Hi community,

  1. I wanted to know why the XGBoost code sticks with using floats as variables. Why not use doubles to enable higher precision data reading, boosting, scoring?

  2. I tried replacing all floats with doubles and building a 100 tree model using GPU hist on Higgs dataset. Found that using floats, the accuracy is around 73% (closer to CPU hist accuracy) and using doubles the accuracy drops to 69%. Why is this so that using doubles across the code, the accuracy drops so much? Is it something related to the way the algorithm works? (Because I expected that having higher precision, would lead to better accuracy)

We actually use double-precision floats in sums and accumulation variables, to prevent accuracy loss when accumulating floating-point values. For example:

Why use single-precision floats for everything else (data, gradient pairs, etc)? Mainly to save memory and increase computational performance. In particular, NVIDIA GPUs typically have a significant gap in performance with single-precision and double-precision arithmetic. Also, many people already complain that XGBoost uses lots of memory. Switching to double-precision arithmetic everywhere will make the problem even worse.