How to set up a native XGBoost cluster

#1

I can find guidance about Distributed XGBoost with Yarn/Kubernetes/Spark/Dask at https://xgboost.readthedocs.io/en/latest/tutorials/index.html. Did xgboost support set up a native XGBoost cluster without 3rd party distributed framework, do we have any guidance about that?

#2

Hello @dding3, I’ve had the most success using AWS ParallelCluster and setting up a SLURM cluster. This is an example script for setting up a master for XGBoost using pcluster.

You’d need to re-enable support for the slurm tracker that was removed from dlmc-core. Also the above used to work about a year ago, not sure if it works any more.