Xgboost 1.7.6 Failed to initialize Rabit

#1

ERROR XGBoostSpark: XGBooster worker 16 has failed 0 times due to
ml.dmlc.xgboost4j.java.XGBoostError: [19:36:53] /workspace/src/collective/rabit_communicator.h:47: Failed to initialize Rabit
Stack trace:
[bt] (0) /opt/tiger/yodel/container/tmp/libxgboost4j2158174565782687335.so(+0x1638f9) [0x7ff54b1598f9]
[bt] (1) /opt/tiger/yodel/container/tmp/libxgboost4j2158174565782687335.so(xgboost::collective::RabitCommunicator::Create(xgboost::Json const&)+0x9a7) [0x7ff54b15ee47]
[bt] (2) /opt/tiger/yodel/container/tmp/libxgboost4j2158174565782687335.so(xgboost::collective::Communicator::Init(xgboost::Json const&)+0x1bc) [0x7ff54b159cfc]
[bt] (3) /opt/tiger/yodel/container/tmp/libxgboost4j2158174565782687335.so(XGCommunicatorInit+0x35) [0x7ff54b126df5]
[bt] (4) /opt/tiger/yodel/container/tmp/libxgboost4j2158174565782687335.so(Java_ml_dmlc_xgboost4j_java_XGBoostJNI_CommunicatorInit+0x550) [0x7ff54b117800]
[bt] (5) [0x7ff885018607]

at ml.dmlc.xgboost4j.java.Communicator.checkCall(Communicator.java:55)
at ml.dmlc.xgboost4j.java.Communicator.init(Communicator.java:82)
at ml.dmlc.xgboost4j.scala.spark.XGBoost$.buildDistributedBooster(XGBoost.scala:306)
#2

I encountered the exact same error log as you did, and the problem has been resolved. You can refer to the following:
SparkEnv: Spark 3.2.1 + Docker on Yarn
Network: IPv4 & IPv6 dual
XGBoost4j-Spark: 1.7.6

This problem may be related to the network environment. In my network environment, when using XGBoostClassifier.fit(), the task will fail if the Tracker host-ip is an IPv4 address, but the task will run stably when the host-ip is an IPv6 address. My solution is to obtain the current host’s IPv6 address during Spark task execution, write it to the xgboost-tracker.properties file, and then call XGBoostClassifier.fit().