Xgb on yarn foreach partition error

#1

Hi,

I’m trying to use XGboost on a quite large Dataset (~500G) on Yarn, and keep getting below error when running stage-1 foreach partition after successfully run stage 0-repartition and will keep retry step 0.

Stage Screenshot:

image
image1912×410 42.2 KB

I’m wondering is this because of the program doesn’t get enough resource from the cluster?

The program ran pretty well on a sample set, can someone take a look at it?

Thanks!!

[jvm-packages] xgboost4j external memory explain
#2

Had the same error, but got no clue yet. See [java-scala]Is XGBoost-Spark training thread-safe?
Really appreciate if somebody could provide some info!

#3

Hi jacob,

I solved my problem by setting larger memory on each task. I think you can try to either increase your memory per executor/ memory overhead, or decrease the number of tasks running parallel.

Jackie.