XGBoost 0.8.2 and Spark 2.4.0: Unable to Save Pipeline Model into AWS S3

#1

I’m trying to save an xgboost pipeline model and I’m gietting this error

xgboostPpipelineModel.write.overwrite().save(pipelineModelPath)

java.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/JsonInput;Z)Lorg/json4s/JsonAST$JValue;
  at ml.dmlc.xgboost4j.scala.spark.params.DefaultXGBoostParamsWriter$$anonfun$1$$anonfun$3.apply(DefaultXGBoostParamsWriter.scala:73)
  at ml.dmlc.xgboost4j.scala.spark.params.DefaultXGBoostParamsWriter$$anonfun$1$$anonfun$3.apply(DefaultXGBoostParamsWriter.scala:71)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
  at scala.collection.AbstractTraversable.map(Traversable.scala:104)
  at ml.dmlc.xgboost4j.scala.spark.params.DefaultXGBoostParamsWriter$$anonfun$1.apply(DefaultXGBoostParamsWriter.scala:71)
  at ml.dmlc.xgboost4j.scala.spark.params.DefaultXGBoostParamsWriter$$anonfun$1.apply(DefaultXGBoostParamsWriter.scala:69)
  at scala.Option.getOrElse(Option.scala:121)
  at ml.dmlc.xgboost4j.scala.spark.params.DefaultXGBoostParamsWriter$.getMetadataToSave(DefaultXGBoostParamsWriter.scala:69)
  at ml.dmlc.xgboost4j.scala.spark.params.DefaultXGBoostParamsWriter$.saveMetadata(DefaultXGBoostParamsWriter.scala:51)
  at ml.dmlc.xgboost4j.scala.spark.XGBoostClassificationModel$XGBoostClassificationModelWriter.saveImpl(XGBoostClassifier.scala:494)
  at org.apache.spark.ml.util.MLWriter.save(ReadWrite.scala:180)
  at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$saveImpl$1.apply(Pipeline.scala:254)
  at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$saveImpl$1.apply(Pipeline.scala:253)
  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
  at org.apache.spark.ml.Pipeline$SharedReadWrite$.saveImpl(Pipeline.scala:253)
  at org.apache.spark.ml.PipelineModel$PipelineModelWriter.saveImpl(Pipeline.scala:338)
  at org.apache.spark.ml.util.MLWriter.save(ReadWrite.scala:180)

Has any one an idea of what could cause this issue ?
Thanks

#2

I don’t think XGBoost 0.82 is compatible with Spark 2.4.x. Can you use Spark 2.3.3 instead?

The next upcoming version of XGBoost (0.90) is planned to work with Spark 2.4.3.

#3

Thanks @hcho3 for your answser.
I used Spark 2.4.x because the documentation said this XGBoost4J-Spark now requires Apache Spark 2.4+.

14%20AM
14%20AM1668×548 74 KB

#4

You should look at the doc for the stable version:
https://xgboost.readthedocs.io/en/release_0.82/jvm/xgboost4j_spark_tutorial.html

#5

Downgrading Spark version from 2.4.0 to 2.3.3 solved the problem.
Thank you @hcho3 :+1: