Can save, but cannot load XGBoost on Spark Scala

Hi there,

I am training Regressor model and then save it to S3. After that I load it into a different variable just to test that it works. Unfortunately, it cannot load:

An error was encountered:
java.lang.IllegalArgumentException: Cannot recognize JSON metadata: {"class":"ml.dmlc.xgboost4j.scala.spark.XGBoostRegressionModel","timestamp":1649302195064,"sparkVersion":"3.0.1-amzn-0.1","uid":"xgbr_514111fca809","paramMap":{"baseScore":0.5,"customEval":null,"customObj":null,"gamma":0.0,"killSparkContextOnWorkerFailure":true,"objective":"rank:pairwise","maxBin":256,"eta":0.05,"allowNonZeroForMissing":false,"useExternalMemory":false,"evalMetric":"ndcg@3","rateDrop":0.0,"maxDepth":6,"colsampleBylevel":1.0,"subsample":0.75,"normalizeType":"tree","batchSize":32768,"maxDeltaStep":0.0,"checkpointInterval":-1,"trainTestRatio":1.0,"scalePosWeight":1.0,"labelCol":"label","missing":"NaN","treeMethod":"auto","cacheTrainingSet":false,"numRound":1,"rabitTimeout":-1,"trackerConf":{"workerConnectionTimeout":0,"trackerImpl":"python"},"seed":0,"predictionCol":"prediction","numWorkers":32,"sketchEps":0.03,"verbosity":2,"alpha":0.0,"timeoutRequestWorkers":1800000,"silent":0,"checkpointPath":"","nthread":1,"skipDrop":0.0,"rabitRingReduceThreshold":32768,"colsampleBytree":0.9,"growPolicy":"depthwise","treeLimit":0,"sampleType":"uniform","numEarlyStoppingRounds":0,"dmlcWorkerConnectRetry":5,"featuresCol":"features","lambdaBias":0.0,"lambda":1.0,"groupCol":"group","minChildWeight":1.0}}.
  at org.apache.spark.ml.util.DefaultParamsReader$Metadata.setParams(ReadWrite.scala:574)
  at org.apache.spark.ml.util.DefaultParamsReader$Metadata.getAndSetParams(ReadWrite.scala:549)
  at org.apache.spark.ml.util.DefaultParamsReader.load(ReadWrite.scala:469)
  at org.apache.spark.ml.util.MLReadable.load(ReadWrite.scala:355)
  at org.apache.spark.ml.util.MLReadable.load$(ReadWrite.scala:355)
  at ml.dmlc.xgboost4j.scala.spark.XGBoostRegressor$.load(XGBoostRegressor.scala:208)
  ... 53 elided

The version I use is 1.3.1.
Why this error happens and what can I do about it?

For some reason, if I try to save model directly via calling:

xgboostModel.write.overwrite().save("s3path")

it the fails to load with an above exception.

But if I add the model as a part of a pipeline and store the whole pipeline:

val pipeline = new Pipeline().setStages(Array(vectorAssembler, xgbRanker))
val model = pipeline.fit(train_dataset_with_group)
model.write.overwrite().save(s"s3path")

Then it is saved and loaded without any errors… Weird, but works.