Xgboost0.72 when tried to compile with spark2.3Cloudera3 errors out

#1

Hi,

I tried to compile xgboost0.72 with spark2.3cloudera3. It is giving me below error. Any suggestion to fix it?

[ERROR] cannot be applied to (java.net.URL)
[ERROR] mapper.readTree(url).findValues(“totalCores”).asScala.map(_.asInt).sum

#2

We have pre-built JARs available on Maven Central.

#3

Can you please point me to the pre-built jars for xgboost0.72 with spark2.3cloudera3. Because I didn’t see jars with this combination.

#4

This one? https://mvnrepository.com/artifact/ml.dmlc/xgboost4j-spark/0.72

#5

@CodingCat Are there multiple variants of Spark? If so, do we support only one of them?

#6

yes, companies like cloudera(hortonworks) have their own distro of Spark…some of them have breaking changes over vanilla version

We can only support vanilla version, that means Apache Spark from github.com/apache/spark

#7

@CodingCat That make sense. We may want to clarify that in the documentation. (“XGBoost4J-Spark requires vanilla Spark”) It would be nice if Cloudera provided their own build of XGBoost.

#8

we actually talked about that “XGBoost4J-Spark is a project aiming to seamlessly integrate XGBoost and Apache Spark by fitting XGBoost to Apache Spark’s MLLIB framework.”

#9

@CodingCat I see now, but previously it wasn’t clear to me (an outsider to Java world) that only vanilla Spark would be supported (or that there are incompatible variants of Spark). I submitted a small patch to make it super clear: https://github.com/dmlc/xgboost/pull/3854.