Build with existing libhdfs.a

chenqin · May 23, 2019, 9:42pm

I tried to build master with HDFS enabled on one of production gateway machine. This point libhdfs to /opt/yarn/binary/lib/native/libhdfs.a and complain. Does it means I need to recompile libhdfs from source?

/usr/bin/ld: /opt/yarn/binary/lib/native/libhdfs.a(hdfs.c.o): relocation R_X86_64_32 against `.rodata.str1.1’ can not be used when making a shared object; recompile with -fPIC
/opt/yarn/binary/lib/native/libhdfs.a: error adding symbols: Bad value

Here is changes in my code

make/config.mk USE_HDFS=1
dmlc-core/make/config.mk USE_HDFS=1
jvm-packages/create_jni.py “USE_HDFS”: “ON”

Here is failed command in maven script.

cq@hadoopgw02-dca1:~/xgboost/jvm-packages$ cmake --build . --config Release
[ 7%] Built target rabit
[ 71%] Built target objxgboost
[ 92%] Built target dmlc
[ 95%] Built target runxgboost
[ 96%] Built target xgboost
[ 98%] Linking CXX shared library …/…/lib/libxgboost4j.so
/usr/bin/ld: /opt/yarn/binary/lib/native/libhdfs.a(hdfs.c.o): relocation R_X86_64_32 against `.rodata.str1.1’ can not be used when making a shared object; recompile with -fPIC
/opt/yarn/binary/lib/native/libhdfs.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
jvm-packages/CMakeFiles/xgboost4j.dir/build.make:172: recipe for target ‘…/lib/libxgboost4j.so’ failed
make[2]: *** […/lib/libxgboost4j.so] Error 1
CMakeFiles/Makefile2:351: recipe for target ‘jvm-packages/CMakeFiles/xgboost4j.dir/all’ failed
make[1]: *** [jvm-packages/CMakeFiles/xgboost4j.dir/all] Error 2
Makefile:129: recipe for target ‘all’ failed
make: *** [all] Error 2

chenqin · May 23, 2019, 9:46pm

cq@hadoopgw02-dca1:~/xgboost/jvm-packages$ cmake … -DUSE_S3:BOOL=OFF -DUSE_OPENMP:BOOL=ON -DUSE_CUDA:BOOL=OFF -DUSE_AZURE:BOOL=OFF -DUSE_HDFS:BOOL=ON -DJVM_BINDINGS:BOOL=ON
– CMake version 3.12.3
– Architecture: x64
– HDFS_LIB_PATHS: /opt/yarn/binary/lib/native
– Hadoop 2.6.0-cdh5.7.2
Subversion Unknown -r Unknown
Compiled by root on 2019-03-18T04:54Z
Compiled with protoc 2.5.0
From source with checksum 9e5ffedfeea2de4cd586f0865c569f8
This command was run using /opt/hadoop/hadoop-cdh5-2.6.0_5.7.2-57/share/hadoop/common/hadoop-common-2.6.0-cdh5.7.2.jar
– HDFS_INCLUDE_DIR: /opt/yarn/binary/include
– HDFS_LIBRARIES: /opt/yarn/binary/lib/native/libhdfs.so
– hdfs_static: /opt/yarn/binary/lib/native/libhdfs.a
– /home/cq/xgboost/dmlc-core/cmake/build_config.h.in -> /home/cq/xgboost/dmlc-core/include/dmlc/build_config.h
– Configuring done
– Generating done
– Build files have been written to: /home/cq/xgboost/jvm-packages

chenqin · May 24, 2019, 6:10pm

I found this https://github.com/dmlc/dmlc-core/issues/10 from long ago. will dig into and see if can solve the problem. wish we have better documentation.

hcho3 · May 24, 2019, 8:31pm

@chenqin I agree that we need better documentation. I’m assuming you are using upstream Spark and not CDH or other 3rd-party distributions?

chenqin · May 25, 2019, 12:23am

Actually, I tried to run on top of yarn example directly without spark at the moment.
looks like we do need to recompile hadoop-common/hadoop-hdfs/src with -fPIC flag to build with HDFS ON.

seems facing similar issue as

github.com/dmlc/xgboost

Get NoClassDefFoundError when running xgboost on yarn demo

opened 09:04AM - 28 Sep 18 UTC

closed 05:18PM - 18 Oct 18 UTC

edwardzjl

environment java: java1.8.0 hadoop: hadoop-2.6.0-cdh5.11.1 CLASSPATH: /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce:/opt/cloudera/parcels/CDH/lib/hadoop/hadoop-common.jar:/usr/java/current/lib:.: LD_LIBRARY_PATH: /root/programs/xgboost/lib:/opt/cloudera/parcels/CDH/lib64:/usr/java/current/jre/lib/amd64/server: problem followed the instructions on tutorial after compiling, test/distributed/runtests.sh works fine, but demo job fails immediately after job submitted. job...

Container: container_e314_1557041842011_2389895_01_000001 on hadoopworker054-dca1.prod.uber.internal_8041

LogType:stderr
Log Upload Time:Fri May 24 20:58:06 +0000 2019
LogLength:940
Log Contents:
Error: A JNI error has occurred, please check your installation and try again
Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/io/DataOutputBuffer
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.DataOutputBuffer
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
… 7 more

LogType:stdout

kumar · September 4, 2019, 7:27pm

@chenqin any resolution for this issue? I’m facing a similar issue now.

chenqin · September 6, 2019, 7:02am

I gave up on this and use xgboost4j-spark, essentially use spark hdfs reader and feed into trainer with dataframe.

kumar · September 17, 2019, 4:44pm

Got it. I gave up on this too. The reason I was trying this was because a job that I have which works with xgboost 0.82, scala 2.11, spark 2.3.2 doesn’t work with xgboost 0.9, scala 2.12, spark 2.4.3. Run into an issue that suggests it is because of HDFS build.