Build with existing libhdfs.a


#1

I tried to build master with HDFS enabled on one of production gateway machine. This point libhdfs to /opt/yarn/binary/lib/native/libhdfs.a and complain. Does it means I need to recompile libhdfs from source?

/usr/bin/ld: /opt/yarn/binary/lib/native/libhdfs.a(hdfs.c.o): relocation R_X86_64_32 against `.rodata.str1.1’ can not be used when making a shared object; recompile with -fPIC
/opt/yarn/binary/lib/native/libhdfs.a: error adding symbols: Bad value

Here is changes in my code

make/config.mk USE_HDFS=1
dmlc-core/make/config.mk USE_HDFS=1
jvm-packages/create_jni.py “USE_HDFS”: “ON”

Here is failed command in maven script.

cq@hadoopgw02-dca1:~/xgboost/jvm-packages$ cmake --build . --config Release
[ 7%] Built target rabit
[ 71%] Built target objxgboost
[ 92%] Built target dmlc
[ 95%] Built target runxgboost
[ 96%] Built target xgboost
[ 98%] Linking CXX shared library …/…/lib/libxgboost4j.so
/usr/bin/ld: /opt/yarn/binary/lib/native/libhdfs.a(hdfs.c.o): relocation R_X86_64_32 against `.rodata.str1.1’ can not be used when making a shared object; recompile with -fPIC
/opt/yarn/binary/lib/native/libhdfs.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
jvm-packages/CMakeFiles/xgboost4j.dir/build.make:172: recipe for target ‘…/lib/libxgboost4j.so’ failed
make[2]: *** […/lib/libxgboost4j.so] Error 1
CMakeFiles/Makefile2:351: recipe for target ‘jvm-packages/CMakeFiles/xgboost4j.dir/all’ failed
make[1]: *** [jvm-packages/CMakeFiles/xgboost4j.dir/all] Error 2
Makefile:129: recipe for target ‘all’ failed
make: *** [all] Error 2


#2

cq@hadoopgw02-dca1:~/xgboost/jvm-packages$ cmake … -DUSE_S3:BOOL=OFF -DUSE_OPENMP:BOOL=ON -DUSE_CUDA:BOOL=OFF -DUSE_AZURE:BOOL=OFF -DUSE_HDFS:BOOL=ON -DJVM_BINDINGS:BOOL=ON
– CMake version 3.12.3
– Architecture: x64
– HDFS_LIB_PATHS: /opt/yarn/binary/lib/native
– Hadoop 2.6.0-cdh5.7.2
Subversion Unknown -r Unknown
Compiled by root on 2019-03-18T04:54Z
Compiled with protoc 2.5.0
From source with checksum 9e5ffedfeea2de4cd586f0865c569f8
This command was run using /opt/hadoop/hadoop-cdh5-2.6.0_5.7.2-57/share/hadoop/common/hadoop-common-2.6.0-cdh5.7.2.jar
– HDFS_INCLUDE_DIR: /opt/yarn/binary/include
– HDFS_LIBRARIES: /opt/yarn/binary/lib/native/libhdfs.so
– hdfs_static: /opt/yarn/binary/lib/native/libhdfs.a
– /home/cq/xgboost/dmlc-core/cmake/build_config.h.in -> /home/cq/xgboost/dmlc-core/include/dmlc/build_config.h
– Configuring done
– Generating done
– Build files have been written to: /home/cq/xgboost/jvm-packages


#3

I found this https://github.com/dmlc/dmlc-core/issues/10 from long ago. will dig into and see if can solve the problem. wish we have better documentation.


#4

@chenqin I agree that we need better documentation. I’m assuming you are using upstream Spark and not CDH or other 3rd-party distributions?


#5

Actually, I tried to run on top of yarn example directly without spark at the moment.
looks like we do need to recompile hadoop-common/hadoop-hdfs/src with -fPIC flag to build with HDFS ON.

seems facing similar issue as

Container: container_e314_1557041842011_2389895_01_000001 on hadoopworker054-dca1.prod.uber.internal_8041

LogType:stderr
Log Upload Time:Fri May 24 20:58:06 +0000 2019
LogLength:940
Log Contents:
Error: A JNI error has occurred, please check your installation and try again
Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/io/DataOutputBuffer
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.DataOutputBuffer
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
… 7 more

LogType:stdout


#6

@chenqin any resolution for this issue? I’m facing a similar issue now.


#7

I gave up on this and use xgboost4j-spark, essentially use spark hdfs reader and feed into trainer with dataframe.


#8

Got it. I gave up on this too. The reason I was trying this was because a job that I have which works with xgboost 0.82, scala 2.11, spark 2.3.2 doesn’t work with xgboost 0.9, scala 2.12, spark 2.4.3. Run into an issue that suggests it is because of HDFS build.