Build with existing libhdfs.a


#1

I tried to build master with HDFS enabled on one of production gateway machine. This point libhdfs to /opt/yarn/binary/lib/native/libhdfs.a and complain. Does it means I need to recompile libhdfs from source?

/usr/bin/ld: /opt/yarn/binary/lib/native/libhdfs.a(hdfs.c.o): relocation R_X86_64_32 against `.rodata.str1.1’ can not be used when making a shared object; recompile with -fPIC
/opt/yarn/binary/lib/native/libhdfs.a: error adding symbols: Bad value

Here is changes in my code

make/config.mk USE_HDFS=1
dmlc-core/make/config.mk USE_HDFS=1
jvm-packages/create_jni.py “USE_HDFS”: “ON”

Here is failed command in maven script.

cq@hadoopgw02-dca1:~/xgboost/jvm-packages$ cmake --build . --config Release
[ 7%] Built target rabit
[ 71%] Built target objxgboost
[ 92%] Built target dmlc
[ 95%] Built target runxgboost
[ 96%] Built target xgboost
[ 98%] Linking CXX shared library …/…/lib/libxgboost4j.so
/usr/bin/ld: /opt/yarn/binary/lib/native/libhdfs.a(hdfs.c.o): relocation R_X86_64_32 against `.rodata.str1.1’ can not be used when making a shared object; recompile with -fPIC
/opt/yarn/binary/lib/native/libhdfs.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
jvm-packages/CMakeFiles/xgboost4j.dir/build.make:172: recipe for target ‘…/lib/libxgboost4j.so’ failed
make[2]: *** […/lib/libxgboost4j.so] Error 1
CMakeFiles/Makefile2:351: recipe for target ‘jvm-packages/CMakeFiles/xgboost4j.dir/all’ failed
make[1]: *** [jvm-packages/CMakeFiles/xgboost4j.dir/all] Error 2
Makefile:129: recipe for target ‘all’ failed
make: *** [all] Error 2


#2

cq@hadoopgw02-dca1:~/xgboost/jvm-packages$ cmake … -DUSE_S3:BOOL=OFF -DUSE_OPENMP:BOOL=ON -DUSE_CUDA:BOOL=OFF -DUSE_AZURE:BOOL=OFF -DUSE_HDFS:BOOL=ON -DJVM_BINDINGS:BOOL=ON
– CMake version 3.12.3
– Architecture: x64
– HDFS_LIB_PATHS: /opt/yarn/binary/lib/native
– Hadoop 2.6.0-cdh5.7.2
Subversion Unknown -r Unknown
Compiled by root on 2019-03-18T04:54Z
Compiled with protoc 2.5.0
From source with checksum 9e5ffedfeea2de4cd586f0865c569f8
This command was run using /opt/hadoop/hadoop-cdh5-2.6.0_5.7.2-57/share/hadoop/common/hadoop-common-2.6.0-cdh5.7.2.jar
– HDFS_INCLUDE_DIR: /opt/yarn/binary/include
– HDFS_LIBRARIES: /opt/yarn/binary/lib/native/libhdfs.so
– hdfs_static: /opt/yarn/binary/lib/native/libhdfs.a
– /home/cq/xgboost/dmlc-core/cmake/build_config.h.in -> /home/cq/xgboost/dmlc-core/include/dmlc/build_config.h
– Configuring done
– Generating done
– Build files have been written to: /home/cq/xgboost/jvm-packages


#3

I found this https://github.com/dmlc/dmlc-core/issues/10 from long ago. will dig into and see if can solve the problem. wish we have better documentation.


#4

@chenqin I agree that we need better documentation. I’m assuming you are using upstream Spark and not CDH or other 3rd-party distributions?


#5

Actually, I tried to run on top of yarn example directly without spark at the moment.
looks like we do need to recompile hadoop-common/hadoop-hdfs/src with -fPIC flag to build with HDFS ON.

seems facing similar issue as

Container: container_e314_1557041842011_2389895_01_000001 on hadoopworker054-dca1.prod.uber.internal_8041

LogType:stderr
Log Upload Time:Fri May 24 20:58:06 +0000 2019
LogLength:940
Log Contents:
Error: A JNI error has occurred, please check your installation and try again
Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/io/DataOutputBuffer
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.DataOutputBuffer
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
… 7 more

LogType:stdout


#6

@chenqin any resolution for this issue? I’m facing a similar issue now.


#7

I gave up on this and use xgboost4j-spark, essentially use spark hdfs reader and feed into trainer with dataframe.