Compiling xgboost on Centos7 against hdfs shared libraries

#1

Hi,
I’m trying to compile xgboost on centos7 with gcc-8 installed from scl-repo.
The xgboost version is 1.0.2.
the commands that I run are:
mkdir build; cd build
HADOOP_HDFS_HOME=/usr/hdp/3.1.0.0-78/hadoop
cmake3 … -DUSE_HDFS=ON -DHDFS_INCLUDE_DIR=/usr/hdp/3.1.0.0-78/usr/include
make

and then I have this error:
Scanning dependencies of target xgboost
[ 80%] Linking CXX shared library …/lib/libxgboost.so
/opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: /usr/hdp/3.1.0.0-78/hadoop/lib/native/libhdfs.a(hdfs.c.o): relocation R_X86_64_32 against .rodata.str1.1' can not be used when making a shared object; recompile with -fPIC /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: /usr/hdp/3.1.0.0-78/hadoop/lib/native/libhdfs.a(exception.c.o): relocation R_X86_64_32 against.rodata.str1.8’ can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: /usr/hdp/3.1.0.0-78/hadoop/lib/native/libhdfs.a(jni_helper.c.o): relocation R_X86_64_32 against hidden symbol hdfsHashMutex' can not be used when making a shared object /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: /usr/hdp/3.1.0.0-78/hadoop/lib/native/libhdfs.a(mutexes.c.o): relocation R_X86_64_32 against.rodata.str1.8’ can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: /usr/hdp/3.1.0.0-78/hadoop/lib/native/libhdfs.a(thread_local_storage.c.o): relocation R_X86_64_32 against `.rodata.str1.8’ can not be used when making a shared object; recompile with -fPIC
/opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/xgboost.dir/build.make:173: …/lib/libxgboost.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:115: CMakeFiles/xgboost.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

Which parameters should I change to make them use libhdfs.so?

#2

This is currently an open issue: https://github.com/dmlc/dmlc-core/issues/552. Right now, XGBoost requires a static lib for libhdfs.

#3

Ok, thank you.
up to 0.90 I was able to compile (without using cmake) with libhdfs.so.

On 0.90 I do this:
export HDFS_INC_PATH="/usr/hdp/$hdp_version/usr/include" # hdfs.h
export HDFS_LIB_PATH="/usr/hdp/$hdp_version/usr/lib" # libhdfs.so.*
sed -i ‘s/USE_HDFS = 0/USE_HDFS = 1/’ make/config.mk
sed -i ‘s/USE_HDFS = 0/USE_HDFS = 1/’ dmlc-core/make/config.mk

make

cd python-package
python3 setup.py bdist_wheel --universal

And this worked for dynamic linking. How do I do static linking to libhdfs.a?