Multithreading doesn't work on linux (openSUSE) in the latest version of xgboost

#1

I am trying to run xgboost with multithreading from a conda/python jupyter notebook. Previously, I did so on Mac OS by installing libomp with homebrew, followed by pip install xgboost. However, after moving to a linux machine that runs openSUSE Leap 15.1, I can’t get the multithreading to work.

Things I tried:

Installing libgomp1 from openSUSE repos
Installing homebrew, then running brew install libomp
Instaling various gcc compilers
Building xgboost from source (by cloning xgboost github, running cmake and pip install -e .)
Installing xgboost with pip
Installing xgboost with conda

While all of the above seems to complete without raising errors, the multi-threading still doesn’t work.

Not sure what to do from here. I feel that any advice you may have would be helpful at this point. Thanks!

UPDATE: whether the multithreading works or not seems to be dependent on the version of xgboost (see below)

#2

Quick update: I got the multithreading to work by installing xgboost via “conda install -c conda-forge/label/gcc7 xgboost”, the gcc7 label being key apparently. The only downside to this is that it installs v0.81, as opposed to v1.0.2 (the latest available via conda, or v1.1.0 available via pip. Is that something I should be concerned about? I am only using XGBClassifier module, has it gone through a lot of changes between those versions? Thank you.

#3

That’s odd. pip install xgboost should give you XGBoost with multi-threading, out of the box.

You can try running the following commands to compile v1.0.2 from the source:

git clone --recursive https://github.com/dmlc/xgboost -b release_1.0.0
cd xgboost
mkdir build
cd build
cmake ..
make
cd ../python-package
python3 setup.py install
1 Like
#4

Hey thanks for your reply. I tried what you suggest with an exception of the last line, where I did “python setup.py develop”. It installed and I was able to import it in jupyter but alas – no multithreading. This problem might be something specific to openSUSE but I was wondering if other people have encountered it on other linux distributions.

#5

Can you paste the log from cmake .. command? The log should say whether OpenMP was enabled or not.

1 Like
#6

Here’s what it says:

image

Also, I sort of confirmed that whether or not multithreading works is dependent on the xgboost version in this case. Multithreading works, if I do “pip install xgboost==0.81”. Version 0.90 works as well but not v1.0.0

Now I just need to figure out what is different about the olderer versions. Could it be that my libomp is outdated?

#7

Can you run make VERBOSE=1 and upload the log here? I want to see if the OpenMP library is properly being linked.

I’ve never personally seen multithreading broken in Linux environment. If it makes any difference, I use Ubuntu distribution.

1 Like
#8

@nijshar28 I just tried creating a fresh EC2 machine (using AWS) running OpenSUSE, and I cannot reproduce your issue.

Here are the steps I took:

  • Go to EC2 console and launch a new instance. Choose c5.4xlarge instance type and choose OpenSUSE Leap OS from the marketplace.
  • SSH into the EC2 machine.
  • Install necessary system packages:
sudo zypper refresh
# Install git, gcc, libgomp and more
sudo zypper install git-core make gcc gcc-c++ libgomp1 glibc htop tmux python3-pip
# Install CMake
wget https://github.com/Kitware/CMake/releases/download/v3.16.5/cmake-3.16.5-Linux-x86_64.sh
sudo bash cmake-3.16.5-Linux-x86_64.sh --prefix=/usr --skip-license
  • Compile XGBoost from the source:
git clone --recursive https://github.com/dmlc/xgboost -b release_1.0.0
cd xgboost/
mkdir build
cd build
cmake ..
make -j32 VERBOSE=1
  • Install XGBoost Python package:
cd ../python-package/
python3 setup.py install --user
  • Install scikit-learn:
python3 -m pip install scikit-learn numpy scipy
  • Run a small test script to verify multithreading:
import xgboost as xgb
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1000000)
dtrain = xgb.DMatrix(X, label=y)
print('Finished generating dataset')

param = {'max_depth': 2, 'eta': 0.001, 'objective': 'binary:logistic', 'eval_metric': 'auc'}
evallist = []
num_round = 100
print('Start training')
bst = xgb.train(param, dtrain, num_round, evallist)

Screenshot:

Cap%202020-03-20%2004-29-49-369
Cap%202020-03-20%2004-29-49-3691664×1300 163 KB

Note that all threads are being used.

1 Like
#9

This is helpful. Thank you.

I noticed that I am missing a few packages you mention in your EC2 setup (like htop and tux). So I ran “sudo zypper install git-core make gcc gcc-c++ libgomp1 glibc htop tmux python3-pip”. Now it gives me the following error message when I run cmake:

image
image1064×698 153 KB

I tried removing linuxbrew, and reinstalling gcc, gcc-c++, and cmake. Hoping it would fix the problem but to no avail. Any advice on how I can repair the symbolic links? Thank you.

#10

UPDATE: I deleted all past xgboost / gcc installs and went through your setup instructions from start to finish. Here’s the cmake log from that:

– The CXX compiler identification is GNU 9.3.0
– The C compiler identification is GNU 9.3.0
– Check for working CXX compiler: /usr/bin/c++
– Check for working CXX compiler: /usr/bin/c++ – works
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Detecting CXX compile features
– Detecting CXX compile features - done
– Check for working C compiler: /usr/bin/cc
– Check for working C compiler: /usr/bin/cc – works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Detecting C compile features
– Detecting C compile features - done
– CMake version 3.16.5
– Performing Test XGBOOST_MM_PREFETCH_PRESENT
– Performing Test XGBOOST_MM_PREFETCH_PRESENT - Success
– Performing Test XGBOOST_BUILTIN_PREFETCH_PRESENT
– Performing Test XGBOOST_BUILTIN_PREFETCH_PRESENT - Success
– xgboost VERSION: 1.1.0
– Setting build type to ‘Release’ as none was specified.
– Found OpenMP_C: -fopenmp (found version “4.5”)
– Found OpenMP_CXX: -fopenmp (found version “4.5”)
– Found OpenMP: TRUE (found version “4.5”)
– Found OpenMP_C: -fopenmp (found version “4.5”)
– Found OpenMP_CXX: -fopenmp (found version “4.5”)
– Looking for clock_gettime in rt
– Looking for clock_gettime in rt - found
– Looking for fopen64
– Looking for fopen64 - not found
– Looking for C++ include cxxabi.h
– Looking for C++ include cxxabi.h - found
– Looking for nanosleep
– Looking for nanosleep - found
– Looking for backtrace
– Looking for backtrace - found
– backtrace facility detected in default set of libraries
– Found Backtrace: /usr/include
– Check if the system is big endian
– Searching 16 bit integer
– Looking for sys/types.h
– Looking for sys/types.h - found
– Looking for stdint.h
– Looking for stdint.h - found
– Looking for stddef.h
– Looking for stddef.h - found
– Check size of unsigned short
– Check size of unsigned short - done
– Using unsigned short
– Check if the system is big endian - little endian
– /home/arseniy/xgboost/dmlc-core/cmake/build_config.h.in -> include/dmlc/build_config.h
– Performing Test SUPPORT_CXX11
– Performing Test SUPPORT_CXX11 - Success
– Performing Test SUPPORT_CXX0X
– Performing Test SUPPORT_CXX0X - Success
– Performing Test SUPPORT_MSSE2
– Performing Test SUPPORT_MSSE2 - Success
– Looking for pthread.h
– Looking for pthread.h - found
– Performing Test CMAKE_HAVE_LIBC_PTHREAD
– Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
– Looking for pthread_create in pthreads
– Looking for pthread_create in pthreads - not found
– Looking for pthread_create in pthread
– Looking for pthread_create in pthread - found
– Found Threads: TRUE
DMLC_ROOT point to /home/arseniy/xgboost/rabit/…/dmlc-core
– Configuring done
– Generating done

Despite the “– Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed” flag, the multithreading works now in all versions, including pip installed ones. Thank you so much for your help.

I think there also might have been an overlaying issue of me using the n_jobs parameter in XGBClassifier. If I set n_jobs = -1 in the newer versions of xgboost, the multithreading still doesn’t seem to work. It wasn’t deprecated, was it?

In any case, the instructions you posted above for setting up the EC2 machine should be the universally accepted xgboost installation instructions for linux.