XGBoost parallelization issue on macOS High Sierra

macOs version: 10.13.5 (17F77)

XGBoost does not work in parallel. I set nthread to 8, but nothing changes. I searched on the internet and learn that it is common problem. This is what I did so far.

I installed gcc by using brew. So, I have gcc-8 now. I followed official documentation, but I still could not make it work.

I saw this example here.

#include <iostream>
#include <cstdint>
#include <omp.h>

    int main(void) {
  const uint64_t N = 100000000000;
  uint64_t sum[8] = {0};
  #pragma omp parallel for
  for (uint64_t i = 0; i < N; ++i) {
    sum[omp_get_thread_num()] += i;
  }
  for (uint64_t i = 0; i < omp_get_max_threads(); ++i) {
    std::cout << sum[i] << std::endl;
  }
  return 0;
}

I tried to compile it by running gcc -o main main.cc -fopenmp -O3 -msse2. It did not work.

Then I tried g++ -o main main.cc -fopenmp -O3 -msse2 . This one worked.

What am I doing wrong?

You can also see the question on stackoverflow that I opened today. That includes more info.

Try using CMake.

mkdir build
cd build
CC=gcc-8 CXX=g++-8 cmake ..
make

I tried it and it still does not work. Is there any way to see why?

➜ build git:(master) CC=gcc-8 CXX=g+±8 cmake …
– The C compiler identification is GNU 8.2.0
– The CXX compiler identification is GNU 8.2.0
– Checking whether C compiler has -isysroot
– Checking whether C compiler has -isysroot - yes
– Checking whether C compiler supports OSX deployment target flag
– Checking whether C compiler supports OSX deployment target flag - yes
– Check for working C compiler: /usr/local/bin/gcc-8
– Check for working C compiler: /usr/local/bin/gcc-8 – works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Detecting C compile features
– Detecting C compile features - done
– Checking whether CXX compiler has -isysroot
– Checking whether CXX compiler has -isysroot - yes
– Checking whether CXX compiler supports OSX deployment target flag
– Checking whether CXX compiler supports OSX deployment target flag - yes
– Check for working CXX compiler: /usr/local/bin/g+±8
– Check for working CXX compiler: /usr/local/bin/g+±8 – works
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Detecting CXX compile features
– Detecting CXX compile features - done
– Found OpenMP_C: -fopenmp (found version “4.5”)
– Found OpenMP_CXX: -fopenmp (found version “4.5”)
– Found OpenMP: TRUE (found version “4.5”)
– Setting build type to ‘Release’ as none was specified.
– Looking for clock_gettime in rt
– Looking for clock_gettime in rt - not found
– Looking for fopen64
– Looking for fopen64 - not found
– /Users/ugurcanlacin/xgboost/dmlc-core/cmake/build_config.h.in -> /Users/ugurcanlacin/xgboost/dmlc-core/include/dmlc/build_config.h
– Performing Test SUPPORT_CXX11
– Performing Test SUPPORT_CXX11 - Success
– Performing Test SUPPORT_CXX0X
– Performing Test SUPPORT_CXX0X - Success
– Performing Test SUPPORT_MSSE2
– Performing Test SUPPORT_MSSE2 - Success
– Found OpenMP_C: -fopenmp (found version “4.5”)
– Found OpenMP_CXX: -fopenmp (found version “4.5”)
– Configuring done
– Generating done
– Build files have been written to: /Users/ugurcanlacin/xgboost/build

After that I ran ‘make’ command.

➜ build git:(master) make
Scanning dependencies of target rabit
[ 1%] Building CXX object CMakeFiles/rabit.dir/rabit/src/allreduce_base.cc.o
[ 3%] Building CXX object CMakeFiles/rabit.dir/rabit/src/allreduce_robust.cc.o
[ 4%] Building CXX object CMakeFiles/rabit.dir/rabit/src/engine.cc.o
[ 6%] Building CXX object CMakeFiles/rabit.dir/rabit/src/c_api.cc.o
[ 8%] Linking CXX static library librabit.a
[ 8%] Built target rabit
Scanning dependencies of target dmlc
[ 9%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/src/build_config.cc.o
/Users/ugurcanlacin/xgboost/dmlc-core/src/build_config.cc:14:6: warning: #warning “Redefining fopen64 with std::fopen” [-Wcpp]
#warning “Redefining fopen64 with std::fopen”
^~~~~~~
[ 11%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/src/config.cc.o
[ 13%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/src/data.cc.o
[ 14%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/src/io.cc.o
[ 16%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/src/recordio.cc.o
[ 18%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/src/io/line_split.cc.o
[ 19%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/src/io/recordio_split.cc.o
[ 21%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/src/io/indexed_recordio_split.cc.o
[ 22%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/src/io/input_split_base.cc.o
[ 24%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/src/io/filesys.cc.o
[ 26%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/src/io/local_filesys.cc.o
[ 27%] Linking CXX static library libdmlc.a
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libdmlc.a(build_config.cc.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libdmlc.a(build_config.cc.o) has no symbols
[ 27%] Built target dmlc
Scanning dependencies of target objxgboost
[ 29%] Building CXX object CMakeFiles/objxgboost.dir/src/c_api/c_api.cc.o
[ 31%] Building CXX object CMakeFiles/objxgboost.dir/src/c_api/c_api_error.cc.o
[ 32%] Building CXX object CMakeFiles/objxgboost.dir/src/common/common.cc.o
[ 34%] Building CXX object CMakeFiles/objxgboost.dir/src/common/hist_util.cc.o
[ 36%] Building CXX object CMakeFiles/objxgboost.dir/src/common/host_device_vector.cc.o
[ 37%] Building CXX object CMakeFiles/objxgboost.dir/src/data/data.cc.o
[ 39%] Building CXX object CMakeFiles/objxgboost.dir/src/data/simple_csr_source.cc.o
[ 40%] Building CXX object CMakeFiles/objxgboost.dir/src/data/simple_dmatrix.cc.o
[ 42%] Building CXX object CMakeFiles/objxgboost.dir/src/data/sparse_page_dmatrix.cc.o
[ 44%] Building CXX object CMakeFiles/objxgboost.dir/src/data/sparse_page_raw_format.cc.o
[ 45%] Building CXX object CMakeFiles/objxgboost.dir/src/data/sparse_page_source.cc.o
[ 47%] Building CXX object CMakeFiles/objxgboost.dir/src/data/sparse_page_writer.cc.o
[ 49%] Building CXX object CMakeFiles/objxgboost.dir/src/gbm/gblinear.cc.o
[ 50%] Building CXX object CMakeFiles/objxgboost.dir/src/gbm/gbm.cc.o
[ 52%] Building CXX object CMakeFiles/objxgboost.dir/src/gbm/gbtree.cc.o
[ 54%] Building CXX object CMakeFiles/objxgboost.dir/src/learner.cc.o
[ 55%] Building CXX object CMakeFiles/objxgboost.dir/src/linear/linear_updater.cc.o
[ 57%] Building CXX object CMakeFiles/objxgboost.dir/src/linear/updater_coordinate.cc.o
[ 59%] Building CXX object CMakeFiles/objxgboost.dir/src/linear/updater_shotgun.cc.o
[ 60%] Building CXX object CMakeFiles/objxgboost.dir/src/logging.cc.o
[ 62%] Building CXX object CMakeFiles/objxgboost.dir/src/metric/elementwise_metric.cc.o
[ 63%] Building CXX object CMakeFiles/objxgboost.dir/src/metric/metric.cc.o
[ 65%] Building CXX object CMakeFiles/objxgboost.dir/src/metric/multiclass_metric.cc.o
[ 67%] Building CXX object CMakeFiles/objxgboost.dir/src/metric/rank_metric.cc.o
[ 68%] Building CXX object CMakeFiles/objxgboost.dir/src/objective/hinge.cc.o
[ 70%] Building CXX object CMakeFiles/objxgboost.dir/src/objective/multiclass_obj.cc.o
[ 72%] Building CXX object CMakeFiles/objxgboost.dir/src/objective/objective.cc.o
[ 73%] Building CXX object CMakeFiles/objxgboost.dir/src/objective/rank_obj.cc.o
[ 75%] Building CXX object CMakeFiles/objxgboost.dir/src/objective/regression_obj.cc.o
[ 77%] Building CXX object CMakeFiles/objxgboost.dir/src/predictor/cpu_predictor.cc.o
[ 78%] Building CXX object CMakeFiles/objxgboost.dir/src/predictor/predictor.cc.o
[ 80%] Building CXX object CMakeFiles/objxgboost.dir/src/tree/split_evaluator.cc.o
[ 81%] Building CXX object CMakeFiles/objxgboost.dir/src/tree/tree_model.cc.o
[ 83%] Building CXX object CMakeFiles/objxgboost.dir/src/tree/tree_updater.cc.o
[ 85%] Building CXX object CMakeFiles/objxgboost.dir/src/tree/updater_colmaker.cc.o
[ 86%] Building CXX object CMakeFiles/objxgboost.dir/src/tree/updater_histmaker.cc.o
[ 88%] Building CXX object CMakeFiles/objxgboost.dir/src/tree/updater_prune.cc.o
[ 90%] Building CXX object CMakeFiles/objxgboost.dir/src/tree/updater_quantile_hist.cc.o
[ 91%] Building CXX object CMakeFiles/objxgboost.dir/src/tree/updater_refresh.cc.o
[ 93%] Building CXX object CMakeFiles/objxgboost.dir/src/tree/updater_skmaker.cc.o
[ 95%] Building CXX object CMakeFiles/objxgboost.dir/src/tree/updater_sync.cc.o
[ 95%] Built target objxgboost
Scanning dependencies of target runxgboost
[ 96%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o
[ 98%] Linking CXX executable …/xgboost
[ 98%] Built target runxgboost
Scanning dependencies of target xgboost
[100%] Linking CXX shared library …/lib/libxgboost.dylib
[100%] Built target xgboost

Finally, I ran install script as mentioned in official documentation

➜ xgboost git:(master) cd python-package; sudo python setup.py install

.
.
some output
.
.
creating ‘dist/xgboost-0.81-py3.6.egg’ and adding ‘build/bdist.macosx-10.9-x86_64/egg’ to it
removing ‘build/bdist.macosx-10.9-x86_64/egg’ (and everything under it)
Processing xgboost-0.81-py3.6.egg
removing ‘/Users/ugurcanlacin/anaconda3/lib/python3.6/site-packages/xgboost-0.81-py3.6.egg’ (and everything under it)
creating /Users/ugurcanlacin/anaconda3/lib/python3.6/site-packages/xgboost-0.81-py3.6.egg
Extracting xgboost-0.81-py3.6.egg to /Users/ugurcanlacin/anaconda3/lib/python3.6/site-packages
xgboost 0.81 is already the active version in easy-install.pth

Installed /Users/ugurcanlacin/anaconda3/lib/python3.6/site-packages/xgboost-0.81-py3.6.egg
Processing dependencies for xgboost==0.81
Searching for scipy==1.1.0
Best match: scipy 1.1.0
Adding scipy 1.1.0 to easy-install.pth file

Using /Users/ugurcanlacin/anaconda3/lib/python3.6/site-packages
Searching for numpy==1.15.4
Best match: numpy 1.15.4
Adding numpy 1.15.4 to easy-install.pth file

Using /Users/ugurcanlacin/anaconda3/lib/python3.6/site-packages
Finished processing dependencies for xgboost==0.81

Can you run make VERBOSE=1 in the build directory?

The output is huge. So, I pasted it here.

https://paste.ofcode.org/sdNNBvNzAM8DLWWyaMNgRz

I can not see any anomaly :confused:

The build flag -fopenmp is there. Sorry that I am unable to help you. I too use MacBook and I get multi-threading just fine.

Can you compile the example snippet with
/usr/local/bin/g++-8 -o main main.cc -fopenmp -O3 -msse2?

Thanks a lot for your help! Yes, it also works.

(xgboost) ➜ ~ /usr/local/bin/g+±8 -o main main.cc -fopenmp -O3 -msse2
(xgboost) ➜ ~ ./main
4338023698911793536
13014071109235380608
3243374445849416064
11919421856173003136
2148725192787038592
10824772603110625664
1054075939724661120
9730123350048248192
(xgboost) ➜ ~

Is there any way to run the algorithm in debug mode and see its logs? Otherwise, I can put some simple print calls or log calls to see running actions. But I do not know where to put. Can you point it?

Edit: I tried to use make -j8 and it made it with 8 thread as well. But, when it comes to run the algorithm with nthread=8, it does only 1 thread.

Try setting environment variable OMP_NUM_THREADS=8

I tried setting that, but it still does not run in parallel, only one thread.

Sorry, I’m out of ideas. You may consider using a Linux machine instead.

Thanks for your help anyway :slight_smile: I will use Linux for it.