No cuda xgboost with celery hangs

I’ve installed xgboost without cuda (pip install xgboost==1.5.0 --no-binary xgboost) on my celery worker.
When running the job via celery the code hangs
In comparison the same code running without celery works as expected.
Additionally running the same code in celery with cuda (installed normally with pip) also works.

I cannot figure out what might be the issue.

It could be an issue with how XGBoost was built on your machine. (Adding --no-binary xgboost causes XGBoost to be compiled from the source.)

Can you try installing XGBoost from Conda? See instructions at Conda lets you install XGBoost while opting out of CUDA.

I’m not keen to mix Conda and pip together in my environment.
In my dockerfile (from 3.9.7-slim-bullseye) I simply added --no-binary xgboost to compile:

RUN printf “# Testing repository - main, contrib and non-free branches\ndeb testing main contrib\ndeb-src testing main contrib\n” >> /etc/apt/sources.list
&& apt-get update
&& apt-get install --no-install-recommends -y -o APT::Immediate-Configure=0 -o APT::Install-Suggests=0

RUN pip install --no-cache-dir --requirement ./requirements.txt --no-binary xgboost

The installation seems to be ok and the file is 5.4M (without CUDA)

I think this has something to do with Celery, do any of you guys have a similar setup?

I won’t be of much help when it comes to celery.