When I just run XGBoost on a server with 24 cores, it uses almost all 24 cores. It uses a process called ZMQbg/1 to run 20+ cores. The “top” command shows one python and one ZMQbg/1 processes. The ZMQbg/1 has like over 2200% CPU utilization. However, if I use cross_val_score to wrap around XGBoost, it shows only 4 python processes, but no ZMQbg/1. I have set n_jobs=-1 for both XGBoost and cross_val_score.
How do I enable the system to run 20+ cores when I use XGBoost and cross_val_score?
If I use cross_val_score with RandomForest, the top command shows 10+ python processes. Somehow with cross_val_score and XGBoost, it has only 4 python proccesses.