Feature_weights does not work properly

#1

Hi there, recently I was playing with the feature_weight parameter to see to reduce the impact of specific features in my model. My understanding is the lower you set the corresponding value in feature_weights for the feature, the less possible it is used as a split in the tree. So I assume by setting the value to 0, it will be equivalent to dropping that feature. However, after I set the feature weight to 0 for that variable, that variable still has a non-zero value from the feature_importances_ variable of the model. Could anyone with more experience with this help me explain?

Thanks!!

#2

Hi! Same problem. The only way I found in order to solve it is to use the new version of xgboost
pip install xgboost==2.0.0

But it seems that this version does not support GPUs

Please let me know if you found another way. Thanks!

#3

@enricorox The latest version of XGBoost should support NVIDIA GPUs. Can you elaborate?

#4

Sure. I have an environment made like this:
conda create -y -n xgb python==3.8
conda activate xgb
yes | pip install xgboost==2.0.0 matplotlib numpy

OS: Ubuntu 20.04
GPU: NVIDIA GeForce GTX 1050 Ti Mobile
Driver: nvidia-driver-525

Take the feature weight example (https://xgboost.readthedocs.io/en/stable/python/examples/feature_weights.html). Now changing the parameter “tree_method” from “hist” to “gpu_hist” or, equivalently, adding “device”: “cuda”, I get /workspace/src/common/algorithm.h:78: Check failed: ctx->IsCPU()

Error

/home/enrico/.miniconda3/envs/xgb/bin/python /home/enrico/PycharmProjects/dna-classifier/feature_weights.py
[09:53:10] INFO: /workspace/src/data/simple_dmatrix.cc:137: Generating new Ellpack page.
Traceback (most recent call last):
File “/home/enrico/PycharmProjects/dna-classifier/feature_weights.py”, line 64, in
main(args)
File “/home/enrico/PycharmProjects/dna-classifier/feature_weights.py”, line 35, in main
bst = xgboost.train(
File “/home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/core.py”, line 729, in inner_f
return func(**kwargs)
File “/home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/training.py”, line 181, in train
bst.update(dtrain, i, obj)
File “/home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/core.py”, line 2049, in update
_check_call(
File “/home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/core.py”, line 281, in _check_call
raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [09:53:10] /workspace/src/tree/updater_gpu_hist.cu:781: Exception in gpu_hist: [09:53:10] /workspace/src/common/algorithm.h:78: Check failed: ctx->IsCPU():
Stack trace:
[bt] (0) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x1ba0be) [0x7f1b681920be]
[bt] (1) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x246f7e) [0x7f1b6821ef7e]
[bt] (2) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x24a1fa) [0x7f1b682221fa]
[bt] (3) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x243485) [0x7f1b6821b485]
[bt] (4) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x6ee0b4) [0x7f1b686c60b4]
[bt] (5) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0xb42885) [0x7f1b68b1a885]
[bt] (6) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0xb46738) [0x7f1b68b1e738]
[bt] (7) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0xb47a91) [0x7f1b68b1fa91]
[bt] (8) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x460609) [0x7f1b68438609]

Stack trace:
[bt] (0) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0xb276ba) [0x7f1b68aff6ba]
[bt] (1) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0xb47d59) [0x7f1b68b1fd59]
[bt] (2) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x460609) [0x7f1b68438609]
[bt] (3) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x4610fc) [0x7f1b684390fc]
[bt] (4) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x4c4e87) [0x7f1b6849ce87]
[bt] (5) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(XGBoosterUpdateOneIter+0x70) [0x7f1b68139d60]
[bt] (6) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/lib-dynload/…/…/libffi.so.6(ffi_call_unix64+0x4c) [0x7f1b992d4630]
[bt] (7) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/lib-dynload/…/…/libffi.so.6(ffi_call+0x22d) [0x7f1b992d3fed]
[bt] (8) /home/enrico/.miniconda3/envs/xgb/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x1084a) [0x7f1b992e884a]

Process finished with exit code 1

No problem if I install py-xgboost-gpu with
conda install -c conda-forge py-xgboost-gpu==1.7.6 but in this case the feature weights (silently) do not work.

#5

@hcho3 ok, it actually works after a reboot. Something strange happened when waking up from suspension. Thanks for pointing it out!

1 Like