"feature_weights " doesn't work

#1

Hi , When I use feature_weights parameter , There is no different in the results…

1)not using feature_weights parameter

model5=xgb.XGBRegressor(objective=‘reg:squarederror’, colsample_bytree = 1, n_estimators =1000,gamma=0,subsample=1,reg_alpha=0.1,tree_method= “exact”)

model5.fit(train_x,train_y)

predictions=model5.predict(test_x)

2)using feature_wieghts parameter

model5=xgb.XGBRegressor(objective=‘reg:squarederror’, colsample_bytree = 1, n_estimators =1000,gamma=0,subsample=1,reg_alpha=0.1,tree_method= “exact”)

model5.fit(train_x,train_y,feature_weights=random_weights)

predictions=model5.predict(test_x)

The two cases have no differences.
I don’t know why ‘feature_weights’ doesn’t work… Please Help…

#2

Please set any of the colsample_* hyperparameters to a value less than 1.0. The feature weights determine how the features are randomly selected by sampling, and if sampling is disabled, setting feature weights won’t have any effect.

1 Like
#3

Thanks. I solved the problem!

#4

Hello @hcho3! I am interested in using feature_weights as well and found this post - I did set all the colsample_* hyperparamters to a value less than 1.0 but a disconnect with the # of columns being samples and their corresponding weights throws an error? Not sure how to proceed - thank you for any advice!

image
image1214×502 174 KB

#5

It appears that you have 110 columns in your data. Please ensure that weights is 110 elements long.

#6

I don’t think that’s what’s happening here: I have also encountered this problem. I think that the set of features is 221 long, but after colsample_bytree, only 110 features are selected. Then when it tries to do sub-subsampling bylevel, the check fails since there’s only 110 remaining features.

I have replicated this problem in the below minimal example:

import numpy as np
import pandas as pd
import xgboost as xgb

total_num_columns = 32
num_rows = 100
X = pd.DataFrame({f’x_{i}’: np.random.normal(0, 1, num_rows) for i in range(total_num_columns)})
y = np.random.normal(0, 1, num_rows)
feature_weights = np.exp(np.random.normal(0, 0.5, total_num_columns))

dm = xgb.DMatrix(X, label=y, feature_weights=feature_weights)

xgb.train({‘colsample_bytree’: 0.75, ‘colsample_bynode’: 0.25}, dm)

XGBoostError: [09:47:43] …/src/common/random.h:92: Check failed: array.size() == weights.size() (24 vs. 32)

#7

The consequence of this seems that it’s impossible to use feature_weights if more than one of colsample_bytree, colsample_bynode, colsample_bylevel is set nontrivially. @hcho3 do you think would be better if xgboost passed a sub-array of just the subsampled feature_weights down to the next level of subsampling?

#8

The bug has been fixed in https://github.com/dmlc/xgboost/pull/8100

#9

Thanks for the update!