CV Shuffling improves performance *a lot*

[DeastinY]

I’ve got a timeseries of multiple parameters, that I’m converting to a static problem by using a moving window which I average over, such that I get a vector with the means of e.g. the last day and my prediction target.

cv = StratifiedKFold(10, shuffle=False)
xgb.cv(dtrain=dtrain, folds=cv, params=params)

Using unshuffled CV I get an ROC AUC of 0.7.

cv = StratifiedKFold(10, shuffle=True)  # this is the only difference
xgb.cv(dtrain=dtrain, folds=cv, params=params)

When I shuffle I do get an ROC AUC of 0.9.

It’s not dependant on the random_state for shuffling either, the results are reliably better when shuffling. From my understanding

1. XGB should be invariant to the order of the training data and
2. XGB assumes iid, but I’ve seen quite some paper use the same approach

Help please, I’m out of ideas.

Somehow related:

XGBoost time-series predictions question

I am having some problems with producing actual values too. I have found this works well as far as getting an ordering of the predictions (I just sort the predictions). I thought I was doing well as far a predicting actual values (and maybe I am if the rmse or mae are any indication). But on this blog I discovered ‘base_score’ and corrected it to the median value of my data (I use the mae metric to control outliers). My ordering remains very accurate (for the type of data) but the predictions …