Running binary classification, the classifier output for being class A peaks on zero for class B, and around 0.5 for class A. Still, it gives decent separation, as evidenced by the ROC curve.
The dataset is unbalanced. When I balance it by using scale_pos_weight, the peak moves from 0.5 to 1. However, this happens only when I run without folds. When I run with folds the peak is still at 0.5. This doesn’t change when I use StratifiedKFold instead of KFold.
My question is: Is having the output peak at 0.5 instead of 1 normal / expected in some way? Or does not peaking on 1 mean that something is definitely wrong with the model, no matter what separation it gives?