Distribution-balanced stratified cross-validation

An improvement to stratified cross-validation for small imbalanced data sets.

Distribution optimally balanced stratified cross-validation (DOB-SCV) partitions a data set into n folds in such a way that a balanced distribution in feature space is maintained for each class, in addition to stratification based on the label.

The real-world effect of using DOB-SCV, instead of stratified cross-validation, is slightly higher testing accuracy. The biggest improvements can be expected on small, class imbalanced data sets.

The implementation can be used as a drop-in replacement for CVPARTITION.

Reference: Study on the Impact of Partition-Induced Dataset Shift on k-Fold Cross-Validation available from https://ieeexplore.ieee.org/document/6226477