Bag of features: how to distinguish between an image containing 1 vs. 2 occurrences of an object

I have successfully trained a classifier to identify different objects in grayscale images. Feature extraction is done using bagOfFeatures and the classifier is generated using SVM.

One interesting finding I learned is that my classifier will frequently confuse an image containing a single occurrence of an object with an image containing two occurrences of that same object (and vice-versa).

For example:

The confusion matrix indicates for testing cases that up to 11% of this (Class OneCat):

is classified as this (Class TwoCat):

And vice-versa.

This seems understandable since BoF will extract features that are not spatially registered and when using most serviceable grid sizes on Matlab (GridStep < 20), I will rarely have a tile that includes features of both objects in image 2 (thus creating a unique feature that distinguishes it from the image 1). I do not see a significant effect so far by expanding my training set (currently n=200 for each class).

Therefore it seems logical that these 2 images would be confused with each other. I suppose the histogram of vectors for each are different regarding count of features due to the higher occurrence of features in the 2nd image.

What would you suggest would be an approach to be able to distinguish between such classes? Any example (pseudo-code) or workflows would be greatly appreciated.