Evaluating performance of different detector/classifier techniques.

Dear all,

I am pretty much confuse in evaluating different detector. i.e if I trained 2 different detector on the same training set, and then evaluating them on the same test set, depending on their features/no of classifier/trees (in case of PBT/Random Forests), I will have different score for each detected bounding box.

So, how do people in Computer Vision compare their detector performance i.e. Precision-Recall curve given that the scores are different? Did they used any normalization i.e. setting to {0,1} as I do understand that precision at certain recall point is obtained via adjusting the threshold e.g score ?

Your kind assistance is highly appreciated.

Many thanks

Dhorl Jr'

Answer 1

[Image Analyst]

You can use ROC curves to evaluate performance, assuming you have ground truth to know what is a correct and incorrect classification. http://en.wikipedia.org/w/index.php?title=Receiver_operating_characteristic&redirect=no

You can use adaboost to combine different classifiers to improve overall performance: http://en.wikipedia.org/wiki/AdaBoost

[Dhorl Jr]

Thanks for your quick reply.

I know those parameters and evaluation method. First let me re quote my question with examples.

Say I train my dataset qith 1) A random forest of 10 trees and depth 20 using features A,B,C 2) An AdaBoost of cascaded 2-stage with 256 and 1024 classfier at each stage using features,A,B,C,D.

Based on different settings, given unseen images A, top detected score using Method 1 is 0.98 (say it normalized already). But an AdaBoost (or PBT) scores could be 256 (maybe for its additive style?).

Put it another way, I also wonder, if let say all the features correctly identified in the images, what would be the scores? e.g. is this is known, then I could simply normalized the score with the maximum score to give me score in a reange of {0,1}

So, my question is that, how you going to compare this two classifiers on the same ROC curve plot (e.g by adjusting your thresholding reference) given your final score (or the way the features giving score) is different.

Let me know if my initial understanding is not well-understood.

Many thanks for your kind assistance.

Best wishes

Dhorl