Doing 2 iterations of reverse image matching with BoW (features)

Hi, I am trying to make a sort of image recognition program that will allow me to do a reverse image search on a database of images, and I wanted to get your opinions on if this is a good way to do it. Here is the project: I have a database of over 30,000 images that are all mostly very similar (they are trading cards). The cards are separated into over 200 different sets and each set has about 200 images.

Sample Images:

Image 1

Image 2

Image 3

Now because the overall framework of the trading cards are the same, if I just used the image matching algorithms I will get a lot of errors or low accuracy.

The way I want to do this is to use a two step method. Since the cards are separated into different sets or categories, I wanted to identify the set/category the card belongs to first, than match it to the actual image next.

For example for Image 2 and Image 3:

These cards are almost identical but are from two different sets. So I would match the set first (Using the set symbol at the right middle section), than I would match the card.

Example Set symbols:

So would using a Bag of Features model with SIFT/SURF/ORB for the set symbols (Using perhaps a shape descriptor for features) to match sets, than useing another bag of features to match cards in the set be an efficient way to do this?

Please let me know if I have not provided enough information or if anything else is an issue.

Thanks in advance for taking the time to look through this.