I can see a few error in the line:
correlationOutput = normxcorr2(mode(:,:A), Image(:,:, channelToCorrelate))
1. The syntax does not make sense, is there supposed to be another comma?
This probably should read:
but you mention that A has imaginary values so you cannot use it to index.
2. If channelToCorrelate is the same as A then you cannot use this to index into Image.
What is it you are attempting to do with this indexing?