I have found an answer, but I still wanted to post it incase anyone else has a similiar situation. I did this:
for n=1:6
C(n,:,:,:)=sum(squeeze(A(n,:,:,:,:)).*B);
end
For me, this method proved to be ~30x faster than my initial method.