I've "done" it in one project where the 3D color gamut was like an ellipsoid that was not aligned with any RGB or HSV axis, and not going along the R=G=B line either. But I was working with a statistician and he said he could do it so I just wrote out all the colors to a N by 3 array and he gave me a formula. Since it was an ellipse, I had PC1 that went along the major axis of the long skinny ellipsoid, and PC2 and PC2 that were perpendicular to that and went out along the short circular cross section of the ellipsoid. PC2 and PC3 didn't have any meaningful interpretation so I just used PC1 to get the gray level. It was basically the amount of the way I was between some sort of blue color, and white. I was wanting to quantify the amount of blue crystals mixed in with a bunch of white crystals. You'd do the same thing - to go to gray you'd want to use the PC1 (the first principal component) only. If you want my opinion, you can post your color image and I could examine the 3D color gamut to see if PCA even really makes sense for your image. I could post the 3D image of your gamut. For example, look at the 3D color gamut of the standard MATLAB demo image "pears":
By looking at the shape of this gamut, you can immediately see how a PCA approach might give a more meaningful grayscale image than the standard book formula like rgb2gray() uses, at least as far as quantification via image analysis goes, though maybe not as far as matching up with the human visual system.
Idub = double(YourImage);
rdub = Idub(:,:,1); gdub = Idub(:,:,2); bdub = Idub(:,:,3);
r2 = sum(rdub(:).^2); g2 = sum(gdub(:).^2); b2 = sum(bdub(:).^2);
rgbtot = sqrt(r2 + g2 + b2);
rfac = sqrt(r2) ./ rgbtot; gfac = sqrt(g2) ./ rgbtot; bfac = sqrt(b2) ./ rgbtot;
YourGrey = cast(rdub .* rfac + gdub .* gfac + bdub .* bfac, class(YourImage));