Good concept, but you are not a MATLAB-thinking guy. Some improvement suggestion
1. try to use sparse() and diag() to form your Gx and Gy, much faster than loop(C/C++ way).
2. Similarly, use sparse() and diag() for G2, or you can form G2 directly without Gx Gy. For God sake, please do that. G2 is a 2NM-by-NM double FULL matrix, if the F is N-by-M matrix. That's (NM)^2*2*8 Bytes, for a normal size image N=M=512, that's 2^40 Bytes = 1024 GB, no one has such RAM.
3. why do you use pinv()? that's very expensive, and I don't see it's neccessary. Try '\'.