Mixtures of Experts, Using Gaussian Mixture Models for the Gate

This code implements the mixture of expert’s using a Gaussian mixture model for the gate.

This code implements using a Gaussian mixture model for the gate. ; the main advantage of this method is that training for the gate uses expected maximization (EM) algorithm or single loop EM algorithm. This is achieved using a Gaussian mixture model for the gate. Other methods use the Softmax Function that does not have an analytically closed form solution, requiring the Generalized Expectation Maximization (GEM) or the double loop EM algorithm. The problems with GEM is that it requires extra computation and the stepsize must be chosen carefully to guarantee the convergence of the inner loop. I used k means clustering for initialization, I find only a small improvement after initialization. If you have any questions or recommendations contact me.