@ Matthew (Jun 28)
I had the same problem (with P growing exponentially). Like you said: this has to do with the Alpha parameter. It has to do with how the Unscented Transform calculates its transformed mean. In certain cases (I think when measurement covariance is very low, and process covariance is a few orders of magnitude greater), there can be some rounding errors in Matlab, which causing the transformed mean to come up short.
To fix this, I changed the UT function to be like this:
function [y,Y,P,Y1] = ut(f,X,Wm,Wc,n,R)
% y=zeros(n,1); % LINE COMMENTED OUT HERE
% y=y+Wm(k)*Y(:,k); % LINE COMMENTED OUT HERE
y = mean([Y(:,1)'; mean(Y(:,2:end)')]); % LINE ADDED HERE