colAUC
Calculates Area under ROC curve (AUC) for a vector or for each column of a matrix.
Author: Jaroslaw Tuszynski

I'm sorry but it is not clear how to get multiclass auc if I have Y as the true label of 1...c classes and Yest as 1...c column vector with predictive score for each instance belong to certain class. The output auc is a c x c x(c-1)/2 combination in your case, take mean(auc) returns c auc values.