@Zach Adams, first, more feedback. Please take some time to digest that. Second, I've improved your code without changing the algorithm and I've shown you how to determine the number of outputs.
Feedback on code
1) What is H and where does it come from? If H is supposed to be a variable, doesn't it cause an error? If H is a script or a function without any inputs, that needs improved for 2 reasons. a) "H" is a really bad function/file name. b) calling a script from a function defeats the advantages of functions - reproducibility and self-containment. If H is a function that merely asks for input you could move that code into his function. If H is a function that is used by other function, it's OK to keep it as it is but the name should really change. Functions should generally contain verbs or describe what it does. Example: getUserInput().
2) I see you have a variable named "max". You should avoid naming variables after common functions for two reasons. a) if you have a variable named max you will not be able to use the max() function anywhere in the same workspace and b) other people reading your code, including your future-self, will get confused because they are used to the "max" being a function. You can rename it to something like "maxC" if you'd like.
3) Use smart-indentation by selecting the entire code (ctrl + a) and then ctrl+i. I've corrected the code in your comment so that it has smart indentation. This makes it MUCH easier to read.
4) Your final for-loop is actually a conditional statement. Instead of "for hypotenuse > A &&..." use "if hypotenuse > A &&..."
5) 'M' is a very bad function name (same reasons as in point #1 above).
Cleaner version of your code
See inline comments.
function [A, B, hypotenuse] = findPythagTriplets(C)
A = ;
B = ;
hypotenuse = ;
for maxC = 1:1:C
for m = round(sqrt(maxC/2)):round(sqrt(maxC))
if m == floor(m)
n = sqrt(maxC - m^2);
A(end+1) = m^2 -n^2;
B(end+1) = 2*m*n;
hypotenuse(end+1) = m^2 + n^2;
if A(end) > B(end)
A(end) = B(end);
B(end) = temp_A;
if hypotenuse(end) > A(end) && hypotenuse(end) > B(end) && hypotenuse(end)^2 == B(end)^2+A(end)^2 && ...
A(end) >= 0 && A(end) == round(A(end)) && B(end) >= 0 && B(end) == round(B(end))
A(end) = ;
B(end) = ;
hypotenuse(end) = ;
Determine number of values in the output
To determine the number of triangles, just call the function with the output and count the number of values in the output variables.
[A, B, hypotenuse] = findPythagTriplets(10)
numberOfTriangles = numel(A)