I really like your first longestword function.
function out = longestword(word1, word2, word3)
out = word1;
if(strlength(word2) > strlength(out))
out = word2;
if(strlength(word3) > strlength(out))
out = word3;
The error that you got when you ran it was because
is one input, not three. Additionally, it doesn't look like there has to be a limit to the number of words that might be in the input, so you may want to write the code knowing that there might be any number of words in the input.
function out = longestword(input_words)
out = input_words(1);
for i = 2:length(input_words)
out = input_words(i);
If you always know there will be three words, you can use the original format, but instead of word1, word2, word3, use word(1), word(2), word(3) to grab the first through third words in the one input array.