How to interpret the outputs of the function "runstest.m"?

A colleague of me and I are having a discussion on how to interpret the results to the runstest function. The following code:

rng shuffle; 
x = rand(1000000,1); 
[h,p] = runstest(x);

Gives as result h = 0, which would mean that the sequence of values is in fact NOT random. That is, if we interpretate the following sentence from the documentation correctly:

"The returned value of h = 0 indicates that runstest does not reject the null hypothesis that the values in x are in random order at the default 5% significance level."

What does h = 0 mean, that the sequence is random or that it is not random?

Answer 1

[John D'Errico]

This is a problem of appreciating the terminology, and the double negative way things get worded. Do two negatives make a positive? Not always. But then, not, not, always either. ;-)

Lets see what happens when we throw a very non-random sequence at runstest.

runstest(cumsum(rand(1,100)))
ans =
     1

runstest(mod(1:100,2))
ans =
     1

The null hypothesis is that the sequence posed is truly random. Can we KNOW a sequence was a random one? Well, you can't KNOW that as fact. You can only test how random it seems to be, or, how non-random. So both cases above are clearly seen to be unlikely events, IF the sequence were truly a random one. Essentially, we reject the null hypothesis, so the a priori assumption that the sequences really are random.

But, when I do this:

runstest(rand(1,1000))
ans =
     0

runstest is not willing to say the sequence seems to be non-random. So, we cannot reject the null hypothesis. Does that mean the sequence is random? NO!!!!!! Only that we cannot comfortably decide the sequence is probably non-random.

So, now lets look at p.

[h,p] = runstest(cumsum(rand(1,100)))
h =
     1
p =
      4.04395443273876e-29

Yep. It is non-random. I'll bet some decent money on that. p is TINY, it reflects a measure of how likely we think this event might be, in context of a truly random sequence.

Now, lets try a sequence of sequences.

x = rand(1,100);

[h,p] = runstest(x)
h =
     0
p =
         0.838684318956303

[h,p] = runstest([x,x])
h =
     0
p =
         0.720356034109542

[h,p] = runstest([x,x,x])
h =
     0
p =
          0.64038079177353

[h,p] = runstest([x,x,x,x])
h =
     0
p =
         0.578177151383335

See that the way I've constructed those sequences, the latter ones are seen to be somewhat less likely to be random in context of the runs seen, although runstest is not designed to detect the totally non-random way I extended the sequences. It cannot detect long lags as I extended the sequence, which in fact, make the longer versions non-random.

If I try a sequence of length 400 directly from rand, p is now larger.

[h,p] = runstest(rand(1,400))
h =
     0
p =
          0.72633274852919

So h==1 says runstest has detected an event that did not seem consistent with randomness. h==0 says it did not detect significant non-randomness, but it cannot tell you to conclude a sequence is truly random, or not. p tells you the strength of what it sees.