dir lists the files and folders in the MATLAB current folder. Results appear in the order returned by the operating system.
So, the order is explicitly documented: it is whatever the OS returns, which is not certain to be any kind of alphanumeric order.
If a file is removed from or added to the directory after the most recent call to opendir() or rewinddir(), whether a subsequent call to readdir() returns an entry for that file is unspecified.
The directory entries for dot and dot-dot are optional. This volume of IEEE Std 1003.1-2001 does not provide a way to test a priori for their existence because an application that is portable must be written to look for (and usually ignore) those entries. Writing code that presumes that they are the first two entries does not always work, as many implementations permit them to be other than the first two entries, with a "normal" entry preceding them.
Basically, then, the order returned may be whatever order the internal directory structure uses to order the files. Unix System V filesystems usually did not sort directory entries on disk, and instead would read as many blocks of the directory as needed to find the file they were after, and when writing a new directory entry would use the first available empty slot. This reflected the reality at the time that writing to hard disks was expensive and that keeping the on-disk entries sorted required rewriting half of the blocks representing the directory (presuming uniform random access.) These days with on-controller disk-write queues that optimize drive head movement, it is usually found more efficient to either sort all the entries each time anything in the directory is touched, or else to use an ordered tree system rather than a linear search system. But the ordered tree system doesn't promise alphabetical order in reading the entries: it could instead choose to return the order of the leaf nodes...
In short: if your code assumes that '.' and '..' are the first two entries in a directory, your code has a bug (even in MS Windows). If your code assumes that directory entries are returned in any sorted order, your code has a bug (in all OS.)