fprintf is printing strange characters instead of numbers

I want to print a vector of unsigned integers to a text file, with a space between each number. But the file I get is just weird symbols. I must be doing some trivial mistake, it's not the first time it happens, I can't remember what could be the fix. It's happening on R2018b (but I remember it happening on older versions as well). Here's sample code below:

clear
data = uint32(zeros(1, 1615));
data(1:2:50) = 1;
output = fopen('output.txt', 'wt');
fprintf(output, '%d ', data);
fclose(output);

Output I get is: ‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‰‰‰‰‰‰‰‰‰ ...

Output I want is: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 ...

[Guillaume]

That's certainly not normal behaviour. Can you attach that output.txt file so we can have a look?

[Paolo Binetti]

Done

[Rik]

I cannot reproduce your problem in R2018b. I suspect there might be an issue with uint32 as an input, but I agree with Guillaume, this should not happen.

[Guillaume]

Looks fine to me. What program are you using to look at its content?

[Stephen23]

Opening the file with Notepad++ shows the 0's and 1's as expected.

Opening the file with MS Notepad shows those unwanted symbols.

What OS are you using?

[Paolo Binetti]

It's happening also if data are floats.

Program is notepad for Windows 7 and it always worked fine with Matlab generated .txt files.

If you can see no mistake in this use of fprintf, could you suggest an alternative way?

[Stephen23]

"could you suggest an alternative way?"

• Don't use MS Notepad. It is very weak on features and vulnerable to pointless newline confusion. There are much better text editors around.
• Save as UTF-8 rather than ANSI. When I opened your file and saved it encoded as UTF-8 it could be opened using MS Notepad without any problems (attached as output1.txt).

[Paolo Binetti]

Which simple and free .txt editor would you recommend, rather?

[Guillaume]

When I opened your file and saved it encoded as UTF-8 it could be opened using MS Notepad

You must have saved it as UTF16. For the three characters '01 ' in the file there is no difference at all between ANSI ad UTF8.

[Stephen23]

"You must have saved it as UTF16. For the three characters '01 ' in the file there is no difference at all between ANSI ad UTF8."

For the characters maybe, but not for the file as a whole: the inclusion of a Byte Order Mark is a significant change. And I notice that excluding the Byte Order Mark made this file unreadable with those same strange characters, even with UTF-8 (as selected by Notepad++).

[Rik]

It is surprisingly difficult to figure out the encoding of a file, even if it is written correctly. There is a reason I wrote my readfile FEX submission in exasperation. I couldn't find any way to read the BOM with Matlab tools.

[Walter Roberson]

Rik, see also my https://www.mathworks.com/matlabcentral/answers/285186-importing-data-without-knowing-number-of-columns#comment_368710 detect_utf_encoding

(I also have internal versions that detect more file types, such as figuring out that a file is an image file or .mat file. I did some research towards trying to figure out which character set a file was probably in, but it looks like I gave up on implementing that.)

[Rik]

That is quite a neat piece of code, thank you for sharing it. I think it is strange that it is so difficult to figure out. Even the reportedly byte-by-byte reading seemingly skips those header bytes.

After testing it, it seems to fail for UTF-8 files I created with Matlab (it detects them as windows-1252). This shouldn't be this hard.

[Walter Roberson]

Was your test file the same as from your FEX submission?

[Rik]

I've attached a zip file with the 4 test files, a tester function and the version of readfile I just checked the files with (slightly tweaked compared to the FEX version, mostly cosmetic changes).

For completeness' sake I also included the version of detect_UTF_encoding you linked to and inserted that in my tester script.

[Walter Roberson]

Yes, unfortunately if there is no byte order mark then it gets more difficult to figure out without false positives.

I have some internal versions of the detection routine that proceed to detect zip and image files and xls and .mat files; my intention was to proceed onwards to detect character encoding such as ISO-8896-1 versus ISO-8896-6 or SHIFT-JIS. I researched that, but once I got into the windows code pages, the number of cases was starting to feel too big for me to bother. Some of the cases could not be told apart (except perhaps by statistics); others had only a single code point difference (that is, one code point was assigned a meaning in one character set but not in the other character set), so if you saw the one codepoint you could disprove a particular character set, but lack of it would not prove the other...

[Rik]

That is why I came close to giving up, even for telling UTF-8 and windows-1252 apart, which should be relatively easy, but turns out to be non-trivial. Notepad++ seems to do a better job than what I can manage with Matlab.

For the files I'm using, my FEX submission works, but I have no idea how future-proof that function is (or past-proof for that matter).

Answer 1

[Guillaume]

The problem is due to Notepad broken encoding detection algorithm. For some reason it assumes that the file is encoded in UTF16, where the byte sequence [49 32] (characters '1' and ' ' in ansi and UTF8) indeed represents the character '‱'. Note that simply adding a space, ' ', before your sequence of numbers completely change the behaviour of Notepad's detection algorithm. With a space at the start, it correctly interprets the file as ANSI or UTF8.

As Stephen said don't use notepad. It's broken (and very limited in functionality). A good alternative is the open source Notepad++. You can also simply use matlab's editor.

edit: another option would be to save your output.txt as UTF16, which would then be read correctly by notepad, particularly, if you insert a BOM at the beginning. However, while matlab can read/write UTF16, it's not documented so there may be some edge cases where it doesn't work appropriately. Additionally, notepad is probably the only software that tends to assume UTF16, everything else tends to assume UTF8 by default, so the display will look odd in almost every other text editor (matlab's included).

[Noam Greenboim]

For adding BOM, see here:

https://www.mathworks.com/matlabcentral/fileexchange/75708-update-bom-for-unicode-text-files

Usage (for example):

BOM('output.txt','UTF-16_LE')