Code covered by the BSD License  

Highlights from
Read Unicode Files

4.0
4.0 | 1 rating Rate this file 15 Downloads (last 30 days) File Size: 76.9 KB File ID: #18956
image thumbnail

Read Unicode Files

by

Vlad Atanasiu (view profile)

 

27 Feb 2008 (Updated )

Reads Unicode strings from file, outputs character array of strings

| Watch this File

File Information
Description

function C = textscanu(filename, encoding, del_sym, eol_sym, wb)
% TEXTSCANU Reads Unicode strings from a file and outputs a cell array of strings
%
% -------------
% INPUT
% -------------
% filename - string with the file's name and extension
% - example: 'textscanu.m.txt'
% encoding - encoding of the file
% - default: UTF-16LE
% - examples: UTF-16LE (little Endian), UTF-8.
% - See http://www.iana.org/assignments/character-sets
% - MS Notepad saves in UTF-16LE ('Unicode'),
% UTF-16BE ('Unicode big endian'), UTF-8 and ANSI.
% del_sym - column delimitator symbol in ASCII numeric code
% - default: 9 (tabulator)
% eol_sym - end of line delimitator symbol in ASCII numeric code
% - default: 13 (carriage return) [Note: line feed=10]
% - on MS Windows use 13, on Unix 10
% wb - displays a waitbar if wb = 'waitbar'
%
% Defaults:
% -------------
% BOM - the first character of the file is assumed to be a
% Byte Order Mark and removed, if it's unicode2native()
% value is 26
% byte_encoding - this value is read from the last two characters
% of the encoding input variable if they are 'LE' or 'BE',
% otherwise 'little endian' is the default for Windows and
% 'big endian' for Unix
% eol_len - number of characters used as end of line markers;
% for a Windows AND a value of 13, eol_len is 2,
% otherwise 1
%
% -------------
% OUTPUT
% -------------
% C - cell array of strings
%
% -------------
% EXAMPLE
% -------------
% C = textscanu('textscanu.txt', 'UTF-8', 9, 13, 'waitbar');
% Reads the UTF-8 encoded file 'textscanu.m.txt', which has
% columns and lines delimited by tabulators, respectively
% carriage returns. Shows a waitbar to make the progress
% of the function's action visible.
%
% -------------
% NOTES
% -------------
% 1. Matlab's textscan function doesn't seem to handle
% properly multiscript Unicode files. Characters
% outside the ASCII range are given the \u001a or
% ASCII 26 value, which usually renders on the
% screen as a box.
%
% Additional information at "Loren on the Art of Matlab":
% http://blogs.mathworks.com/loren/2006/09/20/
% working-with-low-level-file-io-and-encodings/#comment-26764
%
% 2. Text editors such as Microsoft Notepad or Notepad++ use
% a carriage return (CR, ascii 13) and a line feed (LF, ascii 10)
% to mark line ends (when you hit the enter key for example),
% instead of just carriage return as usual on Unix or
% Microsoft Word.
%
% In textscanu use ascii 13 as delimitator in the case of
% end lines marked with the CR/LF combination. Since the LF
% is beyond the end of a given line and not part of the next,
% it is disregarded by the function.
%
% 3. If you get spaces inbetween characters, try changing
% the encoding parameter.
%
% -------------
% BUG
% -------------
% When inspecting the output with the Array Editor,
% in the Workspace or through the Command Window,
% boxes might appear instead of Unicode characters.
% Type C{1,1} at the prompt or in Array Editor click
% on C then C{1,1}: you will see the correct string
% if you have an a Unicode font for the appropriate
% character ranges installed and enabled for the Command
% Window and Array Editor (File > Preferences > Fonts).
%
% However, up to Matlab R2010a at least, Unicode
% characters display as boxes in figures, even if
% data is correctly stored in Matlab as Unicode.
%
% -------------
% REQUIREMENTS
% -------------
% Matlab version: starting with R2006b
%
% See also: textscan
%
% -------------
% REVISIONS LOG
% -------------
% 2015.01.06 - [fix] eol_len now set for all number of input arguments
% 2014.05.04 - [fix] attempt to close figure only if it exists
% 2011.01.17 - [new] support for Unix
% - [new] automatic detection of BOM presence
% 2010.12.31 - [new] no requirement anymore not to end the
% file with end of line marks
% - [fix] define default waitbar handle value
% and make the message more informative
% 2010.10.04 - [fix] upgrade to Matlab version 2007a
% 2009.06.13 - [new] added option to display a waitbar
% 2008.02.27 - function creation
%
% -------------
% CREDITS
% -------------
% Vlad Atanasiu
% atanasiu@alum.mit.edu, http://www.waqwaq.info/atanasiu/

Acknowledgements

Information Based Similarity Toolbox inspired this file.

This file inspired Information Based Similarity Toolbox.

Required Products MATLAB
MATLAB release MATLAB 7.3 (R2006b)
MATLAB Search Path
/
/__MACOSX
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (3)
02 May 2014 Brad Stiritz

Very helpful function, thank you! One small issue needs to be fixed, however. Running the function with one argument only (the file name) generates a run-time error..

------------------------------------------
>>textscanu(file_spec);
Error using close (line 136)
Invalid figure handle.

Error in textscanu (line 141)
close(h)
------------------------------------------

This error results from:
(a) setting the waitbar handle to default value (0) in the switch block beginning on line 59.
(b) not checking (h) against the "magic" default value (0) before trying to close in line 141.

The simplest fix is to replace line 141 with the following code block:

------------------------------------------
% close waitbar if created
if h ~= 0
close(h)
end
------------------------------------------

However, I should point out that it's not a good choice to use (0) as the default value for (h), since (0) can be a valid handle in general. Better to use (NaN) as the default value, since:

------------------------------------------
>> ishandle(0)
ans = 1

K>> ishandle(NaN)
ans = 0
------------------------------------------

A safer way to code the function would be to initialize (h) to (NaN) before the initial switch block, then re-assign h = waitbar(..) within case 5. The conditional block at the end of the function should then be:

------------------------------------------
% close waitbar if created
if ishandle(h)
close(h)
end
------------------------------------------

21 Jun 2008 Vlad Atanasiu

Siyi: Check the format of the input file. Each row must have as many strings as the other rows, the strings have to be tab-delimited, and have carriage-returns at the end of rows. See the sample file zipped with textscanu.m. / Hope this helps. / Vlad

Comment only
28 Feb 2008 Siyi Deng

I got this error message:

??? Undefined function or variable "eos".

Error in ==> textscanu at 75
sos = eos + 2; eos = crt(n) - 1;

Comment only
Updates
01 Mar 2008

The function now supports single column files.

13 Jun 2009

Added option to display a waitbar showing the progress of data reading.

01 Jan 2011

[new] no requirement anymore not to end the
file with end of line marks
[fix] define default waitbar handle value
and make the message more informative
[fix] upgrade to Matlab version 2007a

08 May 2014

[fix] attempt to close figure only if figure exists

19 Nov 2014

Published as Matlab toolbox (.mltbx file).

06 Jan 2015

[fix] eol_len now set for all number of input arguments

Contact us