MATLAB Answers

Help using textscan on .csv files

4 views (last 30 days)
Koustubh Gohad
Koustubh Gohad on 20 Jul 2016
Answered: Thorsten on 25 Jul 2016
Hello,
I'm writing a script to process a number of CSV files generated each day and make a mat file from them. I use readtable first, but if there is an formatspec error due to some inconsistent data, readtable doesn't work. In that case, I'm using textscan to read a chunk of data, skip the bad row, and again read from the next row till I reach the end of the file.
Is there a way to replace the cells which are out of spec with NaNs and read them? These rows only have one bad cell, so I would want to retain information in the other cells if possible. I don't want to use auto-generated code for this, as I will be using this script for other files also, which have different parameters.
This is the inconsistency:
For a column of numbers, there is '3402823593150348600000000000000â…¦äŸå†ï°¸â…¦ï´´â…¦ì¬Ü…' in some rows. I want to replace it with 'NaN' or something else which is convenient to process, and read the rest of the row. I can put it in 'TreatAsEmpty', but I don't know if I'll have the same expression every time. Any help would be greatly appreciated.
Thanks,
Koustubh

  2 Comments

per isakson
per isakson on 23 Jul 2016
"Is there a way to replace the cells which are out of spec with NaNs and read them?" &nbsp Yes, I think it is possible to read the good data of the bad rows, but how that depends on the circumstances.
Questions:
  1. Is speed crucial?
  2. Does an entire file fit comfortable in memory (RAM)?
  3. Is the data comma separated? Does comma, ",", appear among the garbage characters.
  4. What do you mean by "I don't want to use auto-generated code for this" ?
  5. Could you upload a sample file.
My first idea is to
  1. read the entire file as text
  2. fix the text with regexp
  3. read and parse the text with textscan
Stephen Cobeldick
Stephen Cobeldick on 25 Jul 2016
@Koustubh Gohad: please edit your question and upload a sample file by clicking the paperclip button.

Sign in to comment.

Answers (1)

Thorsten
Thorsten on 25 Jul 2016
To check if your string contains any invalid characters, i.e., non-digits, you can use
~isempty(regexp(s, '\D'))
and then set the cell to NaN;

  0 Comments

Sign in to comment.

Sign in to answer this question.