Read non-tabular text file using datastore or mapreduce

1 view (last 30 days)
How can I read non-tabular text file using datastore or mapreduce? Thanks

Answers (2)

yashwanth annapureddy
yashwanth annapureddy on 19 Nov 2014
Hi Darek,
As you might already know datastore is primarily meant for large tabular data sets. Non-tabular/blocked reading is something we want to look at also in the near future.
However you should be able to play with the TextscanFormats to read the data you care about just as you would using textscan for reading non-tabular data. For example, as you may already know, you can set the TextscanFormats as '%q' and read in a table with one variable name with the block of text for further processing.
I could probably help more, if you provide a sample text file?
hope this helps.

Darek
Darek on 19 Nov 2014
I tried this approach, but Matlab is poor at managing large files and runs out of memory quickly. This is my sample text file. Thanks
##################################
0|FRM|FM MLLD FILE|20141001|
1|FRM|A4014X|20051201|
5|FRM|A4014X000005|8K0ER0|C|R|SF|O|01|WA
5|FRM|A4014X000006|8K0ER0|P|R|SF|O|01|WA
8|FRM|A4014X|000000002|
90|FRM|FM MLLD FILE|20141001|
##################################
  3 Comments
Darek
Darek on 24 Nov 2014
This approach with mapreduce works, but it's extremely SLOW. Just to count records in small file (3GB, 8 million rows) it takes 20+ minutes on my new PC. Similar project can be done using other tools (R, Powershell, Python) in seconds. Thanks for your help.
yashwanth annapureddy
yashwanth annapureddy on 4 Dec 2014
By records if you mean lines in the file, datastore is probably not the right tool to count the number of records in a file. Lower or even system level functions like the ones mentioned in this thread might be useful for this kind of analysis.

Sign in to comment.

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!