Extracting huge data from csv file

2 views (last 30 days)
Aditya
Aditya on 23 Jan 2012
I am working on huge csv file of around 100-200 Mb's. I am trying to extract the data using COM server and activex application. But for the very first time when i fetch the data it is extracted without any problem but when I try to re extract I get an error stating that ' Not enough memory to complete this operation '. My C drive has around 5 Gb of free space. Do I have to clear the data from COM server first and then proceed. If it is then how do I do that

Answers (2)

Barry Greene
Barry Greene on 23 Jan 2012
This is usually a problem with RAM rather than hardrive space. Suggest reading your file in chunks using fread within a for/while loop and concatenating the results.
Hope this helps

owr
owr on 23 Jan 2012
Agree on the RAM comment from Barry. If fread is too tedious, you can also try textscan. Specifying a format for each line of the csv file (which columns are numeric vs. string, ignoring columns that you may not need using something like %*f) has drastically increased the efficiency of textscan for me in the past.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!