Out of memory error for big data
Show older comments
The masterDataMatrxOffset, contains 464780 rows and 256 columns of complex numbers. I use meshgrid to get 256X256 matrix of the real and imaginary and store it in a cell, but I get this error: Is there a way to overcome the out of memory?
Out of memory.
Error in meshgrid (line 62)
yy = repmat(ycol,size(xrow));
Error in auto_encoder (line 34)
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
More information
tic;
clear; clc; close all;
%% Load Files
load masterDataMatrxOffset;
%% Extract Parameters
iAntenna = size(masterData,2) - 16;
data = masterData(:,1:iAntenna);
power = normalize(real(data),'range');
phase = normalize(imag(data),'range');
Region = masterData(:,261);
iReceiver = length(data);
iRegion = max(Region);
%% Initializations
xTrn = cell(1,max(Region));
xTst = cell(1,max(Region));
datamat = cell(1,size(data,1));
%% Split into training and testing
for idx = 1:iRegion
IDX = find(Region==idx);
train = sort(randperm(size(IDX,1),ceil(0.8*size(IDX,1))));
trnidx = IDX(train);
tstidx = setdiff(IDX,trnidx);
xTrn{idx} = trnidx;
xTst{idx} = tstidx;
end
xTrnidx = cell2mat(xTrn');
xTstidx = cell2mat(xTst');
Trainreg = zeros(numel(Region), max(Region));
Trainreg(sub2ind(size(Trainreg), 1:numel(Region), Region')) = 1;
tTrain = Trainreg';
for obsrv = 1:size(power,1)
sprintf('observation = %d',obsrv);
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
datamat{obsrv} = Po.^2 + Ph.^2;
end
toc;
4 Comments
Sharif Khalil
on 29 Dec 2020
You still can store them in a cell array
myCell = {};
myCell{1} = tall(1:100); myCell{2} = tall(101:200);
myCell{3} = myCell{1}.^myCell{2};
myCell =
1×3 cell array
{1×100 tall} {1×100 tall} {1×100 tall}
Sharif Khalil
on 31 Dec 2020
Edited: Sharif Khalil
on 31 Dec 2020
Answers (0)
Categories
Find more on Large Files and Big Data in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!