Path: news.mathworks.com!newsfeed-00.mathworks.com!newsfeed2.dallas1.level3.net!news.level3.com!postnews.google.com!r37g2000prr.googlegroups.com!not-for-mail
From: Rune Allnor <allnor@tele.ntnu.no>
Newsgroups: comp.soft-sys.matlab
Subject: Re: sort large amount of data
Date: Wed, 21 Jan 2009 02:27:10 -0800 (PST)
Organization: http://groups.google.com
Lines: 22
Message-ID: <0d6ff837-2778-4524-a1af-5753c05b9b99@r37g2000prr.googlegroups.com>
References: <gl6ojl$1ag$1@fred.mathworks.com>
NNTP-Posting-Host: 77.17.103.233
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Trace: posting.google.com 1232533631 31003 127.0.0.1 (21 Jan 2009 10:27:11 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Wed, 21 Jan 2009 10:27:11 +0000 (UTC)
Complaints-To: groups-abuse@google.com
Injection-Info: r37g2000prr.googlegroups.com; posting-host=77.17.103.233; 
	posting-account=VAp5gAkAAAAmkCze5hvZtMeedpZWNthI
User-Agent: G2/1.0
X-HTTP-UserAgent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET 
	CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022),gzip(gfe),gzip(gfe)
Xref: news.mathworks.com comp.soft-sys.matlab:512868


On 21 Jan, 10:07, "Theodor Zouk" <reb...@hotmail.com> wrote:
> Hi
> I have very large amount of data (reell values of class double) that is r=
egistrered over the chronological appearence and saved in many .mat files. =
I want to sort this data in a ascending order. Now, SORT do this, but it ca=
n only do it per each .mat-file. The ideal would be if all the .mat files c=
ould be saved into one .mat file and then loaded into workspace and then so=
rted. But this can not be done when there is not enough memory. Any tips ho=
w i should 'loop' through all the mat files and then at the end have many m=
at files where the maximum value in one mat file is lower than the minimum =
value of the next consecutive mat file.

Two ideas come to mind:

1) Loop through each file and build a *total* histogram
   of the values that appear in the total data set.
   Use this histogram to separate the data set by value
   into intermediate files. Sort these files, and merge.
2) Find a copy of Knut's "The art of Computer Programming:
   Sorting and Searching" and read.

Rune