Thread Subject:

Recoding a vector to string (a big one)

Dear all,
I need to recode vector v into string array given give a recoding
table lev and str. I used to do something like that:
v = [1 1 2 2 3];

lev = [1 2 3];
str = {'a' 'b' 's'};

[~, idx] = ismember(v, lev);
v_recoded = str(idx)

Once number of levels hits hundreds of thousands this becomes too slow
to live with it. I would really prefer to keep lev sorted in advance
so I could reduce time for searching elements within this array. I
looked at the map object and the dataset objects in the statistical
toolbox but I cannot find desired functionality.
Do I need to write custom code or I am missing something here?
Thank you in advance,
Kirill Andreev

"Kirill" <kirillandreev@gmail.com> wrote in message
news:d080f2d7-2789-409e-9dde-b0a471ab89fd@l6g2000vbn.googlegroups.com...
> Dear all,
> I need to recode vector v into string array given give a recoding
> table lev and str. I used to do something like that:
> v = [1 1 2 2 3];
>
> lev = [1 2 3];
> str = {'a' 'b' 's'};
>
> [~, idx] = ismember(v, lev);
> v_recoded = str(idx)
>
> Once number of levels hits hundreds of thousands this becomes too slow
> to live with it. I would really prefer to keep lev sorted in advance
> so I could reduce time for searching elements within this array. I
> looked at the map object and the dataset objects in the statistical
> toolbox but I cannot find desired functionality.
> Do I need to write custom code or I am missing something here?

Are all the levels positive integer values? If so, just use indexing
directly.

str = {'a', 'b', 's'};
v = [1 1 2 2 3];
c = str(v)

--
Steve Lord
slord@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

On May 23, 11:08 am, "Steven_Lord" <sl...@mathworks.com> wrote:
> "Kirill" <kirillandr...@gmail.com> wrote in message
>
> news:d080f2d7-2789-409e-9dde-b0a471ab89fd@l6g2000vbn.googlegroups.com...
>
>
>
>
>
> > Dear all,
> > I need to recode vector v into string array given give a recoding
> > table lev and str.   I used to do something like that:
> > v = [1 1 2 2 3];
>
> > lev = [1 2 3];
> > str = {'a' 'b' 's'};
>
> > [~, idx] = ismember(v, lev);
> > v_recoded = str(idx)
>
> > Once number of levels hits hundreds of thousands this becomes too slow
> > to live with it.  I would really prefer to keep lev sorted in advance
> > so I could reduce time for searching elements within this array.  I
> > looked at the map object and the dataset objects in the statistical
> > toolbox but I cannot find desired functionality.
> > Do I need to write custom code or I am missing something here?
>
> Are all the levels positive integer values? If so, just use indexing
> directly.
>
> str = {'a', 'b', 's'};
> v = [1 1 2 2 3];
> c = str(v)
>
> --
> Steve Lord
> sl...@mathworks.com
> To contact Technical Support use the Contact Us link onhttp://www.mathworks.com- Hide quoted text -
>
> - Show quoted text -

Thank you, Steve. The idea is to attach a database-like index to lev
variable to make lookup faster so ismember() function could take
advantage of it. This would be useful for very large lev vectors/
matrices/datasets. Maybe it’s been implemented already in Statistics
Toolbox – I need to check it out.

Kirill

Kirill <kirillandreev@gmail.com> wrote in message <b0478a76-c879-4594-a92f-c202ad701aca@v31g2000vbs.googlegroups.com>...
> Thank you, Steve. The idea is to attach a database-like index to lev
> variable to make lookup faster so ismember() function could take
> advantage of it. This would be useful for very large lev vectors/
> matrices/datasets. Maybe it’s been implemented already in Statistics
> Toolbox – I need to check it out.
>
> Kirill
- - - - - - - - - -
  If Steve's indexing suggestion cannot be achieved, perhaps the following might serve. Instead of using 'lev' as it is, prepare 'u' and 'm' in advance:

 [u,m] = unique(lev);

Then for each vector v you can do this:

 [~,p] = histc(v,u);
 v_recoded = str(m(p));

  This should (in theory at least) be faster than 'ismember', since 'histc' can count on 'u' being in ascending order, whereas in your use of 'ismember' it must assume that 'lev' is unsorted (even if it is actually sorted.)

Roger Stafford

On May 31, 7:41 pm, "Roger Stafford"
<ellieandrogerxy...@mindspring.com.invalid> wrote:
> Kirill<kirillandr...@gmail.com> wrote in message <b0478a76-c879-4594-a92f-c202ad701...@v31g2000vbs.googlegroups.com>...
> > Thank you, Steve.  The idea is to attach a database-like index to lev
> > variable to make lookup faster so ismember() function could take
> > advantage of it.  This would be useful for very large lev vectors/
> > matrices/datasets.  Maybe it’s been implemented already in Statistics
> > Toolbox – I need to check it out.
>
> >Kirill
>
> - - - - - - - - - -
>   If Steve's indexing suggestion cannot be achieved, perhaps the following might serve.  Instead of using 'lev' as it is, prepare 'u' and 'm' in advance:
>
>  [u,m] = unique(lev);
>
> Then for each vector v you can do this:
>
>  [~,p] = histc(v,u);
>  v_recoded = str(m(p));
>
>   This should (in theory at least) be faster than 'ismember', since 'histc' can count on 'u' being in ascending order, whereas in your use of 'ismember' it must assume that 'lev' is unsorted (even if it is actually sorted.)
>
> Roger Stafford

Thanks, Roger. This may work -- I will try it.
Kirill

On May 20, 11:58 pm, Kirill <kirillandr...@gmail.com> wrote:
> Dear all,
> I need to recode vector v into string array given give a recoding
> table lev and str.   I used to do something like that:
> v = [1 1 2 2 3];
>
> lev = [1 2 3];
> str = {'a' 'b' 's'};
>
> [~, idx] = ismember(v, lev);
> v_recoded = str(idx)
>
> Once number of levels hits hundreds of thousands this becomes too slow
> to live with it.  I would really prefer to keep lev sorted in advance
> so I could reduce time for searching elements within this array.

That's the wrong solution.

The problem statement calls for some sort of search tree.
In C++ I'd use std::map, I have no idea if or how this
could be done in matlab.

So be prepared to either ditch matlab in favour of a
high-performance programming language, or accept the
poor run-time.

Rune

On Jun 1, 12:44 am, Rune Allnor <all...@tele.ntnu.no> wrote:
> On May 20, 11:58 pm, Kirill <kirillandr...@gmail.com> wrote:
>
> > Dear all,
> > I need to recode vector v into string array given give a recoding
> > table lev and str.   I used to do something like that:
> > v = [1 1 2 2 3];
>
> > lev = [1 2 3];
> > str = {'a' 'b' 's'};
>
> > [~, idx] = ismember(v, lev);
> > v_recoded = str(idx)
>
> > Once number of levels hits hundreds of thousands this becomes too slow
> > to live with it.  I would really prefer to keep lev sorted in advance
> > so I could reduce time for searching elements within this array.
>
> That's the wrong solution.
>
> The problem statement calls for some sort of search tree.
> In C++ I'd use std::map, I have no idea if or how this
> could be done in matlab.
>
> So be prepared to either ditch matlab in favour of a
> high-performance programming language, or accept the
> poor run-time.
>
> Rune

In Matlab one can actually create map containers that provide a very
fast way for data retrieval (according to documentation, of course).
I used them recently to cache results of sql queries and they worked
as expected. Maybe this would be a road to explore.

Kirill