Why memory consumption while using parfor in Windows and Linux so differs

Hello,

I test local parfor MATLAB R2017a in Windows (Server R2012 R2 x64) and Ubuntu (16.04.1-Ubuntu x86_64).

I have different results in memory consumption using simple test:

parpool('local', 4, 'SpmdEnabled', false);

tic;
for j = 1:1
    parfor i = 1:4
        a = randi(255, 2000000, 10); b = randi(255, 2000000, 10);
        [ia, ib] = ismember(a, b, 'rows');
    end
end
toc;

1.1. In windows ( on parpool start):

1.2. And after script execution (listing above):

2.1. In Linux ( on parpool start):

In Linux System Monitor on Russian - units for memory is KiB.

2.2. And after script execution (listing above):

Why after operation (ismember) memory on Linux is not freed?

Thank you.

Answer 1

[Walter Roberson]

The default MS Windows memory allocators and Linux memory allocators differ in how they handle giving back memory to the operating system.

The traditional Unix allocators would check their free tables for available memory, ask the OS for more memory if they needed it, and then afterwards free the memory to its internal lists, more or less figuring that if you needed that much memory once then you will probably need it again.

There are other available Unix memory allocators, some of which keep track in a per-pool basis, and when there is nothing remaining in the pool, return the pool to the OS. The use of these is typically restricted to specific chunks of work where it would be expected that at the end of the chunk there would be no further need for any of the allocated memory.

This would not normally be at the individual mathematics operation level, but rather at the algorithm level, where a routine has a well-defined interface. For example it could be done at the level of a routine that takes in input and calculates a bunch of things and writes outputs to a file, returning no new memory. Such a routine might require that the address of all final output memory be passed in to the routine so that the routine can know for sure that anything it needs to allocate for internal processing can be done in the pool and then the pool can be deleted, after copying out any outputs from the pool into the given output buffers.

This is not MATLAB public semantics: MATLAB public semantics with its copy-shares-on-write says that once memory is allocated, the pointer to it gets passed around as many times as needed including on output: if memory got allocated out of a pool then the pool might have to live indefinitely, until the final end of all variables and modifications there to. There are circumstances under which arrays can be modified in-place instead of creating a new array the same size, reading from the old, writing modified to the new, and then at the end deleting the old, so any memory allocated from a pool could live on indefinitely -- at least under the public semantics. We do not know what semantics Mathworks has chosen internally: they could have chosen to make some routines be strict "here's the output buffer, use a pool for any internal calculations and be sure to delete the pool afterwards."

There is not a lot of information publicly available about the extent to which MATLAB deliberately uses pools. It is known that MATLAB keeps at least one small-block pool of fixed-sized blocks, and that if the memory request fits into one of those that the fixed-sized block is handed over and later the entire block will be returned; and that for other requests that do not fit (either larger than the fixed-sized blocks or they have run out of fixed-sized blocks) then a more general purpose allocator is used.

Thus, this could just be done to a difference in which memory allocator is linked against.

[Jonas]

Walter, thank you for detailed answer.

[Dmitriy Ogureckiy]

Walter, I have the same results, but I need to know how I can free memory after simulation, because my MATLAB broke?

[Walter Roberson]

@Dmitriy Ogureckiy

There is no longer any user-accessible way to ask MATLAB to clean up. You can clear variables and delete graphics objects and kill processes if needed (though MATLAB generally manages those), but you can no longer ask it to condense its memory usage.

In the past there was pack to request that MATLAB clean up its memory. However after the switch to 64 bits, and improvements in the execution engine, Mathworks believes that MATLAB already manages memory well enough, so it removed the pack function.