How to store and retrieve multiple data sent from worker nodes to client node in MATLAB?

q = parallel.pool.DataQueue;

parfor i = 1:20  
    data = ['test - ', num2str(i)];
    send(q,data);
end

This is the code I used to send data from the worker nodes to the client node. But I don't know if it helps me store the data I send from the workers to the client. The Queue length happens to be 20 which is correct but if it contains the data, I don't know how to retrieve it. I tried using a PollableDataQueue and poll() but it retrieves only one value which I guess is the last value.

So my questions are :

• Is the method I'm using to store data sent from the workers correct?
• If yes, how do I retrieve the stored data?
• If no, how do I store data sent from the workers in the client?
• Is the method I'm using to send data from the workers to the client correct?

Answer 1

[Walter Roberson]

https://www.mathworks.com/help/distcomp/aftereach.html

[Viswanath Hariharan]

I did try using aftereach() without success. The queue length shows to be 20. How do I retrieve these items from the queue? How do I access the data in data queue? Could you please guide me to use aftereach() to store data in let's say a stack of cells with an example code or by adding a line of code to mine?

Thank you so much.

[Walter Roberson]

aftereach() automatically calls the given function on each queue entry as it comes in.

If you use a pollable data queue then you can look at QueueLength to see if anything is there, and you can use poll() to retrieve a value.

There is no provision to access the queue members out of order.

function testQueue

    q = parallel.pool.DataQueue;
    stack_of_cells = {};
    afterEach(q, @store_on_stack)

    parfor i = 1:20  
        data = ['test - ', num2str(i)];
        send(q,data);
    end

    disp('done')
    celldisp(stack_of_cells)

    function store_on_stack(data)
      stack_of_cells{end+1} = data;
    end
  end

[Viswanath Hariharan]

Thank you so much. That was really helpful. One last question, aftereach() calls the given function on each queue entry as it comes in. Is there any command/method to retrieve, in this case all 20 string data from the queue at once as opposed to retrieving data after each entry?

[Walter Roberson]

If you

p_struct = struct(p);
Qimpl = p_struct.Queue;

then Qimpl will be a java.util.concurrent.LinkedBlockingQueue that you could theoretically apply the drainTo() method on, after having constructed an appropriate java Collection to store into.

https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/LinkedBlockingQueue.html

[Viswanath Hariharan]

I tried executing the 2 lines of code above. I got a warning -

"Warning: Calling STRUCT on an object prevents the object from hiding its implementation details and should thus be avoided. Use DISP or DISPLAY to see the visible public details of an object."

Should I be worried about this warning?

[Viswanath Hariharan]

I tried the code you suggested, the one using aftereach(). I got an error :

Warning: Undefined function or variable 'stack_of_cells'.

I got this in all 20 iterations. I don't think the function store_on_stack() gets the cell variable. Am I doing something wrong?

[Walter Roberson]

In R2017b, the code works exactly as-is for me.

The warning about calling struct is to be expected. You can avoid having it printed out using code such as

                   old_warning_state = warning('off', 'MATLAB:structOnObject');
                   values = struct(values);
                   warning(old_warning_state);

[Viswanath Hariharan]

It works now. It was my mistake I guess. I didn't define the code in a function i.e. I didn't type in function testqueue. After I did that, everything worked. Thank you.

[Walter Roberson]

Ah, yes. The testQueue I posted relies upon shared variables and nested functions; I did not mention that.

[Viswanath Hariharan]

Is there a way I can do this without relying on shared variables and nested functions? Because when I use this for my application, it produces a logical error that I am not able to clear.

[Walter Roberson]

http://matlab.wikia.com/wiki/FAQ#How_can_I_share_data_between_callback_functions_in_my_GUI.28s.29.3F

You should be able to push your parfor call into a function that you pass all appropriate variables to: indeed, isolating the parfor is recommended, as it can make it easier for the parser to classify the variables, by reducing the scope of what needs to be analyzed as possible inputs and outputs.