Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Connection pooling & serializing connection for parfor

Subject: Connection pooling & serializing connection for parfor

From: yash

Date: 21 Jul, 2010 17:36:11

Message: 1 of 5

Hello,

I'm writing a matlab program that fetches lots of data from a
database. Given the structure of the tables and the nature of the
data, I'm forced to run multiple queries multiple times. For instance,
I may decide that I want to get a dozen attributes for each of 1000
stocks. Getting each attribute requires a separate DB query.

In order to get better performance, I'm using a parfor loop to loop
through the 1000 stocks and fetch data. Since I'll have multiple
parallel loops getting data, I don't want to have them all use a
single connection, and so I create a connection in the parfor loop and
close it before exiting.

As you can imagine, the repeated opening and closing of connections is
very costly and slows down the program. Ideally I'd love to leverage
some kind of connection pooling to avoid the connect/disconnect. I see
that there's no native connection pooling in Matlab and so wrote my
own class to do so.

The issue now is that I believe Matlab serializes objects when passing
data to workers using parfor. The connection object can't be
serialized. So, when the code in my parfor loop tries to request a
connection from the pool, it fails since the pool would have to be
serialized for the worker to see it, and connections can't be
serialized.

Thus, I can't use a connection in the parfor loop that was created
outside the loop, and creating connections in the loop is very costly.
Does anyone have any ideas on how I can implement some kind of
connection pooling in this scenario, or some other way to speed things
up by avoiding the frequent connect/disconnect.

FYI - the database is SQL 2008, which does not implement connection
pooling on the server side. If it did, I could leave the pooling to
SQLServer and not have this issue...

Thanks!

Subject: Connection pooling & serializing connection for parfor

From: Abhishek

Date: 26 Jun, 2012 21:27:07

Message: 2 of 5

@Yash, were you able to find a solution on how to pass the connection objects into a parfor loop? I have a very similar issue. Thanks.

Subject: Connection pooling & serializing connection for parfor

From: Markella

Date: 2 Aug, 2013 13:07:07

Message: 3 of 5

Hello,
I can't get fetch to work in a parfor loop - even if I initiate the connection within it, how did you manage to get the results back?

Regards
Markella

Subject: Connection pooling & serializing connection for parfor

From: Edric M Ellis

Date: 6 Aug, 2013 13:19:13

Message: 4 of 5

"Markella " <markella.skempri@onzo.com> writes:

> Hello, I can't get fetch to work in a parfor loop - even if I initiate the
> connection within it, how did you manage to get the results back?

I'm not sure what you mean: please could you post an example of some
code that you think should work and doesn't.

Cheers,

Edric.

Subject: Connection pooling & serializing connection for parfor

From: Markella

Date: 6 Aug, 2013 14:19:05

Message: 5 of 5

Thanks Edric - i didn't realise that in the end I had to pass and load the drive to every worker in order to initiate a connection. I have now included it in the worker startup file and I am waiting to try it out.
Regards

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us