Thread Subject: FEX file types

The MATLAB FEX seems to have a new policy on acceptable file types. Mex files and dlls are no longer acceptable as part of a submission.
Any views?

"Malcolm Lidierth" <ku.ca.lck@htreidil.mloclam> wrote in message <gge93p$7vi$1@fred.mathworks.com>...
> The MATLAB FEX seems to have a new policy on acceptable file types. Mex files and dlls are no longer acceptable as part of a submission.
> Any views?

IMHO, this is as it should be.

Precompiled code will very often not run under any fixed
choice of OS or MATLAB release.

Worse, suppose someone chooses to slip something nasty
(and possibly viral) into the code?

John

John

Point taken, but pure m-code can pose security problems also. System, dos, unix, ! etc can all tucked into an m-file by those that choose, so "user beware" applies there too.

Mex-files should not be posted without reason, but sometimes there is good reason.
In my specific case, the zip contains dlls supplied by the relevant manufacturers as 'redistributables' to read proprietory file formats. It also contains mex-files to link to these dlls where loadlibrary can not work (e.g. with non-scalar structures which are fairly common in file headers). Of 559 m-files in the zip, only two are shadowed by a mex for speed reasons and these are supplied for Win32, Mac and Linux. All mex files are accompanied by source code.
Mex-files/dlls should be discouraged. But should they be barred?

Regards
Malcolm

"Malcolm Lidierth" <ku.ca.lck@htreidil.mloclam> wrote in message <ggebi1$ach$1@fred.mathworks.com>...
> John
>
> Point taken, but pure m-code can pose security problems also. System, dos, unix, ! etc can all tucked into an m-file by those that choose, so "user beware" applies there too.
>
> Mex-files should not be posted without reason, but sometimes there is good reason.
> In my specific case, the zip contains dlls supplied by the relevant manufacturers as 'redistributables' to read proprietory file formats. It also contains mex-files to link to these dlls where loadlibrary can not work (e.g. with non-scalar structures which are fairly common in file headers). Of 559 m-files in the zip, only two are shadowed by a mex for speed reasons and these are supplied for Win32, Mac and Linux. All mex files are accompanied by source code.
> Mex-files/dlls should be discouraged. But should they be barred?


Are you absolutely sure that you are allowed to redistribute their redistributables yourself? I would doubt it.

Steve
Yes. Most of the manufacturer supplied dlls are part of a "collaborative, vendor-neutral" project "dedicated to public domain standards":
http://neuroshare.sourceforge.net/mission.shtml
partly funded by the US National Institute for Neural Disorders and Stroke . All manufacturers have been told what I am doing and have raised no objections.
The dlls are all available on the web in any case - I include them in the zip for the convenience of the end-user.

I have a specific reason for including mex/dll files. I was hoping to gauge how many others also need that facility.

ML

"Malcolm Lidierth" <ku.ca.lck@htreidil.mloclam> wrote in message <ggef4f$3qi$1@fred.mathworks.com>...
> Steve
> Yes. Most of the manufacturer supplied dlls are part of a "collaborative, vendor-neutral" project "dedicated to public domain standards":
> http://neuroshare.sourceforge.net/mission.shtml
> partly funded by the US National Institute for Neural Disorders and Stroke . All manufacturers have been told what I am doing and have raised no objections.
> The dlls are all available on the web in any case - I include them in the zip for the convenience of the end-user.
>
> I have a specific reason for including mex/dll files. I was hoping to gauge how many others also need that facility.

Interesting. What's the position if one of these DLLs stops working? Do you direct your users to the DLL manufacturer? Do they then blame you for breaking it? It gets messy very quickly.

"Malcolm Lidierth" <ku.ca.lck@htreidil.mloclam> wrote in message <gge93p$7vi$1@fred.mathworks.com>...
> The MATLAB FEX seems to have a new policy on acceptable file types. Mex files and dlls are no longer acceptable as part of a submission.
> Any views?

Yes,
Users without the ML compiler cannot use mexs prepared by others.

Steve
OK. There are two options:
[1] Write an m-file library to support each file type by doing binary reads. That is what I did for the first file type that I included. This has the advantage of being platform-independent but it is a big job for these, often complex, file formats . It has many disadvantages. Lots of code = lots of opportunities for bugs/exceptions. Also, what happens when the manufacturer updates the file format?
[2] Call a 'high level' function in the manufacturer's dll. That is easy. If the file format gets updated, so (hopefully) will the dll - and by the manufacturer. In fact, most of the manufacturers actively encourage the use of the dlls for this very reason.

Let's take a more general-interest example. Micah Richert's excellent mmread function on the FEX calls a mex/dll that in turn uses Windows DirectX to read multimedia files. Is this a useful contribution to the FEX: plainly yes. But would you argue that Micah should instead have written m-code to read each of the supported formats?

The ML Compiler is not needed to run mex/dll files within MATLAB- only to create stand-alone code.

"Steve Amphlett" <Firstname.Lastname@Where-I-Work.com> wrote in message <ggefqv$ed9$1@fred.mathworks.com>...
> "Malcolm Lidierth" <ku.ca.lck@htreidil.mloclam> wrote in message <ggef4f$3qi$1@fred.mathworks.com>...
> > Steve
> > Yes. Most of the manufacturer supplied dlls are part of a "collaborative, vendor-neutral" project "dedicated to public domain standards":
> > http://neuroshare.sourceforge.net/mission.shtml
> > partly funded by the US National Institute for Neural Disorders and Stroke . All manufacturers have been told what I am doing and have raised no objections.
> > The dlls are all available on the web in any case - I include them in the zip for the convenience of the end-user.
> >
> > I have a specific reason for including mex/dll files. I was hoping to gauge how many others also need that facility.
>
> Interesting. What's the position if one of these DLLs stops working? Do you direct your users to the DLL manufacturer? Do they then blame you for breaking it? It gets messy very quickly.

Steve, I fail to see how this gets any messier than if I package a tool of mine with some GPL-licensed code (m, c, whatever), and that GPL-code breaks.

The security concerns I understand and agree with, if there is only source code I can at least search through (or hope that others do) for malicious content.

Can't you submit the source code for the dlls?

Bjoern

Bjoern
The source code is included for all mex/dlls files where available (including all those written by me).
I include compiled files for the convenience of the end-user - not all of the target audience will have a C++ compiler installed or be familiar with mex -setup.
ML

"Malcolm Lidierth" <ku.ca.lck@htreidil.mloclam> wrote in message <ggehue$mnn$1@fred.mathworks.com>...
> Steve
> OK. There are two options:
> [1] Write an m-file library to support each file type by doing binary reads. That is what I did for the first file type that I included. This has the advantage of being platform-independent but it is a big job for these, often complex, file formats . It has many disadvantages. Lots of code = lots of opportunities for bugs/exceptions. Also, what happens when the manufacturer updates the file format?
> [2] Call a 'high level' function in the manufacturer's dll. That is easy. If the file format gets updated, so (hopefully) will the dll - and by the manufacturer. In fact, most of the manufacturers actively encourage the use of the dlls for this very reason.
>
> Let's take a more general-interest example. Micah Richert's excellent mmread function on the FEX calls a mex/dll that in turn uses Windows DirectX to read multimedia files. Is this a useful contribution to the FEX: plainly yes. But would you argue that Micah should instead have written m-code to read each of the supported formats?

Malcolm, Please don't get me wrong. I understand the technical benefits of using a manufacturer DLL through a well defined interface. Indeed, we provide pre-compiled libs for accessing our binary files and for co-simulating with our programs. I've been on both sides of the redist game: both redistributing 3rd party libs and also seeing my own libs being redistributed by a 3rd party. In the former case, the libs are largely compiler and/or OS libs (in the same way TMW provide os libs). But in the latter case, the redistributor was a large company and the agreement we signed was very clear about responsibilities to joint customer(s).

My message is really one of caution.

"Malcolm Lidierth" <ku.ca.lck@htreidil.mloclam> wrote in message <gge93p$7vi$1@fred.mathworks.com>...
> The MATLAB FEX seems to have a new policy on acceptable file types. Mex files and dlls are no longer acceptable as part of a submission.
> Any views?

Some of the most powerful submissions in the FEX are .dll or .mexw32.

a programmer wrote:
> "Malcolm Lidierth" <ku.ca.lck@htreidil.mloclam> wrote in message <gge93p$7vi$1@fred.mathworks.com>...
>> The MATLAB FEX seems to have a new policy on acceptable file types. Mex files and dlls are no longer acceptable as part of a submission.
>> Any views?
>
> Some of the most powerful submissions in the FEX are .dll or .mexw32.
>
>

I'm not on the MATLAB Central team, but I follow the goings-on with
interest. So I went looking for information about the issue of MEX-file
submissions to the File Exchange.

It's true that you can no longer submit MEX-file binaries to the File
Exchange. There is a File Exchange FAQ that mentions this:

http://matlabwiki.mathworks.com/Using_File_Exchange

I've been told that the MATLAB Central team plans to add information to
the submission form, and maybe other places, to make this clear.

However, you can still submit MEX-file source code. Your users will
have compile the MEX source themselves. C and C++ compilers are
available for free on just about every MATLAB platform (gcc on Linux and
Mac; Visual Studio C++ Express Edition on Windows; I don't know about
Solaris).

---
Steve Eddins
http://blogs.mathworks.com/steve/

Personnaly I am very disapointed by this new FEX rule even if I understand perfectly. I agree that the compiled mex files are usually working on a subset of existing plateform. However, in my submissions, I usually included optimized compiled mex-filles for win32 plateform; compiled for example with the last build of the Intel Compiler that not everyone can reach. Of course each user can recompile mex-files for their own plateform.

Now I think that a lot of previous submissions can't be update right now..... (with 3thrd party dll for example). What's a pity

Sebastien

Steve Eddins <Steve.Eddins@mathworks.com> wrote in message <ggk1tm$8r2$1@fred.mathworks.com>...
> a programmer wrote:
> > "Malcolm Lidierth" <ku.ca.lck@htreidil.mloclam> wrote in message <gge93p$7vi$1@fred.mathworks.com>...
> >> The MATLAB FEX seems to have a new policy on acceptable file types. Mex files and dlls are no longer acceptable as part of a submission.
> >> Any views?
> >
> > Some of the most powerful submissions in the FEX are .dll or .mexw32.
> >
> >
>
> I'm not on the MATLAB Central team, but I follow the goings-on with
> interest. So I went looking for information about the issue of MEX-file
> submissions to the File Exchange.
>
> It's true that you can no longer submit MEX-file binaries to the File
> Exchange. There is a File Exchange FAQ that mentions this:
>
> http://matlabwiki.mathworks.com/Using_File_Exchange
>
> I've been told that the MATLAB Central team plans to add information to
> the submission form, and maybe other places, to make this clear.
>
> However, you can still submit MEX-file source code. Your users will
> have compile the MEX source themselves. C and C++ compilers are
> available for free on just about every MATLAB platform (gcc on Linux and
> Mac; Visual Studio C++ Express Edition on Windows; I don't know about
> Solaris).
>
> ---
> Steve Eddins
> http://blogs.mathworks.com/steve/

Dear all!

There have been important examples on the FEX, which are stored as DLL, e.g. "uigetfiles":
http://www.mathworks.com/matlabcentral/fileexchange/331
I was always wondering, why the source code has not been included in addition.
Nevertheless, if the source code is available, it is not sure, that the compiler installed on the user's machine creates a mex file with identical behaviour. The best example are the usage of Intel-compilers with high optimization.

Matlab includes the LCC-Compiler for Windows, but it cannot find the library for the i.e. "stat" command in Matlab 2008b.
Another example is the comparison with NaN's with the Borland BCC5.5 compiler:
NaN>0 is TRUE, NaN<Inf is FALSE. This is not a bug! It is conform with the specifications. But the LCC compiler uses a different definition: any comparison with a NaN is false!
Sharing a function, which uses a fast MEX file, should not expect a user to find such problems and fix them using a variety of compilers!

The danger of downloading a virus seems to be the main problem: While "!format C:" in the Matlab source can be found in theory by any Matlab user reading the source, functions hidden in DLLs are harder to analyse. Nevertheless, PDFs can include Java code also and security problems can appear for HTML also (by forwarding calls to phishing sides).

However, if anyone really wants to add a binary compiled file, it can be stored as bin2hex or base64 string in the comments of an M-file. The M-file needs a simple de-compressing method then. But it is definitely not my opinion that such tricks are helpful for the scientific sharing of programs on a public platform as the FEX!

So long, my personal conclusion:

Dear MathWorks!
Please allow the distribution of compiled MEX files in the FEX, if the source code is included also. For some functions this can be fundamental e.g. to control the different behaviour of compilers. In opposite to MEX files, I agree that P-files should be excluded from the public FEX.

Thanks, Jan Simon

Jan Simon wrote:
> Another example is the comparison with NaN's with the Borland BCC5.5 compiler:
> NaN>0 is TRUE, NaN<Inf is FALSE. This is not a bug! It is conform with the specifications.
> But the LCC compiler uses a different definition: any comparison with a NaN is false!

Citation please?

The classic paper "What Every Computer Scientist Should Know About Floating-Point Arithmetic"
http://www.validlab.com/goldberg/paper.pdf
has this to say on page 38 of the PDF:

  Since comparing a NaN to a number with <, <=, >, >=, or = (but not =/ ) always
  returns false

(Note: =/ is the not-equal operator)

NaN>0 is an instance of "comparing a NaN to a number using the > operator", so if
Goldberg's paper is correct (and one would think that many many people have looked
it over and would have found a flaw by now), then the result of NaN>0 must be *false*,
not *TRUE*.

Is this a fundamental behaviour that changed in IEEE 754-2008 ? Unfortunately the
IEEE 754 standards themselves are not free, so I cannot download them to cross-check.
Please provide a specific reference for your claim.

--
.signature note: I am now avoiding replying to unclear or ambiguous postings.
Please revi
ew questions before posting them. Be specific. Use examples of what you mean,
of what you don't mean. Specify boundary conditions, and data classes and value
relationships -- what if we scrambled your data or used -Inf, NaN, or complex(rand,rand)?

Let's striking :) ....


"Jan Simon" <matlab.THIS_YEAR@nMINUSsimon.de> wrote in message <ggmjs6$nu0$1@fred.mathworks.com>...
> Dear all!
>
> There have been important examples on the FEX, which are stored as DLL, e.g. "uigetfiles":
> http://www.mathworks.com/matlabcentral/fileexchange/331
> I was always wondering, why the source code has not been included in addition.
> Nevertheless, if the source code is available, it is not sure, that the compiler installed on the user's machine creates a mex file with identical behaviour. The best example are the usage of Intel-compilers with high optimization.
>
> Matlab includes the LCC-Compiler for Windows, but it cannot find the library for the i.e. "stat" command in Matlab 2008b.
> Another example is the comparison with NaN's with the Borland BCC5.5 compiler:
> NaN>0 is TRUE, NaN<Inf is FALSE. This is not a bug! It is conform with the specifications. But the LCC compiler uses a different definition: any comparison with a NaN is false!
> Sharing a function, which uses a fast MEX file, should not expect a user to find such problems and fix them using a variety of compilers!
>
> The danger of downloading a virus seems to be the main problem: While "!format C:" in the Matlab source can be found in theory by any Matlab user reading the source, functions hidden in DLLs are harder to analyse. Nevertheless, PDFs can include Java code also and security problems can appear for HTML also (by forwarding calls to phishing sides).
>
> However, if anyone really wants to add a binary compiled file, it can be stored as bin2hex or base64 string in the comments of an M-file. The M-file needs a simple de-compressing method then. But it is definitely not my opinion that such tricks are helpful for the scientific sharing of programs on a public platform as the FEX!
>
> So long, my personal conclusion:
>
> Dear MathWorks!
> Please allow the distribution of compiled MEX files in the FEX, if the source code is included also. For some functions this can be fundamental e.g. to control the different behaviour of compilers. In opposite to MEX files, I agree that P-files should be excluded from the public FEX.
>
> Thanks, Jan Simon
>

Dear Walter Roberson!

>> Another example is the comparison with NaN's with the Borland BCC5.5 compiler:
>> NaN>0 is TRUE, NaN<Inf is FALSE. This is not a bug! It is conform with the specifications.
>> But the LCC compiler uses a different definition: any comparison with a NaN is false!

You are right: Goldberg publication is not equivocal:
> comparing a NaN to a number with <, <=, >, >=, or = (but not =/ )
> always returns false

You are right: IEEE-754 includes an exact definition of NaNs, and this has not been chnaged in 2008!
> Is this a fundamental behaviour that changed in IEEE 754-2008 ? Unfortunately the
> IEEE 754 standards themselves are not free, so I cannot download them to cross-check.

> Citation please?
> Please provide a specific reference for your claim.
I do not think that citations and references help here. As you, I cannot see any lacks in the **theory**.

For the **practice** let me cite: Prof. W. Kahan, Lecture Notes on the Status of IEEE Standard 754 for Binary Floating-Point Arithmetic, 1997.
Kahan: "NaN must not be confused with “ Undefined.” On the contrary, IEEE 754 defines NaN perfectly well even though most language standards ignore and many compilers deviate from that definition."

Example:
#include "mex.h"
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
  double x, y;
  x = mxGetNaN();
  y = mxGetInf();
  
  if (x < 0)
    mexPrintf("NaN < 0\n");
  if (x > 0)
    mexPrintf("NaN > 0\n");
  if (x > y)
    mexPrintf("NaN > Inf\n");
  return;
}

Borland C++ 5.5.1 for Win32 Copyright (c) 1993, 2000 Borland
  NaN < 0
  NaN > Inf

LCC shipped with Matlab 6.5.1: Logiciels/Informatique lcc-win32 2.4.1 Mathworks patch 1.29, compiled 11-Jul-2003
  NaN > 0
  NaN > Inf
(same for the free version 3.8, compiled 24-Jul-2008, or LCC shipped with Matlab 2008b)

Expected according to IEEE 754:
  no output, because each comparison must reply FALSE.

And this is the point where FEX users should not be left alone. If I publish C-source on the FEX, I do like to add a compiled version, which is tested exhaustively.

On the other hand, any cute FEX distribution containing compiled MEX scripts should include a unit test function, which checks the reactions to Inf, NaN, NULL (occuring as element of a cell after "cell(1, 1)") etc.! But assume, such a unit test routine find deviations from the expected behaviour -- i.e. caused by lazy implementations of the IEEE standards in different compilers: how could the reliable sharing of programs work without sharing compiled binaries ?!

Kind regards, Jan

In article <ggmjs6$nu0$1@fred.mathworks.com>,
 "Jan Simon" <matlab.THIS_YEAR@nMINUSsimon.de> wrote:

> There have been important examples on the FEX, which are stored as DLL

[snip]

> Nevertheless, if the source code is available, it is not sure, that the
> compiler installed on the user's machine creates a mex file with identical
> behaviour. The best example are the usage of Intel-compilers with high
> optimization.

I agree.


[snip]

> Sharing a function, which uses a fast MEX file, should not expect a user to
> find such problems and fix them using a variety of compilers!
>
> The danger of downloading a virus seems to be the main problem: While
> "!format C:" in the Matlab source can be found in theory by any Matlab user
> reading the source, functions hidden in DLLs are harder to analyse.
> Nevertheless, PDFs can include Java code also and security problems can
> appear for HTML also (by forwarding calls to phishing sides).

Such a program is called a Trojan horse, but I agree with your
observations.

[snip]

> Dear MathWorks!
> Please allow the distribution of compiled MEX files in the FEX, if the source
> code is included also. For some functions this can be fundamental e.g. to
> control the different behaviour of compilers. In opposite to MEX files, I
> agree that P-files should be excluded from the public FEX.


Unfortunately, there is no guarantee that the supplied mex file was
compiled from the supplied source code.

There is nothing stopping you from making a mex file available on your
own web site with the URL included with the source code -- for the brave
and/or trusting.

--
Doug Schwarz
dmschwarz&ieee,org
Make obvious changes to get real email address.

Doug Schwarz <see@sig.for.address.edu> wrote in message
>
> Unfortunately, there is no guarantee that the supplied mex file was
> compiled from the supplied source code.
>
> There is nothing stopping you from making a mex file available on your
> own web site with the URL included with the source code -- for the brave
> and/or trusting.

Yep,

Sourceforge is well known for distributing kilopounds of trojan, virus, bacteria
in their .exe and .dll hosted files.

I have the same problem as Sebastien above - I'd quite like to submit some mex files which are compiler specific. I've got the same compiler running on a number of different architectures and OS's, so could provide source code, plus a range of precompiled MEX files.

But unless someone has the Intel Fortran Compiler, they wouldn't be able to use any of the submissions I could make...

(I know, I should write code that's compiler independent.... but why would I when IFC has such good and convenient math libraries???)

It's also a lot more convenient... any beginners to MATLAB may find compiling MEX files to be a daunting task, particularly if they encounter compiler dependancies.

My vote:
- Allow MEX files
- Require source code too
- Acknowledge a Mathworks disclaimer that MEX files may contain harmful content, and use at your own risk.

Cheers for a thought provoking thread...

Tom