|
Jeremy Smith wrote:
> Where does a researcher have to go to find information about
> changes that significantly impact code execution?
Mathworks keeps a user-accessible bug database since R14.
However, not all information is in the database, only those
bugs that someone feels "might" be of interest to users.
This differs noticeably from Cisco's approach (for one),
in which all bugs go in the same database but some bugs
might be -specifically- marked as being non-public
for security reasons (or if the bug report details would
expose customer information that the customer would prefer stayed
private.) Default private (Matlab) vs default public (Cisco).
The Mathworks bug database does not appear to offer a way
to generate reports about bugs present in particular releases
or about bugs solved in particular releases.
The Mathworks release notes for any Matlab release are
brief summarizes of new features, and often do not mention
much (if anything) about bug fixes. Cisco release notes,
on the other hand, include two sections: one of bugs
that were fixed in the release (with bug id and short short
summary in the release notes); the other of bugs that are known
to still be open in the release (again with information that
can be used to look up the bug details.)
> How is
> anyone suppose to check all of their old code to make sure
> it works with newer versions of Matlab?
How do you know that your code works with your existing
Matlab release? If you don't have unit tests or other similar
test methodologies, then you don't know whether a change
is a problem or an improvement.
I work in a research environment myself, so I know that
it is -very- common in research environments that one does
not actually -know- what the "right" answer for a program is.
You might feed in artificial or well-understood datasets and
observe the program behaviour, but unless you go through some
hefty mathematical proofs (which are often not possible) and
then do some hoary numerical analysis, you probably don't -know-
that your code is producing the "right" answer even for the
artificial datasets -- you might know that the answers are
"about right" for your test data, but once you introduce
real datasets that might have kinds of noise and data errors
you never thought of (or which you thought too improbable
to be worth spending time on) then chances are high that you
don't know whether a change is bad or not.
For example, suppose that Mathworks started deciding that
numerical accuracy was more important than speed for basic
operations such as summations, and so replaced the sum()
operator with one that took care to sort the values by
numeric significance in order to produce the highest resolution
answer it could. Unless you've already put that layer of
accuracy in your code, the answers produced by your code
would very likely be different in the new version. Which answers
would be the "right" ones? If Matlab pushed automatic parallization
into multiple cores and so ended up producing different answers for
sum() because the numeric cancellations fell at different points (on different
machines!) then which is the "right" answer?
If you already have complete test frameworks that somehow
have the "highest possible accuracy" answer coded in to them,
then when you get a new Matlab version, you just re-run your tests.
If you don't already have those rigid test frameworks, then you
probably don't really know that your existing code is "right" anyhow,
so knowing the details of every change wouldn't help you much.
> We need a tool which can check an m-file and give a nicely
> formatted report on what changes have been made to all its
> functions between two versions of Matlab. This includes
> argument syntax, default settings, phasing out functions,
> and speed.
So you want what in the software business would probably be
called a "conformance suite report". Conformance suite software
commonly runs from $100,000 to $250,000, per license (because
it is a lot of work to build the software and there are
surprisingly few people who care about complete accuracy
of their programs, so the work put in to the suite
has to be recaptured over a small market.) Even the C programming
language has only 2 or 3 vendors of compiler conformance suites
and one would -think- that with the wide variety of C compilers around
that a lot of people would be very concerned about accuracy of their
C compilers.
My personal opinion is that if you need the kind of tool that
you mention above, then you are writing in completely the
wrong programming language. Look at the documentation of most any
of the Matlab major routines, and notice that they contain *no*
discussion of numeric analysis, and notice that they contain
no options to allow you to increase accuracy at the expense of
program speed. The people who write the routines might -individually-
be quite concerned with accuracy, but Matlab as a product is not
and never has been. And just like you can never graft in security
into a program as an after-thought, you can never graft in accuracy
to a large tool such as Matlab as an after-thought.
A question: have you tested your routines with JIT (Just In Time)
optimization 'on' versus 'off'? Mathworks changes the details of JIT
in every release, so complex programs are likely to get different
answers in every release. If that is not a factor you have already
specifically taken into account, then the tool you describe would
be gilding the rotten lily.
I am not without sympathy for the difficulties of moving between
different releases, but I think that it would be sufficient for
Mathworks to beef up their release notes. On the other hand, presenting
the change notes for every routine would probably be far far too much
information to be readily absorbed.
--
Q = quotation(rand);
if isempty(Q); error('Quotation server filesystem problems')
else sprintf('%s',Q), end
|
Feed for this Thread