Path: news.mathworks.com!not-for-mail
From: <HIDDEN>
Newsgroups: comp.soft-sys.matlab
Subject: Re: MATLAB Central Spring Contest
Date: Thu, 15 May 2008 18:34:01 +0000 (UTC)
Organization: McMaster University
Lines: 80
Message-ID: <g0hvmp$t2v$1@fred.mathworks.com>
References: <fv82i0$nql$1@fred.mathworks.com> <fvsk1d$m84$1@fred.mathworks.com> <fvsmhv$se6$1@fred.mathworks.com> <fvsot0$gr9$1@fred.mathworks.com> <fvsqp9$8gg$1@fred.mathworks.com> <g066qs$drt$1@fred.mathworks.com> <g08tt2$c2b$1@fred.mathworks.com> <g09kgq$ldb$1@fred.mathworks.com>
Reply-To: <HIDDEN>
NNTP-Posting-Host: webapp-03-blr.mathworks.com
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
X-Trace: fred.mathworks.com 1210876441 29791 172.30.248.38 (15 May 2008 18:34:01 GMT)
X-Complaints-To: news@mathworks.com
NNTP-Posting-Date: Thu, 15 May 2008 18:34:01 +0000 (UTC)
X-Newsreader: MATLAB Central Newsreader 220730
Xref: news.mathworks.com comp.soft-sys.matlab:468673


I want to add a few comments to what Steve Hoelzer has
already said about the suggested changes for the Matlab
contest ...

"Steve Hoelzer" <shoelzer@gmail.com> wrote in message
<g09kgq$ldb$1@fred.mathworks.com>...

> Ideas (Steve) liked:
>
>     4. Can't view entries until they are scored.

I would suggest a variation on this idea.

During most of the Daylight phase, allow entries in the
queue to be viewed (even before they are scored).  I think
this adds to the interest and excitement by allowing all
competitors to learn what various tricks people are trying,
and most of the time there is no benefit for waiting until
the entries are scored. ... and I wouldn't like to see any
incentive for people clogging the queue, just to delay
others from seeing their not-yet-scored entry further down
the queue.

During the final hour of the contest, go into a kind of
"Late Twilight" where entries can't be viewed until they are
scored.  This will enable competitors to submit what they
think might be a winning improvement, without needing to
wait until the last 5 minutes, and without needing to submit
100 variations of essentially the same thing.

I also think many people have real life constraints on their
schedules (at work or home, depending on time zone) that
makes it difficult for them to be at their computer
precisely moments before the deadline to trigger some
specialized software to submit 500 entries at a rate of 10
per second, or to click submit on several dozen Firefox
browser tabs they set up in advance ...




>     5. Mid-contest and end-of-contest analyses.


I agree these are really valuable and worth the effort. 
They help competitors "catch up" and understand how the key
algorithms form Darkness/Twilight are working, ... thus
increasing more effective participation in the later stages
of the contest.  They also have lasting educational value
for students who want to how to use Matlab to solve problems
similar to the Contest puzzles.

> 
> Ideas Steve was not sure about:
>     1. One hour twilight before all contest deadlines.

I agree we should keep the frenzy and free-for-all
atmosphere for all the mid-contest deadlines, ... but the
one hour twilight before the final deadline does makes sense.


>     2. Random rand seed.

I think the repeatability of scoring the entries is
critically important, so it would be a mistake to vary the
random seed.

Randomization does have a legitimate role in many realistic
algorithms.  In the Wiring Contest, there were often several
"equivalent" choices for different wire paths.  An effective
strategy was to have a very fast algorithm so you could
solve the problem two (or more) times, choosing different
"random" paths each time, and then selecting the better
solution.

We don't know the actual test suite, and we don't know the
actual random number seed.  Both are part of the same
phenomenon of "over-fitting" in the leading entries.