From: "Alan Chalker" <alanc@osc.edu>
Path: news.mathworks.com!newsfeed-00.mathworks.com!webx
Newsgroups: comp.soft-sys.matlab
Subject: Re: now that was interesting
Message-ID: <ef2fb57.141@webx.raydaftYaTP>
Date: Thu, 13 Apr 2006 16:09:22 -0400
References: <ef2fb57.133@webx.raydaftYaTP> <ef2fb57.134@webx.raydaftYaTP> <ef2fb57.135@webx.raydaftYaTP> <ef2fb57.136@webx.raydaftYaTP> <ef2fb57.137@webx.raydaftYaTP>
Lines: 99
NNTP-Posting-Host: 192.148.249.171
MIME-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
Xref: news.mathworks.com comp.soft-sys.matlab:344799


Ned:

I'm glad you are thinking about making changes. A few comments:

-The 'voting' on obscufated entries to remove is an interesting
thought, but unfortunately ripe for abuse. You suggested tracking it
by IP address. According to the statistics page, there were 175
different usernames used in this contest (albeit some are the same
person with different aliases). To be effective, 'n' would need to
be some fraction of that, let's say 25-50. Unfortunately, many
people have easy access to 25-50 IP addresses with a little bit of
work, be it through forcing a DHCP renewal on a commercial broadband
provider, to different computers in a university computer lab, to
using open-proxies, to using co-workers computers after hours. So
someone could malicously remove entries from the queue. Perhaps a
final 'gut-check' by one of your staff once an entry is flagged for
removal would address this. I also think that a similar IP-based
thing in the submission queue might be somewhat prone to the same
issues.

-I like the test-suite submission idea. Perhaps one way to deal with
it is to have a new 'pre-darkness' time frame during which the
problem rules and test suite are released but the testing queue
remains closed. During that time people could play with the problem
and test suite and then submit to the contest team suggestions for
additional tests. Your team would make a judgement somehow on
whether to include them when the queue opens up, but wouldn't
necessarily need to notify us until after the contest what was
included in the queue. Or perhaps you solicit them throughout the
contest but don't add them until certain milestones are met.

-An alternative idea regarding the test-suite is to only actively
test against a random subset of the suite in any given run. This
would make 'tweaking' to the suite a lot harder since we wouldn't
know if the score changes were due to code changes or suite changes.
At each major milestone, you could stop and have the top X entries in
the rankings run against the full suite to get the official scores.

-I'd also like to make a suggestion regarding the timing variations.
They aren't controllable or repeatable to a certain degree. I suggest
you round those results off to something about the uncertainity,
maybe a tenth or a fifth of a second. Maybe as an alternative or as
a tie-breaker you could use something like the total calls column in
the profiler, with the assumption that less total calls is more
efficient code.

-One final suggestion, what about making the contest a full 'day'
cycle. Start with darkness, then a 'dawn/twilight' phase, then
daylight, then 'dusk/twilight', and finally darkness again. We
essentially already have the last 2 steps compressed into a few hours
at the end due to the queue spamming, but by drawing them out,
flooding the queue at the end wouldn't be needed. The contest also
won't come down to whomever lucked out and choose the correct entry
to tweak in the last few minutes the queue was open.

Ned Gulley wrote:
>
>
> Alan Chalker wrote:
>> I agree it'd be nice to have contests more than
>> semi-annually, but I suspect that is asking too
>> much of The MathWorks team.
>
> You're right Alan that it's hard for us to do them much more often,
> but we definitely enjoy hosting them twice a year. Someday I can
> imagine letting people set up their own contests whenever they
> want,
> but it might be a while before we get there.
>
> Lots of good ideas here. We probably will modulate the twilight
> time
> one way or another in the next contest. I want to float two ideas
> to
> see what people think.
>
> One is to introduce subjective voting of some kind. You know how
> some
> picture sites have a button that says "This picture has
> objectionable
> content"? We could have one of those, and if n people from n
> different IP addresses (choose appropriate value of n) clicked on
> "This entry has been maliciously obfuscated" we could yank it. That
> gets us (the contest team) out of the business of passing judgment
> on
> code.
>
> The other idea is less well thought out, but suppose one phase of
> the
> contest involved letting people submit new tests to the test suite
> instead of solutions. That might prevent (or delay) the onset of
> hard-wired algorithm brittleness. There are obvious problems with
> simply submitting tests; clearly anyone with knowledge of the tests
> can use that information unfairly. At the same time, modifying the
> test suite would definitely exercise the entries in a new way. So
> I'm
> hoping that others will have some useful twists on this idea.
>
> -Ned Gulley.
> The Contest Team