Thread Subject: MATLAB Programming Contest: November 29-December 6, 2006

The next MATLAB Programming Contest runs from Wednesday November 29th
at 9AM EST through Wednesday, December 6th at 5PM EST. For complete
details see the website:

 <http://www.mathworks.com/contest/>
    
As in recent contests, there will be three stages, darkness,
twilight, and daylight. During darkness, both the code and scores for
all entries will be hidden. In twilight, the scores will be visible
but the code will still be hidden. When daylight arrives, all scores
and code are visible. We'll also be holding several mid-contest prize
giveaways of MATLAB bags, caps, and other good stuff, so don't wait
until the last minute to participate.
    
Please use this thread to talk about the contest, strategies, or to
ask related questions. If you have an "administrative" type of
question that you don't feel applies to anyone else, e-mail us at
contest@mathworks.com.
 
We've also started a Contest Blog where we will post the latest news,
so be sure to check it out:
 
 <http://blogs.mathworks.com/contest/>
 
Good luck everyone!
    
- The MATLAB Contest Team -

The MATLAB Contest Team wrote:
>
>
> The next MATLAB Programming Contest runs from Wednesday November
> 29th
> at 9AM EST through Wednesday, December 6th at 5PM EST.

Good luck to all obfuscators, those who will grumble about too much
light and not enough darkness, and everyone else!

Sadly, I don't have access to Matlab anymore. I probably won't be
able to resist trying some tweaks, but I think the mantle will be
passed.

I believe Mathworks still own us an explanation why obfuscation gives
an advantage, even with the JIT compiler.

Good luck all.

snake wrote:
>
>
> I believe Mathworks still own us an explanation why obfuscation
> gives
> an advantage, even with the JIT compiler.
>
> Good luck all.
  

I agree. I'm still curious about this too.

How is the final score computed? Is it the sum of 'Error' and 'Beam
Count'? The final result still depends exponentially on the time
spent in finding the solution? Or, is this left out of the rules on
purpose?

Thank You.

DreadNox wrote:
>
> How is the final score computed? Is it the sum of 'Error' and 'Beam
> Count'? The final result still depends exponentially on the time
> spent in finding the solution? Or, is this left out of the rules on
> purpose?

When I wrote this I hadn't seen the code yet. The score is 'Error'
plus 0.1 times 'Beam Count'.

My bad.

when dealing with bounary cases, do you think the definitions of
'deflection' and 'reflection' are a little bit confusing?

Thinking about these cases:

For the pictorial example in the rule page, if you shine a beam from
the bottom(1st column) as [r,c,s] = beam(0,-1), what you will get for
r c s?

or, you may test the first test example in the testsuite by shining a
beam from the top (first column again) as [r,c,s]=beam(0,1), while
there is a '6' at position (1,2). I might inteprete that a
'deflection' will happen, since by definition "if the laser passes
right next to an object, it is deflected 90 degrees as shown above."
However, the returned result from beam() will be [r=0,c=1,s=6], which
has no such effect of "deflected 90 degrees" at all, but is more
likely a 'reflection'.

websiteng wrote:
> when dealing with boundary cases, do you think the
> definitions of 'deflection' and 'reflection' are a
> little bit confusing?

The boundary case you describe is indeed a little confusing.
"Physically" the beam(0,-1) would be deflected, but for the purposes
of reporting the result (and since there is nowhere for the deflected
beam to go) this situation is simply defined as a reflection. It's
sort of like saying factorial(0) = 1. It's a matter of definition.

So beam(0,-1) results in [0,-1,2].

-Ned Gulley.
The MATLAB Contest Team

I am using MATLAB 6.5 R13. Can anyone post changes required in code
to run program

I assume that multiple rules can be applied in sequence, so for
example it is possible for a ray to be absorbed after being deflected
(though not vice versa)?

It will probably become obvious looking through the code, but from
the description it is not clear what happens when multiple rules
apply at the same time. For example, when a beam hits an object in
the near edge of the box (=> absorbed), when there is also an
adjacent object on the edge of the box (=> reflected), which
happens?

I would some help making this work on R13 as well. I get the
following error...

>> runcontest
??? Error: File: C:\Documents and
Settings\Administrator.ARTISTS\Desktop\blackbox\beam.m Line: 175
Column: 17
"]" expected, "identifier" found.

Error in ==> C:\Documents and
Settings\Administrator.ARTISTS\Desktop\blackbox\runcontest.m
On line 33 ==> beam(testsuite{k});

snake and Alan, I did some investigating after the last contest to see if I
could explain the strange timing behavior that obfuscation was producing. I
have to say the inner workings of the interpreter is not my area, but I
think I know what was going on.



The main effect came from jamming all the code together onto long lines.
Last contest, when we upgraded our testing computer, we moved to 64-bit
Linux. The problem was that in the then-current version of MATLAB, the JIT
wasn't as effective on 64-bit as it was in our 32-bit platforms. (JIT is
shorthand for our just-in-time compiler technology that generates
machine-code instructions on the fly.) The code would run faster on most
contestants' machines when each line of code was on its own line because the
JIT prefers it this way. On the contest machine however, where the JIT was
less effective, mashing the lines of code together reduced the overall
runtime because the interpreter has an overhead associated with each line of
code executed, and reducing the number of lines of code had a significant
impact.



In the latest release of MATLAB, which we're using for this contest, the
64-bit JIT is now on par with the 32-bit JIT. This should negate the
performance benefit that obfuscation was providing.

In the example provided on the contest rules page, I don't understand
why entry from [2,0] is deflected but the entry from [-2,0] is
reflected. I expected that the entry from [-2,0] is also deflected by
90 degrees towards the south.
Can you clarify this? Thanks.

The MATLAB Contest Team wrote:
>
>
> The next MATLAB Programming Contest runs from Wednesday November
> 29th
> at 9AM EST through Wednesday, December 6th at 5PM EST. For
> complete
> details see the website:
>
> <http://www.mathworks.com/contest/>
>
> As in recent contests, there will be three stages, darkness,
> twilight, and daylight. During darkness, both the code and scores
> for
> all entries will be hidden. In twilight, the scores will be visible
> but the code will still be hidden. When daylight arrives, all
> scores
> and code are visible. We'll also be holding several mid-contest
> prize
> giveaways of MATLAB bags, caps, and other good stuff, so don't wait
> until the last minute to participate.
>
> Please use this thread to talk about the contest, strategies, or to
> ask related questions. If you have an "administrative" type of
> question that you don't feel applies to anyone else, e-mail us at
> contest@mathworks.com.
>
> We've also started a Contest Blog where we will post the latest
> news,
> so be sure to check it out:
>
> <http://blogs.mathworks.com/contest/>
>
> Good luck everyone!
>
> - The MATLAB Contest Team -

Tracker wrote:
>
>
> In the example provided on the contest rules page, I don't
> understand
> why entry from [2,0] is deflected but the entry from [-2,0] is
> reflected. I expected that the entry from [-2,0] is also deflected
> by
> 90 degrees towards the south.
> Can you clarify this? Thanks.

The [-2,0] beam seems to be covered by the specific rule that if a
beam enters the box adjacent to an object at the edge of the box,
then it is reflected.

Let's say I have the following grid.

     0 0 0 0
     0 0 0 0
     1 0 0 0
     0 2 0 0

If I send a beam into the bottom right corner from the left, i.e.
beam(4,0), it should be reflected b/c there is an object on the edge
adjacent to it. Ditto for sending it from the bottom, i.e.
beam(0,-1). The scores are different, though (1 & 2, respectively).
Can someone explain why?

The MATLAB Contest Team wrote:
>
>
> The next MATLAB Programming Contest runs from Wednesday November
> 29th
> at 9AM EST through Wednesday, December 6th at 5PM EST. For
> complete
> details see the website:
>
> <http://www.mathworks.com/contest/>
>
> As in recent contests, there will be three stages, darkness,
> twilight, and daylight. During darkness, both the code and scores
> for
> all entries will be hidden. In twilight, the scores will be visible
> but the code will still be hidden. When daylight arrives, all
> scores
> and code are visible. We'll also be holding several mid-contest
> prize
> giveaways of MATLAB bags, caps, and other good stuff, so don't wait
> until the last minute to participate.
>
> Please use this thread to talk about the contest, strategies, or to
> ask related questions. If you have an "administrative" type of
> question that you don't feel applies to anyone else, e-mail us at
> contest@mathworks.com.
>
> We've also started a Contest Blog where we will post the latest
> news,
> so be sure to check it out:
>
> <http://blogs.mathworks.com/contest/>
>
> Good luck everyone!
>
> - The MATLAB Contest Team -

When a beam is deflected (reflected) the score returned is the sum of
all the particles it touched, in the first case the beam is deflected
by the 1 (you see it as a reflection becuase it happens that that
particle is in the edge), in the second case the 2 deflects the beam.

Lucio

 Michael Oczkowski wrote:
>
>
> Let's say I have the following grid.
>
> 0 0 0 0
> 0 0 0 0
> 1 0 0 0
> 0 2 0 0
>
> If I send a beam into the bottom right corner from the left, i.e.
> beam(4,0), it should be reflected b/c there is an object on the
> edge
> adjacent to it. Ditto for sending it from the bottom, i.e.
> beam(0,-1). The scores are different, though (1 & 2,
> respectively).
> Can someone explain why?

the code seems to prioritize absortions when multiple rules apply.
For example in the matrix

1 1 0 0
0 0 0 0
0 0 0 0
0 0 0 0

when the ray input is from the top left in the downward direction
(i.e. input 0,1) the answer is (0,0,0) (instead of (0,1,1) if a
reflection had occurred)

Is this always the case? (this is unclear from the rules)

Thanks

It's interesting to note that the problem is impossible to solve if
too many of the spaces within the black box are filled (even if you
use the "high" intensity beam to empty slots). For example, using
the available tools it would be entirely impossible to determine the
entries in ones(3). Any low intensity beam is absorbed immediately
and any high intensity beam destroys information which cannot be
recovered.

Alfonso:

You are right, absortions occurs always before any possible
deflection or reflection.

Lucio

Alfonso Nieto wrote:
>
>
> the code seems to prioritize absortions when multiple rules apply.
> For example in the matrix
>
> 1 1 0 0
> 0 0 0 0
> 0 0 0 0
> 0 0 0 0
>
> when the ray input is from the top left in the downward direction
> (i.e. input 0,1) the answer is (0,0,0) (instead of (0,1,1) if a
> reflection had occurred)
>
> Is this always the case? (this is unclear from the rules)
>
> Thanks

Mike:

We tried to design the testsuite in such a way that all puzzles
should be possible to solve, although it may be likely that there are
some that have no solution.

Lucio

 Mike Bindschadler wrote:
>
>
> It's interesting to note that the problem is impossible to solve if
> too many of the spaces within the black box are filled (even if you
> use the "high" intensity beam to empty slots). For example, using
> the available tools it would be entirely impossible to determine
> the
> entries in ones(3). Any low intensity beam is absorbed immediately
> and any high intensity beam destroys information which cannot be
> recovered.

I am getting the same error. Please help! Time is running out!

Luigi

Dan Pearve wrote:
>
>
> I would some help making this work on R13 as well. I get the
> following error...
>
>>> runcontest
> ??? Error: File: C:\Documents and
> Settings\Administrator.ARTISTS\Desktop\blackbox\beam.m Line: 175
> Column: 17
> "]" expected, "identifier" found.
>
> Error in ==> C:\Documents and
> Settings\Administrator.ARTISTS\Desktop\blackbox\runcontest.m
> On line 33 ==> beam(testsuite{k});

add a comma in "[turns,charge]"
Good luck

 Luigi Sorbara wrote:
>
>
> I am getting the same error. Please help! Time is running out!

Well it works to fix one, but it changes the error to something
involving conv2

??? Error using ==> conv2
Function 'conv2' is not defined for values of class 'single'.

I guess we should just accept that this is just a contest that old
school license holders can't play :¬(

for we the older crowd: remove "single", change "persistent" to
"global",
and change "isscalar(r)" to "(prod(size(r))==1)",
I think that should be all...

 Dan Pearce wrote:
>
>
> Well it works to fix one, but it changes the error to something
> involving conv2
>
> ??? Error using ==> conv2
> Function 'conv2' is not defined for values of class 'single'.
>
> I guess we should just accept that this is just a contest that old
> school license holders can't play :¬(

Alfonso Nieto a écrit :

> and change "isscalar(r)" to "(prod(size(r))==1)",

"prod(size(r))==1" can also be "numel(r)==1"

Jérôme

Thanks very much Alphonso. Seems to work well. I have uploaded an R13
compatible version for download if anyone needs it and can't be
bothered to read the whole page

 <http://tinyurl.com/yjadch>

If anyone finds more problems, please email me at
daniel(dot)pearce(at)brunel(dot)ac(dot)uk
and I will modify the files,

I know this sounds a bit lame. But can there be an extension to the
dark period. I couldn't work on the problem for all of day 1 until
the corrections were posted. And I also fear that using the old
version will mess with the results. Please let me know.

Luigi
 
Dan Pearce wrote:
>
>
> Thanks very much Alphonso. Seems to work well. I have uploaded an
> R13
> compatible version for download if anyone needs it and can't be
> bothered to read the whole page
>
> <http://tinyurl.com/yjadch>
>
> If anyone finds more problems, please email me at
> daniel(dot)pearce(at)brunel(dot)ac(dot)uk
> and I will modify the files,

Luigi Sorbara wrote:
>
> I know this sounds a bit lame. But can there be an extension to
> the dark period.

I second that! Stupid bug in my would-be-winning code that I didn't
discover until one minute past deadline.

Aargh! ;-)

/Per

The MATLAB Contest Team wrote:
>
>
> The next MATLAB Programming Contest runs from Wednesday November
> 29th
> at 9AM EST through Wednesday, December 6th at 5PM EST.

The "Newsgroup Thread" link on the contest page sends me to the
thread of a previous contest. I assume that is not just me.

Thanks, I fixed the link.

How long is twilight? I need to choossse my ssstrategy.

snake wrote:
>
>
> How long is twilight? I need to choossse my ssstrategy.
  
Twilight ends Friday noon, EST.

Wow, this contest converged quickly. Two guys are down to about 11%
of the result already. Wonder how people will impriove that.

snake wrote:
>
>
> Wow, this contest converged quickly. Two guys are down to about 11%
> of the result already. Wonder how people will impriove that.

Don't worry. I know for sure my code isn't optimal.

I can think of a few things that would significantly improve the
score, a few other things that are marginal but which I'm sure will
show up during the tweaking (daylight) phase, and then there are the
random modifications that may give a better result due to luck with
the test suite.

In all I'm pretty sure the top score will keep improving right until
the end of the contest.

/Per

The rankings display seems to have stop updating at 2:11 AM. (A few
hours ago.) /P

Per Rutquist wrote:
>
>
> The rankings display seems to have stop updating at 2:11 AM. (A few
> hours ago.) /P
  
I like the fact that the top Failing Entry is also emblazoned with
the bold red #1. It's like the scarlet letter.

Per, the rankings look OK on this end. If it was down earlier, it
seems OK now.

the cyclist wrote:
>
>
> snake wrote:
>>
>>
>> How long is twilight? I need to choossse my ssstrategy.
>
> Twilight ends Friday noon, EST.
  
Not much pre-twilight activity. Most people waiting for daylight to
tweak, I guess.

A few tips to people who might be thinking of tweaking Aargh :-)

1) Rick's code for handling rotations (in the Petunia series) is both
faster and better-looking than mine. Substituting faster code will
probably only change the last decimal place of the score, but it will
take you to the top of the list!

2) The line "for rot = [1 2 3 0]" near the top of the code determines
the order that the rotations are tested. Changing the order will have
a random effect on the score. Sometimes making it better. Numbers
from 4 to 7 can also be used. (4 is the mirror image of 1, etc.)

Good luck everyone!

Congrats Per!

I tried to get down below 13000 before the deadline, but couldn't do
it. I'll have to poke through your code to see what you came up
with. :)

Rick.

Nice code, Per,
Something is going wrong, but I don't know what. (It is in the first
while loop of rotsolver.) Sometimes blocks are set on places that
should be empty (for example a 19 on test 19 of the testsuite (((is
19 an unlucky number for this program?))) with rot==1;)

Solving that bug should give a clear improvement. (Other examples of
faults are on 52 and 113.)

Stijn

The MATLAB Contest Team wrote:
>
>
> The next MATLAB Programming Contest runs from Wednesday November
> 29th
> at 9AM EST through Wednesday, December 6th at 5PM EST.

"You've been Welched!"

Brilliant. Time for more commands to be banned.

Thanks. For analyzing code, I strongly suggest putting a few "plot"
commands into the beam function, (and a "spy" in the beginning and
"drawnow" in the end). Lets you see what the code does without having
to read through it.

The Aargh code may well have bugs, and sometimes it intentionally
makes guesses. For debugging, it might be better to start from the
penultimate contribution (scored about 132 I think). That one has
comments in it.

Rick's code seems similar to mine, while Cobus' code manages to do
very well without trying all different rotations. Suggests that
there's potential for synnergy effects.

Midnight EST is 6 a.m. Paris time, so I won't participate in this
one. (In general I tend not to be so active during the "daylight"
phase.)

Happy coding,

/Per

The MATLAB Contest Team wrote:
>
>
> The next MATLAB Programming Contest runs from Wednesday November
> 29th
> at 9AM EST through Wednesday, December 6th at 5PM EST.

The scoring algorithm must have gotten an interesting overhaul. My
entry with ID 32377 had the same result and smaller CPU time than
Stijn's entry with ID 32373, but they have the same score.

Will people start tweaking "Welched codes" (for people not knowing
what it is (as far as I know) : called after Brian Welch, a hacker in
a couple of contests).
Looking how the code can be hacked is nice, but I don't think that's
code to be tweaked!

Hehe, nice hack guys! :-)

regexp('i', 'i(?@clear beam)');

Markus wrote:
>
>
> Hehe, nice hack guys! :-)
>
> regexp('i', 'i(?@clear beam)');
  
Thanks, but I just realised I actually just got lucky. I assumed that
TMW uses some form of static analysis to check whether any banned
functions are used BEFORE executing it (e.g. using depfun).

It looks like they are actually using some form of runtime checking
so the regexp trick doesn't work. HOWEVER as it turns out, clear
wasn't banned after all, so it worked anyway.

I'm really curious about the runtime checking mechanism (for one
thing how it affects timing results). Details anyone? ;-)
Cheers
H

Briliant hack!

There's some interesting information in the results of the welched
code. The listed score is for a beam count of zero. Hence it can be
deduced that only 0.14 points remain to be gained from reducing the
error (and 0.0037 points for reducing the computation time).

The remaining 128 points are for reducing beam count, so that's a
good place to focus efforts.

I grabbed your code and ran it, and before I even noticed the hack, I
saw the 'disp(...' commands slowing it up. I just commented them and
submitted it, shocked I was the first to notice them. I then took a
second look at the score and realized something must have been
hacked. 8^)

Silliness.

 Hannes Naudé wrote:
>
>
> Markus wrote:
>>
>>
>> Hehe, nice hack guys! :-)
>>
>> regexp('i', 'i(?@clear beam)');
>
> Thanks, but I just realised I actually just got lucky. I assumed
> that
> TMW uses some form of static analysis to check whether any banned
> functions are used BEFORE executing it (e.g. using depfun).
>
> It looks like they are actually using some form of runtime checking
> so the regexp trick doesn't work. HOWEVER as it turns out, clear
> wasn't banned after all, so it worked anyway.
>
> I'm really curious about the runtime checking mechanism (for one
> thing how it affects timing results). Details anyone? ;-)
> Cheers
> H

Just writing "clear beam" is even more fresh :-)

Please do not spam the queue with those hacked entries, the contest
team might have problems to keep up removing them.

Per Rutquist wrote:
>
>
> Briliant hack!
>
> There's some interesting information in the results of the welched
> code. The listed score is for a beam count of zero. Hence it can be
> deduced that only 0.14 points remain to be gained from reducing the
> error (and 0.0037 points for reducing the computation time).
>
> The remaining 128 points are for reducing beam count, so that's a
> good place to focus efforts.
  
Yep, that's what I was seeing from some of my earlier submissions.
It seems we are getting the solutions correct (maybe a few minor
errors), but beam count is driving the score. The trick will see
what algorithms can be found that can minimize the use of the 'high'
mode and still achieve the right solution.

Rick.

Markus wrote:
>
>
> Just writing "clear beam" is even more fresh :-)
>

True, but the "clear" command is disallowed, which makes the hack so
cool

> Please do not spam the queue with those hacked entries, the contest
> team might have problems to keep up removing them.

Agree with you here.

Per Rutquist wrote:
> Briliant hack!

Hi guys:

Yes, we've been hacked, and we've also added a new word to the
contest lexicon: welched. I don't see Brian in the queue so far this
contest, but certainly his memory lives on.

We've shut down the queue for now while we investigate this scoring
problem. Administratively we were running a little lean this
afternoon (everybody on the contest team has a day job!), but now
we've got people working on it and should be up and running before
too long.

You can certainly expect that the current low scores will not last,
so if you're tweaking and hacking, keep in mind what the current best
legitimate entries are.

-Ned Gulley.
MATLAB Contest Team

Excellent!

I know I certainly didn't deserve to be in first place.

 Ned Gulley wrote:
>
>
> Per Rutquist wrote:
>> Briliant hack!
>
> Hi guys:
>
> Yes, we've been hacked, and we've also added a new word to the
> contest lexicon: welched. I don't see Brian in the queue so far
> this
> contest, but certainly his memory lives on.
>
> We've shut down the queue for now while we investigate this scoring
> problem. Administratively we were running a little lean this
> afternoon (everybody on the contest team has a day job!), but now
> we've got people working on it and should be up and running before
> too long.
>
> You can certainly expect that the current low scores will not last,
> so if you're tweaking and hacking, keep in mind what the current
> best
> legitimate entries are.
>
> -Ned Gulley.
> MATLAB Contest Team

With the hacking going on, I thought I'd try the other way.

Check my score here:
 <http://www.mathworks.com/contest/blackbox.cgi/view_submission.html?id=32388>

and then in my next entry:
 <http://www.mathworks.com/contest/blackbox.cgi/view_submission.html?id=32389>

Steve

Thy cup runneth over. 8^D

 Steve wrote:
>
>
> With the hacking going on, I thought I'd try the other way.
>
> Check my score here:
> <http://www.mathworks.com/contest/blackbox.cgi/view_submission.html?id=32388>
>
> and then in my next entry:
> <http://www.mathworks.com/contest/blackbox.cgi/view_submission.html?id=32389>
>
> Steve

Hannes Naudé took advantage of a funky feature of MATLAB's REGEXP in order
to sneak a command by our security. Dynamic regular expressions allow you
to execute MATLAB commands from within a regular expression, an unusual
feature that's been useful to me in the past when doing replacements.
Embarrassingly, we weren't even trying to block CLEAR because it didn't
matter in past contests. REGEXP and CLEAR are now blocked. No real damage
to the contest was done and we're still on-track to award a Midnight Madness
prize tonight to the best-scoring entry submitted before midnight EST.

the cyclist, we haven't mucked with the scoring. I think the reason you're
seeing identical scores is because the times are way down on the flat part
of the exponential time penalty. The scores are recorded in full precision,
but displayed with fewer significant figures.

Matthew Simoneau wrote:
>
>
> the cyclist, we haven't mucked with the scoring. I think the
> reason you're
> seeing identical scores is because the times are way down on the
> flat part
> of the exponential time penalty. The scores are recorded in full
> precision,
> but displayed with fewer significant figures.
>
>
>
  
OK, but if it is a precision issue, it is still strange that my entry
was listed below Stijn's, with the identical result and better CPU.
(I am fairly sure the result is identical, because there was no
algorithm change.)

Looks like the scoring formula coefficients are the same as last
time:
k1 = 0.01
k2 = 0.001
k3 = 0.125
 
As I pointed out in a blog response during the Blockbuster contest (http://blogs.mathworks.com/contest/?p=38#comments)
this puts the ‘elbow’ somewhere in the 75 - 95 second range,
depending upon how you define it. The curve is flat until 60-65
seconds, which means the current top results hovering in the 8 to 9
second range can spare a significant amount of time if it even
reduces incrementally the result.

The MATLAB Contest Team wrote:
>
>
> The next MATLAB Programming Contest runs from Wednesday November
> 29th
> at 9AM EST through Wednesday, December 6th at 5PM EST.

Problem with the queue as we approach Madness?

Steve wrote:
>
>
> With the hacking going on, I thought I'd try the other way.
>
> Check my score here:
> <http://www.mathworks.com/contest/blackbox.cgi/view_submission.html?id=32388>
>
> and then in my next entry:
> <http://www.mathworks.com/contest/blackbox.cgi/view_submission.html?id=32389>
>
> Steve
  
This got me reminded me of something that I thought of earlier. What
happens with NaN or Inf scores? Using Matlab sort they end up at the
bottom of the rankings, but maybe another sort is used on the Web
server. Easy to try, simple and harmless.

Well, not that harmless it seems. My NaE seems to have hung the
queue. Sorry guys :-(

Hannes Naudé wrote:
>
>
> Steve wrote:
>>
>>
>> With the hacking going on, I thought I'd try the other way.
>>
>> Check my score here:
>> <http://www.mathworks.com/contest/blackbox.cgi/view_submission.html?id=32388>
>>
>> and then in my next entry:
>> <http://www.mathworks.com/contest/blackbox.cgi/view_submission.html?id=32389>
>>
>> Steve
>
> This got me reminded me of something that I thought of earlier.
> What
> happens with NaN or Inf scores? Using Matlab sort they end up at
> the
> bottom of the rankings, but maybe another sort is used on the Web
> server. Easy to try, simple and harmless.
>
> Well, not that harmless it seems. My NaE seems to have hung the
> queue. Sorry guys :-(
  

I see you also might have discovered another security bug with your
"str2num('evalc(''clear beam'')');" test.....

Hannes Naudé wrote:
>
>
> Steve wrote:
>>
>>
>> With the hacking going on, I thought I'd try the other way.
>>
>> Check my score here:
>> <http://www.mathworks.com/contest/blackbox.cgi/view_submission.html?id=32388>
>>
>> and then in my next entry:
>> <http://www.mathworks.com/contest/blackbox.cgi/view_submission.html?id=32389>
>>
>> Steve
>
> This got me reminded me of something that I thought of earlier.
> What
> happens with NaN or Inf scores? Using Matlab sort they end up at
> the
> bottom of the rankings, but maybe another sort is used on the Web
> server. Easy to try, simple and harmless.
>
> Well, not that harmless it seems. My NaE seems to have hung the
> queue. Sorry guys :-(
  
I expect it is a coincidence that your entry was next in the queue
when the processing stopped.

I like your evalc hack (er, uh, research project). I know that eval
was banned in a past contest, but it will be interesting to see if
they caught evalc as well.

the cyclist wrote:
>
> I like your evalc hack (er, uh, research project). I know that
> eval
> was banned in a past contest, but it will be interesting to see if
> they caught evalc as well.

Yes I'm pretty sure it's banned, but I'm guessing the security
machinery won't know that I've used it.

Current best guess at how the security works is as follows: Two
pronged approach consisting of filefilt and execfilt.

Filefilt runs a static dependency check (using depfun presumably).
Because evalc is used dynamically it doesn't show up in the static
dependencies. I used to think this is all there was to it, but then
my first "You've been welched" entry got caught by execfilt.

This entry used the banned function evalin but used it dynamically so
depfun should have missed it. I'm guessing they have some check that
runs every time a function is called (no idea how one would do this)
that checks whether the function is on the banned list.

But it needs to be more complex than this, because many legit
functions use non-legit functions (e.g. eval is used a lot by the
minimization functions). So execfilt presumably checks the stack to
see where the function call originated from and only blocks banned
functions that are invoked from solver. Therefore evalc should get
past execfilt because it is not being called from solver.

If anyone has any other ideas how it may work, I'd be keen to hear
them. Also I'd be very interested how something like execfilt can be
implemented (assuming my theory about execfilt is correct).

I think I'll back off on the hacking, at least for the weekend.
Presumably the contest team can fix the chaos that ensues much more
easily if I pay them the courtesy of hacking during their office
hours ;-).

Hannes

The MATLAB Contest Team wrote:
>
>
> The next MATLAB Programming Contest runs from Wednesday November
> 29th
> at 9AM EST through Wednesday, December 6th at 5PM EST.

Would it be possible to show the results and scores to a bit more
precision, given that such tiny results differences seem to be
distinguishing the top entries?

It looks like the queue is hung again......It's not even showing new
submissions to it this time, nor are the rankings or chronological
listing being updated.

Thanks, Lucio, for the analysis, and the visualization is nice
(especially nice after looking at my own crude versions of the same
sort of thing). However, it is not true that the leading solvers are
generally memoryless, many contain a global variable called BEAMlog
(or more recently, 'alijooj' or something) whose purpose is to record
the results of low intensity beams so that they can be looked up
rather than repeated. It appears to me that all of them clear the log
whenever a high intensity beam is fired, but you make the good point
that if you know the location of the object removed, it may not be
necessary to clear the whole log, just the ones which pass near the
removed object. On the other hand, you may not know which beams pass
near the object if there is complex bouncing going on, so it
certainly is safer to clear the whole log. Perhaps there's a
reasonable compromise.

Mike Bindschadler wrote:
>
>
> Thanks, Lucio, for the analysis, and the visualization is nice
> (especially nice after looking at my own crude versions of the same
> sort of thing). However, it is not true that the leading solvers
> are
> generally memoryless, many contain a global variable called BEAMlog
> (or more recently, 'alijooj' or something) whose purpose is to
> record
> the results of low intensity beams so that they can be looked up
> rather than repeated. It appears to me that all of them clear the
> log
> whenever a high intensity beam is fired, but you make the good
> point
> that if you know the location of the object removed, it may not be
> necessary to clear the whole log, just the ones which pass near the
> removed object. On the other hand, you may not know which beams
> pass
> near the object if there is complex bouncing going on, so it
> certainly is safer to clear the whole log. Perhaps there's a
> reasonable compromise.

In case people are wondering what in the world I'm talking about,
there's a midcontest analysis available at <http://www.mathworks.com/contest/blackbox/midcontest.html>

It's on the list of links to the right of the page when you're
looking at an existing solver submission, but not, at this point,
from any other pages.

Lucio, could you post your drawpuzzle function from the midcontest
analysis on the MATLAB central file exchange? I had assumed it was
already there, but if so, I can't find it. Thanks!

>> there's a midcontest analysis available at <http://>>>www.mathworks.com/contest/blackbox/midcontest.html
In this midcontest analysis it is stated :
"...we know that the fact that a beam appears exactly
on the opposite side of the box does not mean it did
not hit any particle..."
But this is easily detected by the score! I hoped to work out a
solution with "more intelligence". (See some of my submissions.) But
I don't know I will work further on it...

Stijn

Is it just me or is the queue / listings hung up again? They all say
they were last generated at 10:15AM, yet it's close to 11:30AM.

Isn't it theoretically possible to send a lot of short programs that
probe some little bit of information about the cases in the test
suite.

The grey probe solutions by Grey Beard made me wonder if someone is
actually trying it. :)

Anders Skjäl wrote:
>
>
> Isn't it theoretically possible to send a lot of short programs
> that
> probe some little bit of information about the cases in the test
> suite.
>
> The grey probe solutions by Grey Beard made me wonder if someone is
> actually trying it. :)
  
If the largest n is 60, send 60 probes to count the cases of size n.
The sizes n with only one case can be totally decided with 2 probes
per size and can then be put in the code.

And this can be optimized a lot. The non-unique sizes must first be
identified by their elements. This is potential trouble.

Anders Skjäl wrote:
>
>
> Anders Skjäl wrote:
>>
>>
>> Isn't it theoretically possible to send a lot of short programs
>> that
>> probe some little bit of information about the cases in the
test
>> suite.
>>
>> The grey probe solutions by Grey Beard made me wonder if
someone
> is
>> actually trying it. :)
>
> If the largest n is 60, send 60 probes to count the cases of size
> n.
> The sizes n with only one case can be totally decided with 2 probes
> per size and can then be put in the code.
>
> And this can be optimized a lot. The non-unique sizes must first be
> identified by their elements. This is potential trouble.
  

A bit more probes needed, but still.

Anders Skjäl wrote:
>
>
> Anders Skjäl wrote:
>>
>>
>> Anders Skjäl wrote:
>>>
>>>
>>> Isn't it theoretically possible to send a lot of short
> programs
>>> that
>>> probe some little bit of information about the cases in the
> test
>>> suite.
>>>
>>> The grey probe solutions by Grey Beard made me wonder if
> someone
>> is
>>> actually trying it. :)
>>
>> If the largest n is 60, send 60 probes to count the cases of
size
>> n.
>> The sizes n with only one case can be totally decided with 2
> probes
>> per size and can then be put in the code.
>>
>> And this can be optimized a lot. The non-unique sizes must
first
> be
>> identified by their elements. This is potential trouble.
>
>
> A bit more probes needed, but still.

The cat's somewhat out of the bag now. I was the one who started
this early in daylight. David Jones caught on early and deciphered
some of my work, which is why the 2 of us are at the top of the # of
submissions stats. Several other people have picked up on this now
as evident by the number of 'probes' being done, such as Grey Beard
and ___ Herring.

Without divulging too much, at this point, I will say I've calculated
the number of tests in the test suite, the distrution of all the
sizes of the tests, the rough number of items in each test, and most
importantly the actual complete layout of some of the tests. I've
tried to obscufate my probes a bit to make it more difficult for
others to reap the benefits of my work.

At this point, I think whoever wins is going to end up using some
'lookup' tables in their code to shave counts of off the number of
times beam is used. However it requires a SIGNIFICANT number of
'probes' to calculate the details and it's quite tedious.

I had sent an email to David Jones asking if he wanted to partner on
this and 'divide and conquer', but he didn't respond. Thus I'll
extend the invitation to anyone interested.

If you want to 'team' with me I'd be happy to provide the 'tricks' of
the trade I've figured out and divide up the work so we can end up
submitting a joint program at the end that will hopefully win the
whole thing. Email me if you are interested.

Alan Chalker wrote:
>
>
> The queue is hung again on a code with some NaNs. The contest team
> should consider giving us some way to easily contact them about
> these
> hangs.....
  

Hmmm... now it's so bad the webserver is giving 'server error 500'
and you can't see any of the entries in any listing...... last
contest the problem was with obscufation, this one it seems to be
even keeping the contest up and going.

Alan Chalker wrote:
> Hmmm... now it's so bad the webserver is giving 'server error 500'
> and you can't see any of the entries in any listing...... last
> contest the problem was with obscufation, this one it seems to be
> even keeping the contest up and going.

I promise it wasn't me this time. :-)

This may be a naive question. But all this mention of probe, etc,
does this mean that the test suite being used it static? Or am I
missing what's going on here?

Thanks

Luigi Sorbara wrote:
>
>
> This may be a naive question. But all this mention of probe, etc,
> does this mean that the test suite being used it static? Or am I
> missing what's going on here?
>
> Thanks
  
The test suite used for scoring entries is indeed static; it is
distinct from the test suite sent out in the file exchange for "home
use".

That is the reason it is possible to optimize certain constants in
nonintuitive ways, and why certain sequences of pseudorandom numbers
are better than others.

Luigi Sorbara wrote:
>
>
> So my question is:
>
> Are the solutions that are being posted / winning the contest
> actually the best solutions? I ran the best solution (at the time)
> against the test suite that was provided and it actually increased
> the score of submitting a solution of all zeroes. That blows my
> mind. Shouldn't this contest be done for the sake of creative
> algorithm design instead of tweaking to obtain a non-meaningful
> score??
>
> Not to say that I've come up with anything .. :) But I would like
> to
> think that the winning solution would work the best for ALL test
> cases.

Well, I think you will agree that operationally, it would be
difficult to test against ALL test cases. 8-)

Having a fixed test suite at least removes a potentially unfair
situation in which two entries are graded against different suites in
which one is easier and one is more difficult.

The MATLAB Contest Team wrote:
>
>
> The next MATLAB Programming Contest runs from Wednesday November
> 29th
> at 9AM EST through Wednesday, December 6th at 5PM EST. For
> complete
> details see the website:
>
> <http://www.mathworks.com/contest/>
>
> As in recent contests, there will be three stages, darkness,
> twilight, and daylight. During darkness, both the code and scores
> for
> all entries will be hidden. In twilight, the scores will be visible
> but the code will still be hidden. When daylight arrives, all
> scores
> and code are visible. We'll also be holding several mid-contest
> prize
> giveaways of MATLAB bags, caps, and other good stuff, so don't wait
> until the last minute to participate.
>
> Please use this thread to talk about the contest, strategies, or to
> ask related questions. If you have an "administrative" type of
> question that you don't feel applies to anyone else, e-mail us at
> contest@mathworks.com.
>
> We've also started a Contest Blog where we will post the latest
> news,
> so be sure to check it out:
>
> <http://blogs.mathworks.com/contest/>
>
> Good luck everyone!
>
> - The MATLAB Contest Team -

Alan Chalker wrote:
>
> The cat's somewhat out of the bag now. I was the one who started
> this early in daylight. David Jones caught on early and deciphered
> some of my work, which is why the 2 of us are at the top of the #
> of
> submissions stats. Several other people have picked up on this now
> as evident by the number of 'probes' being done, such as Grey Beard
> and ___ Herring.
>
> Without divulging too much, at this point, I will say I've
> calculated
> the number of tests in the test suite, the distrution of all the
> sizes of the tests, the rough number of items in each test, and
> most
> importantly the actual complete layout of some of the tests. I've
> tried to obscufate my probes a bit to make it more difficult for
> others to reap the benefits of my work.
>
> At this point, I think whoever wins is going to end up using some
> 'lookup' tables in their code to shave counts of off the number of
> times beam is used. However it requires a SIGNIFICANT number of
> 'probes' to calculate the details and it's quite tedious.
>
> I had sent an email to David Jones asking if he wanted to partner
> on
> this and 'divide and conquer', but he didn't respond. Thus I'll
> extend the invitation to anyone interested.
>
> If you want to 'team' with me I'd be happy to provide the 'tricks'
> of
> the trade I've figured out and divide up the work so we can end up
> submitting a joint program at the end that will hopefully win the
> whole thing. Email me if you are interested.
  
I'd like to speak up against this strategy. Although clever, and
perhaps fun to implement, I feel that it goes against the spirit of
the contest. In particular, since this strategy could in principle
be applied to every future contest with the current framework
(scores/ranks revealed in daylight), then it seems that all Matlab
programming contests will become tests of who can wring the most
information out of the test suite, rather than whatever topic they
are ostensibly about. The Matlab folks have gone to the effort to
come up with an interesting contest problem for us, and this seems a
poor way to repay them for their trouble. Granted, some information
will always leak out in terms of tuned constants, etc., but these
sorts of probes seem to bring it to a different level.

Do I sound cranky? Must be lack of sleep! ;-)

Alan Chalker wrote:
>
> The cat's somewhat out of the bag now. I was the one who started
> this early in daylight. David Jones caught on early and deciphered
> some of my work, which is why the 2 of us are at the top of the #
> of
> submissions stats. Several other people have picked up on this now
> as evident by the number of 'probes' being done, such as Grey Beard
> and ___ Herring.
>
> Without divulging too much, at this point, I will say I've
> calculated
> the number of tests in the test suite, the distrution of all the
> sizes of the tests, the rough number of items in each test, and
> most
> importantly the actual complete layout of some of the tests. I've
> tried to obscufate my probes a bit to make it more difficult for
> others to reap the benefits of my work.
>
> At this point, I think whoever wins is going to end up using some
> 'lookup' tables in their code to shave counts of off the number of
> times beam is used. However it requires a SIGNIFICANT number of
> 'probes' to calculate the details and it's quite tedious.
>
> I had sent an email to David Jones asking if he wanted to partner
> on
> this and 'divide and conquer', but he didn't respond. Thus I'll
> extend the invitation to anyone interested.
>
> If you want to 'team' with me I'd be happy to provide the 'tricks'
> of
> the trade I've figured out and divide up the work so we can end up
> submitting a joint program at the end that will hopefully win the
> whole thing. Email me if you are interested.
  
I'd like to speak up against this strategy. Although clever, and
perhaps fun to implement, I feel that it goes against the spirit of
the contest. In particular, since this strategy could in principle
be applied to every future contest with the current framework
(scores/ranks revealed in daylight), then it seems that all Matlab
programming contests will become tests of who can wring the most
information out of the test suite, rather than whatever topic they
are ostensibly about. The Matlab folks have gone to the effort to
come up with an interesting contest problem for us, and this seems a
poor way to repay them for their trouble. Granted, some information
will always leak out in terms of tuned constants, etc., but these
sorts of probes seem to bring it to a different level.

Do I sound cranky? Must be lack of sleep! ;-)

This is not probing anymore, and even not "random tweaking"--- it is
just pure spam. What can be the reason for submitting I don't know
how much equal (and fully dummy) submissions?
(I mean the submissions of "Don Antonio Coimbra de la Coronilla y
Azevedo".) If people don't like this contest(/game), just go away...

Nick Howe wrote:
> I'd like to speak up against this strategy. Although clever, and
> perhaps fun to implement, I feel that it goes against the spirit of
> the contest.

I agree completely. Probing for constants is part of the contest, and
tweaking seems unavoidable, but probing for the entire testsuite is
just plain wrong. The contest is to submit code that solves the
problem, not code that outputs the correct solution.

Those who want to show off their probing abilities should do it on
one testcase, and submit the perfect solution for that case. (Briefly
taking the lead, and forcing their code to be included in every
future entry. This happened already in the Mars Surveyor contest.)
Repeating the process for every case in the testsuite proves nothing.
(And it clobbers up the queue.)

By the way, I believe it would take me about two entries to
completely map one of the smaller (say n=5) testcases. Spamming the
queue with hundreds of entries lacks elegance.

/Per

At least this way you can get you name out as the “most active
participant” :-)

On another topic:
What do people think about using random number generators in code?
Seems to me one always can “improve” code by inserting rand and
submitting it
zillion time hoping for randomly better score.
I just want to know if people consider it be inside boundaries of
“gentlemen rules”
 

Stijn Helsen wrote:
>
>
> This is not probing anymore, and even not "random tweaking"--- it
> is
> just pure spam. What can be the reason for submitting I don't know
> how much equal (and fully dummy) submissions?
> (I mean the submissions of "Don Antonio Coimbra de la Coronilla y
> Azevedo".) If people don't like this contest(/game), just go
> away...

Sergey Yurgenson (SY) wrote:
>
>
> At least this way you can get you name out as the “most active
> participant” :-)
>
> On another topic:
> What do people think about using random number generators in code?
> Seems to me one always can “improve” code by inserting rand and
> submitting it
> zillion time hoping for randomly better score.
> I just want to know if people consider it be inside boundaries of
> “gentlemen rules”
>

Assuming this contest works like past ones, every entry is run with a
fixed pseudorandom sequence. So unless you change the code, you get
the same result every time even using rand.

Stijn Helsen wrote:
> This is not probing anymore, and even not
> "random tweaking"--- it is just pure spam.

Greetings contestants:

I'm sure you're wondering where the contest team has been.

First of all, let me acknowledge that our machinery broke down badly
over the weekend and we will work to clean it up both for the next
few days of the contest and over the longer term. Thanks for your
patience, and we are very sorry for the interruption of your contest
fun.

These contests are not a product; we do not charge for them. A very
small team (there are three of us!) puts them on in the hopes that
they will be an entertaining diversion for the community of people
that loves fiddling around with MATLAB code. By design there is very
little at stake: a t-shirt, a mug, a baseball cap. We know our
contest machinery is not industrial strength. And as a result of this
contest in particular, we are all too aware of that.

All this is by way of saying: if you systematically seek to pick
apart our contest machinery, you will probably succeed! And if you
seek to take it down by rapid-fire spam entries, you will probably
succeed. In fact, that seems to be what happened last night, although
we are still working on exactly what happened.

We're really happy that people are so enthusiastic about the contest,
but we would ask for a little restraint, particularly when it comes
to some of the brute force attacks.

Once again, we're sorry for the downtime. We're working to put things
right this morning.

-Ned Gulley.
The Contest Team

PS: We've been consistently impressed with the MATLAB skills on
display here. You may want to consider a job at the MathWorks...

 <http://www.mathworks.com/company/jobs/opportunities/>

We had a long discussion about what to do about queue spamming in the
newsgroup thread during the last contest "blockbuster". Unfortunately
the contest team did not pick any of the suggestions to change the
rules. Or let me better say: they did not find the time for doing it.

When submitting this text to the newsgroup, I have to do a "word
verfication", i.e. reading some letters from an image and typing them
into a text field. I guess if this functionality would be built into
the code submission form, the spamming would have an abrupt end
(maybe some people would try to find the letters by image processing,
but that is supposed to be a much harder problem to solve).

A question to the contest team: How much effort would it mean to put
that word verification stuff into the code submission form?

Regards
Markus

Stijn Helsen wrote:
>
>
> This is not probing anymore, and even not "random tweaking"--- it
> is
> just pure spam. What can be the reason for submitting I don't know
> how much equal (and fully dummy) submissions?
> (I mean the submissions of "Don Antonio Coimbra de la Coronilla y
> Azevedo".) If people don't like this contest(/game), just go
> away...
  

My guess is that it isn't an attack.
He's probably working on an automated solution submitter in order to
spam the queue with iterative probes machanically.

cute, but it destroys the intent of the contest.
so fine, probers win, now go away and let us see who gets second
place with an actual algorythm

Tom
(hypocrite who collaborated to map single mars surveyor solution)

Lemke wrote:
>
>
> Stijn Helsen wrote:
>>
>>
>> This is not probing anymore, and even not "random tweaking"---
it
>> is
>> just pure spam. What can be the reason for submitting I don't
> know
>> how much equal (and fully dummy) submissions?
>> (I mean the submissions of "Don Antonio Coimbra de la Coronilla
y
>> Azevedo".) If people don't like this contest(/game), just go
>> away...
>
>
> My guess is that it isn't an attack.
> He's probably working on an automated solution submitter in order
> to
> spam the queue with iterative probes machanically.
>
> cute, but it destroys the intent of the contest.
> so fine, probers win, now go away and let us see who gets second
> place with an actual algorythm
>
> Tom
> (hypocrite who collaborated to map single mars surveyor solution)
  

Don Antonio did this last contest as well, so his automated submitter
clearly works well. Perhaps this is just a way of trying to increase
his name's page rank in google or something like that?

Per Rutquist wrote:
>
>
> Nick Howe wrote:
>> I'd like to speak up against this strategy. Although clever,
and
>> perhaps fun to implement, I feel that it goes against the
spirit
> of
>> the contest.
>
> I agree completely. Probing for constants is part of the contest,
> and
> tweaking seems unavoidable, but probing for the entire testsuite is
> just plain wrong. The contest is to submit code that solves the
> problem, not code that outputs the correct solution.
>
> Those who want to show off their probing abilities should do it on
> one testcase, and submit the perfect solution for that case.
> (Briefly
> taking the lead, and forcing their code to be included in every
> future entry. This happened already in the Mars Surveyor contest.)
> Repeating the process for every case in the testsuite proves
> nothing.
> (And it clobbers up the queue.)
>
> By the way, I believe it would take me about two entries to
> completely map one of the smaller (say n=5) testcases. Spamming the
> queue with hundreds of entries lacks elegance.
>
> /Per
  

I respectfully disagree regarding about the differences between
probing the testsuite and for constants. With most of the entries
that aren't 'probing' people are just tweaking the random number
generation, which is in effect over fitting the code to the
particular test suite. There isn't a lot of skill involved in that.


At least with the probing I'm doing, I've had to develop some
algorithms to efficiently query the appropriate board and code back a
result. I've had to go through several versions of the algorithms.
I think this is more within the spirit than just tweaking since it
actually involves code development and significant understanding of
what's going on with the test suites. I don't see any way to do even
a small board in 'two entries', although I've been able to
significantly reduce the number of probes needed over my initial
attempts.

I've also pulled together some obscufation techniques that make it a
little harder to read my code without making it impossible, so that
anyone who wants to 'reap the benefits' of my work has to put in a
little effort themselves, which is again within the spirit of the
competition in my view.

And just to show I'm not just probing, I've also contributed some
actual improvements to the leading algorithms and held the lead for a
little while, mainly via speed improvements. I was the one who got
rid of the global variables by going to nested functions and also cut
down on the number of times the beamlog is overwritten with -1s
(which was taking up about 30% of the processing time).

Finally, within the 'spirit' of the competition I offered to team
with people and reduce the amount of probing going on. I've gotten
responses from 2 people who are reoccuring participants in this
contest and will be forming a team with them. This contest is more
about learning new things and collaboration than anything else, so I
think all of this is well within the scope of it.

PS: Although nobody else has really expressed this, I want to give a
heartfelt 'thank you' to the small team at The MathWorks that makes
this possible. It really is an enjoyable diversion to the other
priorities in my job yet helps to build skills that are directly
applicable. I realize this is a bit of a burden on you and
appreciate your contributions to the MATLAB community.

Hi all:

For those interested on the visualization functions, now they are
availabe in the file exchange server:

 <http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=13161&objectType=FILE>

just as usual try "runcontest(true)" or the "seesolver" function with
something like:

load sample_testsuite
seesolver(@mysolver,testsuite{10})

Lucio

OK. Moggy has find effective way to communicate testcase back.
Now what?

 Alan Chalker wrote:
>
>
> Per Rutquist wrote:
>>
>>
>> Nick Howe wrote:
>>> I'd like to speak up against this strategy. Although
clever,
> and
>>> perhaps fun to implement, I feel that it goes against the
> spirit
>> of
>>> the contest.
>>
>> I agree completely. Probing for constants is part of the
contest,
>> and
>> tweaking seems unavoidable, but probing for the entire
testsuite
> is
>> just plain wrong. The contest is to submit code that solves the
>> problem, not code that outputs the correct solution.
>>
>> Those who want to show off their probing abilities should do it
> on
>> one testcase, and submit the perfect solution for that case.
>> (Briefly
>> taking the lead, and forcing their code to be included in every
>> future entry. This happened already in the Mars Surveyor
> contest.)
>> Repeating the process for every case in the testsuite proves
>> nothing.
>> (And it clobbers up the queue.)
>>
>> By the way, I believe it would take me about two entries to
>> completely map one of the smaller (say n=5) testcases. Spamming
> the
>> queue with hundreds of entries lacks elegance.
>>
>> /Per
>
>
> I respectfully disagree regarding about the differences between
> probing the testsuite and for constants. With most of the entries
> that aren't 'probing' people are just tweaking the random number
> generation, which is in effect over fitting the code to the
> particular test suite. There isn't a lot of skill involved in
> that.
>
>
> At least with the probing I'm doing, I've had to develop some
> algorithms to efficiently query the appropriate board and code back
> a
> result. I've had to go through several versions of the algorithms.
>
> I think this is more within the spirit than just tweaking since it
> actually involves code development and significant understanding of
> what's going on with the test suites. I don't see any way to do
> even
> a small board in 'two entries', although I've been able to
> significantly reduce the number of probes needed over my initial
> attempts.
>
> I've also pulled together some obscufation techniques that make it
> a
> little harder to read my code without making it impossible, so that
> anyone who wants to 'reap the benefits' of my work has to put in a
> little effort themselves, which is again within the spirit of the
> competition in my view.
>
> And just to show I'm not just probing, I've also contributed some
> actual improvements to the leading algorithms and held the lead for
> a
> little while, mainly via speed improvements. I was the one who
> got
> rid of the global variables by going to nested functions and also
> cut
> down on the number of times the beamlog is overwritten with -1s
> (which was taking up about 30% of the processing time).
>
> Finally, within the 'spirit' of the competition I offered to team
> with people and reduce the amount of probing going on. I've gotten
> responses from 2 people who are reoccuring participants in this
> contest and will be forming a team with them. This contest is more
> about learning new things and collaboration than anything else, so
> I
> think all of this is well within the scope of it.
>
> PS: Although nobody else has really expressed this, I want to give
> a
> heartfelt 'thank you' to the small team at The MathWorks that makes
> this possible. It really is an enjoyable diversion to the other
> priorities in my job yet helps to build skills that are directly
> applicable. I realize this is a bit of a burden on you and
> appreciate your contributions to the MATLAB community.

On a different subject: I now have code that can solve puzzle #41
and others without destroying all the objects. Sadly, I find that
the savings from doing so don't appear to justify the costs
(increased probing to determine whether the problem is soluble in
this manner). Is anybody else facing this problem?

Do you mean #41 from distributed set or from “unknown testset”.
Seem to me now everybody knows all puzzles from testset

 Nick Howe wrote:
>
>
> On a different subject: I now have code that can solve puzzle #41
> and others without destroying all the objects. Sadly, I find that
> the savings from doing so don't appear to justify the costs
> (increased probing to determine whether the problem is soluble in
> this manner). Is anybody else facing this problem?

Sergey Yurgenson wrote:
>
>
> OK. Moggy has find effective way to communicate testcase back.
> Now what?

That was a very clever hack on Moggy's part. Even more clever was
hiding the array output in the error message past what's shown on the
summary listing tables. (For those of you interested in what I'm
talking about look at entry #34847)

However it was clearly using a function not intended to be accessible
to us (arrayviewfunc) which has now been disabled. Regardless, I
applaud him for knowing some very esoteric MATLAB debugging
techniques.

I agree, it was quite ingenious. However just by using direct and
simple approach I can
probe and report one puzzle in probably 10-20 min. It does really
concern me, because I want to find good algorithm, not to “cheat”.
Temptation…
I still think that final score has to be calculated using new set of
puzzles.
 

Alan Chalker wrote:
>
>
> Sergey Yurgenson wrote:
>>
>>
>> OK. Moggy has find effective way to communicate testcase back.
>> Now what?
>
> That was a very clever hack on Moggy's part. Even more clever was
> hiding the array output in the error message past what's shown on
> the
> summary listing tables. (For those of you interested in what I'm
> talking about look at entry #34847)
>
> However it was clearly using a function not intended to be
> accessible
> to us (arrayviewfunc) which has now been disabled. Regardless, I
> applaud him for knowing some very esoteric MATLAB debugging
> techniques.

Nick Howe wrote:
> On a different subject: I now have code that can solve puzzle #41
> and others without destroying all the objects. Sadly, I find that
> the savings from doing so don't appear to justify the costs
> (increased probing to determine whether the problem is soluble in
> this manner). Is anybody else facing this problem?

puzzle#41 (of the testsuite!) is indeed not so easy. I also have
code that can solve that kind of puzzles. That the probing is giving
more costs, I don't see that (my sore is 6.5, so 65 beams, and that
is even not optimal, while the last "matlab5.2-converted code" gave a
score of 18.1)
The problem with these contests is always that the bigger puzzles (in
this case more blocks with higher "values") have a much higher weight
than smaller ones. So, the improvement from 18 to 6 or lower is not
much, compared to many bigger ones.

But, I beleive that starting from a puzzle solved in "a
block-friendly way", would give a good starting point for further
solutions.
I'm affraid that time will be too short to make use of it, because
it's not so easy to combine current winning algorithms with "nicer
ones"......

It is up to the Matlab Contest Designers to design
a programming problem that does not "leak"
unintended secrets about the test suite.

The test suite is static, and the winning submission
will always be the one that gets the best score
on the specific test suite.

The Contest Organizers can run a test after the contest
to award a "Generality Prize" for the submission
that gets the best score on an entirely new test suite.
I know they have done this in the past,
because I won the Generality Prize in the Ants contest.

The probes are not hurting the contest
because the execution time per probe
is only 10 seconds, and the queue is not getting
backed up dramatically. Plus, ... in order for probes
to work, the person doing the probing has to be able
to read and interpret the results, ... so there is a
natural disincentive to letting the queue get long.

-- David Jones

 Per Rutquist wrote:
>
>
> Nick Howe wrote:
>> I'd like to speak up against this strategy. Although clever,
and
>> perhaps fun to implement, I feel that it goes against the
spirit
> of
>> the contest.
>
> I agree completely. Probing for constants is part of the contest,
> and
> tweaking seems unavoidable, but probing for the entire testsuite is
> just plain wrong. The contest is to submit code that solves the
> problem, not code that outputs the correct solution.
>
> Those who want to show off their probing abilities should do it on
> one testcase, and submit the perfect solution for that case.
> (Briefly
> taking the lead, and forcing their code to be included in every
> future entry. This happened already in the Mars Surveyor contest.)
> Repeating the process for every case in the testsuite proves
> nothing.
> (And it clobbers up the queue.)
>
> By the way, I believe it would take me about two entries to
> completely map one of the smaller (say n=5) testcases. Spamming the
> queue with hundreds of entries lacks elegance.
>
> /Per

Sergey Yurgenson wrote:

> I still think that final score has to be calculated using
> a new set of puzzles.

That's what I used to think too, but then I realized that one of the
many "tweak" entries will still probably be optimized for that new
set of puzzles just by chance. Sigh...

I still think one of the best ways for holding these contests is by
having multiple twilight phases during which the code from past
phases are revealed but not the code from the current phase. I think
it justifies a lot of purposes and makes everybody happy.

FGH wrote:
> I still think one of the best ways for holding these contests is by
> having multiple twilight phases during which the code from past
> phases are revealed but not the code from the current phase. I
> think
> it justifies a lot of purposes and makes everybody happy.
But the duration of the contests is already too long. So, let's keep
it like this....

Stijn Helsen wrote:
>
>
> FGH wrote:
>> I still think one of the best ways for holding these contests
is
> by
>> having multiple twilight phases during which the code from past
>> phases are revealed but not the code from the current phase. I
>> think
>> it justifies a lot of purposes and makes everybody happy.
> But the duration of the contests is already too long. So, let's
> keep
> it like this....
I am not sure what you mean by the duration of the contests being too
long but the contests could still have the same length and each day
or two can be one twilight phase.

Sergey Yurgenson wrote:

> It does really
> concern me, because I want to find good algorithm, not to
“cheat”.

Well, since I dont wanna cheat, here are the infos that i got until
Mathworks took away my toy:

suitenr: hash;n;index1,value1,index2,value2,...

01: 141;10;26,56,47,5,75,70,
02: 1462;17;5,77,6,5,8,35,9,77,10,13,13,78,44,80,64,8 ...
03: 856;22;42,73,50,14,51,4,73,33,76,11,108,6,170,20, ...
04: 3000;30;12,31,13,31,18,31,20,7,24,27,29,28,43,12, ...
05: 144;14;6,8,11,14,44,13,50,6,55,1,78,12,80,12,96,1 ...
06: 75;7;12,1,21,17,23,30,38,20,
07: 197;33;412,38,508,38,578,38,673,12,674,38,
08: ?;37;304,80,366,76,371,80,380,80,415,80,447,80,46 ...
09: ?;23;33,54,53,22,59,46,68,43,72,43,86,43,93,21,10 ...
10: ?;4;8,1,9,3,

a colon behind a line means, that the infos are complete, a '...'
means, that there are more object but the field was too short to show
the entire string.

Hope that helps you guys to go below a score of 100!

Marko

Marko Stefanovic wrote:
> Hope that helps you guys to go below a score of 100!

Seems like this isn't needed anymore :-)
Jan

Jan Langer wrote:
>
>
> Marko Stefanovic wrote:
>> Hope that helps you guys to go below a score of 100!
>
> Seems like this isn't needed anymore :-)
> Jan

Now THAT is a clever hack. Blindingly obvious in hindsight, yet
no-one thought of it until now.

Well done M+E.

Hannes Naudé wrote:
> Now THAT is a clever hack. Blindingly obvious in hindsight, yet
> no-one thought of it until now.
Or no-one who wants to find ways to cheat. Some (I hope many) people
just want to "play the game".

> Or no-one who wants to find ways to cheat. Some (I hope many)
> people
> just want to "play the game".
  
There is a simple way to plug the hole. If beam.m at the contest
server checks that the dbstack doesn't contain "solver" when
initializing the box, then it should be ok.

/M&E

Stijn Helsen wrote:
>
>
> Hannes Naudé wrote:
>> Now THAT is a clever hack. Blindingly obvious in hindsight, yet
>> no-one thought of it until now.
> Or no-one who wants to find ways to cheat. Some (I hope many)
> people
> just want to "play the game".
Hi Stijn
  
If by "play the game" you mean twiddling random seeds and sending in
1000+ entries to find the structure of the test set, yes lots of
people want to "play the game". I see more application of brainpower
in what Magnus + Erik did than in the 100s of probes sent in by our
favourite queue spammers.

I know that you personally contribute proper algorithmic advances and
I have a lot of respect for that. That is also the way I played this
game in earlier contests, unfortunately Daylight is heavily tilted in
favour of the tweakers and probers. If you disagree with this go
check how the last few contests ended.

I would like to draw the attention of those who are interested in
cheating, probing, or their countermeasures to the submission ID
36535. I have no intention to use this, but to publish another weak
point. Although I got only 41 of the 122 characters expected, it's
still a lot of information retrieved.

Heinrich Acker

Heinrich Acker demonstrates a method for extracting information out
of the test suite and provides a good commentary:

 <http://www.mathworks.com/contest/blackbox.cgi/view_submission.html?id=36535>

He is right, of course, and we've had some good discussion about this
in past contests.

As long as we provide any feedback on the entries at all, it's
possible to learn about the test suite. Even if we just indicated if
an entry passed or failed, a contestant could rig a sequence of
submissions to pass and fail when certain conditions are met. This
was part of the motivation behind Darkness- to some part of the
contest where this was true.

Also, we like to show some information when an entry fails for
debugging purposes. As Heinrich demonstrates, this widens the pipe
to send information out.

In this contest, the problem is exacerbated because of the relatively
low information content of the test suite. This is definitely
something we'll keep more directly in mind when defining future
contests.

With all that in mind, we're interested in your feedback about what
changes to make, especially with respect to future contests.

Matthew Simoneau wrote:

> With all that in mind, we're interested in your feedback about what
> changes to make, especially with respect to future contests.
  
More queue problems? It seems to be stuck at 7:18 AM.

will there be generality prize this contest?

Brilliant! (I had another hack in mind, which I believe would also
work, but at about a third of the data rate that Mr. H.Acker
obtained.)

Now that it has been shown that there is an elegant way of extracting
lots of information from the testsuite quickly, we can assume that
the probers will go away, as it is no fun doing something that has
already been done more elegantly by someone else.

/Per
 
 Matthew Simoneau wrote:
>
> Heinrich Acker demonstrates a method for extracting information out
> of the test suite and provides a good commentary:

Per Rutquist wrote:
>
>
> Brilliant! (I had another hack in mind, which I believe would also
> work, but at about a third of the data rate that Mr. H.Acker
> obtained.)
>
> Now that it has been shown that there is an elegant way of
> extracting
> lots of information from the testsuite quickly, we can assume that
> the probers will go away, as it is no fun doing something that has
> already been done more elegantly by someone else.
>
> /Per
>
> Matthew Simoneau wrote:
>>
>> Heinrich Acker demonstrates a method for extracting information
> out
>> of the test suite and provides a good commentary:

While this is another 'elegant' way of doing this, the use of error
messages isn't allowed within the rules (which explicitly state no
debugging commands or error commands). I think this is just another
item that 'fell through the cracks' like the other hacks.

Conversely, the techniques I use, while slow, don't go against the
rules..... Also, he glossed over a few important steps in his
commentary for actually getting useful information. I've found I've
had to develop some interesting algorithms and code in order to
obtain the useful information in a reasonable fashion... And there
are still serious algorithmic challenges to overcome, such as
extracting information from tests that aren't a singleton n (i.e.
there exists more then one test with that n number).

I'm not planning on going away because I feel that the development of
those types of probing algorithms is just as interesting as the
development of the solver alogrithm.

Hannes Naudé wrote:
>If you disagree with this go
> check how the last few contests ended.
I agree that the "random tweaking" can't be called "programming", so
in that way it isn't really part of a "programming contest". But
still, the hacks (some easy, some more complex), are more at (or
over) the sideline of the contest goals.
Regarding the last hack. I know they (the Mathworks) don't, but I
would expect that they would use another beam-routine (in this case)
than that that they offer us to test, especially in the way of
initializing. I would make the initialiser with some unknown (coded)
input, so that it couldn't be hacked as easily. When I saw the
routine I thought there could be problems, but didn't mind.

Stijn

Stijn Helsen wrote:
> I agree that the "random tweaking" can't be called
> "programming", so
> in that way it isn't really part of a "programming contest".

I don't have much to say in favor of the tweakers; their presence is
one reason I don't play much in Daylight. But in my mind there is
still a great difference between submitting an entry that one hopes
will have the best score, and submitting one deliberately designed to
reveal information that can only be useful on future submissions.

I'm glad that people have fun hacking the contest and probing the
test suite. But using that that to win the contest essentially
forces everybody to play that game or give up on the competition.
Perhaps there could be a separate competition for the probers? Run
their entries on a modified testbed that disables the beam function,
and see who can get the best score based on their "remarkable
intuition".

Alan Chalker wrote:
>
> While this is another 'elegant' way of doing this, the use of error
> messages isn't allowed within the rules (which explicitly state no
> debugging commands or error commands). I think this is just
> another
> item that 'fell through the cracks' like the other hacks.

This sounds like hairsplitting to me. But if you want to judge it by
command categories: Debugging commands or error commands are those
that are listed in the respective categories in the Matlab
documentation. I use none of them, so the method I proposed is not
less legitimate than yours - only better, i.e. more effective and
simple.
 
> Conversely, the techniques I use, while slow, don't go against the
> rules..... Also, he glossed over a few important steps in his
> commentary for actually getting useful information. I've found
> I've
> had to develop some interesting algorithms and code in order to
> obtain the useful information in a reasonable fashion... And there
> are still serious algorithmic challenges to overcome, such as
> extracting information from tests that aren't a singleton n (i.e.
> there exists more then one test with that n number).

My example includes using the result of a beam call to distinguish
these cases. Perhaps this is a challenge for you, I call this the
most basic part of my example. Maybe you have to treat it like art
and science to justify what you do.

> I'm not planning on going away because I feel that the development
> of
> those types of probing algorithms is just as interesting as the
> development of the solver alogrithm.

If you have to work so hard on these algorithms, it only shows that
your method is inferior, not that it is a desirable goal.

Heinrich

This is just slightly off topic but...

I agree with all of you that "random tweaking" isn't really
programming, but I don't see it as meritless. It's just another form
of optimization at your disposal. It behooves you to write an
effective algorithm that also has inherent "tweakability." You need
both aspects in the code for it to stay on top in any version.

Originally, when the contest started, I was working on a predictive
algorithm that didn't use any high beams, but used the 4*n possible
shots (since I wasn't destroying anything) as inputs to a neural net
that output the solution. Obviously the neural net has to be
scalable with n, so I was using a 20th order Fourier series on n as a
basis function for the neural net connection matrix. Then I used a
genetic algorithm to evolve the parameters of the Fourier series,
embedding the predictive rules into those
parameters....Aaaaand...then I realized that the high beam penalty
adds to your beam count and NOT the results directly, meaning it was
significantly more advantageous to destroy all the objects in the
process of getting right than getting it wrong.

And now I mostly tweak.

I'm not in favor of the ridiculous queue spamming that's been going
on, but I wanted to make the point that tweaking still has a
competitive value.

 Nick Howe wrote:
>
>
> Stijn Helsen wrote:
>> I agree that the "random tweaking" can't be called
>> "programming", so
>> in that way it isn't really part of a "programming contest".
>
> I don't have much to say in favor of the tweakers; their presence
> is
> one reason I don't play much in Daylight.

Matt McDonald wrote:
>
>
> Originally, when the contest started, I was working on a predictive
> algorithm that didn't use any high beams, but used the 4*n possible
> shots (since I wasn't destroying anything) as inputs to a neural
> net
> that output the solution. Obviously the neural net has to be
> scalable with n, so I was using a 20th order Fourier series on n as
> a
> basis function for the neural net connection matrix. Then I used a
> genetic algorithm to evolve the parameters of the Fourier series,
> embedding the predictive rules into those
> parameters....Aaaaand...then I realized that the high beam penalty
> adds to your beam count and NOT the results directly, meaning it
> was
> significantly more advantageous to destroy all the objects in the
> process of getting right than getting it wrong.
>

I started off overestimating the benefit to avoiding the high beam
also. What stopped me was discovering that there are lots and lots
of configurations where it's impossible to determine much of the
puzzle using only the low beam, because there are regions of the
interior of many puzzles which cannot be reached by low beam. About
this time, daylight broke and I discovered that algorithms which blew
everything up did almost 10-fold better. So, I started tweaking a
bit also (algorithmic, not random number).

Yeah, I think it was the wording of the rules...the rules made it
sound more costly than it actually is if you look at the scoring.

Also, personally I think they should give a prize for the algorithm
that does the best without using highbeams. Doing well under
circumstances where it may be impossible to know the full puzzle is
actually more interesting to me than blowing everything up and
getting it right.

Glad I wasn't the only one to originally misinterpret the scoring.
8^)

Matt

 Mike Bindschadler wrote:
> I started off overestimating the benefit to avoiding the high beam
> also. What stopped me was discovering that there are lots and lots
> of configurations where it's impossible to determine much of the
> puzzle using only the low beam, because there are regions of the
> interior of many puzzles which cannot be reached by low beam.
> About
> this time, daylight broke and I discovered that algorithms which
> blew
> everything up did almost 10-fold better. So, I started tweaking a
> bit also (algorithmic, not random number).

To the contest team: Couldn't you just permute the order of the
puzzles randomly before a submission is tested? This would mean a
simple change in the testing code but would make probing lots
harder...

Markus wrote:
>
>
> To the contest team: Couldn't you just permute the order of the
> puzzles randomly before a submission is tested? This would mean a
> simple change in the testing code but would make probing lots
> harder...
  
I'm not sure why you think that would impact probing at all. The
probing doesn't rely upon the order of the puzzles themselves, just
the size of the puzzles. For example, there is only one puzzle in
the test suite that has size n=60. It doesn't matter where it is in
the order since the probe code just escapes from the main code when
it sees that size.

However, such a permutation WOULD affect the much of the random
number tweaking we see going on. People working on that are
essentially trying to find just the right sequence of random numbers
that fits best with the fixed order of the tests. Rotating the order
of the tests would make that task VERY difficult.

He suggested it because it is more effective to rely on random
numbers order to uniquely tag puzzles then use puzzle size.

 Alan Chalker wrote:
>
>
> Markus wrote:
>>
>>
>> To the contest team: Couldn't you just permute the order of the
>> puzzles randomly before a submission is tested? This would mean
a
>> simple change in the testing code but would make probing lots
>> harder...
>
> I'm not sure why you think that would impact probing at all. The
> probing doesn't rely upon the order of the puzzles themselves, just
> the size of the puzzles. For example, there is only one puzzle in
> the test suite that has size n=60. It doesn't matter where it is
> in
> the order since the probe code just escapes from the main code when
> it sees that size.
>
> However, such a permutation WOULD affect the much of the random
> number tweaking we see going on. People working on that are
> essentially trying to find just the right sequence of random
> numbers
> that fits best with the fixed order of the tests. Rotating the
> order
> of the tests would make that task VERY difficult.

Mike Bindschadler wrote:
> I started off overestimating the benefit to avoiding the high beam
> also. What stopped me was discovering that there are lots and lots
> of configurations where it's impossible to determine much of the
> puzzle using only the low beam, because there are regions of the
> interior of many puzzles which cannot be reached by low beam.
> About
> this time, daylight broke and I discovered that algorithms which
> blew
> everything up did almost 10-fold better. ....
Although I will not have any program ready, I believe a nice program
can be made with only "smart high beaming". I'm curious if anyone
will be ready, and I believe the chance is high!!! (maybe not the
people who spend a lot of time to looking to other submissions, or
reading this log list of messages...)
Stijn

Nick Howe wrote:
>> I'm glad that people have fun hacking the contest and probing
the
> test suite. But using that that to win the contest essentially
> forces everybody to play that game or give up on the competition.
> Perhaps there could be a separate competition for the probers? Run
> their entries on a modified testbed that disables the beam
> function,
> and see who can get the best score based on their "remarkable
> intuition".

Having done both probing and tweaking in past contest (and won a
"Mars Surveyor" T-shirt while doing so), I can't really condemn that
behaviour. At present I mostly play in darkness/twilight, and I
wouldn't have the time to keep playing through daylight even if I
wanted to.

I think it would be an excellent idea to have two contests. One
where, by gentlemen's agreement, probing and tweaking is disallowed.
The other one, where "anything goes" will likely be won by an entry
that outputs the perfect solution in no time for the entire test
suite. Better put them on different servers, though, as the second
one is liable to be overloaded from time to time.

Can we please have another Matlab golf contest? That was so much fun!

/Per

It looks like Hannes successfully crashed the queue
again....Hopefully it can be restarted soon...

Alan Chalker wrote:
>
>
> It looks like Hannes successfully crashed the queue
> again....Hopefully it can be restarted soon...
I'd be very amazed if that entry of mine crashed the queue. I think
the contest team pauses it from time to time for administration tasks
(such as updating the blog).

Alan Chalker wrote:
>
>
> It looks like Hannes successfully crashed the queue
> again....Hopefully it can be restarted soon...
  

Please monitor the queue these last few hours. Hannes stole
development time from us.

Anders Skjäl wrote:
>
>
> Last time I checked 3.3748/0.5073 wasn't >7. :)
  
They fixed the entry... it was >6

Please, no more attacks against the contest infrastructure. I can't watch
the queue constantly.

snake wrote:
>
>
> Eduan Sennah (read it backwards) blocked the queue again. I believe
> the correct punishment is to remove all his earlier prizes
> retroactively.
Alan & Snake
Don't be such cry babies. You don't like how I play the game? Well
guess what, I don't like how you play it either.

By the way if you were doing real "development", the status of the
queue would make absolutely ZERO difference to the amount of
"development time" you have left.

Lastly, I think you will find that my entries are not crashing the
queue. Each time a new one reaches the front, the contest team pauses
the queue and updates their filters. So, yes they are monitoring the
queue.

My intention was to publish the entire test set to force a change of
test set and give the algo guys a chance against probers like
yourself. I've now given up on that and am doing this purely to piss
you off.

Regards
Hannes

The queue is hung again....on Lobotomy XXI

I guess some people have the childish attitude that if they can't win
the game they aren't going to let anyone else play at all.....

Gerbert Myburgh wrote:
>
>
> #1. I just want to make it clear that one wasn't mine.
>
> #2. Your can't complain. The total time your 800 entries took was
> way
> more than the time of a short queue stall.
>
> #3. The point of generality was what again? To send
> 30 entries just so you might get lucky?
>
> Alan Chalker wrote:
>>
>>
>> The queue is hung again....on Lobotomy XXI
>>
>> I guess some people have the childish attitude that if they
can't
>> win
>> the game they aren't going to let anyone else play at all.....

Gerbert:

1. Sorry to imply it was you who did that entry. The comments in it
explicitly refer to Hannes using your name and thus my comment was
aimed at him.

2. I'm not complaining at all about the 'lost' time the queue is hung
(someone else made those comments). I was just trying to notify the
contest organizers in case they hadn't noticed. They explictly asked
for people to not hack the system today since they can't watch it
constantly. Regarding my entries, if you look at the chronological
listing, most of them are during periods of time nobody else was
submitting entries. My personal submissions have had minimal impact
on everyone else.

3.My generality attempts are all different if you care to inspect
them. I went through the leader code from various phases of the
contest and utilized those in my submissions in the idea that some of
the earlier solvers weren't as fitted to the test suite.

The MATLAB Contest Team wrote:
>
>
> The next MATLAB Programming Contest runs from Wednesday November
> 29th
> at 9AM EST through Wednesday, December 6th at 5PM EST.

There was a nice calm before the storm, and then quite a storm of
entries.

It will be nice that this time the queue will probably not take until
1:00 AM tomorrow to process, since the top codes all run in just
several seconds.

Good luck to all. I don't think any of my usual small time tweaks
will carry the day this time. Congrats to whomever the new Overall
Contest Winner is.

Alan: Is "The ultimate end" the best one? In that case I added one of
the n==18 cases with 8 seconds to spare. :)

 the cyclist wrote:
>
>
> The MATLAB Contest Team wrote:
>>
>>
>> The next MATLAB Programming Contest runs from Wednesday
November
>> 29th
>> at 9AM EST through Wednesday, December 6th at 5PM EST.
>
> There was a nice calm before the storm, and then quite a storm of
> entries.
>
> It will be nice that this time the queue will probably not take
> until
> 1:00 AM tomorrow to process, since the top codes all run in just
> several seconds.
>
> Good luck to all. I don't think any of my usual small time tweaks
> will carry the day this time. Congrats to whomever the new Overall
> Contest Winner is.

Never mind, I guess you planned those failed executions. Good idea to
slip the real end in between.

Nope. I knew people would be 'gunning' for me so I submitted several
'fake' ones and several that built up to the best one. The best one
should be "End of it all". Congrats on probing one of the singleton
cases. Feel free to copy the submission 'Final Lookup Tables' and
add it to it. I thought it'd be nice to document what we know about
the test suite for anyone who is interested.

For those of you interested in seeing a non-obscufated version of the
winning entry, check out "That's it" (#38346). I've also posted a
version with the final lookup tables right up at the top of the code
under "Final lookup tables" (#38364).

I agree with the cyclist that it was nice not having this end at
midnight EST and then waiting for the queue to clear for hours.
Thanks all for an exciting competition. Look forward to
participating again in the Spring.

Congratulations to Alan and all the winners of the other prices and
thanks to all contestants It was really fun to participate and learn.

Hope to see you all in spring.

srach

Since they asked in the blog entry, I'd like to make a few humble
suggestions to the contest organizers regarding potential changes in
future contests:

1. A CAPTCHA on the entry submission page would help slow down the
queue spammers. You've got them on the newsgroup and blog pages so I
assume there is just some technical challenge to adding them to the
entry submission page. I suggest this is the highest priority
effort. (Note that I didn't use any sort of automated form filler or
submission code in any of my entries. It was all done by hand except
for the routine that obscufated the code.)

2. Display enough significant digits on the results and time to
understand why certain entries score better then others. For example
the 10th best entry has a better time than the 8th best entry yet the
results and score display as the same.

3. Along the same lines, perhaps just roundoff all times to the
nearest tenth of a second. There is huge variations in on the order
of tenths of seconds on identical entries, resulting in slightly
different scores. Such a fix will help reduce the number of people
who just resubmit entries over and over again at the contest close
hoping for a fluke improvement.

4. Introduce a delay of a few minutes during daylight between when a
submission is run/submitted and when you can view the code of the
submission. This should help reduce the mad copying at contest
deadlines as people try to resubmit other people's codes.

5. As the initialization of the runcontest suite, initialize the rand
number generators with the current time in order to reduce the
attempts at finding just the right constants to match the sequence
rand generates

6. Announce the dates of the contest more than just a day or two in
advance so people can 'clear their calendars' if they so desire.

7. Add some contests / prizes that encourage group efforts /
collaborations. I tried to reach out and get a team formed, but both
people who initially responded ended up not following through with it
for whatever reason.

I'd like to finish by thanking the contest organizers for all the
effort they put into this semiannual event. I'm sure there is
significant work in coming up with a contest idea and then monitoring
the flow of the contest. I know it helps me improve my MATLAB
knowledgebase and I hope it helps provide insights and feedback that
you can apply to your 'day jobs' of continually improving and
spreading the word about MATLAB. Thank you so much.

Congrats Alan, and all the other winners.

Thanks for the Contest team, as always the tourney was great.

As for changes to the format.

I know this comes up every time, but a longer Twilight might improve
the general algorithms, simply because it's impossible to develop
algorithms as fast as the combined tweaking effects in daylight.

Someone else mentioned it, but intermitted twilight might also be
interesting. So you get a glimpse of what everyone else is doing from
time to time but then you have to work on your own again to make
improvements.

And lastly, at least fix it so that you can't access/edit entries
while they are still waiting in the process queue.

See you all in 6 months then. ;)

Alan Chalker wrote:
>
>
> Since they asked in the blog entry, I'd like to make a few humble
> suggestions to the contest organizers regarding potential changes
> in
> future contests:
>
> 1. A CAPTCHA on the entry submission page would help slow down the
> queue spammers. You've got them on the newsgroup and blog pages so
> I
> assume there is just some technical challenge to adding them to the
> entry submission page. I suggest this is the highest priority
> effort. (Note that I didn't use any sort of automated form filler
> or
> submission code in any of my entries. It was all done by hand
> except
> for the routine that obscufated the code.)

Aye

> 2. Display enough significant digits on the results and time to
> understand why certain entries score better then others. For
> example
> the 10th best entry has a better time than the 8th best entry yet
> the
> results and score display as the same.

No need if you do #3 instead.

> 3. Along the same lines, perhaps just roundoff all times to the
> nearest tenth of a second. There is huge variations in on the
> order
> of tenths of seconds on identical entries, resulting in slightly
> different scores. Such a fix will help reduce the number of people
> who just resubmit entries over and over again at the contest close
> hoping for a fluke improvement.

Aye!

> 4. Introduce a delay of a few minutes during daylight between when
> a
> submission is run/submitted and when you can view the code of the
> submission. This should help reduce the mad copying at contest
> deadlines as people try to resubmit other people's codes.

Aye, in principle yes. But it will also benefit the people with too
much spare time, pursuiting their great intuition.

> 5. As the initialization of the runcontest suite, initialize the
> rand
> number generators with the current time in order to reduce the
> attempts at finding just the right constants to match the sequence
> rand generates

Aye, get them rand-tweakers.

> 6. Announce the dates of the contest more than just a day or two in
> advance so people can 'clear their calendars' if they so desire.

Aye!

> 7. Add some contests / prizes that encourage group efforts /
> collaborations. I tried to reach out and get a team formed, but
> both
> people who initially responded ended up not following through with
> it
> for whatever reason.

Nay, the great intuitionists would gang up.

And last. Try making the contests less probeable. Without singleton
cases there would have been less probing, even though someone did
some stuff with n==10 too.

Thank you!

Matthew Simoneau wrote:
>
>
> Heinrich Acker demonstrates a method for extracting information out
> of the test suite and provides a good commentary:
>
> <http://www.mathworks.com/contest/blackbox.cgi/view_submission.html?id=36535>
>
> He is right, of course, and we've had some good discussion about
> this
> in past contests.
>
> As long as we provide any feedback on the entries at all, it's
> possible to learn about the test suite. Even if we just indicated
> if
> an entry passed or failed, a contestant could rig a sequence of
> submissions to pass and fail when certain conditions are met. This
> was part of the motivation behind Darkness- to some part of the
> contest where this was true.
>
> Also, we like to show some information when an entry fails for
> debugging purposes. As Heinrich demonstrates, this widens the pipe
> to send information out.
>
> In this contest, the problem is exacerbated because of the
> relatively
> low information content of the test suite. This is definitely
> something we'll keep more directly in mind when defining future
> contests.
>
> With all that in mind, we're interested in your feedback about what
> changes to make, especially with respect to future contests.
  
Probing itself could be an interesting contest topic, i.e. within 3
min limit probing a number of boxes to identify contents inside these
boxes with very limited information feedback. But in this contest,
when I saw so many look-at tables in top entries, I lost my
interesting.

I also lost interest when I saw all the look-up tables. And I didn't
know about the contest until daylight.

Most of the contests have had two main interests (for me):

1) an interesting and difficult problem
2) the ability to combine parts of multiple approaches to create
better approaches

I have not investigated the evolution of the contest closely but I
suspect 2) is lacking. Once you use a beam, that's it.

I congratulate Alan on his victory. It is good to see the individual
who worked the hardest win. But, the idea of probing the test suite
seems much less interesting to me than the "real" problem.

I support the word verification. I would also support registration
of participants (c'ya Don).

Two other quick suggestions that hopefully would be easy to
implement:

1. Add the ability to search by entry number as the search term on
the 'Search the entries page'. Yes I know we can manually enter a
corrected formated URL for this, but it's a bit of a pain.

2. Add the ability to easily do a diff versus the "based on code",
perhaps via a 'diff' link next to the "based on code" link on each
'View Submission' page. Again, I know we can manually do this, but
it would make it easier on us with what should be a minor HTML
addition.

> 1. A CAPTCHA on the entry submission page would help slow down the
> queue spammers.

Yes! Yes! Yes!

> 3. Along the same lines, perhaps just roundoff all times to the
> nearest tenth of a second.

I totally agree. You should also run the entries that are on the
first places at a phase end again for ten or twenty times and average
the processing time over all runs. This might delay the queue a bit,
but who cares about that some minutes after the deadline for a prize
passed.
 
> 4. Introduce a delay of a few minutes during daylight between when
> a
> submission is run/submitted and when you can view the code of the
> submission.

That's a good idea.

Even better would be to return to twilight for the last day of the
contest! This was suggested already some times in the past. If the
last contest day would be in twilight, one would have to make a real
improvement to the code to win the grand prize. Look at the last
contest, blockbuster. Hannes & Cobus came up with a completely new
algorithm that performed better, but the cyclist came and grabbed the
grand prize away with a small tweak that many other contestants did
before on other algorithms. Ok, it was ingenious to find the
promising code in the queue, but the grand prize should only be given
to a grand code improvement.

> 5. As the initialization of the runcontest suite, initialize the
> rand
> number generators with the current time.

Or completely ban the random functions. But on the other hand,
mod(pi^n, 1) is also some sort of a random number, so this would not
stop the random number tweakers.

> 6. Announce the dates of the contest more than just a day or two in
> advance

Of course!

> 7. Add some contests / prizes that encourage group efforts /
> collaborations.

I can't think of any way to do this. Until the end of the contest, we
are all only a name (or nickname) and mail address. No one can tell
if there is a group or a single person behind it

8. Go through the newsgroup thread of the blockbuster contest. We
discussed there much about how to penalize obfuscation and how to
avoid spamming.

Also my thanks to the three poor guys of Matlab who had a lot of work
keeping the queue running...

Markus

Alan made some interesting suggestions.
My vote:
1 – yes
2/3 – yes
4 – yes
5 – will not help (one can reinitialize rand in code). Probably
random permutation of puzzles is better. However any “random rand”
will encourage people to submit the same code multiple times if rand
is used in code.
6 – yes, it would be nice to know also schedule and deadlines of all
intermediate challenges and if generality competition will be held.

Thanks to all organizers and participants for good game.
See you in six months.

Sergey (SY)

Alan Chalker wrote:
>
>
> Since they asked in the blog entry, I'd like to make a few humble
> suggestions to the contest organizers regarding potential changes
> in
> future contests:
>
> 1. A CAPTCHA on the entry submission page would help slow down the
> queue spammers. You've got them on the newsgroup and blog pages so
> I
> assume there is just some technical challenge to adding them to the
> entry submission page. I suggest this is the highest priority
> effort. (Note that I didn't use any sort of automated form filler
> or
> submission code in any of my entries. It was all done by hand
> except
> for the routine that obscufated the code.)
>
> 2. Display enough significant digits on the results and time to
> understand why certain entries score better then others. For
> example
> the 10th best entry has a better time than the 8th best entry yet
> the
> results and score display as the same.
>
> 3. Along the same lines, perhaps just roundoff all times to the
> nearest tenth of a second. There is huge variations in on the
> order
> of tenths of seconds on identical entries, resulting in slightly
> different scores. Such a fix will help reduce the number of people
> who just resubmit entries over and over again at the contest close
> hoping for a fluke improvement.
>
> 4. Introduce a delay of a few minutes during daylight between when
> a
> submission is run/submitted and when you can view the code of the
> submission. This should help reduce the mad copying at contest
> deadlines as people try to resubmit other people's codes.
>
> 5. As the initialization of the runcontest suite, initialize the
> rand
> number generators with the current time in order to reduce the
> attempts at finding just the right constants to match the sequence
> rand generates
>
> 6. Announce the dates of the contest more than just a day or two in
> advance so people can 'clear their calendars' if they so desire.
>
> 7. Add some contests / prizes that encourage group efforts /
> collaborations. I tried to reach out and get a team formed, but
> both
> people who initially responded ended up not following through with
> it
> for whatever reason.
>
> I'd like to finish by thanking the contest organizers for all the
> effort they put into this semiannual event. I'm sure there is
> significant work in coming up with a contest idea and then
> monitoring
> the flow of the contest. I know it helps me improve my MATLAB
> knowledgebase and I hope it helps provide insights and feedback
> that
> you can apply to your 'day jobs' of continually improving and
> spreading the word about MATLAB. Thank you so much.

> will encourage people to submit the same code multiple times

I think submitting the same code more than once should be forbidden
in general. However, checking if submitted algorithms differ would
require an intelligent parsing routine, because, for example, setting
a variable to a value that is used nowhere changes the source but not
the algorithm.

However, if all code that is a candidate to win a prize would be
re-run some times (as I suggested before), and the computation time
would be rounded, then resubmitting the same code would be completely
useless. If two entries attain the same score, of course the code
submitted first should win the prize.

Markus

my vote (in order of preferene):

1: yes! (simple and effective)

?: (the last comment by snake, about making the testsuite less
probeable by increasing its information content, or by careful design
e.g.: ants contest was much less prompt to probing)

5: Another variant, just initialize rand to a fixed seed value for
EVERY puzzle of the testsuite (instead of just at the initialization
of the testsuite). In this way if your code would benefit of a little
randomness that's fine, yet this will be useless for tagging
individual puzzles.

4: (Markus variant, reintroduce twilight) controversial but yes!

6: obviously

2/3: problematic, if one increases the number of relevant digits
relevant for the time or score result it would encourage resending a
winning code with minimal or no changes at all trying to beet it, if
one decreases the number of relevant digits it would become more
difficult to understand how your entry performs comparing to others.
I would suggest showing as many digits as humanly possible, but also
introducing a small "score bonus" to the entry that makes it to the
first place (in that way trivial differences in time won't beet a
"genuine" first entry).

Thanks everybody for a great contest!

> 2/3: problematic, if one increases the number of relevant digits
> relevant for the time or score result it would encourage resending
> a
> winning code with minimal or no changes at all trying to beet it,
> if
> one decreases the number of relevant digits it would become more
> difficult to understand how your entry performs comparing to
> others.
> I would suggest showing as many digits as humanly possible, but
> also
> introducing a small "score bonus" to the entry that makes it to the
> first place (in that way trivial differences in time won't beet a
> "genuine" first entry).

Good idea. I thought about suggesting some kind of threshold for an
improvement to take the lead. Your suggestion sounds like a good way
to do it.

How about letting the winner be determined by some formula involving
total number of submitted entries? For example:

Total score = sum(improvements on log{score during daylight}) /
(total entries during daylight + 10) + small bonuses for winning
darkness and/or twilight.

This way, submitting multiple identical entries or random
modifications will likely increase the denominator more than it
increases the numerator, hence lowering the total score.

It would also be fun to have a scoreboard, where you can see the
contestants moving up or down depending on their overall
contribution, compared to the current case where anyone can take the
lead by resubmitting the current leading entry.

Of course there also needs to be a gentleman's agreement against
probing, since that can ruin any contest, and is impossible to guard
against using technical means (except during darkness).

/Per

This was my first contest to participate in, actually I observed more
than participated, but I really enjoyed the problem. I came
primarily to improve my MATLAB skills. I know that sometimes a final
analysis which primarily discusses the algorithm but could someone do
an explaination of their coding technique? Answer question like, why
they they coded a certain section that way, etc. Or at least could
someone point me to some well commented code?

Thanks,

RJA