FAQ

General Questions

1. The queue is backing up - is the server down?
2. Do comments hurt performance?
3. Will there be an analysis of the contest once it ends?
4. Can I see the test suite? Why doesn't my entry pass your test suite; it does perfectly well on the cases I've tried?
5. Why do identical submissions have different scores?
6. How was the scoring method determined? I see two entries; one has a better time and performance measure, yet it doesn't have the better score. Why?
7. It's frustrating to have someone modify your code just a bit and receive a better score. It seems as if the best entries won't be submitted until the last minute, since no one will want to have his/her entry beaten by a slight modification. Why is the contest run this way?
8. What hardware and software to you use to score the entries?

The queue is backing up - is the server down?

From time to time during the contest we will pause the queue to perform some maintenance on the database used to store contest results. At these times, the contest queue will be put on hold. We don't expect these downtimes to be very long, and will try to announce them in the Message of The Day (MOTD) at the main contest page.

If there are any problems with the queue which affect the performance of any entries, after correcting the problem we will automatically rerun the affected contest entries to produce valid scores.

Do comments hurt performance?

Comments do not hurt performance. Each entry is Pcoded (see 'helpwin pcode') before it is run. This removes the effects comments would have on execution time.

Will there be an analysis of the contest once it ends?

At the end of the contest we'll put together an analysis of how the entries evolved over time. This includes various methods of analyzing score vs. contest time and an analysis of the "lineage" of various entries. We'll also put together a description of the workings of the winning entry, and what gave it the edge over other entries. We'll also make available the test suite that we used to run the contest.

Can I see the test suite? Why doesn't my entry pass your test suite; it does perfectly well on the cases I've tried?

The test suite is not the same as the examples we've made available for you to try out. It's possible that your code is having trouble with some of our test cases.

We will release the test suite at the end of the contest, at which time you can check the entries that failed to see what happened. However, we don't plan to release any other information about the test cases during the contest.

Why do identical submissions have different scores?

If there were no timing variability, we'd expect identical entries to have the same CPU time and score. However, because of the variability present in timing entries, there is some scoring variability between runs of the same code, even with the same test case. This slight difference in the CPU time of identical entries is caused by the "noise" present when measuring CPU time. It's possible that this timing variability will affect your score slightly.

How was the scoring method determined? I see two entries; one has a better time and performance measure, yet it doesn't have the better score. Why?

The scoring system was chosen to weight the performance parameters in a way that we felt was appropriate. This is somewhat arbitrary; as what's appropriate isn't always objective; it's likely that the rankings you see would be different if a different scoring function were used.

Your score is the average of the scores for each individual test point. The time and performance statistic displayed on the contest rankings are the average time and average blank space ratings over all the test points. They are listed to give you an idea of how your entry performed, but they are not the numbers used to compute your score. It may be that the highest ranking entry does not have the lowest average time or lowest average performance rating since in general:

score = sum(f(score,time)) != f(sum(score),sum(time)).

It's frustrating to have someone modify your code just a bit and receive a better score. It seems as if the best entries won't be submitted until the last minute, since no one will want to have his/her entry beaten by a slight modification. Why is the contest run this way?

One of the goals behind the MATLAB Contest is to learn more about MATLAB programming. Viewing and modifying other entries, as well as having your own entries be modified, is a great way to learn how to write better code.

From previous contests we've seen a recurring pattern - a particular algorithm is submitted and then "optimized", and then suddenly a contestant submits a better algorithm to take the top spot. It's then optimized, and the cycle continues. The top entries typically have been repeatedly modified by several contestants.

Though tweaking of entries that you see can be frustrating, it's part of the evolution of the entries. The tweaking portion doesn't represent the end of the contest but is just an intermediate phase; the brainstorming and tweaking never really ends!

At the end of the contest we'll post an analysis of the entries showing how the entries developed over the course of the contest. This analysis includes highlights of the tweaking that went on in successful entries. At this point you can more clearly see if allowing code tweaking has hindered or helped the creativity involved in the contest. From what we've seen, the tweaking has made the contest more of a learning experience.

What hardware and software to you use to score the entries?

Here are the specs: Dual Pentium D (64 bit, dual core), 2.80GHz, 1024 KB cache, 1GB RAM. We're running the latest version of MATLAB for 64-bit Linux.