Blockbuster: Contest Evolution

By Matthew Simoneau

Over the course of eight days, hundreds of contestants submitted thousands entries. With so much activity, it is hard to follow the action. This report will pick out some statistics and draw some pictures that give some indication of the progress.

Contents

Submissions Over Time

It's impressive to look at the sheer volume of entries. This area plot shows how the total number of entries grew as the contest progressed. The green area represents the entries that passed the test suite, and the red area shows those that failed.

Activity by Hour

The previous area plot shows a lot of "lumpiness". A steepening slope shows an increase in activity. A histogram shows this more directly. Each bar represents an hour's worth of entries.

The contest had three major phases. The first day was in "darkness", where contestants could submit entries but they couldn't see any of the entries or their scores. To win this phase, an entry must be general and robust. The second day was "twilight", where we showed the scores but not yet the code. This allowed contestants to develop their algorithm without anyone else being able to leverage their ideas.

On the histogram, the darkness and twilight phases are the two large blue boxes on the left. The boxes call out two other time periods: the Sunday Push (we offered a prize to the greatest cumulative improvement to the score in 24 hours) and the Leap (where we recognized the biggest single improvement). The vertical gray lines show the other deadlines

Activity of the winners

Looking at the activity of the contest winners shows their different participation styles. Each winner is listed in order and a red line marks the deadline for the phase he won.

Most active participants

This bar plot shows the number of entries submitted by our most prolific authros.

Participants per Day

We know that one participant may submit hundreds of entries. Let's look at the number of unique participants on each day of the contest.

The first few days draw entries from the most participants. Perhaps this shows that more people prefer the Darkness and Twilight phases of the contest, or that the cost of entry seems higher once the top entry gets complicated.

Score

This is the most useful diagram for visualizing the contest. It shows the dramatic improvements that occurred over time. Each passing entry is a dot, with its submission time on the x-axis and it's score on the y-axis. Since a lower score is better, the dots push down further as time goes on. All entries that took the lead are colored red and the red lines marks the best score at any time. The sample entry is the leftmost red dot and the winning entry is the last red dot in the lower right.

Some of the leading entries are circled and labeled with the author's name. They show who was making the biggest improvements in score (represented in this plot as a vertical drop in the red line) at any point in the contest.

The improvement in score happens over a huge dynamic range. Early in the contest, it is easy to make big improvements in the score. As the algorithms get better, improvements become increasingly difficult. To show this, we normalize the scores so the best (smallest) score is 1 and the worst score is some power of 10. Then we plot them on a logarithmic scale. This exaggerates the improvements at the end of the contest. By increasing the number of decades we spread the scores over, we increasingly exaggerate the smaller improvements made at the end of the contest.

Percent improvement

Here is a plot of the percent improvement generated by each new leader relative to the previous leader. This lets us see who is responsible for the biggest changes over the contest. The upper frontier of this plot is a sort of hall of fame, and someone whose name appears there more than once managed to make several significant improvements to the score.

results vs. cpu time

One of the interesting aspects of the contest is that entries needed to minimize two things at once. Getting the best possible answer must be weighed against the time an entry takes to run. As discussed above, the entry's score is a combination of these two factors. Plotting these two against each other yields a very different picture of the contest.

The leader line is shown in red again in this picture. The gray contours show lines of constant score. In general, the best score is somewhere along the lower-left frontier of shortest time and lowest results. Big improvements tend to move down and to the right, and they are followed by tweaking battles in which the new algorithm claws its way back down the time axis.

Entry length

Here we look at a plot that shows how many characters of code are in each of the leading entries. In regions where you see entries of more or less the same length there are very few differences from one entry to the next. In other places you can see the code getting shorter or longer. The density of the lines also shows how often the lead is changing. It's impressive to see that several times during the contest shorter code ousted longer code from the top spot, either by pruning unneeded computation or by introducing new algorithms. The red line at the top shows the cap on entry length.

Contributions per day

These plots and statistics show the total contribution of each contestant to each day's improvement in score. They show which contestants did the heavy lifting for each day. 

 Wed, 05-Apr-2006
 
 48.29%  2 Tom
 16.62%  1 Imre Polik
 12.69%  1 Hannes Naudé & Cobus Potgieter
  9.87%  1 Colin Ross
  7.41%  1 Niilo Sirola
  3.00%  2 Nicke Carlsson
  2.12%  3 Other
 
 
 Thu, 06-Apr-2006
 
 100.00%  3 Hannes Naudé & Cobus Potgieter
 
 
 Fri, 07-Apr-2006
 
 38.07%  2 Markus
 19.14%  1 Hannes Naudé & Cobus Potgieter
 18.07%  3 Alan Chalker
 10.58%  1 Nick Howe
  4.67%  3 Stijn Helsen
  3.51%  5 David Jones
  5.95% 14 Other
 
 
 Sat, 08-Apr-2006
 
 46.60% 13 David Jones
 25.79% 16 Don Antonio Coimbra de la Coronilla y Azevedo
 10.92%  4 Jim Mikola
  6.44%  3 Stijn Helsen
  4.06%  2 Rick St.Pierre
  2.18%  1 Anders Skjäl
  4.01% 10 Other
 
 
 Sun, 09-Apr-2006
 
 29.21%  3 David Jones
 25.26%  3 DrSuess
 25.10% 12 Jan Langer
 17.23%  4 Anders Skjäl
  2.15%  3 Don Antonio Coimbra de la Coronilla y Azevedo
  1.05%  2 XeF
 
 
 Mon, 10-Apr-2006
 
 59.16% 12 David Jones
 35.38%  5 volkan
  4.33%  2 Yi Cao
  0.38%  1 DrSuess
  0.37%  3 Don Antonio Coimbra de la Coronilla y Azevedo
  0.33%  1 DreadNox
  0.04%  1 Ismail Meric Can Uygan
 
 
 Tue, 11-Apr-2006
 
 45.02% 10 David Jones
 33.65%  1 Markus
  8.43% 11 Don Antonio Coimbra de la Coronilla y Azevedo
  3.22%  2 Rick St.Pierre
  2.59%  1 Anders Skjäl
  1.91%  1 Windy Miller
  5.19% 11 Other
 
 
 Wed, 12-Apr-2006
 
 35.56%  3 Hannes Naudé & Cobus Potgieter
 18.53%  9 David Jones
 13.09%  3 Windy Miller
  8.79%  1 Yi Cao
  7.88%  1 utr
  5.37%  2 the cyclist
 10.78%  7 Other

Conclusion

For analysis, including a listing of all the leaders, the author's contributions to the score by day, and other metrics, see the Statistics page.

Thanks for participating, whether by entering the contest many times, once, or merely checking out the site every now and again.

Be sure to sign up on our notify list so you'll be ready to play the next contest: Send an e-mail to lists@mathworks.com with "subscribe contest-announce" in the body.

Colophon

This contest analysis was calculated and published entirely from MATLAB. We used the Database Toolbox to pull the information directly from the contest database. This HTML document was automatically generated from a MATLAB script using "Publish to HTML", a feature introduced in R14.


Published with MATLAB® 7.2