Darkness 2002-11-05 00:00:00 UTC
Twilight 2002-11-06 00:00:00 UTC
Daylight 2002-11-07 00:00:00 UTC
Finish 2002-11-12 00:00:00 UTC

Protein Folding - Statistics

Contest Analysis

There were 2437 entries in the Protein Folding contest, with nearly two hundred participants (the exact number is hard to determine since people don't always report their names consistently).


Submissions Over Time

This area plot shows how the entries added up as the contest wore on. The green area represents the number of files that passed the test suite, and the red area shows those that failed. Overall 1881 of 2437 entries passed the test suite, for an impressive pass rate of 77.2% (the highest pass rate so far for a MATLAB contest).

This histogram shows when contest activity was occurring. Obviously there was a big spike just as the contest was drawing to a close.

Some people entered the contest once or twice, while others entered it literally dozens or even hundreds of times. On average, contestants submitted a dozen entries. Let's look at when two of our most active participants, Lucio and Stijn Helsen, were submitting their entries. They both had very busy weeks.

Here is a bar plot of our most active participants. Lucio, Stijn, Yuval, and Colin all had more than 100 entries.

Score vs. Submit Time

This is the most useful diagram for visualizing the contest: here we see the score of the entries logged against the time at which they were submitted. The result is a scatter plot of all passing entries in the contest. Since a lower score is better, the lowest part of this data set decreases steadily over time as the lead entry gets better. All leaders are shown connected by a red line. Therefore the winning entry by MR Keenan is found in the lower right corner of the plot: the lowest score at the end of the contest.

We are doing a log plot in which the lowest score has been subtracted in order to exaggerate the difference between the final entries.

Here's another way to look at the same information. Since some of the entries are related to one another, we can show this relationship with a line. By coloring all connected family members, we can see "clans" emerge. For instance, shown in red here are all the descendants of Carlos Colon's entry quickfix19.

Corner Points

What points deserve special credit for making significant improvements? Here is the score vs. submission time plot with certain entries highlighted. We call these "corner points" because they represent a significant step change in the score plot. Often these corner points (particularly early in the contest) indicate big changes to the code. As the contest progresses, it becomes harder and harder to make big improvements. The glory associated with making those big improvements rises correspondingly.

Percent Improvement

Here is a plot of the percent improvement generated by each new leader relative to the previous leader. The largest gains are made near the beginning of the contest. This plot lets us see who is responsible for the biggest changes over the contest. The upper frontier of this plot is a sort of hall of fame, and someone whose name appears there more than once managed to make several significant improvements to the score.

Result vs. CPU Time

Two numbers determine the final score for each entry: the folding result in the molecule and the length of time required by the calculation. Here is a scatter plot that shows these two values for each entry with the red leader line superimposed. In general the solutions got more CPU intensive over time as the result improved, but as the knee in the scoring curve came into play, it made sense to trade off results for speed. The gray contours show lines of constant score. Lower left is best.

Unfortunately, this plot doesn't show the order in which the entries arrived. In order to get a feel for the trends in the contest, let's show a day-by-day version of the result vs. CPU time plot. Here we are looking at each of the seven days of the contest. All entries for all days are shown in light blue for context, while the entries for the day in question appear as red dots.

The Solutions

Here we take one puzzle and show how it was solved by each leading entry. Not including our simple straight line solution, there were only nine unique configurations returned by the leading entries. They are shown here.

Entry Length

Here we are looking at a plot over time that shows how many lines of code are in each of the leading entries. In regions where you see entries of more or less the same length there are very few differences from one entry to the next. In other places you can see the code getting shorter or longer.

We are also showing here in the lower plot the score vs. submission time for reference. Leaders are shown in red.

Leader Statistics

Here is some information about each of the leading entries. 215 different entries share the distinction of having led at one point or another. This bar plot shows total time-in-lead by person. Stijn Helsen was in the lead for almost two whole days, and Lucio spent nearly 24 hours in the top slot.

That's the analysis for the Protein Folding contest. Thanks for participating, whether by entering the contest many times, once, or merely checking out the site every now and again. Be sure to sign up on our notify list so you'll be ready to play the next contest: Send an e-mail to lists@mathworks.com with "subscribe contest-announce" in the body.

Appendix: All the Leaders

Here, for the record, is a chronological list of the contestants who held the number one spot and the amount of time they were in the lead. There were 215 leaders, so item 215 on this list is the winning entry. The entry that remained in the lead the longest was Lucio's LT78, which stayed on top for 9.54 hours.

1. MATLAB Contest Team, Sample (0.34 hrs)
2. Stijn Helsen, just a first test (0.32 hrs)
3. Heinrich Acker, just a 2nd test (0.08 hrs)
4. Stijn Helsen, turn around (0.38 hrs)
5. Pat ''n'' Rob, SFMB 3 (0.33 hrs)
6. Stijn Helsen, turn around 4 (0.23 hrs)
7. Claus Still, Simple2 (0.34 hrs)
8. Lucio, another spiral (0.71 hrs)
9. Lucio, spiral&grid (0.54 hrs)
10. Stijn Helsen, spiral&grid without warning (0.08 hrs)
11. Claus Still, Simple3 (0.18 hrs)
12. Leandro Barajas, Ramdom Thing (1.82 hrs)
13. Stijn Helsen, Random thing 3 (1.05 hrs)
14. E. Brian Welch, Eccentric (0.07 hrs)
15. E. Brian Welch, More Eccentrics (0.18 hrs)
16. Yuval Cohen, Variable Zig 2 (0.01 hrs)
17. E. Brian Welch, More Eccentric + Variable Zig (0.08 hrs)
18. Justine Tester, hyperp#2 (0.70 hrs)
19. E. Brian Welch, Eccentic Spirals + Variable ZigZig + Speedups (0.03 hrs)
20. Stijn Helsen, hyperb_zigzag3 (0.08 hrs)
21. E. Brian Welch, hyperb_zigzag3 + speedups (0.28 hrs)
22. Archimedes, hyperb_zigzag3,energic (0.11 hrs)
23. Stijn Helsen, hyperb_ZZ3_faster? (0.07 hrs)
24. Stijn Helsen, hyperb_ZZ_speedups^2 (0.29 hrs)
25. E. Brian Welch, Just a few shells (0.15 hrs)
26. E. Brian Welch, A few more shells (0.19 hrs)
27. save-energy, save-energy (0.47 hrs)
28. E. Brian Welch, One more shell (0.01 hrs)
29. Stijn Helsen, save-energy2 (0.04 hrs)
30. Stijn Helsen, save-energy3 (0.03 hrs)
31. Stijn Helsen, save-energy4 (0.19 hrs)
32. Stijn Helsen, save-energy6 (0.44 hrs)
33. Yuval Cohen, Less Zigs 3 (0.09 hrs)
34. Stijn Helsen, Less Zigs 3_2 (0.16 hrs)
35. Philip Top, Zigging Zig-Zag (0.60 hrs)
36. Yuval Cohen, Even less zigs (0.03 hrs)
37. Philip Top, Zigging Zig-Zag2 (0.13 hrs)
38. Philip Top, Even Less Zigs update (0.65 hrs)
39. Leandro Barajas, Small fix 2 (0.41 hrs)
40. Justine Tester, hyperp#4 (0.97 hrs)
41. Jefferson D, tweak#1 (0.32 hrs)
42. Philip Top, Diamond (0.51 hrs)
43. Leandro, Test 03a (0.16 hrs)
44. Leandro, Test 04 (0.48 hrs)
45. Philip Top, Diamond 2 (0.66 hrs)
46. D Holmes, added interleaved spiral (0.20 hrs)
47. E. Brian Welch, Interleaved Spiral + (1.82 hrs)
48. Stijn Helsen, Interleaved spiral ++ (0.01 hrs)
49. Stijn Helsen, Interleaved spiral ++(real) (0.49 hrs)
50. Cory Sharp, bastij1 oops (0.84 hrs)
51. Stijn Helsen, bastij1_1 (0.26 hrs)
52. Stijn Helsen, bastij1_2 (0.50 hrs)
53. Stijn Helsen, bastij1_3 (0.34 hrs)
54. Stijn Helsen, bastij1_4 (0.14 hrs)
55. Eli Horn, Just fixing a small bug for all of you (0.36 hrs)
56. Per Rutquist, More Dum (0.47 hrs)
57. Stijn Helsen, bastij1_8 (1.12 hrs)
58. Stijn Helsen, bastij1_11_1 (0.27 hrs)
59. Heinrich Acker, bastij1_11_1ct (0.44 hrs)
60. Stijn Helsen, bastij1_13 (0.19 hrs)
61. Heinrich Acker, bastij1_13ct2 (0.14 hrs)
62. Heinrich Acker, bastij1_13ct3 (0.46 hrs)
63. Stijn Helsen, bastij1_14_1 (0.89 hrs)
64. Heinrich Acker, bastij1_14_1ct2 (0.20 hrs)
65. Claus Still, bastij1_14_cs1 (0.18 hrs)
66. Heinrich Acker, bastij1_14_cs1ct (0.05 hrs)
67. Jack Snoeyink, bastij1_14_newsnake (0.46 hrs)
68. Ju, h11_2 (1.71 hrs)
69. Philip Top, diamond zig-zags (1.05 hrs)
70. E. Brian Welch, New Pattern (0.83 hrs)
71. Stijn Helsen, New pattern_tune (0.11 hrs)
72. Stijn Helsen, RLEsnake again (0.73 hrs)
73. Alex Ch., modif1 (0.05 hrs)
74. Philip  Top, Zig-Zag (0.10 hrs)
75. Alex Ch., modif2 (0.39 hrs)
76. Per Rutquist, commentout (0.01 hrs)
77. Stijn Helsen, RLEsnake nrxx (0.44 hrs)
78. Stijn Helsen, RLEsnake again yyy (1.44 hrs)
79. Alex Ch., m17 (0.55 hrs)
80. Stijn Helsen, rlesnake_5 (0.09 hrs)
81. Stijn Helsen, rlesnake_52 (0.21 hrs)
82. Stijn Helsen, rlesnake_53_corr (1.53 hrs)
83. Heinrich Acker, rlesnake_53_corr_tw (0.80 hrs)
84. Philip Top, Random Walk 6 (5.79 hrs)
85. Stijn Helsen, Random walk 6_sh2 (0.21 hrs)
86. Stijn Helsen, Random walk 6_sh3 (0.61 hrs)
87. Carlos R. Colon, quickfix19 (0.44 hrs)
88. Per Rutquist, Ow (0.21 hrs)
89. Stijn Helsen, quickfix19_1 (0.19 hrs)
90. Stijn Helsen, quickfix19_2 (0.38 hrs)
91. Per Rutquist, test_ns (0.02 hrs)
92. Stijn Helsen, quickfix19_3 (0.34 hrs)
93. Per Rutquist, Farewell_spirals (1.19 hrs)
94. Stijn Helsen, quickfix19_4 (0.50 hrs)
95. Nicke, quickfix19_4 (shell-out edition) (0.40 hrs)
96. Nicke, quickfix19_4 (on a diet, fix) (0.20 hrs)
97. Nicke, quickfix19_4 (on a diet++) (0.32 hrs)
98. Colin Ross, SlimChance (0.65 hrs)
99. Stijn Helsen, quickfix_19_5_2cor (0.03 hrs)
100. Stijn Helsen, quickfix19_5_4cor (0.56 hrs)
101. Colin Ross, Jim''ll Fix Me (0.48 hrs)
102. Stijn Helsen, quickfix19_6 (0.52 hrs)
103. Colin Ross, QuirkyEnergy (0.30 hrs)
104. Colin Ross, A Quirk too Far? (0.16 hrs)
105. Colin Ross, Never Too Quirky (0.64 hrs)
106. Leandro Barajas, test: fix i (0.66 hrs)
107. sragner, opttest1 (0.62 hrs)
108. Stijn Helsen, quickfix19_12 (0.24 hrs)
109. Stijn Helsen, quickfix19_13 (0.05 hrs)
110. pb, quickfix19_12_1 (0.10 hrs)
111. Colin Ross, QuirkTastic (1.09 hrs)
112. sragner, opttest5 (1.07 hrs)
113. Ju, h12_1 (1.18 hrs)
114. Stijn Helsen, zigzagoptim03 (0.84 hrs)
115. nqh, Minute Rice 2 (0.12 hrs)
116. Yuval Cohen, rice_2 (0.96 hrs)
117. Yuval Cohen, EB 2 (0.72 hrs)
118. Colin Ross, Back to Quirk (0.84 hrs)
119. Colin Ross, Silence of the Quirks (0.89 hrs)
120. Joshua, joshua TT2 (7.10 hrs)
121. Anders, ana004 (0.31 hrs)
122. Stijn Helsen, ana004-copy (5.79 hrs)
123. Stijn Helsen, zigzagoptim10 (0.81 hrs)
124. pb, zigzagoptim10_1 (0.32 hrs)
125. Colin Ross, Quirkly Does It (2.01 hrs)
126. Colin Ross, A Quirk of Two Halves (0.09 hrs)
127. Colin Ross, OOH!  Quirky! (0.56 hrs)
128. Per Rutquist, Iron Sulphide 3 (1.14 hrs)
129. Archimedes, Disulphide 0 (0.70 hrs)
130. Stijn Helsen, TNT_2 (0.23 hrs)
131. Yuval Cohen, Permutator 12_5 (0.26 hrs)
132. Yuval Cohen, Permutator 13 (0.72 hrs)
133. Philip Top, lhc 5 (3.28 hrs)
134. Yi Cao, Remove_a_Bug (0.19 hrs)
135. Jack Snoeyink, repleat (0.84 hrs)
136. Yi Cao, minenergy (4.95 hrs)
137. Yuval Cohen, Trim here trim there (0.80 hrs)
138. Eli Horn, Stupid Hilbert (0.68 hrs)
139. Stijn Helsen, Trim xxx (0.91 hrs)
140. Yuval Cohen, Trick or trim (2.17 hrs)
141. Stijn Helsen, zigzagoptim13_cor (5.09 hrs)
142. pb, zigzagoptim13_cor_2 (0.35 hrs)
143. pb, zigzagoptim13_cor_3 (0.18 hrs)
144. Stijn Helsen, zigzagoptim13_cor_3--- (4.50 hrs)
145. Yuval Cohen, Tweakzag (0.39 hrs)
146. pb, Yada yada (0.56 hrs)
147. Yuval Cohen, Tweakzag 2 (0.39 hrs)
148. Yuval Cohen, I need some sleep (0.11 hrs)
149. pb, Yada yada 4 (0.36 hrs)
150. Yuval Cohen, I still need some sleep (4.76 hrs)
151. jeff, should be studying (3.58 hrs)
152. Eli Horn, Just looking for score improvement (0.13 hrs)
153. E. Brian Welch, Better?? (0.93 hrs)
154. Per Rutquist, No, but faster! (0.92 hrs)
155. Per Rutquist, Even faster! (0.50 hrs)
156. pb, Yada yada 5 (1.51 hrs)
157. Yi Cao, SpeedUp (0.21 hrs)
158. Heinrich Acker, SpeedUp_0 (1.55 hrs)
159. pb, Yada yada 7 (0.96 hrs)
160. Lucio, LT20 (0.36 hrs)
161. Stijn Helsen, LT20_1 (0.17 hrs)
162. Lucio, LT23 (0.19 hrs)
163. Lucio, LT29 (0.58 hrs)
164. Yuval Cohen, Early termination 3 (0.61 hrs)
165. Lucio, LT30 (0.18 hrs)
166. Lucio, LT31 (0.23 hrs)
167. pb, Yada yada 8 (0.40 hrs)
168. Lucio, LT37 (0.20 hrs)
169. Lucio, LT38 (0.41 hrs)
170. Lucio, LT42 (0.17 hrs)
171. Lucio, LT43 (0.60 hrs)
172. Stijn Helsen, LT43_2 (0.37 hrs)
173. Lucio, LT44 (0.15 hrs)
174. Lucio, LT46 (0.65 hrs)
175. pb, Yada yada 19 (0.37 hrs)
176. pb, Yada yada 20 (0.23 hrs)
177. MR Keenan, stupid 1 (0.55 hrs)
178. pb, Yada yada 21 (0.31 hrs)
179. Lucio, LT53 (1.10 hrs)
180. Lucio, LT56 (0.45 hrs)
181. Lucio, LT60 (0.23 hrs)
182. Nicke, LT56_n#1 (0.00 hrs)
183. Lucio, LT62 (0.13 hrs)
184. Nicke, LT62_meaculpa (0.38 hrs)
185. Lucio, LT68 (0.44 hrs)
186. Lucio, LT70 (0.28 hrs)
187. Lucio, LT71 (0.09 hrs)
188. E. Brian Welch, LT71 - adjacents (0.21 hrs)
189. Heinrich Acker, LT71c (0.26 hrs)
190. Yuval Cohen, Hypermutator II (0.38 hrs)
191. Lucio, LT78 (9.54 hrs)
192. sragner, opttest6 (0.13 hrs)
193. sragner, opttest7 (0.46 hrs)
194. sragner, opttest10 (2.34 hrs)
195. sragner, opttest20 (0.59 hrs)
196. Stijn Helsen, CopyBase1_1 (1.46 hrs)
197. Yi Cao, Remove_Inefficient_Code (0.02 hrs)
198. Yi Cao, Remove_Inefficient_Code_1 (1.89 hrs)
199. Stijn Helsen, copybase1_4_1 (8.69 hrs)
200. CrazyWeasel, CrazyWeasel (3.58 hrs)
201. Mr. Bond, flipper (3.21 hrs)
202. Stijn Helsen, flipper_1cor (0.94 hrs)
203. sragner, opttest31 (1.96 hrs)
204. sragner, opttest32 (0.50 hrs)
205. Roger Stuckey, weaselsearch 1.05 (0.06 hrs)
206. Rob M, Return of the Super Faster Master Blaster (0.32 hrs)
207. Roger Stuckey, weaselsearch 1.07 (1.93 hrs)
208. Jack Snoeyink, weasel1.07+large cases (0.59 hrs)
209. Lucio, LD04 (4.18 hrs)
210. Lucio, DD02 (0.65 hrs)
211. Peter, should be tuned (1.08 hrs)
212. Lucio, FF01 (1.59 hrs)
213. Lucip, FF19 (0.06 hrs)
214. Claus Still, csq4 (0.18 hrs)
215. MR Keenan, Santa Claus (0.16 hrs)


This contest analysis was calculated and published entirely from MATLAB. We used the Database Toolbox to pull the information directly from the contest database, and this HTML document was automatically generated from a MATLAB script.