For Cody challenges, should the best solutions be based on the least number of characters used in one's code?
Mingji Chen
on 30 Aug 2020
It's pretty odd how a solution that uses more characters than usual can be the "leading solution" of a Cody problem and have the least size. Compare these two codes that find the sum of integers from 1 to 2^x, which one uses fewer characters, thus should be the better solution?
function y = sum_int(x)
regexp '' '(?@y=sum(1:2^x);)'
end
function ans = sum_int(x)
sum(1:2^x)
end
11 Comments
Time Descending
Thank you, @John D'Errico, for your insightful answer and comments. Having used Cody extensively, I agree with your sentiments.
Cody has some unusual aspects. I hate to call the idea flawed, but there are some aspects I would like to change, with absolutely no idea how to make it better. It is sadly easy to game the Cody scoring algorithm, at least once you learn some tricks.
The problem is you want Cody to reward the "best" code possible as the best solution. But the truly best code for any problem is often dependent on some feature of the problem. It may even depend on the user, or the computer system it will be solved on.
One user might not care about the time, but is severely memory constrained. For this person, they cannot afford a solution that uses large memory as a tradeoff for time.
Another person may have plenty of memory, but must solve the problem millions of times. Therefore time is the factor we need.
Finally, solution complexity is always an important feature. By teaching students to write unreadable code, that just happens to optimize the Cody scoring algorithm, we create programmers who write terrible code, thinking it is good, without a clue as to the bad habits they have learned.
Having said all of that, Cody is itself a good idea, in that it does teach people to learn programming skills. It teaches them creativity in solving problems, a HUGELY important thing to learn for programming. It can teach you to look for different ways to solve a problem, recognizing that not every algorithm is always optimal for every problem.
My personal goal when solving a Cody problem is to not give a tinker's damn about the score. I know when I've done it well. For example, suppose I want to solve the problem posed? Thus compute the sum of the integers from 1 to 2^x, where x is an input parameter?
I'm sorry, but codes like this:
function ans = sum_int(x)
sum(1:2^x)
end
are just bad programming style, for several reasons. The hack where ans is used to give a slight bump to Cody score is a pure hack. But it makes for poor practice when someone starts to write real code, perhaps for a job, where readability of the code is an important factor. You need to learn to write code that can be easily debugged, used by others, including your successors.
Next, if x will be even remotely large, so 20 or 25, now you are summing possibly millions of numbers. At the same time, it is trivial to write the sum of the integers from 1 to N, as
N*(N+1)/2
This costs almost no computation time, very few flops, regardless of the size of N.
Therefore, a function to compute the sum of the integers from 1 to 2^x might be written as
function S = sum_int(x)
x2 = 2^(x-1);
S = x2*(2*x2+1);
This is always as efficient as possible. It could even be easily vectorized. Better code would be friendly. It would check to see if x is a non-negative integer. It would check to see if x was too large, causing overflows. And those error checks would be huge downgrades to the Cody score. But they would be upgrades to my personal score.
Would the above code score well? Surely not very highly. It is readable. It raises 2 to a power only once, not twice. It is efficient, not requiring millions of flops to compute a result for some values of x.
I want to make it perfectly clear - The Cody scoring algorithm has flaws. Can they be corrected? You could do many things, but any scheme you choose will still probably be possible to game the scoring. Does that mean Cody should not be used? Of course not! Cody is a great tool, if used properly.
I might as well say that money is a bad thing, terribly flawed, because it encourages people to rob banks to get more of it, to steal, to embezzle, etc. Money is fine, as long as we don't abuse it, as long as we understand that it should not be the purpose of our being.
And yes, you could say that I am being hypocritical. For someone with a lot of site rep to say that site rep is unimportant might seem that way. But when I die, I doubt that anyone will mention my site rep anywhere in my obituary, at least I hope not. But some people might remember the many tools I have provided, the many people I have helped at all levels. They might remember the lessons in numerical analysis or mathematics I was able to teach here.
SO PLEASE USE CODY! Just use it to learn good skills, and not bad habits. Don't worry about site rep. Site rep happens as a gradual consequence of your doing good work, not from gaming the system. And you can learn some nice programming skills, some nice mathematics from solving Cody problems. Just don't worry about the scoring algorithm. Once you start to do that, you focus on the wrong things.
It also requires the writer of the challenge to know the best solution in advance, which is probably bad design. By now I'm confident in claiming I'm fairly proficient, but 'even' I still see functions sometimes in solutions by others that I never knew anything about. It then turns out that that those functions have been part of Matlab for a decade (or two).
If we consider some very simple task, like calculating the average.
Out of perfect score of 100%
- Correct result 50%
- Correctly used functions 50% (Using them, in most cases, code will be understandable, short and fast)
These correctly used functions would be in this case sum, since it's simple, and it's the only function that user needs to use, so it would be 50% worth. In more complex tasks, if user should use two or three or more functions, then each would be 25%/16%/x%
Add negative scores, general examples:
- -25% when user uses for to loop over elements in array, adding them and dividing by number of elements instead of using sum
- -5% when size of a variable changes in a loop (when it's not initialised beforehand)
- -1000% when eval is used
- Other examples relevant to the specific problem
This might be hard to implement this for all problems, especially for user written ones, but it would be good for introductory ones.
Yes, this was one of the things that deterred me from proceeding with Cody. I would get frustrated at knowing that there were better solutions (ie, higher scores) but not having access to them. After knowing that people can and do cheat the system, it became even more frustrating since now ya gotta wonder if the higher scores are really better solutions or just cheats. But if you put the competition aside, it's really fun to be exposed to so many different problems that you wouldn't normally get involved with. The Answers forum also provides that outlet.
I would really love to see the option to have a look at a solution that is better than mine. I evidently can write a solution that is as good as or worse than mine. So if I were to be shown the two solutions better than mine (excluding the best score solution), that would already improve it as a teaching tool and might get me back to Cody.
I agree. Cody is best intended to be a fun way to learn. And it does serve that purpose. I would love for there to be a simple metric that would encourage maximally good code. The problem is, good code is very difficult to quantify. The supreme court definition of good code would seem to apply: I know it when I see it. And since the best code will vary according to the problem parameters, this becomes even more difficult to measure.
The problem in my eyes is there are some coding practices that Cody does teach that are antithetical to good code. And these things happen when people learn to game the scoring metric to try for the best score, instead of the best code.
It is not a bad idea to offer a few solutions as examples of well written solutions. Actually, I like the idea. Of course, make them visible to the user only after a correct solution is submitted. I'll float the idea at a future CAB meeting.
Another idea might be to allow the user to indicate how each solution will be scored. That is, would time be the measure? Solution complexity, thus the McCabe cyclomatic complexity? Number of characters? Memory usage? Some overall aggregation of all those measures?
https://en.wikipedia.org/wiki/Cyclomatic_complexity
I want to briefly add on to this.
Cody is a fun place to learn coding and have a small competition with others to write the shortest code to solve a problem, which does not necessarily means it's the best solution.
Firstly, it's a learning place, and it would be great to see other approaches in solving the problem without focusing on the number of characters your solution has. There's a possibility of doing this after solving the problem, but I believe 99% of the players are not checking other solutions. So after successfully solving the problem, it would be useful to show one or two hand-picked solutions for selected examples on the landing page or next to the 'Solution stats' tab.
:)
I wish I knew a way to provide a perfect scoring system for Cody. Of course, if I did, I would have pushed to implement it long ago. Even good code is not always the right code, because it must depend on who is using it and under which conditions it will be used. And that means there cannot be a perfect scoring scheme for Cody. Were I to try to invent a scoring scheme, I would want it to have multiple aspects. I think my scheme would be some sort of weighted measure, based on:
- Measure the time required to run a problem. Good code would have fast run times of course.
- Short code would be good, thus a complexity metric seems to make soem sense.
- Scalability. Does the code work reasonably well on problems of various "sizes"?
With some time, I could surely add a few more. Even those measures are surely not trivial to implement, but I could probably find some way to do so. And even then, anything I did could still be gamed.
And while I say a complexity metric makes sense, good code is also debuggable code, readable code. I've said before that good code must be readable by the person who inherits my code, just in case I get run over one day by the crosstown bus. Someone else will one day probably inherit my code. And if I make it impossible to debug and maintain, then my successors will be forced to re-write my code from scratch when I am gone.
And of course there are other measures of good code that are almost impossible to even try to measure automatically.
- Good documentation is in my eyes almost as important as the code itself. The best code, made without documentation to explain its use? That is useless code.
- Good code should be friendly code. It should provide a soft landing so that the user can figure out how to resolve a problem. I recall the old style IBM fortran, where a coded error was returned and no more. To figure out the problem, you then thumbed through a manual with hundreds of pages of error codes, looking up the error code, to then find a one sentence explanation of the error message, which was itself hardly decipherable. Yes, we learned the common errors by heart.
Now, suppose I were desirous of changing the Cody scoring algoruithms? (Since I'm on the CAB, I could at least have influence.)
For example, I would probably want to add a penalty for every time ans gets used, since this is a common hack. That is, even for my example code, this version would probably score better:
function ans = sum_int(x)
2^(x-1);
ans*(2*ans+1);
And honestly, I consider that variation to be terrible code, since each line relies on the automatic creation of ans from the previous line, then reuses ans in a compeltely different context.
I'd then want to add Cody score modifications to prevent the misuse of eval and regexp, as can be done. (But sometimes, regexp would be the correct tool for some problems. Perhaps even eval.)
Anyway, for every hack I could disincentivise, there would be someone who could find another ingenious hack, just to get their name on the top of the scoring list. This would become a game of spy versus spy that would never end.
So as much as I would love to see the perfect metric for Cody, it does not exist in my eyes. That does not mean Cody, despite this flaw, is not a good tool that can be used to teach skills in MATLAB and mathematics. We all have flaws at some level, but we learn to live with them and choose to accept those flaws.
Good points, have some site rep from me ;)
I hadn't thought of memory constraints, input validation, and vectorization as other possible criteria.
As far as the current system: I have no clue what exactly it is that Cody is rewarding and I refuse to learn how to game the system.
As far as suggestions for improvements: in some competitions the winning code is secret, but the rest is published for everybody to learn from.
My personal opinion: no. I would favor a different metric: time.
If you go for character count, that will only drive the use of short variable names, which doesn't teach good coding practices. A mad hunt for the best perfomance is probably also not optimal, but at least you can see a benefit outside Cody. It should teach you ways to speed up your code, instead of teaching you all the stupid ways you can circument the blocking of eval with regexprep.
Sign in to participate
