All of these cross-language benchmarks inevitably end up like this.
They tend to tell you as much or more about the cleverness/knowledge/available free time of the people writing the specific implementation in each language as they tell you about the languages’ inherent differences.
I don’t think it’s a solvable problem though, as there’s no perfect definition of “how anyone would realistically write the code”. Realistically, people are going to write the first naïve implementation they can think of until it becomes a performance problem and someone has time/resources to throw at speeding things up. In the case that a piece of code is a bottleneck and performance matters enough to devote engineering effort, folks will start doing all kinds of crazy “unrealistic” optimizations to real-world code.
I think the benchmarks would tell us much more if they compared implementations of text book algorithms (and data structures) instead of comparing solutions to text book problems.
I realize that this would not necessarily make the code any more realistic or idiomatic. On the contrary, it could make the code extremely contrived for some algorithms in some languages. Some combinations might even be unimplementable.
But I don't care about that, because, as you say, what is and isn't idiomatic is very subjective and it gets thrown out anyway once we decide that some piece of code does need to run fast.
What makes these benchmarks pretty useless is that you don't know whether the choice of algorithm or the choice of programming language dominates a particular result.
Sorry, my mistake. I thought that they did not mandate one particular algorithm, but apparently I was wrong. They say:
"We are trying to show the performance of various programming language implementations - so we ask that contributed programs not only give the correct result, but also use the same algorithm to calculate that result."
Well, "we ask" is not the same as "we receive" :-)
Also, because of the differences between such a wide range of programming languages, there has to be some slop and flexibility in the way the programs are allowed to be written -- otherwise it really would be C written in Haskell.
They tend to tell you as much or more about the cleverness/knowledge/available free time of the people writing the specific implementation in each language as they tell you about the languages’ inherent differences.
I don’t think it’s a solvable problem though, as there’s no perfect definition of “how anyone would realistically write the code”. Realistically, people are going to write the first naïve implementation they can think of until it becomes a performance problem and someone has time/resources to throw at speeding things up. In the case that a piece of code is a bottleneck and performance matters enough to devote engineering effort, folks will start doing all kinds of crazy “unrealistic” optimizations to real-world code.