4

What resources are available that use benchmarks for comparing programming languages? I am interested in both

  • How quickly a program in a given language can execute a given benchmark?

  • How many lines of code are required in a given language to implement a given benchmark?

There is a long-standing web site called the Computer Language Benchmarks Game, originally created by Doug Bagley as the "Great Computer Language Shootout". (You can view a little history at Portland Patterns Repository.)

Is anyone aware of other resources that enable programmers to compare performance and size of programs written in different languages?

igouy
  • 2,547
  • 17
  • 16
frooyo
  • 1,863
  • 3
  • 19
  • 21
  • Performance and LOC are both poor measures. Typically performance is highly dependent on having a skilled programmer who is familiar with the language, while productivity is related to complexity and raising the level of abstraction, which is only slightly correlated to LOC. – Ben Voigt May 23 '11 at 00:13
  • @Ben Voigt - Poor measures of what? Poor measures for what purpose? – igouy May 23 '11 at 16:30
  • @Ben Voigt - Typically we all try to hire skilled programmers who are familiar with the languages being used on our project ;-) – igouy May 23 '11 at 16:31
  • @igouy: Practically speaking, there's probably a pre-existing team at least as often as the freedom to hire programmers for the particular project. I'm not talking about whether you can find a skilled programmer, in the chosen language, however, I'm pointing out that **the benchmark writers are more skilled in one language than another, which invalidates the performance comparison**. And LOC is a poor measure of complexity, ease of debugging, or maintainability. LOC is probably the single most worthless metric in existence (even bug count -- Dilbert writes a new minivan -- isn't as bad). – Ben Voigt May 23 '11 at 20:25
  • @Ben Voigt - "And LOC is a poor measure of complexity, ease of debugging, or maintainability" - So you claim but do not show. Actually, there is some evidence that complexity metrics don't provide any more information than SLoC or LoC http://my.safaribooksonline.com/book/software-engineering-and-development/9780596808310/general-principles-of-searching-for-and-using-evidence/herraiz_hassan_metrics And, actually the benchmarks game website does not show LOC – igouy May 23 '11 at 21:34
  • @igouy: You ever seen a JAPH? That's all the proof I need that LOC is a poor measure of complexity, ease of debugging, and maintainability. – Ben Voigt May 23 '11 at 21:45
  • 1
    This is not a great question, but benchmarks are definitely *on* topic for programming. If closed, I will vote to reopen. – Norman Ramsey May 23 '11 at 22:17
  • @frooyo: Please provide *evidence* that the site is permanently down, or else rephrase your question. – Norman Ramsey May 23 '11 at 22:18
  • @Ben: The benchmarks aren't written by neutral third parties. They've typically gotten rabid partisans to get the best performance they know how. The alioth site has usually shown what a language expert can do. – Norman Ramsey May 23 '11 at 22:35
  • @Ben Voigt - "That's all the proof I need that LOC is a poor measure of complexity, ease of debugging, and maintainability" - Nope, it's just an example that LoC *can be* a poor measure when applied to bizarre programs. The reference I provided looked at 300,000 program files from the ArchLinux repository - check it out. – igouy May 23 '11 at 22:59
  • @Norman Ramsey - alioth.debian.org now seems to be available again after 3 or 4 days absence - and there's a default web page showing at the benchmarks game. Incidentally, you could add this link to the history you edited into the question http://c2.com/cgi/wiki?GreatComputerLanguageShootout – igouy May 23 '11 at 23:06
  • @igouy: Half of each page was missing, but it seems like they limited themselves to C programs. Whereas this question suggests comparing line counts between different languages, which is something else entirely. If a language gets the task done in fewer lines, it might be a higher level of abstraction (python vs C), or it might be higher density (e.g. lisp or perl). – Ben Voigt May 23 '11 at 23:50
  • @igouy, OK, edited. and now my work here is done. – Norman Ramsey May 24 '11 at 01:56
  • @Ben Voigt - "they limited themselves to C programs" - even if they did, according to you LoC is unequivocally a poor measure of complexity, but that is not what their research shows. (Incidentally, some public libraries provide Safari access as part of library membership - and that would let you read all of the book.) – igouy May 24 '11 at 13:54

3 Answers3

1

Alternatives

After a quick google search, I found a couple other sites where benchmarks for various languages have been done. Some other sites mention the programming language shootout site that is currently down.

There is a CPAN module for Perl that uses the same code found on that site.

Google has a directory where pages on this topic can be found. I have not found any yet that are as comprehensive as the page you speak of, but there are certainly other resources out there for comparisons.

Archived / Cached Page

If you're only seeking some information there, you can view archived pages of the site using the Wayback Machine or Google's cached version. Try searching Google with "site: shootout.alioth.debian.org" and click on the "Cached" links for the pages you find.

Find the Author?

Perhaps the best option is to try to contact the owner of the old site and find out what happened. The author mentioned in the BSD licence on this page is "Brent Fulgham". He may or may not be the one to contact.

Wait until Alioth is Fixed

As @ioguy found out, Debian's Alioth server that hosts the site in question is currently under maintenance. I would suggest subscribing to the debian-devel-announce mailing list for updates, and an idea of when it may be fully functional again.

If you find problems in the future, you can probably post to the debian-user list.

TrinitronX
  • 4,959
  • 3
  • 39
  • 66
  • >>Some other sites mention the programming language shootout site that is currently down.<< No,those other sites DO NOT mention that the programming language shootout site is currently down. Those other blog posting were not even written this year. – igouy May 23 '11 at 16:05
  • 1
    "mention the site that is down" != "mention that the site is down". English fail. – Ben Voigt May 23 '11 at 20:26
  • The Wayback Machine links you've posted HTTP Status 404 – igouy May 25 '11 at 14:41
  • @ioguy : Thanks for finding this... Fixed the links! (Looks like the `*` chars were getting converted to their HEX equivalent `%2A`. Fixed it by changing them to html entities `*` ) – TrinitronX May 27 '11 at 15:48
0

The benchmarks game website is now back to normal!

From Friday 20 May 2011 through Monday 23 May 2011, ALL alioth.debian.org subdomains were down - because the alioth admins were upgrading "in every way we can find: kernel, Debian release, FusionForge software, hardware, and so on."

In addition, making the benchmarks game website work again required:

  • installation of the GD library on the new server, for chart generation

  • basic information about changes to ssh use on the new servers

  • basic information about the project cvs repository on the new servers

  • basic information about the project /htdocs location on the new servers

  • replacement of the long deprecated $HTTP_GET_VARS by $_GET in a couple of dozen PHP scripts

Since the performance benchmark site for Programming Languages (aka Programming Language "Shootout" & shootout.alioth.debian.org) is permanently down ...

The original question was predicated on a false premise.

igouy
  • 2,547
  • 17
  • 16
  • the site is still down. You're answer is not applicable then. – frooyo May 24 '11 at 14:36
  • @frooyo - No, the site was not still down - if you were unable to connect to the benchmarks game website then it was a problem with your ISP or your machine (try refreshing your browser cache). – igouy Jun 01 '11 at 20:13
0

  1. Each year there are two or three isolated blog posts that claim to compare performance and size of one or two programs written in different languages.

    As a resource the blog posts fail for obvious reasons, most obviously:

    • not updated with newer versions of the language implementation

    • not updated with better programs

  2. Every couple of years someone dissatisfied with something about the benchmarks game (often some detail about the code repository or website technology) starts a project that will fix everything they dislike about the benchmarks game.

    As a resource the most obvious problem with those projects is that they never seem to get close to publishing performance data.

  3. Every year some group of programmers campaigns to have language X included in the benchmarks game, while some other group demands that some program is included (or excluded).

    Sadly, they rarely accept that among the resources provided by the benchmarks game are

    • scripts they can use to make and publish language performance measurements

    • examples of which basic information (language version, build commands, run commands, measurement techniques, ...) is required to provide context for the measurements.

    They rarely accept that they are empowered to create what they wish to see.

igouy
  • 2,547
  • 17
  • 16