16

Universities and companies conduct, administer, or provide programmer aptitude tests. A number of people have expressed both skepticism as well as agreement that they may be useful with the hiring process as claimed by one company,

From a practical point of view, the PAAT definitely identifies those people with a high aptitude for programming and for systems analysis. It also identifies those individuals who are high-risk hires (low scores). This is the true value of the test. It minimizes the costs associated with a poor employment decision.

Is there any evidence that this is a reliable mechanism for selection?

Glorfindel
  • 1,452
  • 1
  • 17
  • 28
Raju
  • 279
  • 2
  • 4
  • 7
    [Welcome to Skeptics!](http://meta.skeptics.stackexchange.com/questions/1505/welcome-to-new-users) We want to focus our attention on doubtful claims that are widely held or are made by notable people. Please [provide some references](http://meta.skeptics.stackexchange.com/questions/882/what-are-the-attributes-of-a-good-question/883#883) to places where this claim is being made - including a specific test. – Oddthinking Jun 11 '13 at 11:27
  • @LarryOBrien I don't think you can retract close votes. –  Jun 11 '13 at 20:08
  • 1
    I could say, as a non-reliable source programmer, that I'm highly sceptical aptitude tests for *hiring* people who has supposedly an education. But separating out those will will do well in a programming course is certainly easier. If you have made it through a university It education, you have at least basic programming aptitude. – Lennart Regebro Jun 12 '13 at 04:15
  • IBM does have a PAT http://www-05.ibm.com/employment/uk/graduates/apply_selection.html it is now called IPAT for initial selection. It consists of two parts. –  Jun 12 '13 at 12:51
  • That it's "not related" and "more generic" doesn't mean its not a good indicator of aptitude for programming. It might in fact be *better* at it. – Lennart Regebro Jun 12 '13 at 20:27
  • @rob: Right, but you seem to argue that just because IBM has stopped their previous aptitude test and replaced it with another, more generic aptitude test, this somehow means IBM now things programming aptitude tests doesn't work. This is how it sounds like when you say that there is "no relation" between the tests, even though they clearly test quite similar things, like logic etc. – Lennart Regebro Jun 12 '13 at 21:53
  • 1
    @rob: Why is it even relevant if they are related or not? That makes no difference. They are both aptitude tests, one for programming, another apparently more generic for logic and math tasks. Hence in some way they are clearly related. IBM not endorsing PAT seems to then, in context, mean that they don't endorse *their* PAT. The one used before. As they have replaced it with the IPAT. – Lennart Regebro Jun 12 '13 at 22:03
  • I summarized all of the information I could find about the IPAT in the footnote of my answer and I couldn't find anything showing that the IPAT is related to the PAT in any way (i.e. as a derivative of the older PAT). In fact I couldn't find much information on it at all and it seems like it might be more of a UK thing than an IBM global as they don't appear to use it as part of the [recruiting process here](http://www-03.ibm.com/employment/us/un_recruit_process.shtml). – rjzii Jun 12 '13 at 22:13
  • @rob: The IPAT test is not testing programming specifically, but logic and reasoning, which are closely related. If it's derived/related or not in that sense is still completely irrelevant. It's clearly related in what they test. – Lennart Regebro Jun 13 '13 at 05:33
  • As a programmer that has also hired a couple of people in my life, I find this method incredibly unreliable for accessing one's skills in terms of ability to produce and deliver code. Most of the coding at this point involve different frameworks and libraries, that not only change on monthly/weekly/daily basis but also require developers and programmers alike to spend a significant part of their time simply looking up documentation references etc. I have not yet seen a test, other than 1 month trial period, that could give valuable information regarding performance in this sense. – Cninroh Jan 22 '18 at 13:37

1 Answers1

13

This is actually a very interesting question since we need to look at this from at least two standpoints: the ability for an aptitude test accurately predict if the test taker is suited to leaning the skill set and the ability for such tests to accurately predict employee performance.

With regards to the use of such tests to predict academic success (i.e. leaning the skill set), it appears that IBM lead the way in this department with their development of the IBM Programmer Aptitude Test (PAT)1, 2, 3 in the 1960's. However, in 1980 a study was published that followed 2,000 students, 1,350 of whom finished and concluded that,

The results indicate that future programming skill is not predictable by the most commonly used written test (IBM's Programming Aptitude Test). The suggestion is that this type of test should not be administered to college level people as the results are unreliable.

In 1984 another study was published that examined the more traditional means of student section to programs (e.g. SAT scores, class rank, high school courses, etc.) which found that there was a positive correlation between these variables and academic success. However, it is important to note that in this situation a aptitude test specific to programming was not administered though. In fact, a 2010 survey paper concluded that, with regards to aptitude tests predicting academic success,

Aptitude tests are not reliable predictors of success, and have not identified any cognitive capacity which is especially significant for learning to program. Evidence relating to predictors such as educational or demographic background, cognitive development or style, or a range of other factors, is also ambiguous or weak. On balance some factors (such as mathematical background/ability, or self-ratings of confidence/comfort as the course progresses) emerge as moderately correlated with success.

The apparent working draft Dehnadi and Bornat paper that was referred to by the articles (1, 2) in the question had it's hypothesis rejected by another paper and in fact the Dehnadi and Bornat later presented (direct link) revised findings where the concluded that their orginal working paper was incorrect, concluding,

We cannot claim to be separating the programming goats from the non-programming sheep: experiment 3 demolishes the notion that consistent subjects will for the most part learn well, and others for the most part won't. And even in the most encouraging of our results, we find a 50% success rate in those who don't score C0 or CM2.
None the less, some of our results indicate that there may be something going on with consistency. There is a case for continuing our investigations.

So with regards to aptitude tests indicating academic success, there doesn't appear to be strong evidence that programming aptitude tests are useful.

This brings us to the question of the use of aptitude tests as a prediction of employee success. For the purposes of this answer, I'm going to assume that you are not referring to tests such as the Oracle Certification Program which tests candidates on a specific topic, but rather more generic ones such as the Walden Programmer Analyst Aptitude Test which attempts to

... simulate what the job applicant is required to do on the job, as well as to determine who is likely to be successful in technical training courses.

In this situation I couldn't find much in the way of recent research that definitively showed a correlation between aptitude tests and employee performance. Much of the literature specific to programmer aptitude tests is either focused on predicting academic success which has already been discussed.

  1. Some early examples of which can be found with great difficultly.
  2. Incidentally at least one journal article claims that that IBM no longer endorses the use of PAT and I was unable to find any mention of it being used any later than about the 1980's. A new version of the PAT could not be found when searching the IBM website.
  3. It appears that IBM is using a new test in the United Kingdom for recruitment that goes by the name IPAT or IPATO which is short for the Information Processing Aptitude Test On-line. There doesn't appear to be any relation between the IPAT and the original PAT nor does it appear to test a candidates ability to acquire programming skills.
Glorfindel
  • 1,452
  • 1
  • 17
  • 28
rjzii
  • 16,884
  • 4
  • 92
  • 102
  • 1
    I don't see how your '2' claim for IBM comes out of the link you use to support it. – Lennart Regebro Jun 12 '13 at 04:12
  • 2
    the "tests" don't measure learning ability as they're not intended to be applied to people who still need to learn, they're intended to be applied to people who already have the skills, and supposedly show whether those skills indeed exist. – jwenting Jun 12 '13 at 13:42
  • 1
    @jwenting - The question is referring to aptitude tests which generally mean tests that are used to access if someone can learn a given set of skills. Proficiency tests are what you would use when testing to see if someone already has the given skills. – rjzii Jun 12 '13 at 13:56
  • @rob as those tests are only applied to those who have already passed the resume screening to determine whether they have the skills needed to do the job (including learning, if they couldn't learn they wouldn't have the skills, if they didn't have the skills they'd never be hired) they don't test for learning ability. If you'd ever done any of them, you'd realise that. SUPPOSEDLY they test "logical thinking ability" and "whether you fit in the team", but they don't do even that. They test whether you can do tests like that and nothing else. – jwenting Jun 13 '13 at 05:23
  • @jwenting: "as those tests are only applied to those who have already passed the resume screening" The references show that this isn't true. "If you'd ever done any of them" - special pleading. The rest begs the question and needs references. – Oddthinking Feb 04 '19 at 13:14
  • Your first link is stale, the [Tukiainen paper is here](https://ppig.org/papers/2002-ppig-14th-tukiainen/). Also, *"a test that would predict [future] [student] success in learning programming, at a local polytechnic school"* is infinitely worse than an actual test that tests actual programming competence in programmers, right now, let alone in a specific language e.g. C, Python, Java, Scala etc.; the latter aren't very good, but the former is garbage. – smci Jan 16 '21 at 21:14