257

John Carmack is a renowned developer and CTO. His Twitter account has over 100,000 followers.

On the occasion of Stack Exchange's 5th anniversary, he quipped on Twitter:

[... Stack Overflow] has probably added billions of dollars of value to the world in increased programmer productivity.

The tweet was widely read and retweeted over 100 times.

Is this claim supported by evidence?

What have I tried?

I've made a quick calculation to see if he was completely off the mark, but the numbers seem to add up to a layman like me.

  • Our visit counter recently overflowed Int32: 2,147,483,647.
    The above is a reference to my own tweet but I can also provide supporting evidence: the site is 5 years old and currently does around 6m visits/day
  • Average pay for a programmer in the UK is 45,000£/annum (source) or 22.5£/h. Let's assume a low pay of 20US$/h. I assume an hour is saved every time an answer is provided.
  • Each visit will potentially save some money, but only 77% of visits land on answered questions
  • There is a cost relative to answering a question, however answers are useful to hundreds of people.
  • There are many other factors I am not calculating: for example, does answering a question make you more knowledgeable? Does finding an answer make you less knowledgeable than you would be otherwise? I am ignoring these as this is a ballpark estimate, certainly not a valid answer to my own question.

To sum it up:

  • value provided = 2 billion page views * 77% answer ratio * 20US$ ≈ 30 billion US$

What kind of answer do I expect?

I expect answers to provide evidence to the value (or lack thereof, or cost!) of the increased (or decreased!) programmer productivity as to ascertain whether it is true that they are "probably billions". I do not really want to see calculations like the one I provided.

Disclosure

I work for Stack Exchange.

A Rather Unprecedented Moderator Note from @Oddthinking

This question has caused a startling amount of interest and discussion. A big welcome to the users from Hacker News and Reddit.

Everyone seems to have an opinion. Unlike those sites though, opinions generally aren't welcome here - as answers or in the comments. We are looking for definitive answers based on empirical evidence. Comments should be directed at improving the question, not at discussion.

This is a tough question. It isn't obvious how economists might measure value to an economy. It isn't obvious how computer scientists might measure productivity. It isn't obvious how much time it takes to answer or how much time is saved for each page view. There may be unintentional side-effects of making answers easy to find. etc. Arguments from incredulity are not a valid response to such difficulty.

Sklivvz has given openly naive answers to those questions, as a back-of-the-envelope calculation, and is asking if anyone has done the calculation professionally. If you can find such calculations in the literature, please answer the question. If all you want to do is point out the naive calculations are naive, let's take that as stipulated; it doesn't help answer the question.

In the meantime, the question has been "protected", so only users with a modicum of rep on the site may answer, and the comments have been cleaned up several times. If you want to comment on this mod note, please take it to meta.

Community
  • 1
Sklivvz
  • 78,578
  • 29
  • 321
  • 428
  • 4
    As a side note: there are [many scientific studies regarding specificaly stack overflow](http://scholar.google.com/scholar?hl=en&q=stackoverflow&btnG=&as_sdt=1%2C5), so it's not impossible that this value has been studied. – Sklivvz Nov 30 '13 at 18:17
  • 1
    Related: http://meta.stackexchange.com/questions/134495/academic-papers-using-stack-overflow-data – Sklivvz Dec 03 '13 at 19:02
  • Related (almost an answer with full-on studies): https://news.ycombinator.com/item?id=6854071 – Sklivvz Dec 05 '13 at 12:38
  • 2
    What is the total number of upvotes of answers on SO? – Paul Dec 08 '13 at 20:39
  • 1
    @Paul around 24 millions currently, limited to non-deleted answers. – Sklivvz Dec 08 '13 at 21:39
  • @Sklivvz I think this question was deleted when Oddthinking cleaned things up, but where would you stand on an answer that draws parallels between studies of other public repositories of information? It wouldn't be specifically on Stack Overflow, but some fairly broad strokes might be able to be made for a soft answer (i.e. a case can be made for equivalency). – rjzii Dec 11 '13 at 20:05
  • 2
    @rob it all depends on how many assumptions we have to make. The more assumptions, the more uncertainty, and the more potential for pointless debate. As a sanity check, if you need to ask, you probably need to make a stronger case before posting. :-) – Sklivvz Dec 11 '13 at 20:39
  • @Sklivvz Well, I think a strong case can be made but you never know how people will react some times and I've been having a devil of a time not getting as many up votes as down votes for some reason lately. I'm thinking that if I do sit down and write something up might might put it over on Meta before making it an official answer. – rjzii Dec 11 '13 at 22:15
  • To truthfully figure out the value added by stackoverflow you need to look at the top competitors and how time consuming finding out answers was before stackoverflow. Assuming that finding out an answer on SO saves an hour of work is at the core of the calculation for the answer but it might be a gross overestimation or an underestimation depending on how long it took to find the answer in the past. (We must also keep in mind that more people are on the internet now than in the past so more answers would be provided on other sites as well) This makes answering this question well a challenge – Xitcod13 Apr 16 '14 at 00:27
  • 9
    This assumption dont account for the time lost using stackoverflow, I've spent days hunting for questions that I can answer (as well as writing the actual answers) and sometimes this has impacted the time I spent doing my actual job – isJustMe May 07 '14 at 23:02
  • 2
    I guess you have to define what 'productivity' means first... and I am a bit skeptical as to whether anything that humans do are truly productive. – Michael Lai Jun 06 '14 at 04:37
  • 9
    This question claims exactly the opposite : http://workplace.stackexchange.com/questions/27555/team-members-spending-too-much-time-on-stack-overflow – isJustMe Sep 29 '14 at 21:50
  • 3
    Just to throw a wrench into things, how about the amount of developers that have learned how to develop due to SE? These developers, a lot without degrees, likely costs significantly less in salary. – Paul Muir Dec 11 '14 at 14:21
  • We are here considering that all of the workers who find an answer here will settle there and carry on work. That does not need to be true. They can be easily distracted to read another question regarding a topic they find interesting, or even try their luck at Code golfing. There are, as stated, just too many variables to confirm or bust this idea. It is likely it has saved a fair sum, but billions is probably stretching it. – Sharain Dec 11 '14 at 14:37
  • 2
    I think you have it all backwards. If SO increased productivity, that means that more programers were left out of a job and that decreased the economy. – Jonathon Dec 12 '14 at 02:01
  • @Jonathon Wisnoski That's not how it works. The amount of work that COULD be done is much larger than what IS done. If productivity is increased, more work is done, not less programmers hired. – ax123man Mar 19 '15 at 11:37
  • @MichaelLai If nothing humans do is productive, how did we go from barely being able to feed and clothe ourselves to what you see around you today? The meaning of productivity is to produce, no? – ax123man Mar 19 '15 at 11:41
  • @ax123man I think sometimes 'progress' might be easily seen as 'productivity'. If you take a look at what humans have taken away from the planet compared to what they have produced it is hard to find an argument that we have added more to this planet than we have taken away, at least in my opinion... – Michael Lai Mar 19 '15 at 12:14
  • [Productivity](http://en.wikipedia.org/wiki/Productivity) is a well-defined economic term, so it does not depend on how much "humans have taken away from the planet", that is actually factored out. – Sklivvz Mar 19 '15 at 13:38
  • @MichaelLai What is it we have "taken"? To take something permanently, an alien would need to fly in here, load up, and leave. If humans die off, in short order (relatively) the planet is going to look a lot like it did a million years ago. The argument in my favor is simple: compare our lives starting with the first bipedal hominids, and for the first several million years to our lives today. Do you want to live a life where you have little clothing or shelter and do nothing but work and sleep to survive for maybe 20 years, if your lucky? I bet most people prefer what we have today. – ax123man Mar 20 '15 at 12:50
  • How many money would be wasted by programmers answering questions instead of doing their job? – Mast Mar 27 '15 at 10:29
  • @Mast no money would be lost: they are doing someone's job anyways, and their contribution will also help more people in the future. – Sklivvz Mar 27 '15 at 11:01
  • @Sklivvz I somehow suspect the direct employer of said programmers wouldn't agree with you. The 'work field' wouldn't lose money, but the company could. – Mast Mar 27 '15 at 11:03
  • @Mast Yes, the question is about money saved by the field. *Someone* will lose money because a programmer needs help, but Stack Overflow *reduces* this loss by connecting expertise with need. – Sklivvz Mar 27 '15 at 11:06
  • Very confused why this question has not been closed as primarily opinion based, due to having no REAL translation from page views, likes, etc. to labor costs saved. – Waterseas May 12 '15 at 18:52
  • You haven't subtracted all the lost productivity from people answering interesting questions when instead they should be working! – ErikE Jan 11 '16 at 21:05
  • Well, I can say it has saved my employers upwards of, say, 50 days of work for me so far; and the people who answered my questions spent much much less time on doing so than I saved. So if you multiply that difference by... a whole lot of people, you get a whole lot of money. However, the "77% of visits get 20 USD benefit" sounds rather bogus, unless you only count at most one visit per unique person per day, in which case fine. The percentage from my experience is more like 7% than 77%. And since the general truth can be determined by my anecdotal individual experience :-) the answer is "no". – einpoklum Jun 18 '17 at 10:33
  • The question lucks a time frame setting. Save billions (1+) ... the last 2 years or the last 2 months or over its lifetime? – Stefanos Zilellis Jul 12 '19 at 10:00
  • @stefanos I meant up to the date of the question of course, but that only matters if the answer is yes :) – Sklivvz Jul 13 '19 at 00:16

1 Answers1

36

One of the few published papers on the relationship between Stack Overflow and productivity finds that active GitHub users "ask fewer questions and provide more answers than others" on the site.1 The authors write that users who ask questions "distribute their work in less uniform way," than those who do not (indicating interruption). But, they also note that "despite interruptions incurred" Stack Overflow activities correlate positively with participation and commit rates on GitHub.

In other words, there's evidence that Stack Overflow tends to cause interruptions (as a stand in for loss of productivity),2 but people who code a lot (on GitHub) also tend to contribute to SO, and the other way around. The relationship between GitHub and SO participation is in this way used to approximate impact on productivity.3

Neither this study nor the studies it cites provide direct evidence for or against Carmack's claims. Fundamentally, the authors conclude that:

Despite the popularity of Stack Overflow, its role in the work cycle of open-source developers is yet to be understood: on the one hand, participation in it has the potential to increase the knowledge of individual developers thus improving and speeding up the development process. On the other hand, participation in Stack Overflow may interrupt the regular working rhythm of the developer, hence also possibly slow down the development process.4

Because the original claim argues for value through productivity, under conditions where the impact of SO on productivity "is yet to be understood" in systematic study, it (the original claim) remains speculative and not supported by evidence beyond anecdote.

  1. Vasilescu, B., V. Filkov, and A. Serebrenik. “StackOverflow and GitHub: Associations between Software Development and Crowdsourced Knowledge.” In 2013 International Conference on Social Computing (SocialCom), 188–95, 2013.
  2. "Goal: Is participation in Stack Overflow related to productivity of GitHub developers?" (p.188)
  3. "Here, following Adams et.al., we look at only one, but representative, facet of productivity."
  4. Emphasis mine.
denten
  • 10,781
  • 4
  • 55
  • 77
  • 2
    Thanks. The thing is, it is not useful to compare productivity and stack overflow behavior, because even if someone is less productive because they are on stack overflow all day, their contributions make many other devs productive. – Sklivvz May 12 '15 at 01:27
  • That could be. I am just sticking to the available empirical evidence as per note from @Oddthinking. – denten May 12 '15 at 02:09
  • 1
    I don't see how this answers the question. I don't even see how this is even related to the question. – George Chalhoub May 12 '15 at 08:28
  • @georgechalhoub the claim is about SO increasing value through productivity. the study is about SO and productivity. – denten May 12 '15 at 14:35
  • @georgechalhoub I added language to make the connection the authors are making more obvious. thanks! – denten May 12 '15 at 14:41
  • 1
    @denten the paper you cite is about Stack Overflow data in 2012, ancient... even in 2013, that would have covered much less than 75% of the data available (Stack Overflow has been growing exponentially since 2009) – Sklivvz May 14 '15 at 08:20
  • 1
    There's no possibility to correlate money with SO. There are so many factors like; are they professionals or just hobbyists? What's the actual money that goes about that project the code is for? What's the person's pay grade? When does a small series of helping answers result in a developer inventing something truly big as a whole, resulting in major profits? The most important thing about SO is tho that there are a lot of reputation hunters. Questions get so many speedy crap-answers and the real interesting "hard" questions often just die a slow death. Bounties don't always work as intended. – User2910293 May 14 '15 at 09:11
  • 1
    @Sklivvz The results would unlikely be different if n was increased by 25%. Their sample size was close to 1.5mil users, with an overlap of around 50k between the two sites. More importantly, the answer is addressed in a scientific peer-reviewed paper, appearing in IEEE, SocialCom--one of most significant publications in the study of online communities, and written by prominent mathematicians and computer scientists in the field. The authors present literature review to contextualize their results, and come to a specific conclusion related to the original claim. – denten May 14 '15 at 12:34
  • 2
    Unless someone can show different results that rise to the same standards of evidence, I consider this question resolved. As per site rules I respectfully suggest all further personal opinion and original calculations be expressed in chat as per mods note. – denten May 14 '15 at 12:39
  • 1
    I honestly can't consider this answer appropriate: it doesn't address value and does not use the word "productivity" in the [sense](http://en.wikipedia.org/wiki/Productivity) which would address the question. – Sklivvz May 14 '15 at 13:01
  • 1
    I updated the answer to reflect the authors' understanding of productivity, which corresponds to the common definition linked in the Wikipedia article. Thank you for the suggestion. – denten May 14 '15 at 17:23
  • 3
    The original claim understands value explicitly as "increased programmer productivity." It is therefore enough to show results on productivity. I updated the answer to make that connection more explicit. – denten May 14 '15 at 17:38
  • I don't understand, "productivity" is "value added over time". Defining "productivity" in terms of value seems circular. – Sklivvz May 16 '15 at 09:22
  • That could be! However, the link posted above defines productivity as "an average measure of the efficiency of production" where produciton is "a process of combining various material inputs and immaterial inputs in order to make something for consumption." More importantly, the original claim is about "value to the world in increased programmer productivity." The claim itself understands value in terms of productivity. Hope that helps. – denten May 16 '15 at 15:13
  • An interesting thing to note is the positive effect that answering in SO may have in the programmer's productivity. I would claim that though being time-consuming, it also allows them to improve their knowledge of any given subject, as teaching reinforces self-learning. Programming is a very methodical activity, and explaining questions and detecting problems may prove to be useful to one's expertise. The balance between this increase and the time that one does not work, and their effect in economical terms is yet undetermined. – JuanRocamonde Jan 24 '19 at 01:01
  • Feedback given to one's answers also allows to detect flaws in concepts that one may have considered established. Furthermore, being in the situation of confronting a problem that one may have never seen can also be regarded as an increase of experience, equivalent to that of the actual workplace. I think this extra detail in the analysis is also significantly relevant. – JuanRocamonde Jan 24 '19 at 01:03
  • The source mentioned does mention product quality, which makes sense. Productivity is increased thus money are gained, but improvements, better software and innovation are halted as a price - a cost that needs to be deducted from the benefit. This cost is more difficult to calculate than benefit itself. – Stefanos Zilellis Jul 12 '19 at 10:25