3

I'm in need of a large repository of open source projects (around 1000 or more, the programming languages don't matter, but a good mix will be nice) for my research work. I thought of downloading projects from Github/SourceForge/Codeplex, but I cannot find the right API's to do it.

Does anyone know whether it is possible to download projects from the aforementioned websites (Like, how Twitter allows us to grab tweets from the public time-line)? Or any other place where I can get a good mix of open source projects?

Gooner
  • 1,380
  • 2
  • 14
  • 25

1 Answers1

5

Pretty much all open source repositories allow remote access via the appropriate source control provider - so the simplest way to download all the projects from Github would be to use git. Even if there's no API for it, all you need to do is find the right URL for each repo, and scraping something like the "explore" page should be easy.

Jon Skeet
  • 1,421,763
  • 867
  • 9,128
  • 9,194
  • I'm looking for around a 1000 projects. "Explore" has very limited numbers. – Gooner Aug 23 '11 at 16:24
  • 1
    @Gooner: Do a search then (https://github.com/search) and page through the results. – Jon Skeet Aug 23 '11 at 16:27
  • Sounds cool. I shall wait for a few others to pop in their opinions before accepting the answers though. Thanks! – Gooner Aug 23 '11 at 16:33
  • @Gooner: Of course there are other options beyond Github too - but many of them have multiple source control types, which would make it slightly harder. For example, Google Code supports svn, git and hg. It's fairly easy to find them though: http://code.google.com/hosting/search?q=label%3aJava (etc) – Jon Skeet Aug 23 '11 at 16:51
  • I thought GoogleCode used svn only! This sounds interesting. Let me check it out. – Gooner Aug 23 '11 at 16:57
  • @Gooner: Nope, we started supported hg a while ago, and git just recently (start of August, I think). – Jon Skeet Aug 23 '11 at 17:04
  • That's very cool, I'll check out both and see what suits my use case best! – Gooner Aug 23 '11 at 17:13