4

We have a build server, which doesn't have access to internet. It gets our source from a git server, then tries to build them.

Currently, it fails(obviously) because this computer doesn't have internet access. I discussed it with IT, and they agree to let us connect it to internet to do some maintenance, but they refuse to let it connected all the time.

My question is:

Is it possible to create some kind of local nuget cache that we can populate when we do changes on packages, and then the build server will try to use this cache before connecting to the server?

J4N
  • 19,480
  • 39
  • 187
  • 340

1 Answers1

4

I can think of several solutions.

Download packages used by a solution

I answered a very similar question recently, where the person asking wanted to get packages to use on an offline computer. A summary of my answer from the other question is you create a nuget.config that sets the global packages folder to a folder that's currently empty, do a restore, then copy all the .nupkg files to your destination. NuGet supports local folder feeds, so just drop those nupkgs on a network share and tell your build server to use that network share as a nuget source.

An obvious problem is that every time any developer installs a new package (or new version of an existing package), the new packages need to be copied to your local feed.

Mirror nuget.org's packages locally

My team mates have a tool that will use NuGet's v3 catalog api find all non-deleted packages (optionally filtering out unlisted packages) and download all 1.4 to 1.8 million packages locally. Their implementation takes about a week to finish downloading everything. Out interest, I wrote something that used Azure Functions to auto-scale the job and managed to download all the packages in 2 hours. The blob storage account is now 2.4 TiB, so you almost certainly don't want to do this unless you're doing it to analyse the packages on nuget.org, rather than using it as a source to restore packages. You will probably want to periodically update your local feed.

Use a read-though caching server

Since your build server is network connected, work with your IT department to get a read-through caching server. If your company already uses a HTTP proxy, work with your IT department to allow your build server to use it, then configure it as the HTTP proxy.

If your company doesn't have a HTTP proxy, there are several nuget servers you can install in your company. Some of them support read-through caching, so have your IT department set it up in the company DMZ and your build server can use it to get packages from a feed on the internet.

zivkan
  • 12,793
  • 2
  • 34
  • 51
  • The 1) would be the closest we can do, we will take a look, thank you! – J4N Dec 13 '18 at 06:09
  • Could you please share how did you implement to download all nuget packages? – nabukhas Dec 15 '19 at 09:49
  • wrote code to read nuget.org's "catalog" (event log) to find a list of every package, write each package into a queue and use Azure Functions with auto-scale to download and save every url in the queue. [This blog post](https://blog.maartenballiauw.be/post/2019/07/30/indexing-searching-nuget-with-azure-functions-and-search.html) goes into more detail on how they did something similar. – zivkan Dec 16 '19 at 21:54
  • 1
    @zivkan which [nuget servers](https://learn.microsoft.com/en-au/nuget/hosting-packages/overview) support read-through caching? I've found only BaGet but there is an opened issue with [read-through not working at all](https://github.com/loic-sharma/BaGet/issues/642). And looks like project is abandoned: pull-requests [are not reviewed and merged for 10 month](https://github.com/loic-sharma/BaGet/pull/717) – bairog Oct 19 '22 at 07:51
  • @bairog as a Microsoft employee working on NuGet, I don't want to show bias, so I won't list ones I know support it in case I miss some. But some I know that support it are not free/open source. I feel for open-source maintainers. It's an often thankless job where people can be demanding of your time. There are articles online discussing open source sustainability. If your company finds OSS useful, consider funding it in some way, even if it's by forking it and maintain what the original author is unwilling to maintain without compensation (or perhaps just lost interest in). – zivkan Oct 22 '22 at 21:59
  • @zivkan as for BaGet, there are several community-driven pull requests that are only waiting to be approved and merged. So original author can only assing some rights to those who desire (and have time) maintain the project. So **_funding it in some way_** is already happening, but for some reason the author does not get in touch (although he is actively contributing to other projects) and does not want to spend a bit of time to assign rights to other people. People want to maintain the project, but the author does not take the slightest step to save it from abandonment. – bairog Oct 25 '22 at 06:16
  • 1
    @bairog maintaining an OSS project requires a lot more effort than just clicking the merge button on pull requests. My experience on NuGet is that contributors often don't run tests locally, so they don't realize how many features they break. They sometimes don't even try to build their contribution locally, so even simple syntax errors aren't caught. My point is that contributors sending pull requests doesn't come close to the effort needed. Someone needs to do project management, and an unpaid person working in their free time owes everyone else nothing. Their personal life is more important – zivkan Oct 25 '22 at 07:01