6

I'm using the excellent Fog gem to access just the Rackspace Cloud Files service. My challenge is that I'm trying to keep the service that is accessing Cloud Files lightweight, and it seems that Fog through its flexibility has a lot of dependencies and code I'll never need.

Has anybody tried to build a slimmed down copy of Fog, just to include a subset of providers, and therefore limit the dependencies? For example, for the Rackspace Cloud Files API exclusively, I'd expect to be able to handle everything without net-ssh, net-scp, nokogiri gems, and all the unused code for Amazon, Rackspace and 20 other providers that are not being used. I'm hoping to avoid upgrading the gem every time one of those unused providers notices a bug, while keeping my memory footprint down.

I'd appreciate any experience anybody may have in doing this, or advice from anybody familiar with building Fog in what I can and can't rip out.

If I'm just using the wrong gem, then that's equally fine. I'll move to something more focused.

Phil
  • 2,797
  • 1
  • 24
  • 30

2 Answers2

10

I'm the maintainer of fog so I'll chime in to help fill in some of the explanation/gaps. The short answer is, yeah it's a lot of stuff, but mostly it shouldn't impact you negatively.

First off, fog grew pretty organically over time, so it did get bigger than intended. One of the ways that we contend with this is that we rather aggressively avoid requiring/loading files until really needed. So although you have to download lots of provider files you won't use to install fog, they shouldn't actually end up in memory. That was the simplest thing we could do in order to keep things "just working" while also reducing the memory usage (and load time).

The release schedule doesn't tend to be too crazy (on average about once a month) and tends to include a mix of stuff across most of the providers. So I'd expect you won't have too much churn here (outside of emergency/security type fixes which might warrant shortening the normal cycle).

So, that hopefully provides some insight in to the state of the art. We have also discussed starting to split things out more in the longer term. I think if/when that happens we would end up with something like fog-rackspace for all the rackspace related things. And then they could share things through fog-core or similar. We have a rough outline, but it is a pretty big undertaking without a huge upside, so it isn't something we have really actively begun on.

Hope that helps, certainly happy to discuss further if you have further questions or concerns.

geemus
  • 2,522
  • 16
  • 22
  • Definitely appreciate the additional insight (and all your effort that goes into maintaining this project). As I mentioned in my comment to @elight, I'll attempt a fork and a second gemspec to achieve what I need. In the future if there is anything I can contribute back to the project I'll let you know. – Phil Jan 04 '14 at 19:29
  • @geemus: I reflected on this a bit shortly after joining my team at Rackspace. For a time, I shared the same line of thought about splittng out fog- and fog-core. But there's a downside. In doing so, we make it more difficult for users of fog. Switching to or interacting with multiple providers would require users to install multiple "fog-" gems which adds complexity. So while as a now maintainer I love this idea, I'm a little concerned for the user community if we went this route. – elight Jan 04 '14 at 20:49
  • @elight: I would be interested if it would be possible to do a quick survey of the users of fog to see whether the majority use it as a convenient (and maybe the only supported) API to access a single vendor or provider inside a project (like me). Or as you say whether the majority use it for multiple providers in a single project. I see three data points here, and it would be great to get a view of a representative chunk of the user community, so that the community continues to find the project truly valuable in the future. Would you be open to this type of survey? Could I help? – Phil Jan 06 '14 at 15:07
  • 3
    @elight my plan, roughly, would be for the fog gem to become just a meta-gem of sorts. It in turn would have all the other gems as dependencies. So people could (and perhaps even by default, would) have all of them installed and they would be compatible. BUT, you could also pick and choose. I haven't gone far enough down that path to ensure it would work out nicely, but that seems to be the best/most balanced solution I'm aware of. – geemus Jan 06 '14 at 16:22
  • @Phil - we could survey, perhaps, but I suspect the single provider access pattern is FAR more common. So far we have not supported it because it would be a ton of work and it mostly seems "good enough", not because we don't think a lot of people would love it. Certainly I would love to see this happen, but it isn't something that is such a big/obvious win that it has been easy to find the not inconsiderable time likely required to go through the exercise. – geemus Jan 06 '14 at 16:24
  • @Phil: I didn't state that the multi-provider case is the dominant one. I'm with geemus: it's likely far outweighed by the single-provider case. Still, I'm concerned with facilitating user flexibility. For instance, I want to make it as easy as possible for rackspace fog users to use whomever they wish. – elight Jan 06 '14 at 19:08
  • @geemus: Ah, ok, I think I see where you're going. So breaking out gems per provider would be more an issue of code organization with perhaps the possibility that users could install a single vendor if that's what they wanted. Sounds good to me! – elight Jan 06 '14 at 19:10
  • geemus & @elight : I completely understand where you are coming from now. I couldn't even guess at how this might be approached, but if there is anything I could contribute to the the effort in the future, let me know (https://github.com/philayres on GitHub) – Phil Jan 06 '14 at 23:48
  • @Phil thanks, I appreciate the offer of help. For now, hopefully it is not too impactful for you (despite obviously being suboptimal). – geemus Jan 07 '14 at 14:54
5

I work for Rackspace on, among other things, our Ruby SDKs. You're using the right gem. Fog is our official Ruby API.

This is possibly something that could be done by introducing another gemspec into the project that builds from only fog core and the Rackspace-specific files. Though this would be unconventional and make @geemus' (the gem maintainer) gem release process more complicated––especially should other providers start to do the same. Longer term, this would serve to divert the fog community away from acting as a unified API.

elight
  • 86
  • 5
  • Thanks for chiming in. You made good points, I've just expanded a bit below. – geemus Jan 04 '14 at 05:32
  • Thanks for the thoughts on the alternative gemspec. I'll look at forking the project and giving this a go. Hopefully I'll be able to manage something on my side that is maintainable, without any impact on the primary fog project. – Phil Jan 04 '14 at 19:27