I'm new to Node.js, but reading the statsd source, it looks to me that if for whatever reason the Graphite backend becomes inaccessible, that statsd just spills its collected stats on the floor, rather than retaining them in a growing cache of stats. Is this correct?
(I'm not worried about discarding stats, but about an ever-growing heap of a statsd that can't contact Graphite. So, if my read of the code is correct, I'm happy!)
Short answer: it's configurable, but the default behavior is the one you want (the cache is dropped between flush cycles).
Long answer: statsd doesn't actually report to graphite. It can however load a plugin that will do so (and it's delivered with such plugin).
The important thing to understand is that while plugins can report success or failure to init, the have no callback or other feedback option for flush events. The graphite plugin registers for 2 events, status and flush (https://github.com/etsy/statsd/blob/master/backends/graphite.js line 230), out of which flush is the only event that report metrics.
The graphite plugin register for the flush event. During a flush event, the node EventEmitter, will call all registered plugins, with A COPY of cached stats data. A different copy is sent to every plugin with no guaranty of order. You can see that happening if you look at https://github.com/etsy/statsd/blob/master/stats.js line 32, where you'll see metrics_hash being created and populated, then later used to emit the flush event (line 122).
The cache itself is dropped immediately after the the copy is populated (lines 77-120). As you can see in the code, you can change this default behavior, for a specific metric type (times, gauges, counters, sets) by adding an entry to the config file. To keep the counter cache for example, you should add deleteCounters = true to your config. Notice that this config will always keep the cached metrics, regardless of the success or failure of any plugin.
