1

Our segment.io streams events into our Snowflake instance. This results in the Segment_warehouse being turned on all day and consuming all our credits

Is this a common problem? It's a huge cost add to our operations. Any suggestions to improve this?

Thank you!

thedre
  • 101
  • 1
  • 4
  • I'm not familiar with segment.io specifically but is it continuously loading data all day into snowflake? – Dean Flinter Oct 12 '21 at 17:28
  • Yes. When someone does something in the website for example, we get it near real time. Doesnt seem like you can restrict how often they sync data – thedre Oct 12 '21 at 17:49
  • 1
    How are you actually loading the data - using Snowpipe? What size warehouse are you using and is it larger than actually required? – NickW Oct 12 '21 at 19:05
  • 2
    It sounds like you aren't using Snowpipe. I would look into leveraging that, rather than a warehouse + COPY command. It will likely save you quite a bit of money, if segment.io can send the data to an S3 bucket instead of loading the data with a warehouse. – Mike Walton Oct 12 '21 at 19:53
  • @MikeWalton we are considering doing the S3 route. Otherwise, its a big indirect cost to using segment! – thedre Oct 12 '21 at 20:42
  • Your other option would be to batch up the data and load it on a scheduled batch to save on the WH cost. – Mike Walton Oct 12 '21 at 21:15
  • Snowpipe is serverless and costs are much less than using a dedicated virtual machine that needs to spin up, underneath Snowpipe is executing a COPY command – patrick_at_snowflake Oct 12 '21 at 23:13
  • @thedre please can you give us more details about how the data are ingested into Snowflake ? What volume, how often etc. A brief look at the segment website indicates that it does a "sync" either once or twice daily. As such, if there are just two bulk-copy statements executing every day then first of all I would ensure that auto-suspend is switched on for the virtual warehouse you are using - also ensure that you use a dedicated warehouse for this job. – Robert Long Oct 14 '21 at 08:56

0 Answers0