1

We're using a pretty vanilla instance of Confluent Cloud for internal testing. Because this is cloud-based, they give you statistics on how much data you're going through as the month goes along. Unfortunately, there aren't detailed statistics - just bytes into their instance, bytes out of their instance, and storage. We've transferred in about 2MB of data that's being stored there, but our transfers out are quite excessive, to the tune of about 4GB per day. We don't have many consumers and they're all up to date - there doesn't seem to be anything strange going on where any of the consumers are repeatedly querying from offset 0 or anything like that. My question is: is this typical behavior? Is it due to polling? Or something else?

Thanks @riferrei for your comment. I am sorry for the confusion. To try and help clarify, please take a look at this image: Bill

This is all I get. My interpretation is that during March, we stored at least 390 KB worth of data, but not much more (390 KB = 1024 * 1024 * 0.2766 GB-Hours / 31 days / 24 hours). We transferred in 2MB (0.0021 GB), and according to the bill, we transferred out 138 GB of data, or approximately 4 GB per day. I'm trying to understand how that could possibly happen.

Charlie
  • 17
  • 7

2 Answers2

0

Charlie,

Your question is a bit confusing so before trying answering it let me try get deeper in what is the real problem here.

  • Are you asking why there are 4GB of data instead of 2MB?
  • What are the typical behavior that you're referring to?

FYI, Confluent Cloud has a set of REST APIs that can be used to better monitor usage. Here is the documentation of it:

https://docs.confluent.io/current/cloud/metrics-api.html

Let us know what the problem really is so we can help accordingly.

Thanks,

-- @riferrei

Ricardo Ferreira
  • 1
  • 1
0

I have received word from Confluent support that: 1) They aren't changing their billing to omit overhead charges. Their billing documentation has been modified to call out the fact that they charge for protocol overhead:

"You are billed for the total amount of data transferred in and out of your cluster, including request overhead associated with the Kafka protocol."

2) They've added a note in the FAQ for the Metrics API, clarifying that it can't currently be used to reconcile billing charges. The plan is to also expose a metric that includes protocol bytes that will help with these concerns, but the details on that are still being worked on.

So, for the time being, the proposed solution to avoid excessive/unexplained data transfers on your Confluent Cloud bill is to adjust fetch.wait.max.ms from its default of 100 to something larger like 5000. This increases the time between consumer polls and will therefore decrease the amount of network overhead due to polling.

Charlie
  • 17
  • 7