-1

Google is charging for dataflow job logs generated by the pipeline itself and we are not generating any logs from the job and we are running dataflow jobs via composer using the python-apache beam.

It doesn't make any sense as we are not generating any logs and we don't have much info on how to turn off logs generated and ingested to cloud logging.

Google mail we received

We’re writing to remind you that starting October 12, 2021, logs that your Dataflow applications generate (and are ingested and stored in Google Cloud Logging) will be charged at the standard Cloud Logging prices. You will still receive the free tier of 50 GB of total (across all log volume) ingested logs per project per month. This is a follow-up to the communication we sent on July 12, 2021.

What do I need to do?

We recommend that you review your usage of Dataflow Job logging in Cloud Logging by performing the following steps before October 12, 2021: View your usage metrics using Metrics Explorer in Cloud Monitoring. Save the chart you used in the previous step into a Cloud Monitoring dashboard or use the Dataflow usage open source dashboard template. Refer to controlling Dataflow log volume for information on optimizing log volume for your Dataflow jobs.

James Z
  • 12,209
  • 10
  • 24
  • 44
Chethan R
  • 11
  • 4
  • Follow this [link](https://cloud.google.com/support/billing#how_to_contact_support) to contact Cloud Billing support for billing queries. Otherwise, you can open a new issue on the [issue tracker](https://cloud.google.com/support/docs/issue-trackers). As you will have to share your project ID, use this [link](https://cloud.google.com/support/docs/issue-trackers) for more details. – Prajna Rai T Oct 25 '21 at 14:09

1 Answers1

0

Review the injected logs in Cloud Logging from Dataflow and search for LOG_ID then modify _Default sink to exclude them.

Default configuration is following:

NOT LOG_ID("cloudaudit.googleapis.com/activity") AND NOT LOG_ID("externalaudit.googleapis.com/activity") AND NOT LOG_ID("cloudaudit.googleapis.com/system_event") AND NOT LOG_ID("externalaudit.googleapis.com/system_event") AND NOT LOG_ID("cloudaudit.googleapis.com/access_transparency") AND NOT LOG_ID("externalaudit.googleapis.com/access_transparency")

Please refert to docs

Szymon Tomczyk
  • 1,219
  • 11
  • 16