3

We have an application that tries to migrate existing application logs of about a year into google cloud logging. I have created a custom logging bucket with a retention time of 400d and a custom sink that writes all the requests for a given logName into this bucket. I also excluded this logName from the _DEFAULT bucket.

Now if i write log entries for the given logName, with timestamps from all over the last year, all entires that are older than 30d are discarded. I even increased the retention time of the _DEFAULT bucket to e.g. 60d but still no logs older than 30d can be written. That means that the write successfully happened for all log entries but only entries that are older than 30d do not show up in the Log Explorer.

According to Routing - Log Retention the bucket should define the retention period:

Cloud Logging retains logs according to retention rules applying to the log bucket type where the logs are held.

Also there doesn't seem to be any quota that should limit this.

Does anybody know why entries with a timestamp older than 30d are silently discarded despite properly configured logging bucket and sink?

Or are there any better solutions to import logs into GCL without having to write a custom app to do so?

jstadler
  • 340
  • 1
  • 3
  • 7

1 Answers1

4

Cloud Logging currently has time bounds on the timestamp of the LogEntries it can ingest in its storage. Logs can only be ingested if the timestamps are within the last 30 days or 1 day in the future. This applies even if your bucket retention period is set to 60 days or more.

This is a current limitation and may change in the future.

Disclaimer: I work in Cloud Logging

Summit Raj
  • 820
  • 5
  • 10
  • Thanks a lot. That helps! You wouldn't know if there are already plans to make adjustments to those limits? And I guess this would be worth documenting e.g. here https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry or here https://cloud.google.com/logging/quotas#logs_retention_periods Because I guess there are also others that would like to migrate existing logs to GCL. And this is clearly a blocker for such usecases... – jstadler Jan 27 '22 at 15:14
  • This limit is documented in the same page in https://cloud.google.com/logging/quotas#log-limits. We do intend to support streaming in late data older than 30 days. The limits come from the current storage layer. In the future we plan to support ingesting old logs up to 5 years. What does your requirement look like? – Summit Raj Jan 28 '22 at 01:48
  • Thanks for documenting it now. But it wasn't documented before. Yes, 5 years would certainly be enough for now. A little over a year (e.g. 400d) would probably also already fit our needs. How soon would you guess that this change could be done? Months? Years? Are those plans somewhere publicly visible? – jstadler Jan 31 '22 at 06:23