Use log filenames as Log names on Google logging

Question

How could I use log filenames as log names on google logging.

I have this on google-cloud-ops-agent/config.yaml

logging:
  receivers:
    nginx_access:
      type: nginx_access
    nginx_error:
      type: nginx_error
    nginx_access_files:
      type: files
      include_paths: [/var/log/nginx/*access*.log]
  service:
    pipelines:
      nginx:
        receivers:
          - nginx_access
          - nginx_error
          - nginx_access_files
metrics:
  receivers:
    nginx:
      type: nginx
      stub_status_url: http://127.0.0.1:80/status
  service:
    pipelines:
      nginx:
        receivers:
          - nginx

The problem is that all logs from folder /var/log/nginx/access.log shows up on google logging as one entity "nginx_access_files". The folder has hundreds of access logs with the site name on them. I would like to have that log filename as the log name on the logging console too.

This I have now on Google logging console:

And I would need to set up it so that the filename shows upon logging console. Like the default, Nginx error does on OPS agent. Or is there another way to route all access logs to google logging with the filenames?

score 0 · Accepted Answer · answered May 31 '22 at 16:05

There is no need to configure the nginx_access_files receiver, if you are using the default path for the access.log file, then the agent automatically pulls the logs and sends them to Cloud Logging.

The agent yaml file should be something like this:

logging:
  receivers:
    nginx_access:
      type: nginx_access
    nginx_error:
      type: nginx_error
  service:
    pipelines:
      nginx:
        receivers:
          - nginx_access
          - nginx_error
metrics:
  receivers:
    nginx:
      type: nginx
      stub_status_url: http://127.0.0.1:80/status
  service:
    pipelines:
      nginx:
        receivers:
          - nginx

Under the nginx_access log name, you will see all the access logs generated and since they are generated with the instance name as a label, you can use it to filter them:

If your access.log file is not in the default path, then you can specify where it is, but there is no need to create a new receiver for that, this is an example:

logging:
  receivers:
    nginx_access:
      type: nginx_access
      include_paths: [/your/log/path/access.log]
    nginx_error:
      type: nginx_error
  service:
    pipelines:
      nginx:
        receivers:
          - nginx_access
          - nginx_error
metrics:
  receivers:
    nginx:
      type: nginx
      stub_status_url: http://127.0.0.1:80/status
  service:
    pipelines:
      nginx:
        receivers:
          - nginx

Finally, I recommend you to take a look at the NGINX Cloud Logging integration tutorial.

I know that tutorial, but somehow by doing with that we can only get access logs of the 127.0.0.1 address. As we have hundreds of sites and all of those have their own access logs on path /var/log/nginx/*. After I included the path I can receive those too. We used to use Legacy logging before and then we have separate logs on logging console too. As we have multiple (10 of them) nginx VM hosts that are generating those logs, it would be easier to sort them by log names. — Timo77, Jun 01 '22 at 07:53
So, what you want, if I correctly understood, is to sort the nginx access logs by "site" right? What I dont understand is why the solution provided does not work for you if all the logs are labeled with the source instance id, you can easily sort them "by site" just selecting the corresponding instance id under the log name tab. — Gabriel Robledo Ahumada, Jun 01 '22 at 14:17

Use log filenames as Log names on Google logging

1 Answers1