getting logs from a file with Ops Agent

Question

I have a python script on a vm that writes logs to a file and I want to use them in the google logging.

I tried this config yaml:

logging:
  receivers:
    syslog:
      type: files
      include_paths:
      - /var/log/messages
      - /var/log/syslog
    etl-error-logs:
      type: files
      include_paths:
      - /home/user/test_logging/err_*
    etl-info-logs:
      type: files
      include_paths:
      - /home/user/test_logging/out_*
  processors:
  etl_log_processor:
    type: parse_regex
    field:       message
    regex:       "(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s(?<severity>INFO|ERROR)\s(?<message>.*)"
    time_key:    time
    time_format: "%Y-%m-%d %H:%M:%S"
  service:
    pipelines:
      default_pipeline:
        receivers: [syslog]
      error_pipeline:
        receivers: [etl-error-logs]
        processors: [etl_log_processor]
        log_level: error
      info_pipeline:
        receivers: [etl-info-logs]
        processors: [etl_log_processor]
        log_level: info
metrics:
  receivers:
    hostmetrics:
      type: hostmetrics
      collection_interval: 60s
  processors:
    metrics_filter:
      type: exclude_metrics
      metrics_pattern: []
  service:
    pipelines:
      default_pipeline:
        receivers: [hostmetrics]
        processors: [metrics_filter]
      error_pipeline:
        receivers: [hostmetrics]
        processors: [metrics_filter]
      info_pipeline:
        receivers: [hostmetrics]
        processors: [metrics_filter]

and this is an example of the logs: 2021-11-22 11:15:44 INFO testing normal

I didn't fully understand the google docs so I created the yaml as best as I understood and with a reference to their main example but I have no idea why it doesn't work

environmen:GCE VM

You want to use those logs in GCP Log Viewer: yes

Which docs did you follow: https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/configuration#logging-receivers

How did you install OpsAgent: in gce I entered each vm instance went to observability and there was the option to install ops agent in cloud shell

What logs you want to save: I want to save all of the logs that are being written to my log file live.

specific applications logs: its an etl process that runs in python and saves its logs to a local file on the vm

Could you share more details about your environment? It's GCE VM? You want to use those logs in GCP Log Viewer? Which docs did you follow? How did you install OpsAgent? What logs you want to save, specific applications logs? — PjoterS, Nov 23 '21 at 09:32
To confirm, you have changed default ops agent configuration in `/etc/google-cloud-ops-agent/config.yaml` and you have restarted agent after changes using `sudo service google-cloud-ops-agent restart` command? Also could you check if everything is ok with OPS Agent using [this troubleshooting docs](https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/troubleshooting)? — PjoterS, Nov 25 '21 at 10:19

score 0 · Answer 1 · edited Jan 31 '22 at 11:52

0

sudo journalctl -xe | grep "google_cloud_ops_agent_engine"

Try out this command it should show you the exact(almost) error

edited Jan 31 '22 at 11:52

Syscall

19,327
10
37
52

answered Jan 31 '22 at 11:52

Cristo Mathew

11
2

getting logs from a file with Ops Agent

1 Answers1