Django server not sending logs to Logstash

Question

I am using ELK stack for centralised logging from my Django server. My ELK stack is on a remote server and logstash.conf looks like this:

input {
    tcp {
    port => 5959
    codec => json
  }
}
output {
  elasticsearch {
    hosts => ["xx.xx.xx.xx:9200"]
  }
}

Both services elasticsearch and logstash are working (checked using docker-compose logs logstash).

My Django server's settings file has logging configured as below:

LOGGING = {
  'version': 1,
  'handlers': {
        'logstash': {
            'level': 'INFO',
            'class': 'logstash.TCPLogstashHandler',
            'host': 'xx.xx.xx.xx',
            'port': 5959, # Default value: 5959
            'version': 0, # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
            'message_type': 'django',  # 'type' field in logstash message. Default value: 'logstash'.
            'fqdn': True, # Fully qualified domain name. Default value: false.
            'tags': ['django.request'], # list of tags. Default: None.
        },
  },
  'loggers': {
        'django.request': {
            'handlers': ['logstash'],
            'level': 'DEBUG',
  },
}
}

I run my Django server and Logstash handler handles the logs as console shows no logs. I used the python-logstash library in Django server to construct the above conf, but the logs are not sent to my remote server.

I checked through many questions, verified that services are running and ports are correct, but I have no clue why the logs are not being sent to Logstash.

Are you sure that "django.request" doesn't filter out all events? — Philip Stark, Aug 06 '17 at 12:28

Daniel Backman · Accepted Answer · 2017-08-11T15:11:01.950

6

Looking at the configuration, logger "django.request" is set to level "DEBUG" and the handler "logstash" is set to level "INFO". My guess is that the handler will not process DEBUG messages. I'm not sure though.

Set the same level for the logger and the handler to test that it works.

What level to use depends on what you want from your logs. In this case I guess level INFO would suffice.

If not already take a look at Django logging

NOTE: From comments it seems not to solve the problem but I hope it is useful anyway.

UPDATE:

I tried the following configuration and it catches 404 and 500 errors in the "debug.log".

LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
    'logfile': {
        'level': 'WARNING',
        'class': 'logging.FileHandler',
        'filename': os.path.join(PROJECT_DIR, 'debug.log'),
    },
},
'loggers': {
    'django.request': {
        'handlers': ['logfile'],
        'level': 'WARNING',
        'propagate': True,
    },
}}

With this test configuration the logstash handler should at least receive the message/logrecord. If no luck I suggest to try debug the logstash.TCPLogstashHandler and the SocketHandler (inherited by TCPLogstashHandler) to make sure they receive the emitted record.

edited Aug 11 '17 at 15:11

answered Aug 07 '17 at 11:27

Daniel Backman

5,121
1
32
37

Your point seems to be correct so what would you suggest for solving this? – Arpit Solanki Aug 07 '17 at 11:36
1

I suggested to use level INFO. Depending on how log messages are created, sometimes it's good to be able to switch DEBUG logging on and off depending which environment is used. For instance django uses it for mail_admins. – Daniel Backman Aug 07 '17 at 21:04
@ArpitSolanki `django.request` logging just `WARNING` and `ERROR` messages. https://docs.djangoproject.com/en/1.11/topics/logging/#django-request-logger – Dzmitry Aug 08 '17 at 11:43
@ArpitSolanki What messages are you trying to log? – Daniel Backman Aug 08 '17 at 14:00
tried this answer with producing errors like 404, 500 but did not work – Arpit Solanki Aug 09 '17 at 16:02
1

@DanielBackman Thanks for your effort but it did not solve the problem. I can still award you the bounty if you add a small note to your answer that it did not solve the problem but helped in logging part of the code – Arpit Solanki Aug 11 '17 at 07:16
1

@ArpitSolanki I'll try to find why the logger is not working as expected. – Daniel Backman Aug 11 '17 at 14:25

Tasneem Haider · Answer 2 · 2019-09-04T05:17:13.393

I faced the same issue but after scrambling around a lot, I made it work by changing django.request to django.server. I got the idea when I used only django as a logger name in the python code and then found out the actual logger_name from log data stored in elasticsearch. Below is the updated code

LOGGING = {
  'version': 1,
  'handlers': {
        'logstash': {
            'level': 'INFO',
            'class': 'logstash.TCPLogstashHandler',
            'host': 'xx.xx.xx.xx',
            'port': 5959, # Default value: 5959
            'version': 0, # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
            'message_type': 'django',  # 'type' field in logstash message. Default value: 'logstash'.
            'fqdn': True, # Fully qualified domain name. Default value: false.
            'tags': ['django.request'], # list of tags. Default: None.
        },
  },
  'loggers': {
        'django.server': {  # Here is the change
            'handlers': ['logstash'],
            'level': 'DEBUG',
      }
  },
}

Refer to this for more detail on django logging https://docs.djangoproject.com/en/1.11/topics/logging/#id3

so later should I do this before logging `logging.getLogger('django.server')` ? the `.` is confusing me — Chau Loi, May 25 '21 at 04:55

score 0 · Answer 3 · answered Mar 18 '19 at 07:48

Your logging configuration is correct. You need to mention index name in elasticsearch config part of logstash conf. Update your logstash config to

input {
    tcp {
    port => 5959
    codec => json
  }
}
output {
  elasticsearch {
    hosts => ["xx.xx.xx.xx:9200"]
    manage_template => false
    index => "djangologs"
  }
}

If you are using google cloud or AWS, open port/update firewall rules.

score 0 · Answer 4 · answered Apr 02 '21 at 04:04

Thank you Tasneem,

This worked on my machine:

mysite/settings.py

LOGGING = {
  'version': 1,
  'handlers': {
      'logstash': {
          'level': 'DEBUG',
          'class': 'logstash.TCPLogstashHandler',
          'host': 'xx.xx.xx.xx',
          'port': 9600, # Default value: 5959
          'version': 1, # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
          'message_type': 'django',  # 'type' field in logstash message. Default value: 'logstash'.
          'fqdn': False, # Fully qualified domain name. Default value: false.
          'tags': ['django.request'],# list of tags. Default: None.
      },
  },
  'loggers': {
      'django.server': {
          'handlers': ['logstash'],
          'level': 'DEBUG',
      },
  },
}

logstash# cat logstash-django.conf

input {
  tcp {
    port => 9600
    codec => json
  }
}
output {
    elasticsearch {
      hosts => ["https://your.elasticsearch.com:port"]
      user => ["xx"]
      password => ["xx"]
      index => "djangotest"
      document_type => "%{type}"
    }
   stdout {
      codec => rubydebug {metadata => true }
  }
}

Django server not sending logs to Logstash

4 Answers4