9

Logstash can make use of a bundled GeoLiteCity.dat database for IP address geographical lookups. Is this database the same as the one provided by MaxMind? MaxMind updates the database on the first Tuesday of every month.

Would it be smart to set up a job to auto-refresh the database instead of waiting for updates to Logstash from ElasticSearch?

EDIT: Dec 1 2014 Here's the bash script I wrote to perform the auto-update of the databases. My read of the source code for this filter is that a service restart is probably required to take up the updated database files.

#!/bin/bash

# Downloads the latest GeoLight DBs from maxmind.
# Updates/replaces the databases that logstash uses.
# These are the IP-to-location databases that logstash uses.
# Maxmind updates them once a month on the first Tuesday of the month.
# See http://dev.maxmind.com/geoip/legacy/geolite/

echo Beginning update of GeoIP databases for logstash.
cd /tmp
rm -f GeoIPASNum.dat.gz GeoIPASNum.dat GeoLiteCity.dat.gz GeoLiteCity.dat
echo Downloading latest files.
wget --quiet --output-document GeoIPASNum.dat.gz http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz || { echo 'Download of GeoIPASNum.dat.gz failed' ; exit 1; }
wget --quiet --output-document GeoLiteCity.dat.gz http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz || { echo 'Download of GeoLiteCity.dat.gz failed' ; exit 1; }

echo Unzipping
gunzip GeoIPASNum.dat.gz
gunzip GeoLiteCity.dat.gz

echo Setting permissions
chmod 664 GeoIPASNum.dat GeoLiteCity.dat
chown logstash:logstash GeoIPASNum.dat GeoLiteCity.dat

echo Replacing existing files and backing up the old.
cd /opt/logstash/vendor/geoip/
mv -f GeoIPASNum.dat GeoIPASNum.dat.bak && mv /tmp/GeoIPASNum.dat .
mv -f GeoLiteCity.dat GeoLiteCity.dat.bak && mv /tmp/GeoLiteCity.dat .

echo Restarting logstash
# Modify for your distro services model.
service logstash restart

echo Done
Larry Silverman
  • 1,043
  • 1
  • 11
  • 30
  • Found the source for the geo filter: https://github.com/logstash-plugins/logstash-filter-geoip/blob/master/lib/logstash/filters/geoip.rb It reminded me that I could set in config an alternate location for these databases, which would likely be a better solution than overwriting the distributed databases. – Larry Silverman Dec 01 '14 at 17:54
  • Its silly to have to restart logstash every time a new GeoLiteCity.dat is dropped into place. I see there is an option to have "periodic_flush" but I am not sure what that does exactly or how often as all it says is: "Call the filter flush method at regular interval" (https://www.elastic.co/guide/en/logstash/2.4/plugins-filters-geoip.html#plugins-filters-geoip-periodic_flush) – totalflux Jan 13 '17 at 00:05

1 Answers1

5

Yes, it's the same database, and yes, you can use updates from maxmind website. I use the geoip-database-contrib package in ubuntu which includes a cronjob to update the database files from maxmind automatically.

I don't how fast the maxmind dataset changes, but since logstash (which includes the database file) has a slow release schedule (current 1.4.2 was released 5 months ago), I use an up-to-date database.

whyscream
  • 669
  • 6
  • 11