1

I am trying to use ElasticSearch as a NoSQL database in my Django project. The goal is to plug Kibana on the other side and be able to visualize my data there.

I do not have a choice of adding another database and using ElasticSearch only for indexing as my project is plugging to an existing infrastructure.

So I have installed the following two modules:

  • django-haystack
  • elasticsearch

I was expecting to not need SQLite (or any other database) for storage and to use ElasticSearch as a NoSQL storage (is that wrong?)

I added the Haystack connections settings to my project settings and then, looked for getting rid of "DATABASES" or replacing it with values pointing to my ElasticSearch with no success :(

Here is what I'm stuck with:

settings.py:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
        'URL': 'http://127.0.0.1:9200/',
        'INDEX_NAME': 'haystack'
    },
}

On all the answers I could find on StackOverflow and code snippets I would never see these DATABASES settings changed, so I assume people were using a database for storage and ElasticSearch only for their search engine/indexing which is not what I am looking for.

Is there a way I can have ElasticSearch as my storage DB (just like I would with MongoDB), not breaking the Models and interface of Django?

Mostafa Ghadimi
  • 5,883
  • 8
  • 64
  • 102
Daniel
  • 1,172
  • 14
  • 31

1 Answers1

2

It's not possbile (officially) to use ElasticSearch as your django database backend (I assume that this is what you want) - currently supported backends are:

  • postgresql
  • mysql
  • sqlite3
  • oracle

The above is official list, unofficial (3rd parties) is as follows:

  • SAP SQL
  • IBM DB2
  • Microsoft SQL
  • Firebird
  • ODBC

Anyway, the people around django are very nice and hard working developers :) And there is package for that (you can give it a try - never used it before):

https://github.com/aparo/django-elasticsearch

One more thing - why do you need django? Is kibana not enough? You gonna make some changes in data on the ES index in your Django App? Or should it be just readonly?

Because if you want use the ES like a normal DB storage - it won't work, as you will wait each time you update/insert data about 1-2 seconds till ES index new data. This is just not the purpose of the ES.

Hope this help, happy coding.

opalczynski
  • 1,599
  • 12
  • 14
  • So, my Django project articulates around two apps, that pull data from the GitHub API and do parsing before putting it in a Database. The aim is to be able to visualize metrics, get insight about the use of GitHub in a big big company :) The DB will only be used by a Bot that is listening for new commits/issues across all the repos – Daniel Dec 03 '16 at 20:53
  • Nice :) So basically I think you do not need django. Maybe whole Elastic stack? Probably you will be able to configure logstash to feed your elasticsearch index and then make a metrics in kibana? – opalczynski Dec 03 '16 at 23:19
  • 1
    @Daniel please take a look: https://www.elastic.co/blog/introducing-logstash-input-http-plugin – opalczynski Dec 03 '16 at 23:27
  • This looks very good but I would like to parse the messages and apply some cognitive logic before dumping them into the database, even use some natural language processing so I'm still wanting to use Django :) What this shows me however is that I can add a hook that points to my Django API :) – Daniel Dec 04 '16 at 16:32