1

There are multiple containers using data from the same indexes in Elasticsearch and Mongodb databases:

  • app is a python app that serves data via API
  • tasks is a python + celery app that perform periodical updates, cleaning, etc. on the same data
  • backend is a python app that use the same data for business-logic

The definitions of the models are elasticsearch-dsl and mongoengine Documents with minimum code for validators. Currently they live in a separate repo on gitub, which is included in requirements.txt of each app, and used when running containers via docker-compose (in dev) or swarm (in production)

While OK for production, this method is very inconvenient for developer, since after each change in any model description, a separate repo should be updated, and then those changes should be populated across all local apps.

Is there any other convenient method for sharing model definitions across multiple containers in the same project (without compromising security)?

funkifunki
  • 1,149
  • 2
  • 13
  • 24
  • aaaaaand this is why NoSQL has never been a good idea and never will be. If you want the benefit of not having to design a schema then don't be surprised that you can't maintain the system. Anyhow, the way one usually decouples a db model from the applications using the db is with an API. It looks like one of your services is an API. the other services should use that. If all your services talk directly to the db then you have shared data between microservices, which is always risky for exactly this reason. You should only ever share data of any kind between microservices if it doesn't change. – Neil Apr 14 '21 at 10:48
  • alternatively publish your model definitions in a library and version it. Your apps can install the library. then just update requirements.txt when you update the models. – Neil Apr 14 '21 at 10:49
  • 1
    Even more brute force, write a bash script that copy-pastes the models into each repo. update the model in one place and then run the bash script. This obviously has hundreds of it's own problems but it can work for simple config that you need to share between services. – Neil Apr 14 '21 at 10:51
  • Of course the micro-service people will tell you that you need a config management service.... probably is the best solution.... if you can afford it... – Neil Apr 14 '21 at 10:51
  • @Neil thanks a lot for the suggestions. i've thought ab/API, but the models are accompanied with some code. versioning models in a separate lib is what we're doing now (described) and it's a burden (( bash script - yeah, i agree this can be problematic, but this actually sounds ok for a small-sized infrastructure. re: config management service - can u pls throw couple named entities here? i haven't heard ab/that before (i'm just a dev, not devops) – funkifunki Apr 14 '21 at 11:16
  • 1
    Something like Spring Cloud Config: https://cloud.spring.io/spring-cloud-config/reference/html/ If your model definitions can be represented in JSON then the easiest is probably just to put the models definitions themselves into a collection in Mongo and have all your applications read it in from there, with the understanding that the schema for that collection can't change. So you read that table first, get the config, and then you know how to read the rest of the db. – Neil Apr 14 '21 at 11:22
  • 1
    @Neil thanks for the suggestion! re: config in a separate collection - i've considered that but in our case there are few validators (method of classes that describe mongoengine Documents). but maybe it is a good idea actually to move validation to the level of db... then the idea of the schema defined in a separate collection is probably the most universal one (doesn't require API, etc.). again - thanks a lot! – funkifunki Apr 14 '21 at 11:29

0 Answers0