1

I'm new to Elixir and working on a distributed system. This system shares several global tables. Writes to this tables are frequent, so they should be pretty fast. Now, I sometimes have to process one whole table and generate some JSON from it, but I need to process the records in this table in the order they were inserted into the database. (I do not need any hard guarantees. Infrequent errors, like two swapped records or even single dropped ones are tolerable.)

Unfortunately, mnesia afaik doesn't have anything like SQL's auto_increment, and now I struggle to implement this functionality myself.

Solution I can think of:

  • Store a unix timestamp (or some other kind of timestamp) as a key for the records. But it feels wrong for me that the correctness of the program depends on something like time. Situation where the time on two machines gets out of sync would be horrible to debug. And retrieval of time is afaik slow (read it somewhere, please correct me if I'm wrong) Unix timestamps's unit is a second, which is far too big.
  • Store a global counter as they key for the records. Might be even slower then the timestamp, and I'm unsure if messing around with mnesia counters is a good idea as they are not suited for distributed systems (again, please correct me if I'm wrong)
  • Find a magic way how to maintain the order the records were inserted in. This is my favourite ;)
nickkoro
  • 374
  • 3
  • 15
  • Would not [`:mnesia.dirty_update_counter/3`](http://erlang.org/doc/man/mnesia.html#dirty_update_counter-3) suffice your needs? It’s blazingly fast and does exactly what you seem to need. – Aleksei Matiushkin Oct 25 '18 at 12:14
  • @AlekseiMatiushkin there are several articles and discussions (for example [here](https://erlang-questions.erlang.narkive.com/GHaRicLJ/mnesia-dirty-update-counter-and-replicated-tables) and [here](https://stackoverflow.com/questions/675608/safe-sequential-and-scalable-counters-in-mnesia)) which state that using mnesia counters in distributed systems isn't a good idea. Altrough they may be enough for what I'm planning... I have to test it myself probably. – nickkoro Oct 27 '18 at 11:08

1 Answers1

0

If you want to implement it yourself, then as @Aleksei mentioned, dirty_update_counter/3 is the way to go.

But if you are fine with using an external library, both Memento and Amnesia support a custom implementation of autoincrement. Here's how it would look like in Memento:

defmodule MyTable do
  use Memento.Table,
    attributes: [:id, :title, :content, :status, :author],
    type: :ordered_set,
    autoincrement: true

end

Also, Mnesia has an eventual-consistency model, so although it is extremely fast, it doesn't make instant consistency guarantees. Whatever implementation you choose for autoincrement, it will definitely have a hit on the performance.

Full Disclosure: I'm the author of Memento

Sheharyar
  • 73,588
  • 21
  • 168
  • 215