-2

Oliver Dony, data protection specialist at Odoo, says that using "serializable transactions is not an "architecture flaw", but it is in fact the only guaranteed way of ensuring the correctness of a concurrent system".

One approach to scaling Postgres in an Odoo deployment that gets implemented from time to time is splitting read and write queries using DB Proxy servers. Odoo was not designed to work this way and Odoo's foremost Data Protection Specialist has already commented that there is a deliberate reason this is not in Odoo's design from the start.

To the question then. Is read/write query splitting, for Odoo, a safe/flawless thing to implement system wide regardless of the modules you are using? Yes or no, and why. (no opinions)

Michael M
  • 8,185
  • 2
  • 35
  • 51
  • Note .. [odony](https://stackoverflow.com/users/1398110/odony) has [commented on this topic elsewhere](https://bugs.launchpad.net/openobject-server/+bug/746620/comments/8) – Michael M May 04 '23 at 11:14
  • 1
    I'm not sure if this belongs here, because you could probably start a big discussion instead of getting any proper answer. But anyways an interesting question. – CZoellner May 04 '23 at 13:34
  • Once we are done taking the premise with a grain of salt, what is left to be answered? – jjanes May 04 '23 at 14:15

1 Answers1

1

a word about transactions first

As I tried to explain on that 12 year old post, the Odoo framework strongly relies on database transactions that have (at least) REPEATABLE READ guarantees [1], by design.

It would be perfectly valid to design a (non Odoo-based) system without the protection of isolated database transactions. For example if you're building a social media platform, you may not care if the number of likes on a post is 100% accurate to the individual click, and focus on other things.

However, the Odoo framework is designed to make writing business code as easy as possible, by automatically caring for all mundane tasks. For example, Odoo App developers are not supposed to write any DDL code to manage the database schema, and they're not supposed to manage database transactions. Except rare circumstances, they're not supposed to care or manage database transactions explicitly, nor detect any transactional conflict or handle them. It's all taken care of by the framework, in a generic way: it starts and stops transactions automatically, handles concurrency issues, retries transactions transparently, all without much intervention of the developer.

That's not a small thing for a framework to do, and it's very different from just focusing on a single application model, e.g. that social media site.

Now a consequence is that Odoo Apps do get written this way. You can achieve a great deal with a surprisingly small amount of Odoo code because of this - and I believe this to be a key factor in building a thriving app ecosystem.

read vs write load-balancing

There could be different ways to implement safe balancing of read and write transactions over database replicas while using the Odoo framework.

One way would be to flag URL/routes/endpoints as being read-only (RO) or read-write (RW), and route the database transaction to a read or write database node accordingly. You could even do that "naively" at the web proxy/lb level, and spin up different RO and RW Odoo servers, each connecting to their respective RO or RW database nodes. The difficult part here is categorising the routes, of course, as sending a RW query to a RO node would fail.

A second way could be to start all transactions as RO, and when a write operation happens (and thus fails), restart that entire transaction automatically on a RW node. That may seem wasteful, as each RW query would require up to 2x the time and resources. But then you could combine this with the first idea, and attempt to send RW queries directly to a RW node, only doing the replay when a RO transaction turns out being RW.

For the record, there are ongoing efforts internally at Odoo [2] to add support for such a read-only/read-write load-balancing system in the framework, to scale up large deployments.

All of this is possible while maintaining the safety and consistency of each Odoo transaction, even though there are inevitable challenges (like "eventual consistency" across transactions)

transaction isolation is essential

What definitely does not seem safe, would be to remove the guarantees of the REPEATABLE READ transaction isolation. Even temporarily. Doing so would remove fundamental guarantees that all developers have taken for granted when writing their Odoo Apps, and that the framework expects.

Of course, stripping those guarantees may appear to work, just like you can drive a car without a seatbelt. But there will be problems, and those problems may be hard to detect, like incorrect accounting or stock reservations, and it may lead to data corruption that is hard/impossible to fix.

There is no limit in terms of number of concurrent users that will lead to problems: as soon as more than 1 transaction is executing at the same time, isolation is needed, and the lack of isolation can lead to errors. Of course, the more concurrent transactions you have, the greater the risk of data consistency problems.

Those same problems can occur when delaying the start of a transaction, as one can imagine. It's even rather likely for a few reasons:

  • the Odoo framework maintains an in-transaction cache of data previously fetched, and tries to keep this cache in sync with what it believes to be the state of the database transaction
  • certain data queries cannot be answered with the in-transaction cache, so the data will be fetched from the database, and must return consistent results
  • most Odoo models use "computed fields", which like Excel formulas, are automatically maintained by the framework whenever a dependency changes. Those values are computed based on the contents of the in-transaction cache, so their correctness depends on the accuracy of those values at the time of computing.
  • as of Odoo 13.0, the ORM was further optimized to delay all write operations (and field recomputations) as late as possible into the transaction, by buffering them [3]. This means that by the time any write operations are being sent to the database, the transaction may be close to completion, and all values already computed in memory.
  • ...

So anyone who decides to remove the database transaction isolation that Odoo expects must do so with a very good understanding of the consequences, and be prepared to deal with those classes of issues.
That may work for some workloads, but it seems quite hard to do so in a generic manner that works with any Odoo App without risk of data corruption.

There are many blog articles and youtube videos giving examples of data corruption phenomenons that can occur when transaction properties are missing (non-repeatable reads, phantom reads, etc).


[1]: Database Isolation: https://en.wikipedia.org/wiki/Isolation_(database_systems)
[2]: See e.g. this draft work: https://github.com/odoo/odoo/pull/112000
[3]: The pull request implementing that ORM optimization: https://github.com/odoo/odoo/pull/35659

odony
  • 4,027
  • 17
  • 27
  • Good answer. Thank you. I think that instead of treating a database proxy as a magic bullet, or a turn key solution, one must realize that a little effort to [improve code or app design](https://stackoverflow.com/a/58503679/390722) will go a lot further. – Michael M May 07 '23 at 04:07
  • Yes, it's simple enough to make a direct query and even to call a complex Postgres function, by executing SQL directly to the DB. This would be a more intelligent way to resolve some slow reads, than to break Odoo like Heimdall could. https://www.youtube.com/watch?v=bAyWziRBERg – Mister Mister May 07 '23 at 04:27