-1

Our (embedded) Linux system has an ext4 file system. Now, one of our apps there needs to modify data files using simple file write APIs. The requirement there is that the file updates should be atomic - not in the sense of parallel writes from different apps (we don't have that), but in the sense that each write can't be partially executed in case of a power failure - it can either be fully executed or not. Is this guaranteed? I'm aware of the fact that file writes may not be executed immediately due to caching, but I'm not sure whether these writes can be split by the cache in a way they may become partial, hence my question.

I can alternatively use a copy-write-rename method to copy the original file to a temporary one, make the changes there and then rename the file back to the original one, counting on the atomic nature of the rename operation. But even then I'm not sure that these operation are guaranteed to be ordered the way I want (especially the write and rename).

dsaada
  • 37
  • 6
  • What happens on data loss. Is the embedded system life critical (e.g. some hospital device?) – Basile Starynkevitch Nov 01 '21 at 14:23
  • What will happen (in the real world) when data is lost? Is you embedded system a consumer vacuum cleaner (loss of data is annoying, but not important) or a neurosurgical robot ((loss of data kills people)? – Basile Starynkevitch Nov 01 '21 at 15:37
  • Data loss is not life critical there. Product is sort of a router. – dsaada Nov 01 '21 at 16:30
  • Please *edit your question* to improve it. I feel that my answer is relevant. Feel free to upvote it. But you want a guarantee, and you cannot have it. Discuss with your boss or client. Consider improving the hardware or lowering the specifications. – Basile Starynkevitch Nov 01 '21 at 16:32
  • Sorry if my question wasn't clear enough, will edit it. I don't need to guarantee the completeness of writes, I just want to understand whether there can be a case it can be partially completed. If there is such a case, then it is one way to design my app. If not, it is another way. – dsaada Nov 01 '21 at 16:59
  • 1
    AFAIK yes, writes can be non-atomic. For instance, imagine that your write spans two pages in the page cache. It would be unreasonable to expect the kernel to remember that they have to be flushed together, or in a particular order. Flushing together may even be impossible, if the hardware can only write out one page or sector at a time. Pretty much the only way to get any certainty about what's on the disk is to use `fsync`. – Nate Eldredge Nov 01 '21 at 22:28
  • Thanks @NateEldredge. This makes sense. – dsaada Nov 03 '21 at 09:03

1 Answers1

1

A possibly might be to use (in your user-mode application) the sync(2) system call. Before that, use fflush(3) if using stdio

To ensure atomicity, you may need to check a lot of code (perhaps even inside the kernel) with static analysis tools like Frama-C, Bismon, or the DECODER project. Of course, this is very costly (above 100k€ or US$ in 2021). Feel free to contact me by email about them. Be aware of Rice's theorem.

At the kernel (or hardware) level, atomicity cannot be guaranteed: for example, a successful write(2) system call of four megabytes (by your application) is very probably involving (on the SATA cable to your hard disk) many frames or packets. If power is lost, data will be lost.

Don't forget that the Linux kernel and GNU libc are open source. You are allowed to study their source code and improve them.

Consider also a hardware approach : adding some UPS.

Another possibility is to extend your C compiler, e.g. coding your GCC plugin, to semi-automatically add calls to sync(2)

Yet another possibility is to generate your C code (e.g. with RefPerSys or GPP or your own C code generator). Jacques Pitrat's last book Artificial Beings, the conscience of a conscious machine explain in details how to do so.

See also my sync-periodically.c program (GPLv3+ licensed; so no warranty).

You could also improve some open source compiler generating C (like Bigloo) to emit at suitable places calls to sync(2).

PS. Things are more complex if your embedded software is multi-threaded (using several pthreads or processes), or if your hardware has several disks or SSD, or is in space (cosmic rays?) or inside a nuclear power station (radioactivity?)

Basile Starynkevitch
  • 223,805
  • 18
  • 296
  • 547
  • Thanks for all the suggestions, however that wasn't my point. Even if I sync/flush automatically (and I could do it in my app anyway after writing), this could still be interrupted by a power failure. I'm OK with that. Only thing I needed to know was whether I could ensure that writes were either fully committed or not, but not partially. HW UPS is not an option (it's not our HW). Multi-threading is not an issue here as well. – dsaada Nov 01 '21 at 16:39