3

I'm working on an ANSI C application that produces file contents in reverse order. That is, the bytes at the end of the file are received first, and those at the beginning are received last. Preferably, due to the amounts of data that may be involved, I would like to write this data directly to the file without first arranging it in a separate memory buffer. Is that possible? How may this be accomplished using ANSI C? If this can only be done with a higher level library that is not ANSI compliant, that would acceptable.

Jim Fell
  • 13,750
  • 36
  • 127
  • 202
  • Very interesting question! A workaround would be to write a temp file in reverse-order, then read the temp file and rewrite it in forwards order. Of course this just takes the memory buffer you're talking about and uses the disk for it instead. Curious to see solutions for this one. – Maximillian Laumeister Aug 16 '15 at 22:17
  • First idea springing to my mind would be to write data to a temp file in "reverse order" first, then moving from back to front with an adequate blocksize reverse to the destination file. – BitTickler Aug 16 '15 at 22:17
  • Do you know beforehand how much data you will need to write? – Tordek Aug 16 '15 at 22:19
  • I hadn't thought of using a temporary file. While it's not exactly what I had in mind, it's certainly better than using a memory buffer. The data my application needs to be able to process could easily outstrip the memory available on most systems. – Jim Fell Aug 16 '15 at 22:20
  • @Tordek No, only that the application is intended to be able to handle a very large amount of data, more than would be available on most systems. – Jim Fell Aug 16 '15 at 22:21
  • Second idea springing to my mind is, to live with the "wrong order" and cope with this fact on the side which uses the file. That would work if the application using the data is under your control (you write that one, too). – BitTickler Aug 16 '15 at 22:21
  • @BitTickler Unfortunately, what I'm asking about in this post is the final output file that is intended to be the "deliverable" of the application. What happens to it is outside my control. – Jim Fell Aug 16 '15 at 22:23
  • Yeah, in that case it seems your only choice is to save-then-reverse; most filesystems are not designed to allow prepending data. – Tordek Aug 16 '15 at 22:24
  • 2
    not ANSI, but the simple way for anything like this is to memory map the file, so you can write in any order you like, mmap on a nix system, truncating when done. – camelccc Aug 16 '15 at 22:25
  • 1
    Oh my I will stop this after offering my third idea (spring tired :) ): Use a block-oriented file format for your raw data in the form: ``{BlockSize : uint32; ReversedFlag: uint8_t;}`` followed by ``BlockSize`` bytes of data. And then you would have the option to "lazily" reverse if your consuming application is not accessing sequentially but in a more random-access-ish manner. @JimFell Sorry - race condition to your reply to my second one :( – BitTickler Aug 16 '15 at 22:26
  • @camelccc idea good. Check out ``CreateFileMapping()`` if you happen to be on Windows. Its the equivalent of mmap() function on *nix. – BitTickler Aug 16 '15 at 22:31
  • Thanks for the great suggestions everyone! I think there may be a way for my application to determine the size of the file, after the first set is received. Then, I think the whole (empty) file can be created on disk, and back-filled as the data becomes available by alternatively setting the file pointer and writing the available data to the file. – Jim Fell Aug 16 '15 at 22:33
  • @JimFell I doubt writing byte-wise and seeking byte-wise positions will result in effective code. Given the expected huge amount of data I would rather use the mmap() idea or declare the reversal as a final post processing step. – BitTickler Aug 16 '15 at 22:34
  • @BitTickler I'm going to give the mapped file option a try. Thanks, everyone! – Jim Fell Aug 16 '15 at 22:54

1 Answers1

0

The solution to this was actually much simpler than I had originally thought. I was able to use fseek to move the file pointer to the end of the file and then incrementally move it backward (decrement the file pointer index) through the file for each write.

Jim Fell
  • 13,750
  • 36
  • 127
  • 202