5

It have often noticed that I would have been able to solve practical problems in C elegantly if there had been a way of creating a ‘virtual FILE’ and attaching the necessary callbacks for events such as buffer full, input requested, close, flush. It should then be possible to use a large part of the stdio.h functions, e.g. fprintf unchanged. Is there a framework enabling one to do this? If not, is it feasible with a moderate amount of effort, on at least some platforms?

Possible applications would be:

  • To write to or read from a dynamic or static region of memory.
  • To write to multiple files in parallel.
  • To read from a thread or co-routine generating data.
  • To apply a filter to another (virtual or real) FILE.
  • Support for file formats with indirection (like #include).
  • A C pre-processor(?).

I am less interested in solutions for specific cases than in a framework to let you roll your own FILE. I am also not looking for a virtual filesystem, but rather virtual FILE*s that I can pass to the CRT.

To my disappointment I have never seen anything of the sort; as far as I can see C11 considers FILE entirely up to the language implementer, which is perhaps reasonable if one wishes to keep the language (+library) specifications small but sad if you compare it with Java I/O streams.

I feel sure that virtual FILEs must be possible with any (fully) open source implementation of the C run-time, but I imagine there might be a large number of details making it trickier than it seems, and if it has already been done it would be a shame to reduplicate the effort. It would also be greatly preferable not to have to modify the CRT code. Without open source one might be able to reverse engineer the functions supplied, but I fear the result would be far too vulnerable to changes in unsupported features, unless there were a commitment to a set of interfaces. I suppose too that any system for which one can write a device driver would allow one to create a virtual device, but I suspect that of being unnecessarily low-level and of requiring one to write privileged code.

I have to admit that while I have code that would have benefited from virtual FILEs, I have no current requirement for it; nonetheless it is something I have often wondered about and that I imagine could be of interest to others.

This is somewhat similar to a-reader-interface-that-consumes-files-and-char-in-c, but there the questioner did not hope to return a virtual FILE; the answer, however, using fmemopen, did.

Community
  • 1
  • 1
PJTraill
  • 1,353
  • 12
  • 30
  • Did you look at [libfuse](http://fuse.sourceforge.net/) ? It allow you to implement your custom filesystem through callbacks (ioctl, read, stat, write,...) – mpromonet Jun 11 '15 at 22:00
  • @mpromonet: No, I am not familiar with libfuse, but your description sounds promising … but when I look I see it sounds a bit different: I want a virtual (C) `FILE` that I can pass to e.g. `fprintf`, not a virtual file (on a storage medium). But I imagine libfuse might use relevant techniques. – PJTraill Jun 11 '15 at 22:08
  • C is *intended* to be designed without I/O. all the I/O is tucked away in libraries. That makes C portable. What a file in C? possibly an array of bytes; but maybe seekable. Semantics may creep in, but only at the semantics level. – wildplasser Jun 11 '15 at 22:21
  • @wild: are you claiming that what I am inquiring after is undesirable? It would also be a matter of library functions with appropriate interfaces, and have no impact on the language. Or am I missing your point? – PJTraill Jun 11 '15 at 22:26
  • yes, you are missing my point. – wildplasser Jun 11 '15 at 22:35
  • This depends very much on the given `libc` implementation, but I wouldn't be surprised if it was possible to do for all of them. However, each one could end up requiring a different approach, and I'm almost certain you'd need to `-include` your header first. Still, this would be a fun and useful project. Go for it! :) – mtijanic Jun 11 '15 at 22:55
  • Another thing to note is that `tmpfile` is allowed not to create an actual file on disk, and just use a buffer. You can't really know whether it will or not, though. You'd still need to read data out of it using `fread`, though. What platforms do you need to support? – mtijanic Jun 11 '15 at 23:05
  • You could consider moving to C++. The `iostream` library doesn't suffer from this problem; people almost have to go out of their way to make I/O code that you can't use with your custom `streambuf` objects. –  Jun 12 '15 at 02:29
  • @mtijanic: where do you get your proscription on '`tmpfile` is not allowed to create an actual file on disk' from? – Jonathan Leffler Jun 12 '15 at 04:46
  • @JonathanLeffler sorry, poorly worded. `tmpfile` may create an actual file, but it doesn't have to. Since a standard conforming C program can't tell the difference (other than performance), all a libc has to do is offer the same interface as an actual file, whatever the underlying implementation. – mtijanic Jun 12 '15 at 10:13

1 Answers1

7

There is no standard C interface for creating virtual FILE*s, but both the GNU and the BSD standard libraries include one. On linux (glibc), you can use fopencookie; on most *BSD systems, funopen (including Mac OS X). (See Note 1)

The two interfaces are similar but slightly different in some details. However, it is usually very simple to adapt code written for one interface to the other.

These are not complete virtualizations. They associated the FILE* with four callbacks and a void* context (the "cookie" in fopencookie). The callbacks are read, write, seek and close; there are no callbacks for flush or tell operations. Still, this is sufficient for many simple FILE* adaptors.

For a simple example, see the two answers to Write simultaneousely to two streams.

Notes:

  1. funopen is derived from "functional open", not from "file unopen".
Community
  • 1
  • 1
rici
  • 234,347
  • 28
  • 237
  • 341
  • Interesting. Thanks! – Jonathan Leffler Jun 12 '15 at 04:44
  • Thank you very much, `fopencookie` sounds like exactly what I mean; the `funopen` link timed out (may be my VPN). I shall take a closer look and give it some time, but I suspect I shall accept this answer. It sounds to me like something that belongs in the standard C11⁺ⁿ library! – PJTraill Jun 12 '15 at 09:42
  • 1
    @PJTraill: Added an apple.com link for man funopen in case your VPN prefers that. IMO, if it were to be added to C or even to Posix, the interface would need to be made more complete. – rici Jun 12 '15 at 22:26
  • Thanks, I have now found `funopen`. Interesting remark about CRT/Posix: I suppose you refer to the `flush`/`tell` operations; do you think this is too much a matter of opinion to include in your answer? P.S. I prefer the `struct` approach of `fopencookie` – the more operations there are the better this looks – though not really the name, not that `funopen` is ideal either. – PJTraill Jun 13 '15 at 10:41
  • @PJTraill: Yes, I don't think SO is an appropriate forum for suggestions about future C/Posix/... standards. But my opinion is that the handling in general of file offsets is a bit of a camel, and virtual interfaces only emphasize the issues. Also, flush is essential for a robust virtualization. But if you are looking for a usable interfaces, both APIs are usable and they may well be sufficient for the next problem you have which could benefit from virtual interfaces. – rici Jun 13 '15 at 21:01