1

I have a client-side gui application which records data received periodically to a file.

There is one such file per component, and there can be ~400 components.

Limits:

I would have thought this would fit well within the soft limits, but it's hitting the limit:

$ grep "Limit\|files" /proc/7469/limits
Limit                     Soft Limit           Hard Limit           Units     
Max open files            1024                 4096                 files     

$ ls -l /proc/7469/fd | wc -l
1025

Showing the open file descriptors with lsof shows that in fact there are 8 file descriptors opened per std::ofstream, half of which seem to be related to gtk and Qt, and the other half directly from my app.

$ lsof | grep comp1.data
gui_app   7469      steve 746w REG 8,2 99024 1049235 /var/log/gui_app/comp1.data
gui_app   7469 7477 steve 746w REG 8,2 99008 1049235 /var/log/gui_app/comp1.data
QXcbEvent 7469 7480 steve 746w REG 8,2 99008 1049235 /var/log/gui_app/comp1.data
gdbus     7469 7481 steve 746w REG 8,2 99008 1049235 /var/log/gui_app/comp1.data
gmain     7469 7482 steve 746w REG 8,2 99008 1049235 /var/log/gui_app/comp1.data
gui_app   7469 7486 steve 746w REG 8,2 99008 1049235 /var/log/gui_app/comp1.data
QDBusConn 7469 7487 steve 746w REG 8,2 99008 1049235 /var/log/gui_app/comp1.data
gui_app   7469 7490 steve 746w REG 8,2 99024 1049235 /var/log/gui_app/comp1.data

Design:

There is only a single reference to each ofstream in the client code.

_ofs = std::make_unique<std::ofstream>(_filename, std::ofstream::out | std::ios_base::app | std::ios_base::binary);

I made the decision to keep each file descriptor open for the duration of the app, since I write to it every second, and made the (perhaps naive) assumption that opening/closing 400 file descriptors every second wasn't the best approach.

Questions:

  • What are all the extra file descriptors for?
  • Is there any way I can cut down on the number without closing/reopening the file descriptor every time I need it?
  • Is opening the fd, writing, closing the fd for 400 different files, each once per second, feasible?
Steve Lorimer
  • 27,059
  • 17
  • 118
  • 213
  • Did you allow for `std::cin` `std::cout`, `std::clog` `std::wcin`, `std::wcout` and `std::wclog`? – Galik Jun 09 '16 at 18:48
  • @Galik *per output file*? Sure I would expect the standard fds to be open, but this is happening for each file I open. 8 file descriptors for each and every `ofstream`, so 400 `ofstreams`, 3200 file descriptors - limit hit! – Steve Lorimer Jun 09 '16 at 18:50
  • 2
    Looks like it's one per thread, since the 3rd column of lsof's output has different values. Maybe open the file *after* spawning threads? Or do all those threads really need to share those file descriptors? – nephtes Jun 09 '16 at 18:54
  • @nephtes - ah yes - that must be it!! Only 1 thread needs the file, so yeah, I could try lazily creating the files, since at the moment they're created on startup. Thanks! – Steve Lorimer Jun 09 '16 at 19:06
  • I can't reproduce this, I get only one file descriptor per `std::ofstream`. Can you provide a minimal example code that produces the effect? – Galik Jun 09 '16 at 19:07
  • @Galik - I can try. I imagine it will involve a fair amount of effort unfortunately. – Steve Lorimer Jun 09 '16 at 19:10
  • @nephtes, according to [this SO answer](http://stackoverflow.com/a/6223808/955273) file descriptors are shared – Steve Lorimer Jun 09 '16 at 19:11
  • @SteveLorimier: Hmmm... good point, and it makes me wonder... are you in fact certain that those are 8 separate file descriptors, or is lsof just reporting one record for each thread? Looking at the 5th column, the FD is 746 everywhere. – nephtes Jun 09 '16 at 19:17
  • 2
    @nephtes yeah, maybe it's that. Red Herring! – Steve Lorimer Jun 09 '16 at 19:19

0 Answers0