Very slow to execute system command

Question

I have a C++ program that records a lot of data to disk continuously for a long time. As such I have a thread that monitors the disk space available and once it hits a certain percentage does some stuff.

This is on a dual quad core x64 CentOS system and the recording is happening on directly connected SATA disks that are used solely for the recording with ext3 filesystem. I am monitoring the disk usage by issuing a "df" command using system() and reading in the result.

Whilst running it last night I noticed in the log files that it took a full 39 minutes to run the command to find the disk usage.

The code that handles the time out is this:

int DiskSpaceMonitor::handle_timeout(const ACE_Time_Value& time_, const void* pFunc_)
{
    LOG4CXX_TRACE(m_logger, "DiskSpaceMonitor timer fired");

    ACE_UINT8 usagePercent = m_diskChecker.getDiskSpaceUsagePercentage(m_monitoredDisk);

    m_fileRecorder->notifyDiskUsage(usagePercent);

    return 0;
}

Which calls this function that does the "df":

ACE_UINT8 DiskSpaceChecker::getDiskSpaceUsagePercentage(std::string diskMountPoint)
{
    std::stringstream usageCommand;
    usageCommand << "df -PH " << diskMountPoint << " | grep -v \"^Filesystem\" | awk '{print $5}' | cut -d'%' -f1 > " << m_mountSpaceFile;
    system(usageCommand.str().c_str());

    std::ifstream inFile(m_mountSpaceFile.c_str(), std::ios::in);
    if (!inFile)
    {
        return 0;
    }

    std::string usageStr;
    inFile >> usageStr;
    int usage = atoi(usageStr.c_str());

    inFile.close();

    std::stringstream rmCmd;
    rmCmd << "rm " << m_mountSpaceFile;
    system(rmCmd.str().c_str());

    LOG4CXX_DEBUG(m_logger, "Disk usage for disk: " << diskMountPoint << " = " << usage << "%");

    return  usage;
}

So between the trace logging statement in handle_timeout() and the debug trace statement in getDiskSpaceUsagePercentage() it took 39 minutes. But the delay really came before the inFile >> usageStr; (because I can see that the read percentage was higher than expected - it should have gone up 1% or less but it jumped more than 16%).

Why the hell should the processing to run the command and read it in take such a huge amount of time?

Now I admit that the disks do get a bit of a hammering whilst they are being written to, but there is only one program writing to them and it is only writing one data file and one index file. So I don't see how this should take so long.

As an alternative is there an easy way to call a system() function and have it return after a timeout period if it is taking too long?

It would be interesting to know what the iowait state was like during this slow response. — j.w.r, Jun 17 '11 at 08:05

score 2 · Accepted Answer · answered Jun 17 '11 at 08:02

2

An alternative to using system is an OS-specific API call, like statvfs.

But I agree it's weird that it's taking that long. Can you recreate this?

answered Jun 17 '11 at 08:02

Assaf Lavie

73,079
34
148
203

Thanks I will give statvfs a go. My testing runs for several hours and I've only seen it once so recreating it is not easy. but I will try doing so. – fwg Jun 17 '11 at 08:34

score 1 · Answer 2 · answered Jun 17 '11 at 08:08

1

I've seen 'df' hang when some NFS fs is mounted but the client cannot access the server. So do you have any network fs mounted?

In any case, as others have already pointed out, you should use statvfs()/fstatvfs() instead of system(). That might also help with the hanging in case the hung fs is not the one you're trying to run statvfs() on.

answered Jun 17 '11 at 08:08

janneb

36,249
2
81
97

There are NFS mounts mounted by the server but they are not used in this recording process or by the `df` command. Perhaps if they go bad the `df` does hang up though. I'll give statvfs() a go and see if that improves things. Cheers. – fwg Jun 17 '11 at 08:51
Yes, df hangs if any mounted fs is hung, as what df essentially does is that it runs statvfs() for every mounted fs. – janneb Jun 17 '11 at 09:33

score 0 · Answer 3 · answered Jun 17 '11 at 08:04

0

Maybe you could try using the fstatvfs()/statvfs() system calls to get the amount of free disk space instead, and see if that makes any difference.

answered Jun 17 '11 at 08:04

iceaway

1,204
1
8
12

Very slow to execute system command

3 Answers3