3

[Sorry for the confusion: The original post had "SSD" instead of "HDD" in the title, but I figured out that I performed the tests on an HDD by accident, as I was accessing the wrong mounting point. On an SSD this phenomenon did not occur. Still interesting that it happens for an HDD though.]

I am using the following code for reading in a loop from a given number of files of constant size. All files to be read exist and reading is successful.

It is clear that varying the file size has an effect on fMBPerSecond, because when reading files smaller than the page size, still the whole page is read. However, nNumberOfFiles has an effect on fMBPerSecond as well, which is what I do not understand. Apparently, it is not nNumberOfFiles itself that has an effect, but it is the product nNumberOfFiles * nFileSize.

But why should it have an effect? The files are opened/read/closed sequentially in a loop.

I tested with nFileSize = 65536. When choosing nNumberOfFiles = 10000 (or smaller) I get something around fMBPerSecond = 500 MB/s. With nNumberOfFiles = 20000 I get something around fMBPerSecond = 100 MB/s, which is a dramatic loss in performance.

Oh, and I should mention that I clear the disk cache before reading by executing:

sudo sync
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

Any ideas what is happening here behind the scenes would be welcome.

Pedram

void Read(int nFileSize, int nNumberOfFiles)
{
    char szFilePath[4096];

    unsigned char *pBuffer = new unsigned char[nFileSize];

    Helpers::get_timer_value(true);

    for (int i = 0; i < nNumberOfFiles; i++)
    {
        sprintf(szFilePath, "files/test_file_%.4i", i);

        int f = open(szFilePath, O_RDONLY);

        if (f)
        {
            if (read(f, pBuffer, (ssize_t) nFileSize) != (ssize_t) nFileSize)
                printf("error: could not read file '%s'\n", szFilePath);

            close(f);
        }
        else
        {
            printf("error: could not open file for reading '%s'\n", szFilePath);
        }
    }

    const unsigned int t = Helpers::get_timer_value();

    const float fMiliseconds = float(t) / 1000.0f;
    const float fMilisecondsPerFile = fMiliseconds / float(nNumberOfFiles);
    const float fBytesPerSecond = 1000.0f / fMilisecondsPerFile * float(nFileSize);
    const float fMBPerSecond = fBytesPerSecond / 1024.0f / 1024.0f;

    printf("t = %.8f ms / %.8i bytes - %.8f MB/s\n", fMilisecondsPerFile,
        nFileSize, fMBPerSecond);

    delete [] pBuffer;
}
Pedro
  • 842
  • 6
  • 16
  • is there any reason to mark it c++, or even any language? – user1810087 Dec 18 '13 at 18:36
  • Is this really a programming question? – undur_gongor Dec 18 '13 at 18:39
  • 3
    This question appears to be off-topic because it is about hardware performance and seems to be unrelated to programming. – Jim Garrison Dec 18 '13 at 18:40
  • Try shuffling the indices of the files you access. Perhaps the first 10000 or whatever are laid out more optimally on the disk. – MischaNix Dec 18 '13 at 18:41
  • I thought it is relevant that it is somewhat low-level access and not an interpreted language such as Java, plus the example code is C/C++. If the tags C and C++ are misleading or not useful, I can remove them though. – Pedro Dec 18 '13 at 18:41
  • There's some latency in opening/closing files, no? – Fiddling Bits Dec 18 '13 at 18:42
  • @alk Only in C's dreams. ;-) – Fiddling Bits Dec 18 '13 at 18:43
  • I will try to shuffle the indices, but I don't think it will have an effect, because when I half the number of files, but double the file size, the phenomenon is the same, which is why I said I it seems like the product, i.e. the total number of bytes, is relevant. – Pedro Dec 18 '13 at 18:44
  • 1
    I know that syntactically it is not C. But the file access is C, which is more relevant than the syntax in this context, I guess. – Pedro Dec 18 '13 at 18:46
  • Perhaps you're exceeding an SSD-internal cache space then. – Mark B Dec 18 '13 at 18:47
  • It is an addition, but with system specific coefficients: `K1 * number_of_write + K2 * number_of_files`. `K1` and `K2` describe the impact in terms of delay imposed per operation. – alk Dec 18 '13 at 18:47
  • It would be nice if someone has an idea, if this is more related to the SSD, or to the controller, or to the operating system. – Pedro Dec 18 '13 at 18:47
  • Another possibility is that cache is still a concern--the kernel may start evicting cached pages beyond a certain number of bytes accessed, reducing performance. – MischaNix Dec 18 '13 at 18:47
  • @alk: Good point. I will look into that. – Pedro Dec 18 '13 at 18:49
  • @MischNix: Good point, too. Thank you everybody for your comments. – Pedro Dec 18 '13 at 18:50
  • @alk: When I close the file, can I assume that operations on that file are completed fully, or can there be some pending internal stuff that is executed by the OS asynchronously? – Pedro Dec 18 '13 at 18:59
  • Is the timing the same if you don't do the `sync; echo 3 > /proc/sys/vm/drop_caches` beforehand? – James Waldby - jwpat7 Dec 18 '13 at 19:11
  • @jwpat7: When I remove that, then it runs at 3000 MB/s, no matter if 10000 oder 20000 files. – Pedro Dec 18 '13 at 23:50
  • POSIX defines I/O using `open()/read()/write()/close()` to be unbuffered, so from the processes' perspective data had been written when `close()` returns. What the lower levels **really** do is unknown. – alk Dec 20 '13 at 07:48

1 Answers1

1

There are several SSD models, especially the more expensive datacenter models, that combine an internal DRAM cache with the (slower) persistent NAND cells. As long as the data you read fits in the DRAM cache, you'll get faster response.

Guntram Blohm
  • 9,667
  • 2
  • 24
  • 31