sendfile64 only copy about 2GB

Question

I need to use sendfile64 to copy about 16GB of files. What I have achieved so far is

#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <fcntl.h>
#include <stdlib.h>
#include <sys/sendfile.h>
#include <sys/stat.h>

int main (int argc, char** argv)
{
  long long src;
  long long dest;
  struct stat64 stat_buf;
  off64_t offset = 0LL;
  long long rc;

  if (argc != 3) {
    fprintf(stderr, "usage: %s <source> <destination>\n", argv[0]);
    exit(1);
  }

  src = open64(argv[1], O_RDONLY);
  if (src == -1) {
    fprintf(stderr, "unable to open '%s': %s\n", argv[1], strerror(errno));
    exit(1);
  }

  fstat64(src, &stat_buf);

  dest = open64(argv[2], O_WRONLY|O_CREAT, stat_buf.st_mode);
  if (dest == -1) {
    fprintf(stderr, "unable to open '%s': %s\n", argv[2], strerror(errno));
    exit(1);
  }

 /* copy file using sendfile */
 rc = sendfile64 (dest, src, &offset, stat_buf.st_size);
 if (rc == -1) {
    fprintf(stderr, "error from sendfile: %s\n", strerror(errno));
    exit(1);
 }
 if (rc != stat_buf.st_size) {
   fprintf(stderr, "incomplete transfer from sendfile: %lld of %lld bytes\n",
           rc,
           (long long)stat_buf.st_size);
   exit(1);
 }

 /* clean up and exit */
 close(dest);
 close(src);

 return 0;
}

I have compiled using

g++ BigCopy2.cpp -o BigCopy2 -D_FILE_OFFSET_BITS=64 -DLARGEFILE64_SOURCE

The problem is I still can't copy more than 2GB of file.

Can someone point me where is my mistake?

Your casting of `stat_buf.st_size` in the `printf` call when the sizes doesn't match, seems to indicate that it's not a `long long` to begin with (even if it should be)? Could there be some problem there? Or is the cast because `stat_buf.st_size` is unsigned and you don't use an unsigned format code for `printf`? — Some programmer dude, Mar 13 '14 at 09:09
You've spelled `_LARGEFILE64_SOURCE` wrong, but `g++` enables that by default anyway. Is this a 32-bit or 64-bit program? I assume 32-bit, because `_FILE_OFFSET_BITS` and `_LARGEFILE64_SOURCE` are unnecessary on 64-bit — Jonathan Wakely, Mar 13 '14 at 09:16
I'm using 64 bit ubuntu 13.10 and thanks for pointing my typo. But still even if I remove it, I can only copy 2GB. I didn't use it at first. It's just I'm so desperate here:) — Ste, Mar 13 '14 at 09:29
If you're on 64-bit then `off_t` and `off64_t` are the same type, you should use the standard types and functions not the `xxx64` versions. If you're on 32-bit and you define `_FILE_OFFSET_BITS=64` then the same applies. — Jonathan Wakely, Mar 13 '14 at 09:35
Does it always stop at the same position or does the amount of transferred bytes change with each run? — scai, Mar 13 '14 at 09:36

nos · Answer 1 · 2014-03-13T10:21:46.880

You should use a loop to copy it all, sendfile() might, for various reasons, not copy all the data with one call. As janneb points out, the return value of sendfile64 is a ssize_t, so we should not pass in more than SSIZE_MAX to sendfile, moreover the last argument to sendfile is a size_t which would be 32 bit on 32 bit platforms.

 /* copy file using sendfile */
while (offset < stat_buf.st_size) {
  size_t count;
  off64_t remaining = stat_buf.st_size- offset;
  if (remaining > SSIZE_MAX)
      count = SSIZE_MAX;
   else 
      count = remaining;
  rc = sendfile64 (dest, src, &offset, count);
  if (rc == 0) {
     break;
  }
  if (rc == -1) {
     fprintf(stderr, "error from sendfile: %s\n", strerror(errno));
     exit(1);
  }
}

 if (offset != stat_buf.st_size) {
   fprintf(stderr, "incomplete transfer from sendfile: %lld of %lld bytes\n",
           rc,
           (long long)stat_buf.st_size);
   exit(1);
 }

Note that you can replace all your 64 bit variants, off64_t, stat64, sendfile64, with off_t, stat, sendfile. As long as you have the -D_FILE_OFFSET_BITS=64 flag, that define will do the right thing and transform off_t to off64_t, sendfile to sendfile64 and so on if those types and functions are not already 64 bits (such as on 32 bit architectures).

Not only is the last argument of type size_t, but the return value is of type ssize_t, which has a smaller max value than size_t. So you need to transfer at most SSIZE_MAX bytes at a time in order to properly track how much each call actually transferred. — janneb, Mar 13 '14 at 09:58
Yes, fair point - updated. At this point I'd rather use a normal, sane, easy to understand read()/write() loop. — nos, Mar 13 '14 at 10:04
Anyone tried this on MacOS? SendFile fails there and sendfile arguments are bit differently ordered so code will require bit of change. But in the end , Sendfile fails — Helena, Apr 14 '22 at 04:48

sendfile64 only copy about 2GB

1 Answers1

Linked