33

Is it possible to know the progress of file transfer with kubectl cp for Google Cloud?

akshitsethi
  • 546
  • 1
  • 7
  • 12
  • One alternative I recently discovered...Google Cloud Shell shows file upload progress if that fits into your use case. – James Jul 09 '20 at 14:21

6 Answers6

29

No, this doesn't appear to be possible.

kubectl cp appears to be implemented by doing the equivalent of

kubectl exec podname -c containername \
  tar cf - /whatever/path \
| tar xf -

This means two things:

  1. tar(1) doesn't print any useful progress information. (You could in principle add a v flag to print out each file name as it goes by to stderr, but that won't tell you how many files in total there are or how large they are.) So kubectl cp as implemented doesn't have any way to get this out.

  2. There's not a richer native Kubernetes API to copy files.

If moving files in and out of containers is a key use case for you, it will probably be easier to build, test, and run by adding a simple HTTP service. You can then rely on things like the HTTP Content-Length: header for progress metering.

David Maze
  • 130,717
  • 29
  • 175
  • 215
17

One option is to use pv which will show time elapsed, data transferred and throughput (eg MB/s):

$ kubectl exec podname -c containername -- tar cf - /whatever/path | pv | tar xf -

14.1MB 0:00:10 [1.55MB/s] [       <=>                                   ]

If you know the expected transfer size ahead of time you can also pass this to pv and it will then calculate a % progress and also an ETA, eg for a 100m transfer:

$ kubectl exec podname -c containername -- tar cf - /whatever/path | pv -s 100m | tar xf -

13.4MB 0:00:09 [1.91MB/s] [==>                          ] 13% ETA 0:00:58

You obviously need to have pv installed (locally) for any of the above to work.

Freshleaf Media
  • 382
  • 4
  • 10
  • This is brilliant and works as advertised. The example copies from a remote container to your local (which has kubectl). How might the inverse operation work? From local to a remote container (which may not have kubectl)? – Chris Beck Oct 09 '20 at 23:11
  • 2
    @ChrisBeck I don't believe there is a way to do the copy, in either direction, without kubectl. Copying from local to remote is simple enough though: `$ tar cf - /whatever/path | pv -s 100m | kubectl exec -i podname -c containername tar xf -` note the addition of the `-i` argument passed to `kubectl exec` – Freshleaf Media Oct 12 '20 at 09:34
  • 1
    More recent `kubectls` prefer `--` as in `kubectl exec podname -c containername -- tar cf - /whatever/path | pv | tar xf -` – Chris Beck Oct 15 '20 at 00:21
  • This also worked for me `cat local_file | pv | kubectl exec -i podname -c containername -- sh -c 'cat > remote_file'` – dan May 06 '21 at 20:34
  • If `tar` is not available (required for `kubectl cp`)on the pod, you can use this technique to copy but you don't get the progress indicator: https://fabianlee.org/2022/09/10/kubernetes-copying-files-into-and-out-of-containers-without-kubectl-cp/#:~:text=The%20'kubectl%20cp'%20command%20is,be%20installed%20inside%20the%20container. – Chris Beck Apr 12 '23 at 16:52
4

I figured out a hacky way to do this. If you have bash access to the container you're copying to, you can do something like wc -c <file> on the remote, then compare that to the size locally. du -h <file> is another option, which gives human-readable output so it may be better

Marcus Buffett
  • 1,289
  • 1
  • 14
  • 32
3

It's not possible, but you can find here how to implement rsync with kubernetes, rsync shows you the progress of the transfer file. rsync files to a kubernetes pod

javierlga
  • 1,409
  • 9
  • 14
3

On MacOS, there is still the hacky way of opening the "Activity Monitor" on the "Network" tab. If you are copying with kubectl cp from your local machine to a distant pod, then the total transfer is shown in the "Sent Bytes" column. Not of super high precision, but it sort of does the job without installing anything new.

enter image description here

Jonathan Lurie
  • 61
  • 1
  • 3
2

I know it doesn't show an active progress of each file, but does output a status including byte count for each completed file, which for multiple files run via scripts, is almost as good as active progress:

kubectl cp local.file container:/path/on/container --v=4

Notice the --v=4 is verbose mode and will give you output. I found kubectl cp output shows from v=3 thru v=5.

Andrew
  • 906
  • 7
  • 9