Gsutil copy/move files in batches

Question

Is there a way to copy or move files using gsutil command in batches? For example If I want to copy 100 files from given folder to another.

Maxim · Answer 1 · 2019-11-01T09:35:34.103

1

Another way of doing it is using the Client libraries. For example in Python:

from google.cloud import storage

storage_client = storage.Client()

bucket_name = 'my_bucket'
bucket = storage_client.get_bucket(bucket_name)

blobs_to_move = [blob for blob in bucket.list_blobs(prefix="folder1/")]

with storage_client.batch():
    for blob in blobs_to_move[:100]:
        # copy to new destination
        new_blob = bucket.copy_blob(blob, bucket, "folder2/" + blob.name[8:])
        # delete in old destination
        blob.delete()

This will move the first 100 files from the folder1 in the GCS bucket my_bucket to folder2.

edited Nov 01 '19 at 09:35

answered Oct 31 '19 at 13:32

Maxim

4,075
1
14
23

Not exactly. What I am looking for is a way to copy only subset of the whole content. For example only 100 files out of 1000. – user101010 Oct 31 '19 at 13:39
Yes, randomly something that the find command does like here - find . -maxdepth 1 -type f |head -500|xargs – user101010 Oct 31 '19 at 14:18
You can generate a list of files on your own and pipe them into gsutil using the `-I` flag. – Travis Webb Oct 31 '19 at 16:27
@user101010 You can also do it using the Python client library. I've updated my answer. – Maxim Nov 01 '19 at 09:36

score 1 · Accepted Answer · answered Oct 31 '19 at 16:33

1

Try this:

gsutil ls gs://bucketA | head -n 100 | shuf | gsutil cp -m -I gs://bucketB

This will get a listing of files from bucketA, take the first 100 items, randomize them with shuf, and pipe them into gsutil to copy to bucketB. The -I flag reads the file list from stdin.

answered Oct 31 '19 at 16:33

Travis Webb

14,688
7
55
109

I think the -m flag needs to be before the cp command here. – dgketchum Sep 29 '20 at 20:02

score 0 · Answer 3 · edited Nov 20 '21 at 06:32

0

Slight modification so it will randomly move 100 files instead of first 100 files from bucketA to bucketB:

gsutil ls gs://bucketA | shuf | head -n 100 | gsutil -m mv  -I gs://bucketB

edited Nov 20 '21 at 06:32

ZygD

22,092
39
79
102

answered Nov 20 '21 at 03:19

user17139793

1

Gsutil copy/move files in batches

3 Answers3