0

Have the following bash codes to deal with filenames passed randomly with mapfile. I want to call rsync on each file and send to a destination path dpath.

mapfile -d '' fl < <(
  find "$dpath" -maxdepth 1 -type f "${inclnm[@]}" |
  shuf -z -n "$nf"
)

Or with shuf working on arguments directly

mapfile -d '' fl < <( shuf -z -n "$nf" -e "${inclnm[@]}" )

How can I modify the two alternatives to run rsync on each file and send to destination?

Nigel
  • 39
  • 4
  • 2
    It's really not clear why you need `mapfile` here. Is your actual question "how can I send files to `rsync` in random order?" What is the role of the other variables you use but never define? – tripleee Sep 04 '21 at 19:51
  • @tripleee `mapfile` is an alternative to `while read`, the latter is straightforward to add an `rsync` statement. Bet with `mapfile` I am struggling. – Nigel Sep 04 '21 at 20:11
  • `find ... | shuf ... | xargs -I {} rsync ... {} ...`? – Cyrus Sep 04 '21 at 20:30
  • @Cyrus Yes, that centainly another way. – Nigel Sep 04 '21 at 21:03
  • 3
    `rsync on each file` How would you use `rsync` with one file or without `mapfile`? Please show an example. I would definitely use `rsync --from0 --files-from=-` instead of arrays anyway... `shuf -z` why would you shuf the files? It will make `rsync` less efficient, instead sort them. – KamilCuk Sep 04 '21 at 21:46
  • @KamilCuk he wants to remotely synchronize a random selection of files instead of all files, probably to minimize some load. Now selecting files randomly could be improved by selecting within files that need to be synchronized only. – Léa Gris Sep 04 '21 at 22:33

1 Answers1

5

As said in comments, you don't need to mapfile an intermediate array. Just stream the null delimited selection of files to rsync like this:

#!/usr/bin/env bash

nf=4
inclnm=( a* b* )
# For testing purpose, destination is local host destfolder inside
# user home directory
destination="$USER@localhost:destfolder"

# Pipe the null delimited shuffled selection of files into rsync 
shuf -z -n "$nf" -e "${inclnm[@]}" |
# rsync reads the null-delimited selection of from files from standard input
rsync -a -0 --files-from=- . "$destination"

If you want to collect the random selection of files and use it with rsync then do:

#!/usr/bin/env bash

nf=4
inclnm=( a* b* )
# For testing purpose, destination is local host destfolder inside
# user home directory
destination="$USER@localhost:destfolder"

# Capture the selection of files into the fl array
mapfile -d '' fl < <( shuf -z -n "$nf" -e "${inclnm[@]}" )

# Pass the fl array elements as sources to the rsync command
rsync -a "${fl[@]}" "$destination"
Léa Gris
  • 17,497
  • 4
  • 32
  • 41
  • I see. With `mapfile` I can store the results in tho variable `fl`, but I can simply pipe to `rsync`, as you say. – Nigel Sep 04 '21 at 23:14
  • @Nigel I added an alternative method with capturing the random files selection into the array with `mapfile`, and using the array with `rsync`. – Léa Gris Sep 04 '21 at 23:28
  • Are there any advantages using `mapfile`, rather than simply using `fl=$( ,,, )`. – Nigel Sep 04 '21 at 23:29
  • @Nigel the other valid alternative to `mapfile` here is reading the file records in a loop and adding them one by one to the array. `mapfile` does it all at once. `using fl=$( ,,, )` will capture the output of the command as a string and not map the command output as individual elements. Splitting the command output string into array elements with the IFS delimiter and globbing is an invalid bad practice and fail in many different scenaries. – Léa Gris Sep 04 '21 at 23:37
  • Correct, tho target of `mapfile` is an array, – Nigel Sep 04 '21 at 23:55