0

URL return list of files separated by comma example

URL : http://sometimesdownload.com/list_of_files

To get list of files

curl -0 http://sometimesdownload.com/list_of_files

Output of above curl command/List returned :

[" file1.csv","file2.csv","file3.csv"...."filen.csv"]

How to download all files in a single run to a specified path in /tmp/downloaded_files directory in linux and delete the file after downloading. File name of files should be same as original files.

CURL GET to get the files, Download the files and then delete the files after downloading using CURL DELETE.

File can be downloaded like this :

http://sometimesdownload.com/list_of_files/file_name

Example:

curl -o http://sometimesdownload.com/list_of_files/file1.csv

curl -o http://sometimesdownload.com/list_of_files/file2.csv

curl -o http://sometimesdownload.com/list_of_files/file3.csv

Thanks

Kamal Chopra
  • 1
  • 1

3 Answers3

0

You can do this in Java without curl, but if you want to use curl, you're going to want to use either Runnable.exec or ProcessBuilder. See pass multiple parameters to ProcessBuilder with a space for an example

ControlAltDel
  • 33,923
  • 10
  • 53
  • 80
0

This could be done as something like

for i in $(curl http://sometimesdownload.com/list_of_files | jq -c '.[]'); then
    # GET file
    curl -O --output-dir /tmp/downloaded_files http://sometimesdownload.com/"$i"

    # DELETE file
    # However this is mostly not working straightforward and need tuning 
    # as no enough information here about how the files are hosted and 
    # configured by the remote server 
    curl -X DELETE http://sometimesdownload.com/"$i"
done

-c is the compact output option in jq to get the plan text output from a json array.

$ echo '["file1.csv","file2.csv","file3.csv"]' | jq -c '.[]'
"file1.csv"
"file2.csv"
"file3.csv"
user20126052
  • 11
  • 2
0

You can use below script. Since list syntax is different in BASH, you have to convert to bash syntax. [abc,bca] to (abc bca)

And then you can go throw files one by one and download via CURL.

Here is example script, I believe you can correct it according your env.

bashParse() {
    listData=$1
    #validateData
    finalData=()
    echo ${listData} | grep ^"\[" > /dev/null && beginScope=0 || beginScope=1
    echo ${listData} | grep "\]$" > /dev/null && endScope=0   || endScope=1
       
    # build our List BASH stntax    
    if [ $beginScope -eq 0 ] && [ $endScope -eq 0 ] ; then
        # Here I will remove [ and ], also replace last SPACE with nothing => sed
        declare -a finalData=($(echo $listData|tr '\"' ' '| tr -d '[]'| tr ',' ' '))
    fi
    
    # Download files one by one by CURL
    for i in "${finalData[@]}" ; do
       echo "curl://<link>/$i"
    done

}
aze2201
  • 453
  • 5
  • 12