I am using python-swiftclient 2.6.0 version 1.60 on one of my amazon cloud. I was trying to download the container from other cloud and I am using swift command line library to do that. However script fails in the middle throwing swit client JSONDecodeError. I was trying to find the root cause. Due to script fail. I have to download data again. Is there any way to ignore the error and makes an log file for future purpose and continue downloading.
I am using following command :
swift -A https://127.0.0.1:443/auth/v1.0 -U swiftops:swiftops -K download container_name -m container_name
And also please let us know is there anyway to start downloading again from the previous stopping point?
After searching long time I am ending up with the following snippet. Here I am dumping entire objects for the specific container and saving it in the text file. And reading it later and passing to the swift command line.
import swiftclient
import os
import sys
import subprocess
user = 'user'
key = 'key'
url = "url"
conn = swiftclient.Connection(
user=user,
key=key,
authurl=url)
input_file_name = "test.txt"
def write_error_log(path):
"""
Input : file path
This module writes the path which is failed to create or write during
downloading.
"""
with open('log.txt', 'w') as f:
f.write("Output not created for " + path)
def main():
"""
Main method is used to read the entire text file and
creates the file and folder based on the returned values
during reading.
It creats an error log file when script fails to create
the file.
"""
for path in open(input_file_name, 'r'):
print("Writing file", path)
download_command = "swift -A {0} -U {1} -K {2} download {3} {4}".format(url, user, key, container_name, path)
try:
return_status_code = subprocess.call(download_command, shell=True)
if return_status_code:
write_error_log(path)
except:
write_error_log(path)
if __name__ == "__main__":
container_name = sys.argv[1]
main()
Please suggest, whether the above code can work smoothly in terms of performance.?
