Using swiftclient to stream large files straight to Openstack

Question

I need to upload files from a file storage service to OSS, alongside a Rabbit MQ queue. Currently, I am downloading files using urllib, and then using uploading them to OSS using python swiftclient. This solution works fine for smaller files, but for larger files the queue loses its connection as it assumes the consumer has died (as it doesn't return a heartbeat).

This is my current uploading/downloading code:

def download_file(file_name, url):
    urllib.urlretrieve(url, file_name)

def upload_file(file_name):
    with open(file_name, 'r') as myfile:
        conn.put_object(container, file_name, contents=myfile.read())

I was wondering it there is a way to speed up this process by avoiding downloading the files to disk. Possibly downloading chunks to memory in some variable and then streaming straight onto OSS?

Thanks for your time :)

score 0 · Answer 1 · answered Nov 15 '21 at 08:12

Technically the WSGI specification doesn't allow chunked request content. There is however an unofficial extension that is supported by some WSGI servers to indicate that request content with an unspecified/mutable length is being supplied. This can be used to enable chunked request content to be handled, albeit that you may need to also configure the WSGI server to allow chunked requests as well.

but with ajax you can uplaod directly to your swift.

$("#fileUpload").change(function () {
  var myFile = $(this)[0].files[0];
  
  ajax = new XMLHttpRequest();
  ajax.onreadystatechange = function () {
    if (ajax.status) {
      if (ajax.status == 200 && ajax.readyState == 4) {
        //To do tasks if any, when upload is completed
      }
    }
  };
  ajax.upload.addEventListener("progress", function (event) {
    var percent = (event.loaded / event.total) * 100;
    //**percent** variable can be used for modifying the length of your progress bar.
    console.log(+percent.toFixed(2) + "%");
  });

  ajax.open(
    "PUT",
    "http://swift:8080/v1/AUTH_ACCOUNT/CONTAINER/" + myFile.name,
    true
  );
  ajax.setRequestHeader("Content-type", myFile.type);
  ajax.setRequestHeader(
    "X-Auth-Token",
    "TOKEN"
  );
  ajax.send(myFile);
  //ajax.send is for uploading form data.
});

int the HTML file we have :

<label class="btn btn-cyan" for="fileUpload">
            <span aria-hidden="true">upload</span>
            <input id="fileUpload" type="file" style="display:none">
</label>

for this solution you have to access the swift-proxy (http://swift:8080), you can request to your host(e.x your horizon host) the proxy pass to your local swift. you can configure your host apache for redirecting in this way:

Listen 8080
<VirtualHost *:8080>
    ProxyPreserveHost On
    ProxyPass / http://swift:8080/
    ProxyPassReverse / http://swift:8080/
#    ServerName localhost
</VirtualHost>

Using swiftclient to stream large files straight to Openstack

1 Answers1