1

I have an Object Storage instance on IBM's Bluemix, and I'm attempting to upload a ~32GB SQLite file. Here is my Python code which uses the OpenStack Swift API (with the credentials removed):

import swiftclient
conn = swiftclient.Connection(key="pw",authurl="url",auth_version='3',os_options={"project_id": "project_id","user_id": "user_id","region_name": "region"})
container_name = 'containerName'
file_name = 'file.sqlite'
with open(file_name, 'rb') as sqlite_file:
    conn.put_object(container_name,file_name,sqlite_file)

I tested this code with a small .html file and it uploaded without a problem. When I changed the file to the SQLite file, it ran for > 5 hours and eventually gave a "requests.exceptions.ConnectionError: [Errno 32] Broken pipe" error. What am I doing wrong?

Ross Lewis
  • 755
  • 2
  • 7
  • 17

1 Answers1

2

You'll need to read-up on Swift DLO/SLO support and manifests. Here's a blog post that might help with context on what manifests are and the differences between Static Large Object and Dynamic Large Object support.

Basically, I'd recommend the following approach:

  1. Download/install the Python-SwiftClient binary
  2. Use its upload command in conjunction with your object storage credentials from the Bluemix service. In the manifest article above, it discusses this approach here. Take note of the upload command, the use of the --use-slo flag and the ability to define the size of the concatenated segments generated. Roughly, the invocation will look like this:

$ swift --os-auth-url=https://identity.open.softlayer.com/v3 --os-user- id=some_hex_value --os-password="weird_characters" --os-project-id=another_hex_value --os-region-name=dallas -V 3 upload my_object_storage_container_name -S int_seg_size_in_bytes my_local_large_file_with_some_extension --use-slo

my_local_large_file_with_some_extension segment 3
my_local_large_file_with_some_extension segment 1
my_local_large_file_with_some_extension segment 2
my_local_large_file_with_some_extension segment 0
my_local_large_file_with_some_extension/1443450560.000000/160872806/52428800/00000002
my_local_large_file_with_some_extension/1443450560.000000/160872806/52428800/00000003
my_local_large_file_with_some_extension/1443450560.000000/160872806/52428800/00000001
my_local_large_file_with_some_extension/1443450560.000000/160872806/52428800/00000000
my_local_large_file_with_some_extension

Good luck.

Sanjay.Joshi
  • 304
  • 1
  • 12
  • It's running now. Some segments are working and some are erroring out: put_object(u'RossL_segments', u'bigfile.sqlite/slo/1438471533.000000/31807655936/200000000/00000007', ...) failure and no ability to reset contents for reupload. Any advice on this? – Ross Lewis Jun 02 '16 at 01:59
  • 2
    @RossLewis Per the swift docs [here](http://docs.openstack.org/cli-reference/swift.html#swift-upload), you may be able to re-run the command using the **--leave-segments** argument to try and get the failures re-uploaded. I've encountered upload segment errors from time to time and usually succeed after retrying. You may also be able to tweak the segment sizing and number of threads to match your network characteristics better. – Sanjay.Joshi Jun 02 '16 at 07:09
  • Thanks again for helping me out on this! Now that I have the file uploaded, can I get rid of all of the segments in the _segment container? – Ross Lewis Jun 02 '16 at 21:30
  • 1
    @RossLewis . Nope. Basically, the _segment container **contains** the actual data in its respective pieces. The file that represents the db.sqlite asset is really a *manifest* file that is a single point virtual representation of the segment files which OpenStack understands as being concatenated together. In other words, it simply holds a list of the segments within it in a way that Openstack Object Storage understands as needing to be concatenated when the object is fetched (downloaded). Glad that you found success. If you're good, please accept the answer ;-) – Sanjay.Joshi Jun 03 '16 at 14:53