0

I'm trying to find a way to speed up the transfer of 95gb of data to my company's Object Storage.

I'm transfering the files via node/aws-sdk but it's too slow. I already tried ManagedUpload and Multipart upload, with same result (too slow).

After that I found s3fs... same thing.

So now I dicovered CrossFTP and DragonDisk that can access S3 services, but both use bucket.site.com style urls and my corporative server uses path-style.

Is there a way to configure one/both of them to use path-style ?

Also anyone knows a faster way to transfer the files ? Or any other Linux s3 FOSS clients ?

If I could upload the metadata with the files that would be a plus.

Edit

Answering @emdee:

AFAIU, some S3 servers access a bucket using a prefix in domain name like bucket.s3domain.com. Others are configured to access buckets using a path AFTER domain name, like s3domain.com/bucket.

My corporative server uses Ceph to provide s3 functionality.

So when I access it with CrossFTP or DragonDisk they try to access bucket.s3domain.com and it doesn't work.

You can see here that there is a s3ForcePathStyle property in S3 constructor which my project sets in order to be able to access site. Also when I mounted s3fs I had to use the following command:

s3fs bucket /mnt/s3/ -o passwd_file=/home/me/.passwd-s3fs,url=https://myserverurl.com,use_path_request_style

You can see the use_path_request_style in the options. When I first tried without it, it didn't work either.

So it's not a share. It's a configuration of the s3 server itself.

I know CrossFTP is trying to access the server that way because of the error message:

R1] S3 Error: -1 (null) error: Request Error: bucket.myserverurl.com: Nome ou serviço desconhecido; XML Error Message: null

I'm obviously changing the bucket and server name in order to hide my company details, but I imagine you understand it.

DrangonDisk in the other hand lists buckets correctly but when I try to access one it puts 

https://bucket.myserverurl.com/

in the url field. So this is where I got the information that both softwares are accessing the server with a configuration that it doesn't support

Hope I clarified.

Nelson Teixeira
  • 225
  • 1
  • 3
  • 15
  • When you say `bucket.site.com` style do you mean like `s3://my-bucket/path/to/file.txt`? When you say `corporate server uses path-style` I assume that's like a file share? What kind? `\\Windows\samba\etc` or `/some/nfs/mount/path` --- Please provide much more details to your question, there is a lot of vagueness here I think specific information will help others understand your problem more. Personally, the fastest I've ever transferred files to S3 is using multipart uploads just using `aws-cli` on a server with fast upload speeds. – emmdee May 13 '20 at 18:13
  • @emmdee see my edit. If you need any other info, just ask. :) – Nelson Teixeira May 13 '20 at 18:47
  • 1
    If you run s3fs with `-f -d` you can get debugging output to further diagnose this issue. – Andrew Gaul Nov 25 '20 at 08:45

0 Answers0