2

I'm trying to configure my Duplicity installation to backup to an Amazon S3 bucket. I keep getting a BackendException error with no further details.

EDIT: I've removed my Duply configuration from this question in an effort to isolate the issue more and switched to a plain duplicity command on a small directory (10 MB, 34 files).

Here's the command I'm trying to run:

duplicity full ./logs "s3://s3-us-east-1.amazonaws.com/bucketname" -v9

And here's the output:

root@ats:/var/ats# duplicity full ./logs "s3://s3-us-east-1.amazonaws.com/bucketname" -v9

Duplicity 0.6 series is being deprecated:
See http://www.nongnu.org/duplicity/

Using archive dir: /root/.cache/duplicity/876c7d0b54276e675d41f6ea6077d52f
Using backup name: 876c7d0b54276e675d41f6ea6077d52f
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.dpbxbackend Succeeded
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.ftpsbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Import of duplicity.backends.~par2wrapperbackend Succeeded
Using temporary directory /tmp/duplicity-sQ3sGs-tempdir
Backend error detail: Traceback (most recent call last):
  File "/usr/local/bin/duplicity", line 1509, in <module>
    with_tempdir(main)
  File "/usr/local/bin/duplicity", line 1503, in with_tempdir
    fn()
  File "/usr/local/bin/duplicity", line 1336, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/local/lib/python2.7/dist-packages/duplicity/commandline.py", line 1062, in ProcessCommandLine
    backup, local_pathname = set_backend(args[0], args[1])
  File "/usr/local/lib/python2.7/dist-packages/duplicity/commandline.py", line 955, in set_backend
    globals.backend = backend.get_backend(bend)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 163, in get_backend
    return _backends[pu.scheme](pu)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 163, in __init__
    self.resetConnection()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 189, in resetConnection
    raise BackendException(err.message)
BackendException

BackendException:

I've tested Boto (which is what Duplicity uses for S3 connections) with this script:

root@ats:/var/ats# python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto.s3
>>> conn = boto.s3.connect_to_region("us-east-1")
>>> bucket = conn.get_bucket("bucketname")
>>> for key in bucket.list(): print key, key.storage_class
...
<Key: bucketname,test.txt> STANDARD
>>> exit()

I've made sure that I'm running the latest appropriate versions of duplicity and python, and I've set the AWS_* variables which work fine in Boto.

Am I missing something? What should I do?

Castaglia
  • 3,349
  • 3
  • 21
  • 42
A.M.K
  • 163
  • 1
  • 9
  • Run duplicity with `-v9` for extra verbosity. – ceejayoz Jul 08 '15 at 20:13
  • @ceejayoz I did "And here's the log I'm getting from Duplicity with the `-v9` flag:", unfortunately that _is_ the extra verbosity. – A.M.K Jul 08 '15 at 20:24
  • My apologies, I missed that in the question. :-( – ceejayoz Jul 08 '15 at 20:25
  • No problem. Thanks for taking a look, I've been stuck on this one a little while. – A.M.K Jul 08 '15 at 20:25
  • smells like the "file too big for s3" issue
    https://answers.launchpad.net/duplicity/+question/253609 ..ede/duply.net     –  Jul 09 '15 at 09:46
  • @ede I just tried on a small directory (10 MB) which also failed, I've updated my question with a reduced test-case. – A.M.K Jul 09 '15 at 16:41
  • have you tried enabling debug in boto? http://boto.readthedocs.org/en/latest/boto_config_tut.html – Fox Jul 10 '15 at 13:09
  • I couldn't find a way to set boto to a higher logging level through duplicity (it worked fine on its own), but I found a solution. I still have no idea what was wrong with the previous one. – A.M.K Jul 10 '15 at 18:42

4 Answers4

1

I don't know why the original command wasn't working, but in the end I figured out that changing the URL format from s3://s3-us-east-1.amazonaws.com/bucketname to s3+http://bucketname fixes the problem.

The reduced working command is now:

duplicity full ./logs "s3+http://bucketname" -v9
A.M.K
  • 163
  • 1
  • 9
1

If you still can't get it to work, try adding: export S3_USE_SIGV4="True" to your duply profile conf file.

This is described in detail here and worked for me.

ErikE
  • 4,746
  • 1
  • 20
  • 27
Joe Hudson
  • 11
  • 1
  • I tried that when I was first having problems and it didn't help. Thanks for the suggestion however! – A.M.K Jul 13 '15 at 22:23
0

My answer is in 2022: I have the latest duply (v2.3.1) and duplicity (v0.8.21). I discovered:

  • The bucket name requires http: s3+http://my-bucket/
  • The bucket name must be non-existent in your Amazon S3 dashboard
  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables must be exported in the terminal before running duply data backup. Putting them inside .duply/data/conf won't help. There's even a note about the latter in .duply/data/conf.
tash
  • 111
  • 1
  • 6
0

Neither the SIGV4 nor the s3+http solutions worked for me.

My solution was to use the region name in the URL, instead of the generic 's3'. In my case, this was eu-west-1 (Ireland).

This looks like:

s3://s3-eu-west-1.amazonaws.com/bucketname

Instead of:

s3://s3.amazonaws.com/bucketname

Even though the second generic URL works in other tools such as s3cmd, it wasn't working with Duplicity / Boto.

You may need to combine this solution with the answer by Joe Hudson if you're using a new region such as Frankfurt or Singapore.

Rafiq Maniar
  • 1,120
  • 9
  • 15