CouchDB dump to file and load from file

Question

I cannot replicate between two couchdb servers, so I would like to dump to file from one server and load from file into the other server.

I used this statement to dump and it worked fine:

curl -X GET http://localhost:5984/<DATABASE_NAME>/_all_docs?include_docs=true > FILE.txt

But when I used this statement to load:

curl -d @FILE.txt -H “Content-Type: application/json” -X POST http://localhost:5984/<DATABASE_NAME>/_bulk_docs

it failed like this:

curl: (6) Could not resolve host: application; Host not found {"error":"bad_content_type","reason":"Content-Type must be application/json"}

Any ideas?

will this backup the revisions of each of the documents ? – user3526 Feb 22 '15 at 11:02 — user3526, Feb 22 '15 at 11:02

Daniele B · Accepted Answer · 2013-06-19T00:55:43.943

15

As said, you should use the " and not the “ as argument of the -H option

If you are a Linux or MacOSX user you can use the couchdb-dump tool, which basically works on bash shell.

It dumps the database on a local file (ASCII text file), formatted as requested by http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API

Then you can restore it with the bulk document upload or with the couchdb-dump restore tool included in the package.

This is the link to the tool: https://github.com/animamea/couchdb-dump

But you can find other tools also:

https://github.com/stockr-labs/couchdbdump

https://github.com/zebooka/couchdb-dump

edited Jun 19 '13 at 00:55

answered Jun 13 '13 at 12:54

Daniele B

3,117
2
23
46

1

It seems a pretty small task to use a tool for. The cURL commands worked perfectly for me :) – Niels Abildgaard Aug 14 '14 at 09:16
https://github.com/animamea/couchdb-dump didn't work for me :( sed magic inside produces invalid jsons even on a simple docs – Max Lobur Jul 26 '16 at 22:40
@MaxLobur same here. – John LaBarge Jan 02 '17 at 01:15
@MaxLobur on what OS? – Daniele B Jan 02 '17 at 08:53
@DanieleB El Capitan – Max Lobur Jan 19 '17 at 17:16

score 10 · Answer 2 · answered May 18 '16 at 08:34

10

You can use the following command line to convert the output of the curl command to the “docs” structure that the _bulk_docs requires:

curl -X GET 'http://localhost:5984/mydatabase/_all_docs?include_docs=true' | jq '{"docs": [.rows[].doc]}' | jq 'del(.docs[]._rev)' > db.json

jq is the name of an excellent command line processor very useful (i.e. in this situation).

Hope it helps.

answered May 18 '16 at 08:34

Roberto

374
6
9

Your example using jq was very helpful, thank you! I also found one of my files was too large to load, and chunking using jq did the trick: `cat db.json | jq '{"docs": .docs[0:30000]}' > db.0-30000.json; cat db.json | jq '{"docs": .docs[30000:]}' > db.30000-.json` – Michael Allan Jackson Jun 15 '16 at 15:14
Is there a way to include the revisions? – Michael Allan Jackson Jun 15 '16 at 21:51
Yes, of course, just drop the `| jq 'del(.docs[]._rev)'` part that removes the revisions. Is that what you need? Note that you wont be able to restore the file if you include the revisions, AFAIK. – Roberto Jun 16 '16 at 18:40
I tried that and many of the records get an error `"error":"conflict","reason":"Document update conflict."` as described here: https://wiki.apache.org/couchdb/HTTP_Bulk_Document_API . I also tried adding `"new_edits": false` field described in that link, but in that case zero records were loaded without error. – Michael Allan Jackson Jun 16 '16 at 18:56
Sorry Michael, I didn't realize you're talking about the import, not the export. That's exactly the reason why I removed the `_rev` fields in the first place, because of the document conflict. I don't know very much about the way MVCC (multiversion version concurrent control) is handled in CouchDB, so I can't help too much here as I haven't got the need to store more than the current version of the documents. Regards – Roberto Jun 18 '16 at 11:44

score 7 · Answer 3 · answered Jul 25 '12 at 04:38

7

The reason for your actual error is that you're using “ instead of plain ASCII " around your -H argument on the command line.

However, the real solution here is to just copy the <DATABASE_NAME>.couch file from the /path/to/var/lib/couchdb directory from one server to the other.

answered Jul 25 '12 at 04:38

smathy

26,283
5
48
68

I tried to copy the *.couch files first, but after tying to access the database entries, I received the error, that the database was build with a wrong Erlang version. – eriq Jul 25 '12 at 17:06
1

sure! How do I accept? Never the less it was a good and correct answer. I'm obviously not too familiar with this system. – eriq Aug 13 '12 at 22:11

score 2 · Answer 4 · edited Aug 31 '15 at 00:56

2

As alternative solution, you may use couchdb-load and couchdb-dump utilities from couchdb-python project.

edited Aug 31 '15 at 00:56

Jonathan Hall

75,165
16
143
189

answered Jul 25 '12 at 09:52

Kxepal

4,659
1
19
16

Warning: those do not work with python 3.x (TypeError: write() argument must be str, not bytes) – Max Lobur Jul 26 '16 at 22:34
@MaxLobur why not to report about that here https://github.com/djc/couchdb-python/issues ? (; – Kxepal Jul 27 '16 at 15:51

score 2 · Answer 5 · answered Jun 23 '16 at 07:31

2

Nolan from the PouchDB team makes some great tools. These will work well to dump and load from CouchDB (including attachments):

Dump/Backup:

https://github.com/nolanlawson/pouchdb-dump-cli

Load/Restore:

https://github.com/nolanlawson/pouchdb-load

answered Jun 23 '16 at 07:31

PaulMest

12,925
7
53
50

score 1 · Answer 6 · answered Feb 06 '22 at 13:11

There's also github.com/danielebailo/couchdb-dump , which might help to clean out old transients, the authors state:

We've seen 15GB database files, containing only 2.1GB of raw JSON, reduced to 2.5GB on disk after import!

There are also hints on how update_seq works if you want do sync clusters.

CouchDB dump to file and load from file

6 Answers6

Linked