1

Currently we have thousands of documents in our Couchbase bucket and we want to export them to csv (or json) format for further use.

As the documents are huge in numbers, we want to split the exported data into multiple files instead of single large file.

Currently we are using cbtransfer utility and Couchbase documentation mentions that we can use extra options for splitting the file ‘-x cbb_max_mb=size’ if it exceeds particular size.

Command we used -

cbtransfer http://127.0.0.1:8091 csv:./data.csv -b bkt-customer -u Administrator -p password -x cbb_max_mb=100

With this, file is getting created but it’s not getting split into smaller chunks of 100 MB as we expected.

Can anyone guide how can we achieve that?

Nachiket Bapat
  • 61
  • 1

1 Answers1

3

The -x options are "special" and do not apply to all of the backends and frontend (backup, CSV, couchstore, etc) cbtransfer support. cbb_max_mb option is only supported when writing backup, the documentation hints at this:

cbb_max_mb=100000 Split backup file on destination cluster if it exceeds the MB.

There are some other options:

  1. Use the Unix/Linux split after cbtransfer is finsihed
  2. Use the --single-node argument on cbtransfer if there are multiple data nodes in the cluster. This will just grab the data from the node specify in the connection string:

--single-node Transfer from a single server node in a source cluster. This single server node is a source node URL.

I would also recommend having a look at cbexport which will at some point replace cbtransfer

Paddy
  • 1,195
  • 9
  • 16