I have set up an arangodb 3.1.10 cluster on a single server with 3 agents, 5 coordinators and 5 primaries. I have an issue with using big batches with sharded collections.
I regularly get the following error in the coordinator responsible for the batch query:
2017-02-15T15:23:28Z [2665] ERROR {cluster} cannot create connection to server 'PRMR-6f3a5704-5500-4fa8-8933-5029e492c25d' at endpoint 'tcp://localhost:4101'
2017-02-15T15:23:28Z [2665] ERROR {cluster} cannot create connection to server 'PRMR-6f3a5704-5500-4fa8-8933-5029e492c25d' at endpoint 'tcp://localhost:4101'
Localhost:4101 is a PRIMARY. But its own log does not show any error:
2017-02-15T14:38:10Z [2582] INFO ArangoDB 3.1.10 [linux] 64bit, using VPack 0.1.30, ICU 54.1, V8 5.0.71.39, OpenSSL 1.0.1e-fips 11 Feb 2013
2017-02-15T14:38:10Z [2582] INFO using SSL options: SSL_OP_CIPHER_SERVER_PREFERENCE, SSL_OP_TLS_ROLLBACK_BUG
2017-02-15T14:38:10Z [2582] INFO Starting up with role PRIMARY
2017-02-15T14:38:27Z [2582] INFO {cluster} Fresh start. Persisting new UUID PRMR-6f3a5704-5500-4fa8-8933-5029e492c25d
2017-02-15T14:38:27Z [2582] INFO file-descriptors (nofiles) hard limit is 65536, soft limit is 65536
2017-02-15T14:38:27Z [2582] INFO WAL directory '/opt/arangodb/primary_4101/data/journals' does not exist. creating it...
2017-02-15T14:38:27Z [2582] INFO created base application directory '/opt/arangodb/primary_4101/apps/_db'
2017-02-15T14:38:27Z [2582] INFO JavaScript using startup '/usr/share/arangodb3/js', application '/opt/arangodb/primary_4101/apps'
2017-02-15T14:38:28Z [2582] INFO Cluster feature is turned on. Agency version: {"server":"arango","version":"3.1.10","license":"community"}, Agency endpoints: http+tcp://127.0.0.1:4002, http+tcp://127.0.0.1:4000, http+tcp://127.0.0.1:4001, server id: 'PRMR-6f3a5704-5500-4fa8-8933-5029e492c25d', internal address: tcp://localhost:4101, role: PRIMARY
2017-02-15T14:38:28Z [2582] INFO using heartbeat interval value '1000 ms' from agency
2017-02-15T14:38:28Z [2582] INFO using endpoint 'http+tcp://0.0.0.0:4101' for non-encrypted requests
2017-02-15T14:38:28Z [2582] INFO bootstrapped DB server PRMR-6f3a5704-5500-4fa8-8933-5029e492c25d
2017-02-15T14:38:28Z [2582] INFO ArangoDB (version 3.1.10 [linux]) is ready for business. Have fun!
2017-02-15T15:22:42Z [2582] INFO created application directory '/opt/arangodb/primary_4101/apps/_db/entities' for database 'entities'`
As long as the collection are not sharded it works fine, but as soon as I add sharding some of the batches just fail. When doing this test my collections all had 5 shards.
Here is the command I used to start the primary:
/usr/sbin/arangod \
--server.endpoint tcp://0.0.0.0:4101 \
--server.threads 8 \
--log.level INFO \
--javascript.v8-contexts 4 \
--database.directory /opt/arangodb/primary_4101/data \
--javascript.startup-directory /usr/share/arangodb3/js \
--javascript.app-path /opt/arangodb/primary_4101/apps \
--log.file /opt/arangodb/primary_4101/arangod.log \
--log.force-direct false \
--server.authentication false \
--cluster.my-address tcp://localhost:4101 \
--cluster.my-role PRIMARY \
--cluster.my-local-info tcp://localhost:4101 \
--foxx.queues false \
--server.statistics true \
--cluster.agency-endpoint tcp://localhost:4000 --cluster.agency-endpoint tcp://localhost:4001 --cluster.agency-endpoint tcp://localhost:4002
