1

I tried to run the yugabyte 2.0.8.0 quick start demo according to these instructions:

https://docs.yugabyte.com/latest/quick-start/install/

https://docs.yugabyte.com/latest/quick-start/create-local-cluster/

with 3 local nodes

./bin/yb-ctl --rf 3 create

And then explore ysql:

https://docs.yugabyte.com/latest/quick-start/explore-ysql/

but the db creation fails:

yugabyte=# CREATE DATABASE yb_demo;
CREATE DATABASE yb_demo;
ERROR:  Timed out: Write(tablet: 00000000000000000000000000000000, num_ops: 1, num_attempts: 9, txn: 00000000-0000-0000-0000-000000000000) passed its deadline 3393.992s (passed: 65.317s): Remote error (yb/rpc/outbound_call.cc:440): Service unavailable (yb/tserver/tablet_service.cc:291): Soft memory limit exceeded (at 93.81% of capacity), score: 0.00

yugabyte=# CREATE DATABASE yb_demo;
CREATE DATABASE yb_demo;
ERROR:  Already present: Keyspace 'yb_demo' already exists

yugabyte=# DROP DATABASE yb_demo;
DROP DATABASE yb_demo;
ERROR:  database "yb_demo" does not exist

yugabyte=# \c yb_demo;
FATAL:  database "yb_demo" does not exist
Previous connection kept

So the db creation failed. I cannot recreate it but it seems that I also cannot remove it nor connect to it. The error message hints that there was not enough memory.

free output is: 

              total        used        free      shared  buff/cache   available
Mem:        4045992     1400276     1419504       64104     1226212     2319448
Swap:       4194300           0     4194300

Questions:

How much memory is required for running YugaByte? In this case the runtime environment is 4GB Ubuntu 16.4 VirtualBox VM with desktop. Your HW requirements say min 2GB https://docs.yugabyte.com/latest/deploy/checklist/.

How can I recover this error? Either drop the failed db or finalize the db creation?

Thank you for your support.

user4955663
  • 1,039
  • 2
  • 13
  • 21
  • When I created only 1 node, the CREATE DATABASE command succeeded. So maybe 3 nodes in one 4GB VM is too much. But how can I recover from the error? – user4955663 Dec 30 '19 at 17:37
  • Yes, as you correctly pointed out, the requirement is 2GB per node (or process instance). BTW, those recommendations are for multi-node clusters that you intend to run for a long time. * If you don't care about the data, just do `yb-ctl destroy` and recreate. * If you want to rescue it, could you please open a github issue? Would be good for us to work thought it there. – Karthik Ranganathan Dec 30 '19 at 22:04
  • You can get more realtime help in our forum as well. That said, I will also ask some folks more knowledgeable to jump in here to help out. – Karthik Ranganathan Dec 30 '19 at 22:06
  • It's just a demo so in this case I will destroy the data, but generally there should be way to recover without losing the data. – user4955663 Dec 30 '19 at 22:40
  • Yes, agree. This is definitely being worked on, we can try to add these types of tests to our regression suite. – Karthik Ranganathan Jan 03 '20 at 05:06

1 Answers1

1

How much memory is required for running YugaByte? In this case the runtime environment is 4GB Ubuntu 16.4 VirtualBox VM with desktop. Your HW requirements say min 2GB https://docs.yugabyte.com/latest/deploy/checklist/.

Indeed, the 2GB minimum is intended to be per node. Since you are running a 3-node cluster on one machine, it should help to increase it to 6GB or more.

Alternatively, if increasing the memory is not an option, you can tune the memory limits for the processes. By default Yugabyte allocates 10% of total memory to the yb-master process (metadata server) and 85% to the yb-tserver process (tablet-server).

In your case, the memory issue is most likely with the yb-master process (CREATE DATABASE command will initialize the metadata for the new database). You can inspect the current memory settings and usage using the Web UI to confirm.

  • yb-master: <node-ip>/7000/mem-trackers
  • yb-tserver: <node-ip>:9000/memz

Note: for a 3-node yb-ctl cluster the node IPs are typically 127.0.0.1, 127.0.0.2, and 127.0.0.3.

To update the yb-master memory limit you can use the memory_limit_hard_bytes flag (same flag can be used for the yb-tserver). For instance, to increase it to 600MB:

./bin/yb-ctl stop && ./bin/yb-ctl start --master_flags="memory_limit_hard_bytes=629145600"

Note: You can also use the same options directly when creating a cluster:

./bin/yb-ctl destroy && ./bin/yb-ctl create --master_flags="memory_limit_hard_bytes=629145600"

How can I recover this error? Either drop the failed db or finalize the db creation?

Regarding the memory issue causing the new database to be left in that state, that can happen in some corner cases in 2.0.8 if the DDL command (CREATE DATABASE fails with a system error in the middle).

For now, the simplest fix, if this is a test cluster, is to destroy the cluster and then recreate it (either on a machine with more memory or with updated flags as shown above). Otherwise you can use the yb-admin tool to clean up the partial metadata.

./bin/yb-admin delete_namespace ysql.yb_demo

Note: Some commits to fix such cases have landed recently [1], [2] and should be in the next release (though I can't confirm if they will fix this particular issue).

Mihnea Iancu
  • 46
  • 2