2

I crerated a simple erlang application which periodically collects required data and puts it in a riak database.

As I start my application it runs smoothly.. but after a period of time it stucks as PUT requests to riak database becomes too slow.. It is logs from my app:

2013-06-26 12:44:09.090 [info] <0.60.0> data processed in [16476 ms]
2013-06-26 12:45:51.472 [info] <0.60.0> data processed in [18793 ms]
...
2013-06-26 12:57:28.138 [info] <0.60.0> data processed in [15135 ms]
2013-06-26 13:07:01.484 [info] <0.60.0> data processed in [488420 ms]
2013-06-26 14:03:11.561 [info] <0.60.0> data processed in [3370075 ms]

In riak crash logs I can see a lot of messages like 

2013-06-26 17:06:20 =CRASH REPORT====
crasher:
initial call: riak_kv_index_hashtree:init/1
pid: <0.13660.7>
registered_name: []
exception exit: {{{badmatch,{error,{db_open,"IO error: ./data/anti_entropy/
    433883298582611803841718934712646521460354973696/MANIFEST-000004: 
    Cannot allocate memory"}}}, [{hashtree,new_segment_store,2,
    [{file,"src/hashtree.erl"},{line,499}]},
    {hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},
    {riak_kv_index_hashtree,do_new_tree,2,
    [{file,"src/riak_kv_index_hashtree.erl"},
    {line,426}]},{lists,foldl,3,[{file,"lists.erl"},
    {line,1197}]},{riak_kv_index_hashtree,
    init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},
    {line,368}]},{riak_kv_index_hashtree,init,1,
    [{file,"src/riak_kv_index_hashtree.erl"},
    {line,225}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},
    {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},
    [{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},
    {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
ancestors: [<0.955.0>,riak_core_vnode_sup,riak_core_sup,<0.129.0>]
messages: []
links: []
dictionary: []
trap_exit: false
status: running
heap_size: 1597
stack_size: 24
reductions: 593
neighbours:

I can see the same behavior on Amazon AWS and local virtual machine. my VM's are quite small 512-1024 mb.. AWS is Micro, so it has the same amount of memory.

There are no cluster currently. Just single node with Riak and my app running on it.

I've checked riak documentation and basic things they recommend to do is to increase ulimit and to update sysctl. So, my server ulimit shows: ulimit -n 65536 AND sysctl updated as recommended.

I've tried bitcask and eleveldb, but result is the same.

Currently, I can't figure out what is broken and why riak Cannot allocate memory.. Thanks.

kikulikov
  • 2,512
  • 4
  • 29
  • 45
  • How big is each put request? How many of them per second? – nanofarad Jun 26 '13 at 16:15
  • @ObsessiveSSOℲ Put requests are small. Actually it is small arrays in json format. There are about 20-30 requests per second. – kikulikov Jun 26 '13 at 16:43
  • Hey! Why do you minus me? – kikulikov Jun 26 '13 at 16:45
  • Anyone can downvote a post. It is most likely due to it being (from the voter's standpoint) poorly researched or poorly written. No single user has control over this. I suggest you improve your post and perhaps you may be able to get upvotes. – nanofarad Jun 26 '13 at 16:49
  • i can see some `IO Errors` , `Failing to allocate memory` e.t.c. Dont you think there is some system limit which is denying riak enough resources ? Also, probably consider sending this question to the riak community. – Muzaaya Joshua Jun 27 '13 at 04:02
  • It does appear Riak is unable to allocate memory. To identify why, some additional information is required. What is the size of your cluster? How much RAM does each server have? Is Riak the only application running on that server? Which backend are you using? What does your app.config file look like? – Christian Dahlqvist Jun 27 '13 at 08:23
  • @Christian Dahlqvist currently there are no cluster actually.. just single node. virtual machines have 512-1024 mbs of memory. I run Riak and my application on the same node. – kikulikov Jun 27 '13 at 17:14
  • 1
    @Muzaaya Joshua thank you.. I will try to get help from Riak community. Unfortunately, they prefer to use mailing lists, which is not very comfortable in use.. – kikulikov Jun 27 '13 at 17:15

1 Answers1

7

1GB RAM is quite small for a Riak node, and even more so as you are also running your application there. The default settings in Riak are targeted at environments with considerably more RAM and processing power, so you will need to tweak the default settings in order to get it to work. Here are a few pointers that may help:

  1. As you only have one node, disable AAE by setting {anti_entropy, {off, []}}.
  2. Reduce the ring size. This will limit your ability to scale out, but is most likely required in order to get it to work. A suitable starting value could perhaps be 16, but possibly even as low as 8.
  3. Change default bucket properties so that you have n_val, r, w, dw and rw all set to 1, as you otherwise will be writing multiple copies of every record to disk. These will need to be increased when you scale out and add more nodes.
  4. As bitcask requires all keys to be kept in memory, it is probably a good idea to instead use leveldb as a backend. You will however most likely need to reduce the size of write buffers as well as the cache significantly. You may need to experiment to find a suitable level.
  5. As this environment most likely is too small to be able to run mapreduce on anyway, you can also set the map_js_vm_count and reduce_js_vm_count configuration parameters to 0 in order to save some additional memory.
Christian Dahlqvist
  • 1,665
  • 12
  • 9
  • thank you! very detailed and helpful answer! Actually, even point (1) resolved my problem. As soon as I disabled anti_entropy Riak runs perfectly even on 512 mb VM. As currently I don't use cluster I can keep that anti_entropy disabled. – kikulikov Jun 28 '13 at 12:22