riak-cs crashing with a lot of requests

Question

I'm testing RIAK-CS as an S3 alternative. I currently have 3 nodes with 4GB RAM and 2GB SWAP each. I pushed 10GB of data to RIAK-CS. Except the high IO, everything was fine.

Then I tried to pull that data with the following python script.

conn = boto.connect_s3(
    aws_access_key_id = access_key,
    aws_secret_access_key = secret_key,
    host = riak_host,
    proxy = riak_host,
    proxy_port = 8080,
    calling_format = boto.s3.connection.OrdinaryCallingFormat(),
    debug = 2,
    is_secure = False,
)

bucket = conn.create_bucket('bucket_name')

for row in rows:
    key=bucket.new_key(valid_key_name)
    if key.exists():
        key.open_read()

Things started to be pulled from RIAK. After couple of minutes, except the huge IO again, I noticed that the riak-cs process is "eating" more and more memory and eventually crashed as it took all 6GB of RAM+SWAP.

If I changed the python script as the following, riak-cs remained at ~2-300 MB. Notice the connect in FOR loop.

for row in rows:
    conn = boto.connect_s3(
        aws_access_key_id = access_key,
        aws_secret_access_key = secret_key,
        calling_format = boto.s3.connection.OrdinaryCallingFormat(),
        debug = 2,
        is_secure = False,
    )

    bucket = conn.create_bucket('bucket_name')
    key=bucket.new_key(valid_key_name)
    if key.exists():
    key.open_read()

Any explanation why this would happen?

Thank you!

can you please provide a gist of the settings you have on riak-cs — WojonsTech, Apr 10 '13 at 06:44
Sure, here it goes: [link](https://gist.github.com/silasistefan/5352379) — silasistefan, Apr 10 '13 at 06:54

score 0 · Answer 1 · answered Oct 03 '13 at 12:45

0

This is possibly related to the issue which is fixed at version 1.4.

answered Oct 03 '13 at 12:45

shino

839
5
9

riak-cs crashing with a lot of requests

1 Answers1