CRASH REPORT Process and show list of bucket very slow in riak

Question

I setup complete riak system. but It's have a problem, when i use s3cmd to list bucket by s3cmd la, then it's show very slow.

I use Riak 1.4.10 combine Riak-cs 1.5.0, This log excerpt from Riak-cs log (/var/log/riak-cs/error.log), It's show following:

    2014-12-17 23:54:52.586 [error] <0.18590.89> CRASH REPORT Process <0.18590.89> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.6047.89>, {req,{rpbgetreq,<<48,98,58,14,235,1,111,70,12,63,166,79,50,217,158,89,177,58,143>>,<<203,229,11,...>>,...},...}, infinity) in gen_server:terminate/6 line 747
2014-12-17 23:55:18.409 [error] <0.20755.89> gen_server <0.20755.89> terminated with reason: no such process or port in call to gen_server:call(<0.14402.89>, block_pbc)
2014-12-17 23:55:18.409 [error] <0.20755.89> CRASH REPORT Process <0.20755.89> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.14402.89>, block_pbc) in gen_server:terminate/6 line 747
2014-12-17 23:55:20.737 [error] <0.21075.89> gen_server <0.21075.89> terminated with reason: no such process or port in call to gen_server:call(<0.14690.89>, block_pbc)
2014-12-17 23:55:20.737 [error] <0.21075.89> CRASH REPORT Process <0.21075.89> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.14690.89>, block_pbc) in gen_server:terminate/6 line 747
2014-12-17 23:56:08.816 [error] <0.24106.89> gen_server <0.24106.89> terminated with reason: no such process or port in call to gen_server:call(<0.17691.89>, block_pbc)
2014-12-17 23:56:08.816 [error] <0.24106.89> CRASH REPORT Process <0.24106.89> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.17691.89>, block_pbc) in gen_server:terminate/6 line 747
2014-12-17 23:56:22.985 [error] <0.25880.89> gen_server <0.25880.89> terminated with reason: no such process or port in call to gen_server:call(<0.13391.89>, {req,{rpbgetreq,<<48,98,58,14,235,1,111,70,12,63,166,79,50,217,158,89,177,58,143>>,<<0,105,226,...>>,...},...}, infinity)
2014-12-17 23:56:22.985 [error] <0.25880.89> CRASH REPORT Process <0.25880.89> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.13391.89>, {req,{rpbgetreq,<<48,98,58,14,235,1,111,70,12,63,166,79,50,217,158,89,177,58,143>>,<<0,105,226,...>>,...},...}, infinity) in gen_server:terminate/6 line 747
2014-12-17 23:57:50.034 [error] <0.31483.89> gen_server <0.31483.89> terminated with reason: no such process or port in call to gen_server:call(<0.25032.89>, block_pbc)
2014-12-17 23:57:50.034 [error] <0.31483.89> CRASH REPORT Process <0.31483.89> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.25032.89>, block_pbc) in gen_server:terminate/6 line 747
2014-12-17 23:57:55.988 [error] <0.31644.89> gen_server <0.31644.89> terminated with reason: no such process or port in call to gen_server:call(<0.25207.89>, block_pbc)
2014-12-17 23:57:55.988 [error] <0.31644.89> CRASH REPORT Process <0.31644.89> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.25207.89>, block_pbc) in gen_server:terminate/6 line 747
2014-12-17 23:58:24.909 [error] <0.1078.90> gen_server <0.1078.90> terminated with reason: no such process or port in call to gen_server:call(<0.27321.89>, block_pbc)
2014-12-17 23:58:24.910 [error] <0.1078.90> CRASH REPORT Process <0.1078.90> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.27321.89>, block_pbc) in gen_server:terminate/6 line 747
2014-12-17 23:58:51.765 [error] <0.3998.90> gen_server <0.3998.90> terminated with reason: no such process or port in call to gen_server:call(<0.29222.89>, block_pbc)
2014-12-17 23:58:51.766 [error] <0.3998.90> CRASH REPORT Process <0.3998.90> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.29222.89>, block_pbc) in gen_server:terminate/6 line 747
2014-12-17 23:59:06.018 [error] <0.5059.90> gen_server <0.5059.90> terminated with reason: no such process or port in call to gen_server:call(<0.30266.89>, block_pbc)
2014-12-17 23:59:06.018 [error] <0.5059.90> CRASH REPORT Process <0.5059.90> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.30266.89>, block_pbc) in gen_server:terminate/6 line 747
2014-12-17 23:59:29.631 [error] <0.6502.90> gen_server <0.6502.90> terminated with reason: no such process or port in call to gen_server:call(<0.31681.89>, block_pbc)
2014-12-17 23:59:29.631 [error] <0.6502.90> CRASH REPORT Process <0.6502.90> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.31681.89>, block_pbc) in gen_server:terminate/6 line 747

From /var/log/riak-cs/console.log

2014-12-15 12:31:46.189 [error] <0.28550.18> gen_server <0.28550.18> terminated with reason: no such process or port in call to gen_server:call(<0.17445.18>, block_pbc)
2014-12-15 12:31:46.190 [error] <0.28550.18> CRASH REPORT Process <0.28550.18> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.17445.18>, block_pbc) in gen_server:terminate/6 line 747
2014-12-15 12:34:54.486 [error] <0.2349.19> gen_server <0.2349.19> terminated with reason: no such process or port in call to gen_server:call(<0.27112.18>, block_pbc)
2014-12-15 12:34:54.487 [error] <0.2349.19> CRASH REPORT Process <0.2349.19> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.27112.18>, block_pbc) in gen_server:terminate/6 line 747
2014-12-15 12:35:24.624 [error] <0.4154.19> gen_server <0.4154.19> terminated with reason: no such process or port in call to gen_server:call(<0.29025.18>, block_pbc)
2014-12-15 12:35:24.625 [error] <0.4154.19> CRASH REPORT Process <0.4154.19> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.29025.18>, block_pbc) in gen_server:terminate/6 line 747
2014-12-15 12:35:39.917 [error] <0.4649.19> gen_server <0.4649.19> terminated with reason: no such process or port in call to gen_server:call(<0.29489.18>, block_pbc)
2014-12-15 12:35:39.917 [error] <0.4649.19> CRASH REPORT Process <0.4649.19> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.29489.18>, block_pbc) in gen_server:terminate/6 line 747
2014-12-15 12:43:26.349 [error] <0.21992.19> gen_server <0.21992.19> terminated with reason: no such process or port in call to gen_server:call(<0.15354.19>, block_pbc)

Any help?

Cheers,

Nghiep

There isn't enough information here to diagnose the problem. Is this log excerpt from the Riak log or the Riak CS log? Did the other log have any entries? Which version of Riak/Riak CS are you using? Is this the only operation that isn't working? — Joe, Dec 15 '14 at 13:41
The additional log info you posted as an answer (please edit the question and add the log lines there) indicate that gen_server:call(Pid,block_pbc) were failing due to the server not being present. Those calls should have been handled by https://github.com/basho/riak_cs/blob/1.5.0/src/riak_cs_riak_client.erl#L208-L214 which implies that the client connection between Riak CS and Riak was not properly established or was interrupted. — Joe, Dec 15 '14 at 23:06
Client connection between Riak CS and Riak was properly established . It's still appear error. — Nghiep Thoi Minh, Dec 18 '14 at 09:57

score 0 · Answer 1 · answered Dec 18 '14 at 17:27

Your logs show 2 types of function call that fail:

gen_server:call(<0.6047.89>, {req,{rpbgetreq,<<48,98,58,14,235,1,111,70,12,63,166,79,50,217,158,89,177,58,143>>,<<203,229,11,...>>,...},...}, infinity)

and

gen_server:call(<0.14690.89>, block_pbc)

The first is a get request for a specific data block of a document called from riakc_pb_socket:call_infinity, called from riakc_pb_socket:get. In this case, the process ID <0.6047.89> is supposed to refer to an already started riakc_pb_socket gen_server.

The second is a request to check out a protocol buffers connection to the underlying Riak cluster, called from riak_cs_riak_client:block_pbc, which is called from several places in riak_cs_block_server. In this case, the process ID <0.14690.89> is supposed to refer to an already started riak_cs_riak_client gen_server, which would in turn have a pool of riakc_pb_socket gen_servers to handle requests.

In both cases, the error no such process or port indicates that the server has failed, stopped, or otherwise gone away. The most common reason for this is the connection from Riak CS to the Riak protocol buffers port was interrupted.

If you are certain your connection is solid, you might try turning up the log level to see if it tells you what happened to those gen_servers.

It's also appear error : [error] <0.12831.36> CRASH REPORT Process <0.12831.36> with 0 neighbours crashed with reason: no case clause matching {ok,{http_error," \n"},<<>>} in mochiweb_http:request/3 line 107 . What's it mean? — Nghiep Thoi Minh, Dec 19 '14 at 03:39

CRASH REPORT Process and show list of bucket very slow in riak

1 Answers1