Performance Issue of Sending Large Object using NetMQ Multipart Message

Question

I'm sending large object using the following code in C# (NetMQ):

var client = new DealerSocket("<connection String>");
var serializedData = new string('*', 500000);
var message = new List<byte[]>();
message.Add(Encoding.UTF8.GetBytes("BulkSend"));
message.Add(Encoding.UTF8.GetBytes(serializedData));
client.TrySendMultipartBytes(TimeSpan.FromMilliseconds(3000), message);

This code is taking 90% of CPU usage if it would be used in a high traffic (for example 10MB message per second). After some research, I've tried the two following codes. First of all, I removed the first frame ("Bulk Send"):

var client = new DealerSocket("<connection String>");
var serializedData = new string('*', 500000);
var message = new List<byte[]>();
message.Add(Encoding.UTF8.GetBytes(serializedData));
client.TrySendMultipartBytes(TimeSpan.FromMilliseconds(3000), message);

Surprisingly, the performance was improved. Secondly, I rearrange two frames. I mean moving the large frame to the first. Like the following:

var client = new DealerSocket("<connection String>");
var serializedData = new string('*', 500000);
var message = new List<byte[]>();
// change the order of two following codes
message.Add(Encoding.UTF8.GetBytes(serializedData));
message.Add(Encoding.UTF8.GetBytes("BulkSend"));

client.TrySendMultipartBytes(TimeSpan.FromMilliseconds(3000), message);

Again surprisingly the performance was improved!

What's the problem? How can I improve the performance? What about using zproto on netmq? Is there any proper document around that?

How are you measuring this performance improvement? Visual observation of the CPU? Total time needed to send the message? Something else? — Eric J., Jul 18 '16 at 16:44
Be very sure you reset everything before you draw these conclusions — Jeroen Heier, Jul 18 '16 at 16:52
@EricJ.: By the visual observation of the CPU (in server task manager). — OmG, Jul 19 '16 at 06:54
@JeroenHeier: Unfortunately I didn't get this. What do you mean "reset everything"? Exactly what should be reset? — OmG, Jul 19 '16 at 06:57
@EricJ.: P.S. Indeed we route part of real traffic into the client and it's a real test. — OmG, Jul 19 '16 at 07:03
@Omg Run each test under the same conditions and do not rely on one single test. Pay attention to all parts: the client, server, network, ... — Jeroen Heier, Jul 19 '16 at 18:49

score 1 · Accepted Answer · answered Jul 20 '16 at 07:09

1

As I found, there is a problem with sending multipart message. You can see this link http://hintjens.com/blog:84. And probably encode your message into one message and then sending it!

answered Jul 20 '16 at 07:09

OmG

18,337
10
57
90

iiwaasnet · Answer 2 · 2016-07-19T14:24:20.127

0

High CPU/low performance might be because of GC and memory allocations. I tried to send the same message size with my framework, which uses NetMQ, with server and client working on same machine. In case the client was waiting for the response before sending another message to server I achieved 60K msg/sec. Nevertheless, if the client was sending hundreds of messages in parallel, CPU was as well high and throughput was tens of messages per sec. At some point in time I've even got OutOfMemoryException. Would be good, if you post the complete code.

UPD: Sorry, I measured it wrong. With with either 100 messages sent in parallel or just one the performance is just 120 msg/sec

edited Jul 19 '16 at 14:24

answered Jul 19 '16 at 11:21

iiwaasnet

51
1
8

Definitely it is not about GC as one large object message was sent in three ways which are the same, except they have different order frames. – OmG Jul 19 '16 at 13:46
The complete code is here! Except created a fake message instead of real serialized message (actually in the same size). Also, our test had the same result for the example too. So, there is no more to release. – OmG Jul 20 '16 at 07:03

Performance Issue of Sending Large Object using NetMQ Multipart Message

2 Answers2