2

Not being an expert on MSMQ or WCF, I have read up a fair bit about it and it sounds and looks great. I am trying to develop something, eventually but first some theory, which needs to be robust and durable.

MSMQ I guess will be hosted on a seperate server.

There will be 2 WCF services. One for incoming messages and the other for outgoing messages (takes a message, does some internal processing/validation then places it on the outgoing messages queue or maybe sending an email/text message/whatever)

I understand with the right configuration, we can have the system so that it can be transactional (no messages are ever lost) and can be sent exactly once, so no chance of duplication of messages.

The applications/services will be multithreaded to process messages, which there will be hundreds and thousands of them.

BUT during the processing of a message or through the services lifetime, what if the server crashes? What if the server reboots? What if the service throws an exception for whatever reason? How is it possible to not lose that message but some how to put it back on the queue waiting for it to be processed again? Also how is it possible to make sure that the service is robust in such a way that it will spawn itself again?

I'd appreciate any advice and details here. There is quite alot to take in and WCF/MSMQ exposes quite alot of options.

Ahmed ilyas
  • 5,722
  • 8
  • 44
  • 72
  • You can use transactions instead... even if something goes wrong, either all the messages are sent before crash occurs or none is sent. – Milee Apr 20 '12 at 10:55

2 Answers2

9

Your assumption:

MSMQ I guess will be hosted on a seperate server.

is incorrect. MSMQ is installed on all machines which want to participate in message queuing.

There will be 2 WCF services. One for incoming messages and the other for outgoing messages

In the most typical configuration, the destination queues are local to the listening service.

For example, your ServiceA would have a local queue from which it reads. ServiceB also has a local queue from which it reads. If ServiceA wants to call ServiceB it will put a message into ServiceB's local queue.

I understand with the right configuration, we can have the system so that it can be transactional (no messages are ever lost)

This is correct. This is because MSMQ uses a messaging pattern called store-and-forward. See here for an explanation.

Essentially the reason it is safe to assume no message loss is because the transmission of a message from one machine to another actually takes place under three distinct transactions.

  1. The first transaction: ServiceA writes to it's own temporary local queue. If this fails the transaction rolls back and ServiceA can handle the exception.
  2. Second transaction: Queue manager on ServiceA machine transmits message to Queue manager on ServiceB machine. If failure then message remains on temporary queue.
  3. Third transaction: ServiceB reads the message off local queue. If ServiceB message handler method throws exception then transaction rolls message back to local queue.

The applications/services will be multithreaded to process messages

This is fine except if you require order to be preserved in the message processing chain. If you need ordered processing then you cannot have multiple threads without implementing a re-sequencer to reapply order.

I thought that MSMQ can be hosted seperately and have x servers share that queue?

All servers which want to participate in the exchange of messages have MSMQ installed. Each server can then write to any queue on any other server.

The reason for my thinking was because what if the server goes down? Then how will the messages get sent/received into MSMQ

If the queues are transactional then that means messages on them are persisted to disk. If the server goes down then when it comes back up the messages are still there. While a server is down it obviously cannot participate in the exchange of messages. However, messages can still be "sent" to that server - they just remain local to the sender (in a temporary queue) until the destination server comes back on-line.

so by having one central MSMQ server (and having it mirrored/failover) then there will be guarentee of uptime

The whole point of using message queueing is it's a fault-tolerant transport, so you don't need to guarantee uptime. If you have a 100% availability then there would be little reason to use message queuing.

how will WCF be notified of messages that are incoming?

Each service will listen on its own local queue. When a message arrives, the WCF runtime causes the handling method to be called and the message to be handled.

how will the service be notified of failures of sending messages

If ServiceA fails to transmit a message to ServiceB then ServiceB will never be notified of that failure. Nor should it be. ServiceA will handle the failure to transmit, not ServiceB. Your expectation in this instance creates a hard coupling between the services, something which message queueing is supposed to remove.

Community
  • 1
  • 1
tom redfern
  • 30,562
  • 14
  • 91
  • 126
  • Thanks! I appreciate that. I thought that MSMQ can be hosted seperately and have x servers share that queue? The reason for my thinking was because what if the server goes down? Then how will the messages get sent/received into MSMQ, ready for the services to read and write to? if hosted on same servers as the WCF Services then what if that server goes down? how can it read/write to the queue, so by having one central MSMQ server (and having it mirrored/failover) then there will be guarentee of uptime. – Ahmed ilyas Apr 20 '12 at 13:03
  • Whilst I have not looked into this but how will WCF be notified of messages that are incoming? or would a Windows Service/console app need to be created to which on a timely manner will seek the queue, obtain the message(s) and invoke the WCF service? how does it work? how will the service be notified of failures of sending messages or when a new message arrives in the MSMQ? – Ahmed ilyas Apr 20 '12 at 13:04
  • Thanks for the update. Like I said, I dont know much about MSMQ but learning every bit so thanks for that. Regarding the failover, what if one of the servers are down? Granted that the messages wont be sent but left in the queue until the destination is online again but if a failover needs to be implemented with mirroring perhaps, I guess this is more of a Windows infrastructure thing rather than configuration of MSMQ correct? I want to guarentee that messages can be sent/received at all times therefore this requires failover/mirroring put in place correct? – Ahmed ilyas Apr 20 '12 at 15:52
  • Having read more and more, MSMQ can be configured to use WAS which then can invoke WCF service to be notified of messages being received to the service. How does this play with threading? I mean, I want the service to be able to process as many messages as possible and not have it in a synchronised manner but rather in an async way - any ideas or suggestions? Or would it be better to make a Windows Service which runs 24/7, looking at the queue and processing messages on threads as and when messages appear in the queue? – Ahmed ilyas Apr 20 '12 at 15:55
  • Its possible to run MSMQ using public queues which are hosted in active directory. I have never needed to use them but I believe that when you do this you can implement a distributor pattern as a kind of load balancer in the way you are suggesting. This will give you better uptime. However this is beyond my understanding. – tom redfern Apr 20 '12 at 17:40
  • Thanks Hugh! you have been great. – Ahmed ilyas Apr 20 '12 at 19:44
0

MSMQ can store messages even if temporary shutdown the service or reboot computer.
Main goal of WCF is transport message from source to destination. Doesn't matter what is the transport. In your case MSMQ is transport for WCF and not obvious to have online / available both client and service simultaneously. But when message is received, it's your responsibility to correctly process it, despite what transport was used to send message.

paramosh
  • 2,258
  • 1
  • 15
  • 23