Looking for advice on message queue > handler design

So i'm looking at implementing a message queue to deal with some system scenarios like sending emails/notifications, dealing with events fired by other components outside of my system's solution (to avoid creating unnecessary dependencies on those systems, and also prevent those systems from creating dependencies on mine; agnosticism). I have some simple design questions about the relationship between the queue especially as it has to do with multi-casting.

It was my understanding that each queue on the queuing system should represent a single 'kind' of message i.e. it should be very focused in its purpose. Likewise, there should be a single handler (not counting additional instances of the handler, when scaling) for that particular queue. That is one handler listens to one queue from which it processes. So if you have Queue 'A' then you'll have n number of Handler 'A' that processes that one queue — let's say that queue is for handling emails; all instances of handler 'A' (1 through n) only deal with emails request messages from that queue.

So now, if I wanted something else to happen when an email message is inserted into the queue, I can use multi-casting; basically I set up a different queue that will receive the same message, but is purposed for something else, call that Queue 'B'. The application that queues the email request message only sends the one message to the queuing system, but that message lands in two different queues, 'A' and 'B'. Now, whatever this additional process is that is processing queue 'B' (maybe it is just submitting the email to some database) can do whatever it wants with it, but it only does that one thing and nothing else.

So that was a lot of jibber jabber, what is my question?

It seems that my understanding of this 1:1 relationship between the queue and the handler implies that we could end up with a 100 different windows services (handlers) that all only do one single thing on a schedule. This makes sense to me but also tickles my "Uh… there's gotta be a different strategy" nerve. So now I'm thinking, ok… well we could probably consolidate a little. For example, instead of just email handlers, we can have a notification handler that deals with email, sms, and push notifications and the message would have a flag, indicating which one it is. That could help… but at the same time, I see that kind of consolidation (or overloading) of services to become problematic in the future in terms of scalability; we'll hit the process cap on a single instance more quickly and if one handler goes down, you lose multiple pieces of functionality and so on.

I'm looking for advice from anyone with experience with large systems and message queues. How did you do your handlers? Single purpose? Overloaded? What did you learn?

by dasjestyr via /r/csharp

Leave a Reply