|
Find out about how to use nServiceBus to efficiently distribute messages over several servers - and get the solution running in a few minutes! You'll also get a deep understanding about how the system works.
nServiceBus includes a Distributor that's capable of distributing messages over several Servers (message subscribers) in a group. The Distributor works like a load balancer: the Distributor sends one message to an available Server as opposed to sending the same message to all available servers.
This article is the first in a series of three articles that discusses using the nServiceBus distributor:
- I discuss how to get the nServiceBus Distributor to handle messages between a client and two servers in this article. All processes are local to one system for ease of deployment and use.
- The next article discusses deploying the solution in the Cloud - specifically on Amazon's Elastic Compute Cloud (Amazon EC2).
- The last article discusses how to span nServiceBus over several locations using the Internet without using WCF.
The demonstrations in this article are based on Julian Birch's article, "Getting the NServiceBus Distributor Working".
Some Background
This article uses the nServiceBus Distributor and FullDuplex sample to walk you through the process of setting up a system that includes one Client that sends messages to a Distributor. The Distributor forwards the Client's messages to one of two Servers - effectively load balancing the Servers.
When a Client sends a message to the Distributor's data bus, the Distributor forwards the message to an available Server's input queue. The Server that receives the message also gets the original Client's computer and private queue name, so that it can reply directly to the Client if necessary. Once the Server finishes processing a message, it sends a ReadyMessage to the Distributor's control bus queue making it available for work again.
The Distributor automatically reconfigures itself as servers join and leave the logical group of available servers. A Server joins the group of available servers when it announces its availability to the Distributor by sending a ReadyMessage to the Distributor's control bus queue.
Understanding the FullDuplex Sample
The FullDuplex sample demonstrates messaging using Send and Reply. The Send and Reply operations are different from Publish and Subscribe, which you're familiar with from the Pub/Sub sample.
Understanding Publish and Subscribe
In Publish/Subscribe the publisher is aware that a subscriber is interested in the message type that the publisher sends. The Publisher uses the intermediate messagebus queue to publish the message so it does not, in theory, know about the subscriber's queue or computer name. In practice, the nServiceBus code handles the details of getting a message from the messagebus queue to the subscriber's queue.
If a Subscriber has never registered its interest in the type of message that the Publisher sends, then the Publisher won't publish the message. Note that it is not important whether the subscriber is running to consume message - it's only important that the subscriptions queue contains a message from a subscriber for the Publish`er's message type.
Understanding Send/Receive
The Send operation sends a message directly to an endpoint (queue@Computer in this case). The recipient process does not have to be running and it does not have to register interest in the sent message. The Send operation does not use an intermediate queue.
The Receive operation simply picks up messages from the InputQueue.
Settingup and Running FullDuplex
The FullDuplex sample is included in the nServiceBus source and sample distribution. This article uses other parts of the nServiceBus solution, so you'll have to get a copy of nServiceBus (version 1.9 or the trunk) if you plan to follow this article. Refer to my previous article "nServiceBus: Building the Solution" for details.
Use the following steps to run the sample:
- If you do not have the necessary queues on your system, run the "msmq install.vbs" script located in the Samples folder.
- Compile the FullDuplex sample (located in the FullDuplex folder).
- Run the Client.
- Run the Server.
- Press Enter on the Client and note the output in the Client and Server windows.
You'll see that the Client sends a message to the Server, the Server receives the message and adds some information, and finally the Client displays the result. The configuration files provide the details about the names of queues. I cover the configuration files in detail, as they relate to running this sample in the context of the Distributor, later in this article.
Building the Distributor
Before you continue, purge all messages from all of the queues on your system. You'll have to visit each queue and Purge all messages to ensure that the system is ready to work. We'll use a new queue, called "ignore" - so create it and set it as a transactional queue.
You're going to use the NServiceBus.Unicast.Distributor.Runner to get things running, so the minimum build you'll need is the output of "build_src.bat" - if you ran a build with or without the strong name, you don't need to build again (Refer to my previous article "nServiceBus: Building the Solution" for details about the build scripts).
With the Distributor built, you can just run it since its configuration is right for our purpose. The Distributor resides in the following folder:
[YourFolder]\trunk\src\distributor\
NServiceBus.Unicast.Distributor.Runner
[YourFolder] is the folder you created to use for downloading the nServiceBus solution. You can replace "trunk" with the name of the branch you are using.
Understanding the Distributor's Configuration
The Distributor's configuration is in the Bin\Release folder. So the (almost) complete path to the file is (I added line breaks to make the path easier to read):
[YourFolder]\trunk\src\distributor\
NServiceBus.Unicast.Distributor.Runner\bin\Release\
NServiceBus.Unicast.Distributor.Runner.exe.config
Again, [YourFolder] is the folder you created to use for downloading the nServiceBus solution. You can replace "trunk" with the name of the branch you are using.
Since configuration files and features can change over time and since this demonstration is using the trunk version (the branch where lots of changes can occur), here is a copy of the key configuration elements followed by some explanation:
<MsmqTransportConfig
InputQueue="distributorControlBus"
ErrorQueue="error"
NumberOfWorkerThreads="1"
MaxRetries="5"
/>
<UnicastBusConfig DistributorControlAddress="" DistributorDataAddress="">
<MessageEndpointMappings />
</UnicastBusConfig>
<appSettings>
<add key="DataInputQueue" value="distributorDataBus"/>
<add key="NumberOfWorkerThreads" value="1"/>
<add key="ErrorQueue" value="error"/>
<add key="StorageQueue" value="distributorStorage"/>
<add key="NameSpace" value="http://www.UdiDahan.com"/>
<add key="Serialization" value="xml"/>
</appSettings>
The MsmqTransportConfig section sets-up the Distributor to use the distributorControlBus queue for its input. Unlike the PubSub sample where clients publish messages to the messagebus queue, the distributorControlBus handles messages from Servers about their availability. Section three describes where the Distributor gets messages from Clients - however, I discuss section two next.
Section UnicastBusConfig configures the message endpoint mappings. You can think of a message endpoint as a queue, so the message endpoint mapping element maps a type to an endpoint (or a message to a queue). An endpoint isn't always a queue, but a discussion of that detail is out of the scope of this tutorial.
The relevant parts of appSettings section are the definitions of the DataInputQueue and the StorageQueue. The DataInputQueue is the name of the queue that receives messages from a Client - this is the queue from which the Distributor gets messages to forward to Servers.
The StorageQueue maintains the state of available Servers. When a Server becomes available for work, the Distributor puts a message onto StorageQueue and removes it when the Server becomes unavailable. The Distributor is the only entity that's aware of the StorageQueue.
Start the Distributor and watch for error messages near the end of the output on the console. Correct any errors that occur - the most common problem for me was having non-transactional queues because I deleted and added queues when I was testing.
Next up - configuring the Client.
Exploring and Modifying Client Configuration
The Client's configuration is much simpler. The path to the Client's configuration file is (I added line breaks to the path to make the path easier to read):
[YourFolder]\trunk\Samples\FullDuplex\Client\
bin\Release\Client.exe.config
The Client's configuration includes the MsmqTransportConfig and UnicastBusConfig sections:
<MsmqTransportConfig
InputQueue="client"
ErrorQueue="error"
NumberOfWorkerThreads="1"
MaxRetries="5"
/>
<UnicastBusConfig
DistributorControlAddress=""
DistributorDataAddress=""
ForwardReceivedMessagesTo="">
<MessageEndpointMappings>
<!-- changed from original -->
<add Messages="Messages" Endpoint="distributordatabus" />
</MessageEndpointMappings>
</UnicastBusConfig>
The MsmqTransportConfig section simply describes the InputQueue and ErrorQueue - this should be familiar if you have looked at the configuration file in the PubSub sample.
The UnicastBusConfig section's MessageEndpointMappings needs to change from the FullDuplex sample's configuration: change the value of the Endpoint attribute from "messagebus" to "distributorDataBus".
The new configuration maps the Messages type to the distributorDataBus queue - so that messages that the Client publishes get stored in the distributorDataBus queue. The distributorDataBus queue is the Distributor's DataInputQueue.
You don't need to change any other configuration settings, so run the Client and as you did with the Distributor, check for errors and correct them before you continue.
Finally bring everything together by configuring the Server.
Modifying Server Configuration Settings
The Server's configuration is straight-forward - the Server's configuration file is here:
[YourFolder]\trunk\Samples\FullDuplex\
Server\bin\Release\Server.exe.config
The Server's configuration also uses the MsmqTransportConfig and UnicastBusConfig sections. The MsmqTransportConfig section follows:
<MsmqTransportConfig
InputQueue="worker"
ErrorQueue="error"
NumberOfWorkerThreads="1"
MaxRetries="5"
/>
Modify the MsmqTransportConfig section's InputQueue by changing the value from "messagebus" to "worker". You are reconfiguring the Server's input queue to the "worker" queue.
The UnicastBusConfig needs a minor change - edit the MessageEndpointMappings so that it is empty, as shown:
<UnicastBusConfig
DistributorControlAddress="distributorcontrolbus"
DistributorDataAddress="distributordatabus"
ForwardReceivedMessagesTo="audit">
<MessageEndpointMappings />
</UnicastBusConfig>
The message endpoint mapping is empty because the Server isn't Sending messages - the Server just replies to messages so the end point mapping is not necessary.
A demonstration that has a Distributor managing one Server isn't very interesting, so use the following steps to create a second server:
1. Create a new folder
2. Copy the contents of the Server's folder to the new folder you created in the previous step.
The Server's files are in this folder:
[YourFolder]\trunk\Samples\FullDuplex\Server\bin\Release\
3. Edit the Server.exe.config file
4. Change the value of the MsmqTransportConfig section's InputQueue to "worker2".
5. Save the file
So now Server1 uses the "worker" queue and Server2 uses "worker2".
Run both servers and correct any errors that may appear in the console window.
Understanding the System
You should have Distributor, Client, and two Servers running at this point. Send a message to the Distributor by hitting Enter in the Client window. You should see some output in the Client, Distributor, and one of the Server windows. So, what's happening?
The process appears to be complicated, but all of the steps are necessary and use a practical approach to ensuring message durability and low overhead. The following steps trace the events that occur from the moment you start a Server to the moment that the Client finishes handling a Server's response message:
[Gaps between the numbered items highlight the transition between the Server, Client,and Distributor]
1. When you start a Server, it sends a ReadyMessage to the DistributorControlAddress, or distributorControlbus queue.
2. The Distributor picks up the ReadyMessage.
3. The Distributor creates a new message for itself, indicating that the Server is available, and pushes it onto its StorageQueue (called distributorStorage).
4. When you send a message from the Client, it sends the message to the Distributor's DataInputQueue or the distributorDatabus queue.
5. The Distributor picks up the Client's message from its DataInputQueue.
6. The Distributor removes a message from its StorageQueue - the message (from step three) provides the Distributor with the queue and computer name of the next available Server.
7. The Distributor sends the Client's message directly to the Server's InputQueue (using the format queueName@ServerName).
8. The Server picks up and processes the message.
9. The Server wants to respond to the Client, so it sends its response directly back to the client by sending a message to its queue using the Reply method.
10. When the Server is finished processing, it sends a ReadyMessage to Distributor's DistributorControlAddress, or distributorControlbus queue.
11. Steps two and three execute again (to record that the Server is available).
12. The Client receives and processes the Server's response message.
All of the preceding steps execute very quickly when all of the processes are running.
When you run two servers along with the Client and Distributor, you'll notice that the Distributor sends two messages to each Server before switching to the next available server - this happens because each Server sends two ReadyMessages to the Distributor's DistributorControlAddress queue so that two sequential 'available' messages end up in the Distributor's StorageQueue. Observe that in steps two, three and five, six that the Distributor picks up messages from the StorageQueue to determine the next available Server.
This approach to managing available servers is very efficient because Servers just announce when they are available and the Distributor essentially consumes the availabilty messages. Once the Distributor consumes an availability message it moves on to the next next server by consuming another availabilty message from its StorageQueue. So, a server does not have to send a message to indicate that it is busy - it just sends a ReadyMessage to the Distributor.
Summary
In this article you learned about the FullDuplex sample and how to configure and use the sample with the Distributor to manage message load between two Servers. You also learned about the sequence of events that occur as a message flows through the system.
The next article discusses running this sample on several instances of Windows deployed on Amazon's Elastic Compute Cloud (EC2). If things go well, I may also get automatic scaling to work - where the Amazon infrastructure automatically scales the solution by adding and removing instances as processing load changes.
More Information
You can get more information from these sites:
|
