MSMQ clustering - odd behavior

[firehawk]

Hi.

so ive been able to create a successful MSMQ clustered environment on VMWare using WS2008R2.

all has been working well. I can create a transactional MSMQ and send messages to it no problems.

my client though, they have also created a cluster of some kind but VERY odd behavior.

when they send messages, it appears to be sent but it is not in the queue. nothing on the outbound queue or even on the clustered incoming queue.

so I took a quick look.
I created a transactional queue and used the same app that I used in my VMWare environment on theirs.

First I figured out that the server name they supplied to send/connect to the MSMQ was incorrect. So I used the "correct" clustered MSMQ server name.
After that, I was able to send messages to the queue. I could see the message on the incoming private queue. great.

I then purged the queue

I then resent messages. This time - no messages! Then took a while and the messages appeared.
I deleted the queue, recreated it, sent messages and then saw the message.
I deleted it again, created the queue once again, sent messages and no messages to be found on outbound or incoming clustered MSMQ!

what on earth is going on? I made the queue transactional to ensure reliability and durability.

anyone know what to look for next? I am leaning on some network issue somewhere. Nothing obvious on their eventlogs.

any help/guidence/hints appreciated! its quite odd. sometimes it works, other times it does not but on my VM Environment works flawlessly. Oh - they are using WS2008 datacentre edition. I am using WS2008R2 enterprise edition (shouldnt make much difference)

 

[PieMaster]

Are they sending the messages from one of the cluster nodes by any chance?

 

[firehawk]

im not sure. I tried to do that and thats how it was working then didnt work..then worked...and now not so much.

on my own environment I did try sending it from the clustered MSMQ itself (well, one of the nodes in the cluster) and worked fine. but also from outside the clustered nodes and that also worked fine.

they do have DTC enabled for inbound and outbound comms and with no authentication required

 

[n3vrmind]

how are you viewing MSMQ? are you opening the MSMQ console from the actions page in the failover clustering console after you selected the cluster or just running it off the admin tools?

edit: are they also clustering DTC? if so you have to update DTC security settings on both the local and clustered DTC instances

 

[firehawk]

from the cluster manager....

right click on the clustered MSMQ then "Manage MSMQ"

 

[n3vrmind]

If they are sending messages and not getting an error then id expect its going into a queue somewhere, if its not in the clustered queue then perhaps in the local queue of one of the nodes failing that id check from their client that is sending the messages as to what server name they are sending to and where that resolves to (in case of host file entries, or bad dns records)

 

[firehawk]

yeh.
I mean, I even tried with the IP Address and still the same problem.

right now, I created a new transcational queue. able to send messages to it.
Then when i run a WCF app to read the queue - it cant read it because apperently either the queue does not exist or I dont have permissions... the app crashes (I expect it to crash)

then when I try to resend messages to the same queue... it appears to be "sent" but they dont exist anywhere.....

 

[n3vrmind]

Well theres a bit of strangeness when you cluster MSMQ, not all things that work on a standalone MSMQ work on a clustered one

Like i scripted the configuration of queues and queue permissions in powershell, works completely fine against a standalone instance but no matter what method i tried (via COM, via .net) I couldnt find a way that would work against a clustered instance

I think the issue for me came down to what kind of binding you used in the formatname string, only some of them work against clusters and the ones I needed were the ones that didnt

The other possibility is the one you mentioned, by default only the account that created the queue has permissions so did you perhaps create it under another account?

 

[firehawk]

I actually didnt create the account - it was the "IT" team.
The format I use to connect to is:

OS:ClusterName\private$\queueName

or if its TCP:

TCP:IP.Address.Goes.Here\private$\queueName

But I dont get it - it works PERFECTLY on my own VM Environment....
the one difference i noticed is that I dont have clustered DTC but they do. shouldnt make much different anyway.