Flex/Actionscript Multiple Consumer Handling Issues
I have recently come across an issue that has been causing me grief for a while now because it was so hard to reproduce. Basically we have quite an advanced application (FusionAnalytics) that waits for data from a JMS channel. We have many different Views of data and each have their own data delivered on the queue. This means that we can have about 1-500+ objects each listening for messages. When a message is received, each object checks to see if it can handle the data it receives and if not ignores it. This has been working fine for a while now (or so I thought).
When an object is waiting for data it turns on a spinner indicator so the user knows that the application is doing something. Various members of the team kept seeing an issue where the spinner would never be turned off, so it looks like that the system never receives a message at all or fails in some other way. Testing it myself the messages are received and handled correctly but other members have this case where randomly it would die. I spent weeks tweaking the application to hopefully fix this issue that I never saw in the first place – so not easy trust me – needles and haystacks should come to mind. Today, my good friend and colleague John Hawksley managed to reproduce it again and placed a ticket in explaining the exact details of how he replicated it (Which was very useful), but when we came to discuss the issue I could not reproduce it on my box, again. It’s so weird that even big boss came over to have a look and see how I could not also be getting the issue. This was testing it against my own dev server so instead I tried to debug the application using John’s server in Germany. Immediately upon connecting I had the “Spinner of Death” on every single view. That is weird I thought – even John didn’t see this on his own client using his own server.
On first glance it would seem that I may have overlooked the weak reference flag in our HashMap, therefore causing the context objects we use to hold information about the data to get GC’d. After changing all this around and altering some other data structures to make sure that it was this I booted up again – BANG – exactly the same thing. That’s very weird I thought. By now I have been debugging this thing for hours and compared it against my local server version. What should happen is that each object waiting should receive each item that comes down the queue.
So lets say I have 10 consumers and are expecting 10 messages we should expect that a handler should be called 100 times = 10 x 10. This was the case for the local environment server but not for the remote server. This is the bit that I have still not worked out – I know it has something to do with time in some way but I cannot tell you how or why and also I have not tested it using multiple queues – perhaps its an issue that affects multiple consumers on the same queue. Each message on the queue is handled at least once, so each message is being received at the client but for some reason it is not being dispatched correctly to all the handlers.
I can guarantee that most of you will never come across this case because the chances are that you are only going to have one, maybe two consumers at a time. The application that we have created is very data intensive as its an advanced analytic tool, plus its also a development framework that means it’s very lousely coupled allowing it to be used for multiple purposes (there is too much about it to write here so you will have to check out the website when it has been released). Any application that you are connected to (you are able to connect to multiple instances of the analytic tool) are running on the one server so it listens on its messaging queue, but each object (view) is channelling different data from the line so they all need to listen on the same message queue.
What was my solution?
I created my own dispatcher for the consumer channel. Like I said, each message is received and handled at least once, it’s just that you cant guarantee that it will be handled by all the objects that need the message. My dispatcher allows each object to post information about the data it is expecting and when that data is received it will dispatch the data to all the handlers that required it. This allows there to be only one consumer channel listening for data on the queue for each application. According to my theory (and I have been testing it for a while now) each message is received (excluding anything like connection errors etc) once so I can then dispatch it myself. So my advice is to remember to use as few consumers as possible in the client – one consumer for each channel. Don’t give youself more work to do later when this thing pops up and believe me, no doubt they will. It’s very hard to track down as it’s so random in everyday use.
I will post back my findings when the application has been in testing for a longer period of time and see if it was the bug casusing the “Spinner of Death”.
P.S. If Adobe would change the RemoteObject request paradigm so that it did not pack multiple requests into one server request then we wouldn’t be having this problem in the first place. When you have some requests that take much longer to execute than the others it really does affect the performance of the application because you have to wait for the longest request to finish before you get a response. Although I can see the idea of it, it would have been nice to be able to switch it off. Very very bad idea not to allow this. There has been a bug committed to adobe about this, but whether they see it as a bug is a discussion for another day.
Update. It seems like this was the issue causing the “Spinner of Death” since it has been in action now for a while and the issue has not reared its ugly head again.
