multiple requests question

BIZ TMS XData
1

My client app needs to submit a batch of requests that will be given to my service that will forward them on to another service individually, get the responses back, pick out some specific data, then return them to the client. Right now I'm processing them synchronously, sending each one to my service and waiting for a response before processing the next one.

I'm getting ready to do them asynchronously, because it will speed things up considerably. The client spends most of its time waiting, and they can be run in parallel by the remote service by sending a bunch of separate async requests. (We're talking up to 100 per batch, but commonly <50.)

I'm wondering if there's any particular reason not to just process them in parallel in my client and blast them all out to my XData service separately to be processed as they are now, but asynchronously? Or if a different approach would be better?

2

Just curious why your client wouldn't just send them all in one batch to the XData server rather than separately? And then wait for the responses (async if possible) in another batch to come back? Can still be async, but less overhead and less back and forth?

3

That's what I'm wondering about. I guess the thing I'm not clear about with that is ... how do I get, say, 50 callbacks from one query? Or do I need to poll the service? Polling seems to defeat the whole purpose. The queries themselves are relatively short.

4

Lots of factors at play here, depending on so many things. Hard to know what's best without more information.

But say your client is looking up weather data for a selected list of cities. It could pass these individually to your XData server, which passes them individually to an external weather service and then returns them individually as part of that call from the client. The XData server is just serving as a middleman in this process, doing nothing but shuttling requests between the client and the external weather service.

I'm thinking it would be better to pass the list of cities in one XData service call, let XData parse the list of cities and send out the individual requests if that's the only option available for communicating with the weather service. Then collect the results and pass them back to the client. Naturally the client would want that to be async as it may take the XData server awhile to collect up all the responses.

Potentially the XData server could even cache the responses from the weather service if appropriate. But there's only one call from the client to the XData server, requesting the cities and returning the data after some delay.

There isn't a way I've heard of to have multiple responses to a single service call, so the data would have to be collected up in the server and returned duing that one service call.

1 Like
5

Depending on the timeliness of the responses, XData could be set to return whatever it could after a set timeout, for example. Or if you have a big picture of the world with temperatues displayed for many citiies, you could batch requests to the XData server or just send a request for citiies that haven't been updated for awhile, rather than sending a complete list and waiting a long time for the request to be satisfied which might result in a poor UI update display (nothing for 10s then everything updates at once vs. regions updating independently, sort of idea).

1 Like
6

Your analogy is reasonable.

The queries typically take 8-12 seconds each. If there's a problem, they have a 30-sec timeout.

If I submit 50 of them serially, that can take quite a while. But since they spend most of their time waiting, it should process the whole batch asynchronously in 20 seconds or so -- a few seconds longer than the longest one.

But I'd like to display updates on the page that shows results as they're completing. It looks more interesting than showing a whirlygig for 20-30 seconds.

Right now I'm sending one request at a time. I'll give it a try to see if a blast of them at once will cause any problems.

But it would be easy to put the queries into a stringlist and send that up to the service, then have it go through that list and blast them out asynchronously, collect the results, then return them when it's all done.

I'm not sure there are any tradeoffs unless the volume of simultaneous requests to the service starts growing steadily and risks saturating the server. But I imagine I can always just add more CPU cores to the account.

7

Mihgt be worth your while to plug in some numbers and see where things are likely to break. Not too many variables in play. So let's go with the weather example, and assume a single client makes 50 requests and it takes 12s for the weather service to respond to each request. Let's also assume for now that the weather service is capable of responding to any volume of requests it receives simultaneously, just that it always takes 12s for each.

Scenario 1. Your original situation where the requests are synchronous. So 50 requests will take 600s or 10 minutes to complete. Practically zero performance impact on the client as it isn't doing anything most of the time. Same for XData. Just an unhappy user waiting for 10 minutes.

Scenario 2. Requests are asynchronous. So the client makes 50 calls to XData, which then makes 50 calls to the weather service and responds with each after 12s. Bit of a performance hit on the client as the requests are queued up, and then nohting while waiting for them to be completed. XData has to handle 50 incoming connections with 50 threads, and then 50 outbound connections. It is likely that this will work very well, as no limits are being tripped and the total time will be slightly more than 12s but probably well under 20s. In my adventures, round trip response times to XData involving a database query are typically less than 200ms, so not really a problem at all.

Scenario 3. Requests are asynchronous and batched together. So the client makes one call to XData. And XData makes 50 calls to the weather service. Might be marginally longer to respond due to the overhead of parsing the list of cities requested and combining the results in the response, but likely in the same ballpark as Scenario 2. But 1/50th of the overhead in terms of threads, CPU, TCP connections, memory, etc.

So then the question is about what is going to break if one of the key variables changes?

Say you have 1,000 users?

In Scenario 1, nothing really changes other than having 1,000 unhappy users instead of 1 unhapy user. I'm thinking XData can plow through 1,000 connections at a time without falling over unexpectedly. It might slow down a bit depending on its resources, but a modern server with a reasonable amount of RAM is probably going to be just fine here.

In Scenario 2, I think things are more problematic. 1,000 users each making 50 connections means more like 50,000 connections. That's probably going to be an unpleasant experience no matter how well equipped XData is and wil likely require multiple servers and a load balancing scheme. Always doable but suddenly much more difficult.

In Scenario 3, it's back to just 1,000 incoming XData requests, so likely not going to be a big problem. A bit more of a hit than Scenario 1, but manageable.

Say you have only 100 users but each makes 500 requests instead of 50?

In Scenario 1, well, hard to imagine a situation where a user would wait more than an hour and a half for data to update. Unlikely to be feasible.

In scenario 2, we're back to 50,000 connections again and bad news all around.

In scenario 3, we're only having to deal with 100 inbound connections to XData, so all should be ok.

Another consideration is where the bottleneck might be in terms of the network. If you've got high-powered desktop clients on the same LAN as the XData server, then not really a concern. Same if your clients are all happy broadband users in the same country as your XData server. But say you've got users in a foreign land using mobile devices on wireless networks?

Scenario 1 is probably fine, so long as the data coming back is relatively small. If it is returning satellite images in the weather data instead of just daily highs/lows, then this might not work so well.

Scenario 2 is not going to work so well as that may be making demands beyond the capabilities of their network connection, and likely many connections would be dropped.

Scenario 3 is probably just fine as well, with the same caveat about the data being small.

Lots to think about, maybe a little bit of measurement, but I think there are quite a few options available to you depending on your specific situation. Would be interested to know what you end up with at the end of the day, and maybe some numbers to show how you got there?

1 Like
8

Oh, and if the weather service isn't ideal and doesn't take kindly to being overloaded with 50,000 requests simultaneously, having XData serve as a cache may be a good idea. Even if it is very short term. Weather data may not be the best example as it is unlikely to ever be updated more than once an hour, so caching is perfectly reasonable. But even market data can be cached at times, like if markets are closed or perhaps for very short periods of time. If market data is delayed by 15mins for example (often the case) then having that cached for a short period of time may be acceptable. Naturally you know your data and whether any caching is acceptable, but if you can, it can make a big difference. Particuarly when the entire batch can be served out of the cache.

Also helps if you can send multiple requests at a time to the weather service. So being able to pass it multiple cities in the same way as we're doing for the client. Or maybe if the service has set batches that are supersets of your data that can be used to extract what you need. For example, maybe it has an interface for individual cities, and a separate interface for countries that return all the cities in that country. It maybe worth the extra effort to check if it makes sense to get a couple of countries' data (at a higher cost) that contain all the cities rather than the cities individually. But our example is getting a bit stretched at this point!

1 Like
9

LOL! Well, the number of customers that would randomly lead to 1000 requests at the same time would be mind-boggling. Even 1000 customers would be great, but the average utilization per custome of this service I expect will be extremely low, a batch of 50-100 requests a few times a week. I guess someone could abuse it, but they're paying for the queries being processed anyway, so I'll deal with it.

I'll probably start out with separate async requests, and if there's more traffic than expected over time, then I can box them up and send them as a single batch request to the XData service. (And nothing needs to be parsed. Just iterate over the requests sent in a stringlist and send them out.)

Thanks for the detailed analysis, tho. :smile:

1 Like
10

You're welcome! It wasn't so much about having 1,000 concurrent users... we should all be so lucky :grin: but more about what is likely to break first and prepare for that, keeping in mind the UI experience you're after. I think there's enough technology and know-how to scale to any number of uers, just a matter of being prepared for them. Easier if they're paying customers of course.

1 Like
11

What would break first? Probably the vendor's server!