So I'm currently using socket.io for a loading screen and it just isn't working so I'm going to try a different solution. I need the following to happen: user opens up my app--this triggers a function on the server side (CPU intensive scraping function that takes about 8 seconds). As soon as function is done, I need to notify the client that the function is done and the client fetches the results (after this result is fetched the app loading screen closes and the user enters the app).

My current implementation is using socket.io emissions and processing the scraping function via Redis queue on a background thread.

Is there a different approach to doing this instead of a socket connection and sending emissions from client -> server and vice versa?

I don't think I can just to a classic GET request, because isn't it bad practice to leave a long running (8 second) request open for so long while waiting for response? Also, I looked into Server-Sent Events but I'm not sure I would be able to use it in this case.

Any insight is much appreciated!

CodePudding user response：

8 seconds is longer than normal for a response from a "healthy" server, but still should be fine. I'd just use a GET from the mobile client, and write it to not block the UI while it is waiting.

Write your UI so that it informs the user that it is waiting for a response, and maybe even give the user an idea of how long the response is likely to take.

CodePudding user response：

A few points to keep in mind.

1. nodejs can usually do a lot of concurrent web-scraping: most web-scraping elapsed time is spent waiting for the scraped servers to respond. nodejs is asynchronous. So worker threads may not help much. They definitely will make your server app more complex. A good choice for scaling this out might be clustering your nodejs app.
2. There's no harm in writing your server to stall the response to a GET request for a few seconds, except harm to the user experience. If your user has something reasonable to look at for those few seconds, you can probably get away with this.
3. If your user is looking at a web page during that stall, you can use xhr or fetch from Javascript code in the page to retrieve that data while you entertain them with a spinner or some such user interface stuff.