My team maintains an application (written on Java) which processes long running batch jobs. These jobs needs to be run on a defined sequence. Hence, the application starts a socket server on a pre-defined port to accept job execution requests. It keeps the socket open until the job completes (with success or failure). This way the job scheduler knows when one job ends and upon successful completion of the job, it triggers the next job in the pre-defined sequence. If the job fails, scheduler sends out an alert.

This is a setup we have had for over a decade. We have some jobs which runs for a few minutes and other which takes a couple hours (depending on the volume) to complete. The setup has worked without any issues.

Now, we need to move this application to a container (RedHat OpenShift Container Platform) and the infra policy in place allows only default HTTPS port be exposed. The scheduler sits outside OCP and cannot access any port other than the default HTTPS port.

In theory, we could use the HTTPS, set Client timeout to a very large duration and try to mimic the the current setup with TCP socket. But would this setup be reliable enough as HTTP protocol is designed to serve short-lived requests?

CodePudding user response：

When your worked with socket and TCPprotocol you were in control on how long to keep connections open. With HTTP you are only in control of logical connections and not physical ones. Actual connections are controlled by OS and usually IT people can configure all those timeouts. But by default how it works is that when you even close logical connection the real connection is no closed in anticipation of next communication. It is closed by OS and not controlled by your code. However even if it closes and your next request comes after that it is opened transparently to you. SO it doesn't really matter if it closed or not. It should be transparent to your code. So in short I assume that you can move to HTTP/HTTPS with no problems. But you will have to test and see.

Also about other options on server to client communications you can look at my answer to this question: How to continues send data from backend to frontend when something changes

CodePudding user response：

One possible solution to this problem would be to use a message queue, such as Apache Kafka or RabbitMQ, to communicate between the containerized batch job and the scheduler. The batch job could publish a message to the queue indicating that it has started, completed, or failed, and the scheduler could consume these messages to determine the status of the job and trigger the next job in the sequence. This approach would allow you to use the default HTTPS port for communication, while still providing a reliable and robust way to manage the sequence of batch jobs.

Another approach would be to use web sockets over the HTTPS port, which are designed to handle long-lived connections and can be used for real-time communication between the batch job and the scheduler. This approach would also allow you to use the default HTTPS port, but it would require more development effort and additional configuration to implement the web socket communication.