background

I'm planning to make a scheduler which runs every minutes. Whenever it runs, different target data will be given and each time it might take about 90 seconds to proceed.

my plan

The process is like :

multiple producers(threads) : 
1. get data from some API
2. using data from 1. parse given data
3. put it into a common queue

↓

common queue

↓

multiple consumers(threads) : 
1. get data from the queue
2. call some API
3. then insert into RDB

(order of each data doesn't matter)

If I understand right, while user threads are running, main thread might finish earlier and the process will keep running till those user threads finish.

problem/question

I want each process has own common queue only for threads which from the same process. Also hope to free the memory of certain queue when each process done.

I mean, is it possible like below? :

1. running process A, only threads from A can access to queue X.
2. running process B, only threads from B can access to queue Y.
3. shutting process A, free the memory of queue X.
4. running process C, only threads from C can access to queue Z.
5. shutting process B, free the memory of queue Y.
6. shutting process C, free the memory of queue Z.

This is my first time dealing with multi-threading. Thank you in advance! :)

CodePudding user response：

Just consider to use java.lang.TreadLocal<T>.

Quote from JavaDoc:

Each thread holds an implicit reference to its copy of a thread-local variable as long as the thread is alive and the ThreadLocal instance is accessible; after a thread goes away, all of its copies of thread-local instances are subject to garbage collection (unless other references to these copies exist).

CodePudding user response：

I would design it this way: Use "Thread Pool" of "Workers" assigned to its own queue. A "Worker" is a class that extends a Thread and have a private reference to its Queue. So, the Thread Pool of Workers is actually a group of threads all listens to the same queue. Then you can define a ProducerWorker that extends a Worker. These producer workers will handle processing the API and put in their common queue. Same way, A ConsumerWorker is a worker in a workers pool which handles consuming "tasks" from its common queue it has reference to. In this design, you have a logical separation of the thread "types" (different workers) so you have a more generic separation not limited to different processes. This is the design. To implement you'll need to read about Thread Pool and Extending a Thread. You can write your own "thread pool" or better use Java's builtin implementation ExecutorService to create pools:
Example:

ExecutorService poolAlpha = Executors.newFixedThreadPool(10);
ExecutorService poolBeta = Executors.newFixedThreadPool(10);
poolAlpha.submit( alphaWorker );
poolBeta.submit( betaWorker );

Here's a pseudo of how the base Worker should look like:

    class Worker extends Thread {
        private Queue myCommonQueue;
        private Runnable myWork;

        // to know when to stop the thread.
        private boolean isRunning = true;

       // Constructor
       public Worker(Queue queue, Runnable work) {
         this.myCommonQueue = queue;
         this.myWork = work;
       }
   }