java Executor not executing all tasks if task count is not known in compile time-CodePudding

I have a very simple java code snippet:

  ExecutorService executor = null;          
      try {
          executor = Executors.newFixedThreadPool(4);             
          for (int i = 0; i < 10; i  ) {                
              executor.submit( () -> processRule(rule_queue.poll()));           
            }
       }

And here is the thing: if I replace the 10 (which is in my case the count of objects in the rule_queue queue with rule_queue.size() then not all tasks will be executed.

It is a very strange behaviour, for 1-2 fixedThreadPool threads it will work, but for 3 and above fixed threads will usually be only 5-7 tasks executed.

The problem is the count of objects in the queue comes from the database so I can't hardcode it into the for loop.

The method processRule do some database inserts/selects so I also don't want to open too many threads/connections, 4-5 SQL Selects simultaneously would be sufficient.

Thanks in adavance for any help how to run all tasks and how to run 4 in parallel and put all the others (could be up to 300) to the executors queue.

Edit: sorry forgot to write that after this code two additional lines are executor.shutdown() and waiting for finish.

CodePudding user response：

I assume you replaced

for (int i = 0; i < 10; i  ) {                
    executor.submit( () -> processRule(rule_queue.poll()));           
}

with

for (int i = 0; i < rule_queue.size(); i  ) {
//                  ^^^^^^^^^^^^^^^^^
    executor.submit( () -> processRule(rule_queue.poll()));           
}

The problem is that rule_queue.size() is being re-evaluated at each iteration. Consider the case where the initial queue size is 2.

Iteration    i   rule_queue.size()   result
---------    -   -----------------   ------
    1        0           2           submit
    2        1           1           exit loop

So only half of your rules will be submitted. What you want instead is:

while(rule_queue.size() > 0) {
    executor.submit( () -> processRule(rule_queue.poll()));           
}

You would have immediately seen this if you had stepped through the code in your IDE debugger.