I have 5 parameterized threads pointing to 5 folders where the parameter would be the name of the folder. Let's say FolderNames are A, B, C, D, and E.

Each folder will have multiple files on which certain operations need to be performed.

The operation to be performed on all the files under each folder will be the same i.e the Task will be the same. This entire thing needs to be constantly running i.e it has to be in a Schedule.

Things I have tried :

1. Scheduled with MultiThreading (i.e 5 parameterized threads working on the same Task) -> but this will cause overlapping of threads as the task is common and desired output won't be generated.

2. Scheduled with MultiTasking i.e by creating an individual class for each folder which implements Runnable and using executor.scheduleAtFixedRate for each class. This will result in Sync operation which means until the operation on the 1st folder isn't over, processing of the other 4 folders won't start. We can't increase the corePoolSize in newScheduledThreadPool as it will create same overlapping issues like Point 1.

So I am looking for help to solve this issue where my threads don't overlap on the underlying tasks.

Summary of above problem in Pictorial Form

Dummy code for scenario 2:

class FolderA implements Runnable{
    private final String fileName;

    FolderA(String fileName){
        this.fileName=fileName;
    }
    @Override
    public void run() {
        ScheduleJob.insideRun(fileName);
    }
}

class FolderB implements Runnable{...}
class FolderC implements Runnable{...}
class FolderD implements Runnable{...}
class FolderE implements Runnable{...}

public class ScheduleJob{
    ScheduledExecutorService executor = Executors.newScheduledThreadPool(1);
    
    FolderA folderA = new FolderA("folderA");
    executor.scheduleAtFixedRate(folderA, 60,60, TimeUnit.SECONDS);
    
    FolderB folderB = new FolderB("folderB");
    executor.scheduleAtFixedRate(folderB, 60,60, TimeUnit.SECONDS);
    
    --and same for Folder  C D and E
    
     public static void insideRun(String folderName){
     
     //Contains call to various operations that need to 
     be perfomed on each file present in Folder A B C D and E
     
     Operation order : Read, Process, Write for each file
     
     }
    
    
}

CodePudding user response：

Your Question is not clear, but I am guessing you want the processing of files in one folder to not hinder or block the processing of files in the other folders.

Multiple ExecutorService objects

Create more than one ExecutorService objects. Each executor service is focused on a single folder. If you have five folders, have five executor services, one executor service per folder.

If you want only one file per folder to be processed at a time make each executor service single-threaded.

Define your task (your Runnable or Callable) as taking an argument indicating which folder to process.

public void FileProcessor implements Runnable
{
    // Constructor
    public FileProcessor( Path path ) { … } 

    // Implement `Runnable`
    @Override
    public void run() { … }
} 

Define your folders.

List< Path > folders = List.of( pathToA, pathToB, … ) ;

Feed those to constructors of executor services.

List< ExecutorService > executorServices = new ArrayList<>() ;
for( Path folder : folders )
{
    ExecutorService es = Executors. newSingleThreadExecutor() ;
    executorServices.add( es ) ;
    es.submit( new FileProcessor( folder ) ) ;
}

Then use that collection executorServices to gracefully shut down all the executor services.

For scheduled executor services, same idea. Change the type from ExecutorService to ScheduledExecutorService. Call Executors.newSingleThreadScheduledExecutor. Change submit method to one of the schedule methods.