How to handle copies of large zip files with Spring Integration-CodePudding

This topic is similar to : Spring integration problem when reading file from ftp but there is no clear answer and the scenario here is not exactly the same.

I have an input directory on a server where zip files are copied into from another remote server. Basically files are copied by an external file synchronization tool (between a source dir on server A to the destination dir on server B), but could also be copied by scp or ftp tools for test purposes.

The zip files are processed by Spring Integration Zip module this way :

@Bean
public IntegrationFlow zipFilePollingFlow(
        @Value("file:///${zip.inputDir}") File in,
        UnZipTransformer unzipTransformer,
        UnZipResultSplitter unZipResultSplitter,
        TransactionManager pseudoTransactionManager) {

    FileInboundChannelAdapterSpec inbound = Files.inboundAdapter(in).autoCreateDirectory(true);

    return IntegrationFlows.from(inbound, pm -> pm.poller(p -> p.fixedDelay(5000)
                .transactionSynchronizationFactory(zipIntegrationTxSynchronizationFactory())
                    .transactional(pseudoTransactionManager))) // Zip process is done transactionnaly
            .handle(fileCleanService) // Deletes files in another directory
            .transform(unzipTransformer)
            .split(unZipResultSplitter)
            .<File, Boolean>route( ... ) // the rest of the flow
            .get();
}

Everything is fine with small zip files. However when large files are copied/transferred (hundred of MBs), we are falling in this error :

o.s.integration.handler.LoggingHandler : org.springframework.integration.transformer.MessageTransformationException: failed to transform message; nested exception is org.springframework.messaging.MessageHandlingException: Failed to apply Zip transformation.; nested exception is java.lang.IllegalStateException: Not a zip file: /.../temp/extract/zip/full-extract-rev.zip

Caused by: org.springframework.messaging.MessageHandlingException: Failed to apply Zip transformation.; nested exception is java.lang.IllegalStateException: Not a zip file: ....

How can we handle and process correctly large zip files with Spring Integration ? Would it be possible for the Zip file transformer to retry the process until the transfer of the source zip file is complete ?

CodePudding user response：

The alternative is to configure that FileInboundChannelAdapterSpec with a LastModifiedFileListFilter:

 * The {@link FileListFilter} implementation to filter those files which
 * {@link File#lastModified()} is less than the {@link #age} in comparison
 * with the current time.

So, it will let the file to pass only when it is old enough.

You also need to combine this filter with an AcceptOnceFileListFilter into a ChainFileListFilter to avoid reading the same file again and again.

Not sure if there is an option to configure a directory on file system to let only an exclusive access to the file (well, it is there on Windows by default), so then we would be able to write a custom filter with an attempt to open the file. If we fail, we don't path the file down to AcceptOnceFileListFilter in the ChainFileListFilter: https://docs.spring.io/spring-integration/docs/current/reference/html/file.html#file-reading