Retrieve latest modified zip files with Spring Integration-CodePudding

I've written an application that downloads files from a Sftp server. What I want to achieve is to download only ZIP files, but to download them only when they've been modified.

I've written a SftpInboundFileSynchronizer and several InboundChannelAdapter. What is weird is that the same file gets downloaded once and again. I know the key is to choose the right filters, but I don't know how to accomplish it.

public static final String SYNCHRONIZER_BEAN_NAME = "synchorinzer-bean-name";
    @Bean(SYNCHRONIZER_BEAN_NAME)
    public SftpInboundFileSynchronizer synchronizer(
            SessionFactory<SftpClient.DirEntry> sf,
            PropertiesPersistingMetadataStore ms,
            AppProps cfg) {
        SftpInboundFileSynchronizer sync = new SftpInboundFileSynchronizer(sf);
        sync.setDeleteRemoteFiles(false);
        sync.setRemoteDirectory(cfg.getFtpRemoteDirectory());
        sync.setPreserveTimestamp(true);
        // sync.setFilter(); ????
        return sync;
    }


    public static final String GIPUZKOANA_OUT_CHANNEL_NAME = "GIPUZKOANA_OUT_CHANNEL";
    public static final String GIPUZKOANA_SYNCHRONIZER_BEAN_NAME = "GIPUZKOANA_FILE_SYNCHRONIZER_BEAN";
    @Bean(GIPUZKOANA_SYNCHRONIZER_BEAN_NAME)
    @InboundChannelAdapter(channel = GIPUZKOANA_OUT_CHANNEL_NAME)
    public MessageSource<File> gipuzkoanaMessageSource(
            @Qualifier(SYNCHRONIZER_BEAN_NAME) SftpInboundFileSynchronizer sync,
            AppProps cfg) {
        SftpInboundFileSynchronizingMessageSource source = new SftpInboundFileSynchronizingMessageSource(sync);
        source.setLocalDirectory(cfg.getGtfsLocalDirSyncGtfs());
        source.setAutoCreateLocalDirectory(true);
        source.setMaxFetchSize(1);
        source.setLoggingEnabled(true);
        source.setLocalFilter(files -> Lists.newArrayList(files)
                .stream()
                .filter(f -> f.getName().equalsIgnoreCase(cfg.getGtfsGipuzkoana()))
                .collect(Collectors.toList()));
        return source;
    }
    // ...

I've tried so far new SftpPersistentAcceptOnceFileListFilter(ms, "gtfs_"), new SftpSimplePatternFileListFilter("*.zip")... but with no luck.

How can achieve what I want?

Thanks!

CodePudding user response：

Try to use something like this:

    ChainFileListFilter<SftpClient.DirEntry> chainFileListFilter =
            new ChainFileListFilter<>()
                    .addFilters(new SftpSimplePatternFileListFilter("*.zip"),
                            new SftpPersistentAcceptOnceFileListFilter(ms, "gtfs_"));
    sync.setFilter();

This way it will check for file extension first and only then check for its previous state after processing.

Not sure about your "only when they've been modified" since this filter cannot know about such a state. You can try with a LastModifiedFileListFilter modification for SFTP to be sure that the file is old enough to be pulled.

CodePudding user response：

To download only ZIP files and only those that have been modified, you can set the appropriate filters on the SftpInboundFileSynchronizingMessageSource.

You can use a combination of AcceptOnceFileListFilter and SimplePatternFileListFilter to achieve this. The AcceptOnceFileListFilter will ensure that a file is only downloaded once, while the SimplePatternFileListFilter will restrict the files that are downloaded to ZIP files only.

Here's an updated implementation of your gipuzkoanaMessageSource method:

@Bean(GIPUZKOANA_SYNCHRONIZER_BEAN_NAME)
@InboundChannelAdapter(channel = GIPUZKOANA_OUT_CHANNEL_NAME)
public MessageSource<File> gipuzkoanaMessageSource(
        @Qualifier(SYNCHRONIZER_BEAN_NAME) SftpInboundFileSynchronizer sync,
        AppProps cfg) {
    SftpInboundFileSynchronizingMessageSource source = new SftpInboundFileSynchronizingMessageSource(sync);
    source.setLocalDirectory(cfg.getGtfsLocalDirSyncGtfs());
    source.setAutoCreateLocalDirectory(true);
    source.setMaxFetchSize(1);
    source.setLoggingEnabled(true);
    source.setFilter(new CompositeFileListFilter<>(
            Arrays.asList(
                    new AcceptOnceFileListFilter<>(new SftpPersistentAcceptOnceFileListFilter(ms, "gtfs_")),
                    new SimplePatternFileListFilter("*.zip"))
    ));
    return source;
}