0

i have configured cached session factory and i have sftp inbound adapter, this downloads the file for some time then stops, even after restarting the process it just hangs and doesn't establish JSCH session with sftp server

@Autowired
@Qualifier("cachedSftpSessionFactory")
private DefaultSftpSessionFactory defaultSftpSessionFactory;

@Bean(name="sessionFactoryCached")
public CachingSessionFactory getCachingSessionFactory(){
    CachingSessionFactory cachingSessionFactory = new CachingSessionFactory(defaultSftpSessionFactory,8);
    cachingSessionFactory.setTestSession(true);
    return cachingSessionFactory;
}

@Bean(name="SftpSessionFactory")
public DefaultSftpSessionFactory getDefaultSftpSessionFactory() throws Exception {
    DefaultSftpSessionFactory defaultSftpSessionFactory = new DefaultSftpSessionFactory();
    defaultSftpSessionFactory.setPort(port);
    defaultSftpSessionFactory.setHost(host);
    defaultSftpSessionFactory.setUser(user);
    defaultSftpSessionFactory.setPassword(password);
    defaultSftpSessionFactory.setProxy(getProxySOCKS5FactoryBean().getObject());
    defaultSftpSessionFactory.setSessionConfig(getPropertiesFactoryBean().getObject());
    return defaultSftpSessionFactory;
}

@Bean(name="sessionConfig")
public PropertiesFactoryBean getPropertiesFactoryBean(){
    PropertiesFactoryBean propertiesFactoryBean = new PropertiesFactoryBean();
    Properties properties = new Properties();
    properties.setProperty("StrictHostKeyChecking", "no");
    propertiesFactoryBean.setProperties(properties);
    return propertiesFactoryBean;
}

@Bean(name="proxySocks5")
public ProxySOCKS5FactoryBean getProxySOCKS5FactoryBean(){
    ProxySOCKS5FactoryBean proxySOCKS5FactoryBean = new ProxySOCKS5FactoryBean(proxyHost, proxyPort);
    return proxySOCKS5FactoryBean;
}

<bean id="remoteMetadataStore"        class="org.springframework.integration.metadata.PropertiesPersistingMetadataStore">
        <property name="baseDirectory" value="/externalSftpFolderMetadata"/>
        <property name="fileName" value="remoteMetadatastore.properties"/>
</bean>
<bean id="localFilter" class="org.springframework.integration.file.filters.AcceptAllFileListFilter"/>

<bean id="compositeCSVFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
        <constructor-arg name="fileFilters">
            <list>
                <bean class="org.springframework.integration.sftp.filters.SftpRegexPatternFileListFilter">
                    <constructor-arg name="pattern" value=".*\.(csv|CSV)"/>
                </bean>
                <bean class="org.springframework.integration.sftp.filters.SftpPersistentAcceptOnceFileListFilter">
                    <constructor-arg name="store" ref="remoteMetadataStore"/>
                    <constructor-arg name="prefix" value="SftpService"/>
                    <property name="flushOnUpdate" value="true"/>
                </bean>
            </list>
        </constructor-arg>
</bean>

<int-sftp:inbound-channel-adapter
        id="ftpInboundAdapter"
        channel="ftpInputChannel"
        session-factory="cachedSftpSessionFactory"
        remote-directory="/data"
        local-directory="/externalSftpFolderSync"
        auto-create-local-directory="true"
        remote-file-separator="/"
        temporary-file-suffix=".writing"
        delete-remote-files="false"
        preserve-timestamp="true"
        local-filter="localFilter"
        filter="compositeCSVFilter"
        auto-startup="true">
    <int:poller fixed-rate="1000"/>
</int-sftp:inbound-channel-adapter>

i have tried keeping int:poller fixed-rate="1000" and testSession= true, but none of these helped. referred :- https://github.com/spring-projects/spring-integration/issues/2605, https://docs.spring.io/spring-integration/docs/5.1.0.RELEASE/reference/html/sftp.html#sftp-jsch-logging

file should be downloaded as in when they are available on remote server, its a long running process and i have multiple inbound adapter as need to download file from multiple directory from remote server, all adapter are using same channel and shares same cachedSessionFactory.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
Uday Singh
  • 155
  • 1
  • 8
  • How does it work if you don't use `CachedSessionFactory`? Sorry, it is very hard to give you an answer since it feels like the problem is out of Spring Integration scope. – Artem Bilan Jun 24 '19 at 17:23
  • @ArtemBilan I am downloading around 100 odd files from around 45 directory, even if i want i can't ignore usage of cached session factory, as from sftp server maximum number of connection i(client) can acquire is 20 i guess, could you please suggest some other approach? can this help https://docs.spring.io/spring-integration/docs/5.1.0.RELEASE/reference/html/sftp.html#sftp-rotating-server-advice – Uday Singh Jun 25 '19 at 10:04
  • i tried that default session factory as you asked got error:- org.springframework.integration.handler.LoggingHandler:182 - org.springframework.messaging.MessagingException: Problem occurred while synchronizing remote to local directory; nested exception is org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is java.lang.IllegalStateException: failed to create SFTP Session... Caused by: org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is java.lang.IllegalStateException: failed to create SFTP Session – Uday Singh Jun 25 '19 at 10:27
  • i get below 2 errors repeatedly while using DefaultSessionFactory 1)Caused by: com.jcraft.jsch.JSchException: Session.connect: java.net.SocketException: **bold** `Socket closed` at com.jcraft.jsch.Session.connect(Session.java:565) at com.jcraft.jsch.Session.connect(Session.java:183) 2)Caused by: java.io.IOException: inputstream is closed at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2911) – Uday Singh Jun 25 '19 at 10:34
  • OK. Try to make it with the `public DefaultSftpSessionFactory(boolean isSharedSession)`. Also, please, consider to use a `fixed-delay` for the poller. It fully doesn't look like you need a parallel sessions for your use-case. It is going to cache all the matching files into memory on a first poll anyway. It really looks like your SFTP server doesn't allow to have too many sessions. – Artem Bilan Jun 25 '19 at 14:04
  • not sure what you meant by "you doesn't need parallel session" i have multiple sftp-inbound-adapter pointing to the same channel which require session from cached session factory so all adapter around 55 downloading 220 files from multiple directory(55 directory from one sftp server),should get parallel connection as per me, as per i understood sharedsession is usefull if we have multiple channel, that's what i read on spring page, let me try what you have suggested then will update. – Uday Singh Jun 26 '19 at 08:19
  • That’s correct, but according your config, you don’t use shared session. And I suggest to use that instead of cache – Artem Bilan Jun 26 '19 at 12:47
  • i have used isSharedSession as true, without the cached session factory(used that default session factory in each adapter) it was giving similar error as when only using DefaultSessionFactory without sharedSession, error above and in addition to that it was saying session dropped at few times. when i used SharedSession true with defaultsession factory passing that to cachedSesssionfactory, it only created one thread and it hanged after few hours. do we need to do some setup in channel as i think adapter stops polling after few hours. – Uday Singh Jun 27 '19 at 07:15
  • what did "single poller" meant in this issue https://github.com/spring-projects/spring-integration/issues/1980 commented by ulmermark, it seems he had similar requirement – Uday Singh Jun 28 '19 at 14:41
  • I'm not sure: that was 3 years ago. There were a lot of other changes around. I think the point is that you too many session for the same SFTP account when there is a limit on the server... – Artem Bilan Jun 28 '19 at 14:52
  • Well, "single poller" over there means that there is only one Source Polling Channel Adapter. Doesn't look like it is your case. – Artem Bilan Jun 28 '19 at 14:54
  • `>it just hangs...` If you don't see an active poller thread, I suspect the thread is "stuck" someplace, either in your code, or waiting for data from the server; you should take a thread dump when this situation occurs to see what the poller threads are doing. – Gary Russell Jun 28 '19 at 15:14
  • so situation is like this:- 1) i have many (55)sftp-inbound adapter polling same sftp server 55 directories, all sftp-adapter publishing to same channel(is it "single poller" for me?) and to one(xyz) folder. 2) then i have file-inbound channel adapter, which is taking (xyz) folder and i have an outbound file adapter which publishes to (abc) folder, both these file inbound and outbound adapter is activate by a service activator. sftp adapters job is to just download and put the files into xyz folder. then these file adapter do the job please suggest which thread dump i should be looking into? – Uday Singh Jun 29 '19 at 15:16
  • it got fixed by remove acceptOcnceFilter from my fileinboundchannel adapter, which started consuming the files from download folder location of sftp-inbound-adapter and subsequesntly all adapter started loading file, earlier files were getting stuck in download location due to TaskRejectedException, and other sftp adapter use to wait as file just use to lie there, there should be consumption also for next adapter to come in play. – Uday Singh Jul 09 '19 at 15:21

0 Answers0