Download a file using http with Spring Integration

Question

Making the plunge into Java from .NET for a long time.

I am looking for is an example on how to periodically download a file, read the text from it and then take some action based on the read using Springs Integration library and the annotation based approach.

I want to pull GTFS formatted zip file from a transit provider. This provider produces a simple text file with a timestamp to indicate the last publishing time.

Specifically the producers of the data publish a text file at:

https://someserver.com/gfts/published.txt

This file has a simple timestamp to indicate when the last time their data file was published.

Then there is the data:

https://someserver.com/gfts/schedule.zip

I have tried to find some examples on how to go about polling the "published" file. Basically I want to periodically download the file and check the timestamp to determine if the schedule should be downloaded.

Most of the examples I have seen are using the XML based configuration with spring - and I barely am holding onto the annotation based. I have also seen examples of downloading a file using FTP / SFTP.

I need to use http AND I also need to include Basic Authorization (in the header).

This is as far as I have gotten. I am not sure how to go about wiring this up?

From the Spring Integration docs - this is how I am supposed to declare an outbound gateway (I think that is what I need?)

The question is now what? I need that HttpRequestExecutingMessageHandler to save the stream (file) a local file so I can read the contents and take some other action?

@Configuration
@EnableIntegration
public class GtfsConfiguration {

@Bean
public MessageChannel fileUpdateChannel () {
    return new DirectChannel();
}

@Bean
@ServiceActivator(inputChannel = "fileUpdateChannel", polling = @Poller(fixedDelay="5000")
public HttpRequestExecutingMessageHandler fileUpdateGateway() {
    HttpRequestExecutingMessageHandler handler = new HttpRequestExecutingMessageHandler("https://someserver.com/gtfs/raw/published.txt");
    handler.setHttpMethod(HttpMethod.GET);
    handler.setExpectedResponseType(byte[].class);
    return handler;
    }

}

Answer 1

If you need to download such a file periodically, you need to use a "fake" Inbound Channel Adapter, for example:

    @Bean
    @InboundChannelAdapter(value = "fileUpdateChannel"
            poller = @Poller(fixedDelay = "1000", maxMessagesPerPoll = "1"))
    public String downloadFileSchedule() {
        return () -> "";
    }

The @ServiceActivator for the HttpRequestExecutingMessageHandler is going to be called every second. You don't need to have there a @Poller on the @ServiceActivator. It isn't going to do anything by itself. Plus your fileUpdateChannel is a DirectChannel, not QueueChannel.

I don't think you need to save a downloaded file locally. I even would say that handler.setExpectedResponseType(String.class); is fully enough to get a file content as a reply message payload for downstream analyze.

The easiest way to configure a Basic Authorization is with the Apache Commons HTTP Client:

CredentialsProvider provider = new BasicCredentialsProvider();
UsernamePasswordCredentials credentials
 = new UsernamePasswordCredentials("user1", "user1Pass");
provider.setCredentials(AuthScope.ANY, credentials);

HttpClient client = HttpClientBuilder.create()
  .setDefaultCredentialsProvider(provider)
  .build();

and use this in the HttpComponentsClientHttpRequestFactory, which you then should inject into the mentioned HttpRequestExecutingMessageHandler via its setRequestFactory(ClientHttpRequestFactory requestFactory).