0

I am executing the below code to pull the .gz file from a URL to a local directory.For small files it goes through fine but for large files it downloads only part of it but does not fail. I get to know the error only when I try to UNZIP it. Can someone throw any light on this on what could be the reason.

public boolean downloadFilemethod(String filePath, String url, String 
decompressFilePath) { 
if (StringUtils.isNotBlank(filePath) && StringUtils.isNotBlank(url)) { 
try { 
FileUtils.copyURLToFile(new URL(url), new File(SRC_BASE_DIR + filePath), 
TIMEOUT_IN_MILLIS, TIMEOUT_IN_MILLIS); 
ingestmethod(filePath,decompressFilePath);   
downloadSuccess = true; 
} 
catch (MalformedURLException e) 
{ LOG.warn("some message"); } 
catch (IOException e) 
{ LOG.warn("some message); 
} 
}
return downloadSuccess 
}
Jonik
  • 80,077
  • 70
  • 264
  • 372
user3072054
  • 339
  • 2
  • 6
  • 17
  • 2
    What library does `FileUtils` belong to? – Sotirios Delimanolis Jan 13 '14 at 21:21
  • There is a tag: `FileUtils` or `org.apache.commons.io.FileUtils` is File manipulation utility from [Apache Commons] :D Read carefully whole question ;) – MariuszS Jan 13 '14 at 21:24
  • Its from Apache commons : org.apache.commons.io.FileUtils – user3072054 Jan 13 '14 at 21:27
  • What does the surrounding code look like? – matt forsythe Jan 13 '14 at 21:27
  • 1
    @matt: Once I download the file into local temp directory I move it into Hadoop and unzip it.When I try to unzip it I face an error "java.io.EOFException: Unexpected end of ZLIB input stream" which means the input .gz file was not fully downloaded. – user3072054 Jan 13 '14 at 21:31
  • I am more interested in actually seeing the surrounding code to verify that there are no exceptions that are being squashed, etc. – matt forsythe Jan 13 '14 at 21:34
  • public boolean downloadFilemethod(String filePath, String url, String decompressFilePath) { if (StringUtils.isNotBlank(filePath) && StringUtils.isNotBlank(url)) { try { FileUtils.copyURLToFile(new URL(url), new File(SRC_BASE_DIR + filePath), TIMEOUT_IN_MILLIS, TIMEOUT_IN_MILLIS); ingestmethod(filePath,decompressFilePath); downloadSuccess = true; } catch (MalformedURLException e) { LOG.warn("some message"); } catch (IOException e) { LOG.warn("some message); } } return downloadSuccess } – user3072054 Jan 13 '14 at 21:39
  • Pleas edit question and put this code inside question. – MariuszS Jan 13 '14 at 22:07
  • Perhaps `try (InputStream stream = new GZIPInputStream(new URL(url).openStream())) { Files.copy(stream, Paths.get(decompressFilePath)); }` will work better? (And as a bonus, no third-party libraries required.) – VGR Jan 15 '14 at 13:44

0 Answers0