How to download and save a file from the internet using Scala?

Question

Basically I have a url/link to a text file online and I am trying to download it locally. For some reason, the text file that gets created/downloaded is blank. Open to any suggestions. Thanks!

    def downloadFile(token: String, fileToDownload: String) {

    val url = new URL("http://randomwebsite.com/docs?t=" + token + "&p=tsr%2F" + fileToDownload)
    val connection = url.openConnection().asInstanceOf[HttpURLConnection]
    connection.setRequestMethod("GET")
    val in: InputStream = connection.getInputStream
    val fileToDownloadAs = new java.io.File("src/test/resources/testingUpload1.txt")
    val out: OutputStream = new BufferedOutputStream(new FileOutputStream(fileToDownloadAs))
    val byteArray = Stream.continually(in.read).takeWhile(-1 !=).map(_.toByte).toArray
    out.write(byteArray)
    }

score 40 · Answer 1 · edited Feb 12 '15 at 21:02

40

I know this is an old question, but I just came across a really nice way of doing this :

import sys.process._
import java.net.URL
import java.io.File

def fileDownloader(url: String, filename: String) = {
    new URL(url) #> new File(filename) !!
}

Hope this helps. Source.

You can now simply use fileDownloader function to download the files.

fileDownloader("http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words", "stop-words-en.txt")

edited Feb 12 '15 at 21:02

David P

3,604
3
37
54

answered Oct 17 '14 at 09:59

Chetan Bhasin

3,503
1
23
36

1

what if I want to use the same code above and instead of downloading and storing it locally I want to store it in a different server whos IP is know to me – Aamir Nov 30 '15 at 13:08
7

Thank you for adding the imports! – Thomas Kaliakos Feb 15 '16 at 12:13
@Aamir depends on how the other server handles upload. In that case you'll the above mention method will not work. – Chetan Bhasin Mar 01 '16 at 17:48
2

Also when the file is not available there seems to be no way to catch the exception. – user833970 Aug 18 '18 at 17:45
@user833970, you can always wrap the entire thing in `scala.util.Try`. – Chetan Bhasin Aug 19 '18 at 10:37
This doesn't follow redirects either :'( – Toby Aug 29 '18 at 09:21

score 11 · Answer 2 · answered Jun 11 '14 at 13:51

11

Here is a naive implementation by scala.io.Source.fromURL and java.io.FileWriter

def downloadFile(token: String, fileToDownload: String) {
  try {
    val src = scala.io.Source.fromURL("http://randomwebsite.com/docs?t=" + token + "&p=tsr%2F" + fileToDownload)
    val out = new java.io.FileWriter("src/test/resources/testingUpload1.txt")
    out.write(src.mkString)
    out.close
  } catch {
    case e: java.io.IOException => "error occured"
  }
}

Your code works for me... There are other possibilities that make empty file.

answered Jun 11 '14 at 13:51

Herrington Darkholme

5,979
1
27
43

1

Tried another link and it works, sigh, something wrong with the url I'm trying. thanks! – slizorn Jun 11 '14 at 13:58
5

But what if the file is in a binary format, then I guess `src.mkString` might fail... – eugen-fried Aug 26 '15 at 08:36

score 10 · Answer 3 · answered Aug 22 '18 at 22:49

Here is a safer alternative to new URL(url) #> new File(filename) !!:

val url = new URL(urlOfFileToDownload)

val connection = url.openConnection().asInstanceOf[HttpURLConnection]
connection.setConnectTimeout(5000)
connection.setReadTimeout(5000)
connection.connect()

if (connection.getResponseCode >= 400)
  println("error")
else
  url #> new File(fileName) !!

Two things:

When downloading from an URL object, if an error (404 for instance) is returned, then the URL object will throw a FileNotFoundException. And since this exception is generated from another thread (as URL happens to run on a separate thread), a simple Try or try/catch won't be able to catch the exception. Thus the preliminary check for the response code: if (connection.getResponseCode >= 400).
As a consequence of checking the response code, the connection might sometimes get stuck opened indefinitely for improper pages (as explained here). This can be avoided by setting a timeout on the connection: connection.setReadTimeout(5000).

Does the last line open a 2nd connection? – charles-allen Mar 19 '20 at 17:16 — charles-allen, Mar 19 '20 at 17:16

Eric Hydrick · Accepted Answer · 2014-06-11T13:03:31.867

6

Flush the buffer and then close your output stream.

edited Jun 11 '14 at 13:03

answered Jun 11 '14 at 12:54

Eric Hydrick

3,467
2
28
41

How to download and save a file from the internet using Scala?

4 Answers4

Linked