how to download external files in gradle?

Question

I have a gradle project which requires some data files available somewhere on the internet using http. The goal is that this immutable remote file is pulled once upon first build. Subsequent build should not download again.

How can I instruct gradle to fetch the given file to a local directory?

I've tried

task fetch(type:Copy) {
   from 'http://<myurl>'
   into 'data'
}

but it seems that copy task type cannot deal with http.

Bonus question: is there a way to resume a previously aborted/interrupted download just like wget -c does?

I've done this before using custom configurations, so that I can declare the resource as a versioned dependency. Then the normal resolution handling does its magic for you — Ben Manes, Jun 16 '13 at 00:12
There's a good answer here: https://stackoverflow.com/a/34327202/2873507 — Vic Seedoubleyew, Jan 16 '18 at 13:37

score 69 · Answer 1 · edited Jul 03 '16 at 19:56

69

How about just:

def f = new File('the file path')
if (!f.exists()) {
    new URL('the url').withInputStream{ i -> f.withOutputStream{ it << i }}
}

edited Jul 03 '16 at 19:56

mkobit

43,979
12
156
150

answered Jun 15 '13 at 13:36

Ryan Stewart

126,015
21
180
199

3

Works well, but can you please suggest any pointers for directory download ? – Ravi Bhojani Dec 12 '16 at 17:11
How to do the same using Kotlin DSL ? – Saurabh Padwekar Aug 16 '20 at 07:07
4

@SaurabhPadwekar in Kotlin DSL, you can use `val f = file("/tmp/file"); uri("http://example.com/index.html").toURL().openStream().use { it.copyTo(FileOutputStream(f)) }` – Dario Seidl Mar 04 '21 at 14:46

score 33 · Answer 2 · edited Jan 03 '18 at 12:45

You could probably use the Ant task Get for this. I believe this Ant task does not support resuming a download.

In order to do so, you can create a custom task with name MyDownload. That can be any class name basically. This custom task defines inputs and outputs that determine whether the task need to be executed. For example if the file was already downloaded to the specified directory then the task is marked UP-TO-DATE. Internally, this custom task uses the Ant task Get via the built-in AntBuilder.

With this custom task in place, you can create a new enhanced task of type MyDownload (your custom task class). This task set the input and output properties. If you want this task to be executed, hook it up to the task you usually run via task dependencies (dependsOn method). The following code snippet should give you the idea:

task downloadSomething(type: MyDownload) {
    sourceUrl = 'http://www.someurl.com/my.zip'
    target = new File('data')
}

someOtherTask.dependsOn downloadSomething

class MyDownload extends DefaultTask {
    @Input
    String sourceUrl

    @OutputFile
    File target

    @TaskAction
    void download() {
       ant.get(src: sourceUrl, dest: target)
    }
}

Fyi all, you can set the ant proxy, using `ant.setproxy(proxyhost: System.props['http.proxyHost'], ...)`. — Nick Grealy, Feb 26 '14 at 00:52
Always nice to see textbook authors answer questions on SO. This is extremely handy. Just used this to point to a binary release JAR file on GitHub. Thanks Benjamin! — tmn, Apr 25 '15 at 02:16
I think with the `kotlin-dsl` you can do `ant.invokeMethod("get", mapOf("src" to sourceUrl, "dest" to target))` because of how the Groovy builder support works. — mkobit, Dec 18 '17 at 21:28

score 30 · Answer 3 · edited Jul 03 '16 at 19:56

30

Try like that:

plugins {
    id "de.undercouch.download" version "1.2"
}

apply plugin: 'java'
apply plugin: 'de.undercouch.download'

import de.undercouch.gradle.tasks.download.Download

task downloadFile(type: Download) {
    src 'http://localhost:8081/example/test-jar-test_1.jar'
    dest 'localDir'
}

You can check more here: https://github.com/michel-kraemer/gradle-download-task

For me works fine..

edited Jul 03 '16 at 19:56

mkobit

43,979
12
156
150

answered May 13 '15 at 11:08

Maksim Kriuchko

301
3
3

what is localDir – jpell Jan 15 '22 at 18:17

score 5 · Answer 4 · answered Aug 09 '16 at 09:23

The suggestion in Ben Manes's comment has the advantage that it can take advantage of maven coordinates and maven dependency resolution. For example, for downloading a Derby jar:

Define a new configuration:

configurations {
  derby
}

In the dependencies section, add a line for the custom configuration

dependencies {
  derby "org.apache.derby:derby:10.12.1.1"
}

Then you can add a task which will pull down the right files when needed (while taking advantage of the maven cache):

task deployDependencies() << {
   String derbyDir = "${some.dir}/derby"
   new File(derbyDir).mkdirs();
      configurations.derby.resolve().each { file ->
        //Copy the file to the desired location
        copy {
          from file 
          into derbyDir
          // Strip off version numbers
          rename '(.+)-[\\.0-9]+\\.(.+)', '$1.$2'
        }
      }
}

(I learned this from https://jiraaya.wordpress.com/2014/06/05/download-non-jar-dependency-in-gradle/).

score 4 · Answer 5 · answered May 14 '19 at 06:30

Using following plugin:

plugins {
    id "de.undercouch.download" version "3.4.3"
}

For a task which has the purpose of only downloading

task downloadFile(type: Download) {
    src DownloadURL
    dest destDir
}

For including download option into your task:

download {
    src DownloadURL
    dest destDir
}

For including download option with multiple downloads into your task:

task downloadFromURLs(){
    download {
        src ([
               DownloadURL1,
               DownloadURL2,
               DownloadURL3
        ])
        dest destDir
    }
}

Hope it helped :)

how to add authentication also in task. If i have i want to pass username/password in my gradle task — Ramz, May 07 '20 at 09:05

roomsg · Answer 6 · 2015-05-29T08:54:33.220

2

just now ran into post on upcoming download task on gradle forum.
Looks like the perfect solution to me.. Not (yet) available in an official gradle release though

edited May 29 '15 at 08:54

answered Sep 30 '13 at 10:57

roomsg

1,811
16
22

Thx for notifying, I updated the url (gradle forum has been recently moved) – roomsg May 29 '15 at 08:54
i'm sure it'll be listed in as "incubating" for years too – spy Sep 10 '16 at 18:32

score 1 · Answer 7 · answered Aug 16 '20 at 07:39

Kotlin Version of @Benjamin's Answer

buildscript {
    repositories {
        jcenter()
        google()
    }

    dependencies {
        classpath("com.android.tools.build:gradle:4.0.1")
    }
}

tasks.register("downloadPdf"){
    val path = "myfile.pdf"
    val sourceUrl = "https://file-examples-com.github.io/uploads/2017/10/file-sample_150kB.pdf"
    download(sourceUrl,path)
}

fun download(url : String, path : String){
    val destFile = File(path)
    ant.invokeMethod("get", mapOf("src" to url, "dest" to destFile))
}

how to download external files in gradle?

7 Answers7

Linked