gradle - download and unzip file from url

Question

What would be the proper gradle way of downloading and unzipping the file from url (http)?

If possible, I'd like to prevent re-downloading each time I run the task (in ant.get can be achieved by skipexisting: 'true').

My current solution would be:

task foo {
  ant.get(src: 'http://.../file.zip', dest: 'somedir', skipexisting: 'true')
  ant.unzip(src: 'somedir' + '/file.zip', dest: 'unpackdir')
}

still, I'd expect ant-free solution. Any chance to achieve that?

Don't forget to wrap the execution part of a task with `doLast { ... }` (same mistake as in your previous question). — Peter Niederwieser, Apr 11 '14 at 22:20
~~BOUNTY~~ Can anyone provide an example for the answer below: "if you do want to benefit from Gradle's dependency resolution/caching features, by pretending it's an Ivy repository with a custom artifact URL" ? — CMPS, Dec 16 '15 at 21:22
netflix has released some plugin: https://github.com/nebula-plugins/nebula-core - not sure, why this is not maintained anymore. — koppor, Aug 19 '19 at 20:12

score 92 · Answer 1 · edited May 18 '21 at 15:54

92

Let's say you want to download this zip file as a dependency:

https://github.com/jmeter-gradle-plugin/jmeter-gradle-plugin/archive/1.0.3.zip

You define your ivy repo as:

repositories {
    ivy {
        url 'https://github.com/'

        patternLayout {
            artifact '/[organisation]/[module]/archive/[revision].[ext]'
        }

        // This is required in Gradle 6.0+ as metadata file (ivy.xml) 
        // is mandatory. Docs linked below this code section
        metadataSources { artifact() } 
    }
}

reference for required metadata here

The dependency can then be used as:

dependencies {
    compile 'jmeter-gradle-plugin:jmeter-gradle-plugin:1.0.3@zip'
    //This maps to the pattern: [organisation]:[module]:[revision]:[classifier]@[ext]         
}

To unzip:

task unzip(type: Copy) {

  def zipPath = project.configurations.compile.find {it.name.startsWith("jmeter") }
  println zipPath
  def zipFile = file(zipPath)
  def outputDir = file("${buildDir}/unpacked/dist")

  from zipTree(zipFile)
  into outputDir

}

optional:

If you have more than one repository in your project, it may also help (for build time and somewhat security) to restrict dependency search with relevant repositories.

Gradle 6.2+:

repositories {
    mavenCentral()
    def github = ivy {
        url 'https://github.com/'
        patternLayout {
            artifact '/[organisation]/[module]/archive/[revision].[ext]'
        }
        metadataSources { artifact() }
    }
    exclusiveContent {
        forRepositories(github)
        filter { includeGroup("jmeter-gradle-plugin") }
    }
}

Earlier gradle versions:

repositories {
    mavenCentral {
        content { excludeGroup("jmeter-gradle-plugin") }
    }
    ivy {
        url 'https://github.com/'
        patternLayout {
            artifact '/[organisation]/[module]/archive/[revision].[ext]'
        }
        metadataSources { artifact() }
        content { includeGroup("jmeter-gradle-plugin") }
    }
}

edited May 18 '21 at 15:54

mkobit

43,979
12
156
150

answered Dec 17 '15 at 05:24

RaGe

22,696
11
72
104

@RaGe how does 1.0.3:class@zip get matched to [revision].[ext] ? – addy Jan 27 '16 at 05:49
@addy `class` in that string is a classifier. It is optional, and here it isn't being used in the ivy url at all. `1.0.3` is revision and `zip` is ext. – RaGe Jan 27 '16 at 12:59
Is this for android or java? I'm trying to use it but it seems like "project.configuration.compile" does not exist? I've tried to add apply java before in the script, but that does not make a difference. Can't find it in the documentation either. Using com.android.tools.build:gradle:2.1.0 and version 2.10 of gradle. – peuhse May 11 '16 at 13:12
@peuhse it should be `project.configurations`, mind the `s` in configuration**s**. The android plugin implicitly adds the java plugin to a project. It shouldn't matter if you add it explicitly again. – RaGe May 11 '16 at 13:51
It was a spelling mistake by me. When I write project.configurations and let the IDE autofill the possibilites the compile isn't one of them. And then I get a build error. Commenting out the line makes it build again. – peuhse May 11 '16 at 13:57
please raise a new question, and refer to this one. – RaGe May 11 '16 at 15:33
3

Unfortunately URLs with query strings cannot be accessed this way, '?' is encoded by https://github.com/gradle/gradle/blob/master/subprojects/resources/src/main/java/org/gradle/internal/resource/ExternalResourceName.java#L115 – Akom Apr 21 '17 at 18:11
2

When I am trying this rather than looking for .zip file it is looking for a .xml file. Any way around for this? I followed exactly the same instruction. – Sourav Bhattacharjee Jun 26 '18 at 10:09
3

Doesn't work for me. Gradle is looking for an ivy.xml file in addition to the artifact. – Victor Lyuboslavsky Jun 29 '18 at 15:42
Does work, as others have stated, Gradle looks for `.xml` file. – Abhijit Sarkar Oct 01 '20 at 02:07
1

This is a great answer. I would only add that the dependency should probably be compileOnly to prevent the raw zip ending up in the runtime artifacts. – Andy Brown Oct 19 '20 at 09:29
gradle 7 now changing configurations - 'compile' removed. adding a configuration in replacement or updating to the new names still worked. – Greg Domjan Aug 12 '21 at 10:48
Doesn't work. Gradle 7.2 is looking for a normal maven dependency. – rios0rios0 Sep 07 '21 at 13:17
@rios0rios0 I'd expect gradle to look in all available repositories for all dependencies. If you want to restrict certain deps to certain repos, see the `optional` section in the answer – RaGe Sep 07 '21 at 15:00
1

I can confirm this works with 7.2, @rios0rios0 have you resolved? – elect Oct 21 '21 at 16:02
@elect I solved it in another way. Because I spent a lot of time trying to work with this approach. Community on Gradle's Slack said a better way to handle my original issue. But thanks for confirming that's working. Here Gradle wasn't recognizing the pattern "/[organisation]/[module]/archive/[revision].[ext]" on dependencies. – rios0rios0 Oct 21 '21 at 18:49

score 14 · Answer 2 · edited May 25 '20 at 14:54

14

    plugins {
        id 'de.undercouch.download' version '4.0.0'
    }

    /**
     * The following two tasks download a ZIP file and extract its
     * contents to the build directory
     */
    task downloadZipFile(type: Download) {
        src 'https://github.com/gradle-download-task/archive/1.0.zip'
        dest new File(buildDir, '1.0.zip')
    }

    task downloadAndUnzipFile(dependsOn: downloadZipFile, type: Copy) {
        from zipTree(downloadZipFile.dest)
        into buildDir
    }

https://github.com/michel-kraemer/gradle-download-task

edited May 25 '20 at 14:54

Sergey

3,253
2
33
55

answered Aug 19 '19 at 11:10

HolyM

301
3
9

1

This is not really a good solution compared to Ant as it is really slow. 26 minutes download instead of 2 minutes with Chrome. – Michael S. Oct 29 '19 at 14:30

score 9 · Accepted Answer · answered Apr 11 '14 at 22:20

9

There isn't currently a Gradle API for downloading from a URL. You can implement this using Ant, Groovy, or, if you do want to benefit from Gradle's dependency resolution/caching features, by pretending it's an Ivy repository with a custom artifact URL. The unzipping can be done in the usual Gradle way (copy method or Copy task).

answered Apr 11 '14 at 22:20

Peter Niederwieser

121,412
21
324
259

4

I would appreciate an example... I'm a total gradle newbie! – xpmatteo Mar 05 '15 at 14:12
4

~~BOUNTY~~ Can anyone provide an example for this answer "if you do want to benefit from Gradle's dependency resolution/caching features, by pretending it's an Ivy repository with a custom artifact URL" ? – CMPS Dec 16 '15 at 21:23
@CMPS why you no groovy? – RaGe Dec 17 '15 at 05:20
@RaGe my current solution is using groovy and it works fine, but I wanted one using repository to benefit from the caching system. – CMPS Dec 17 '15 at 05:41
@CMPS Ah, caching! Posted an answer below. – RaGe Dec 17 '15 at 05:42

score 7 · Answer 4 · edited Mar 20 '18 at 20:03

7

Unzipping using the copy task works like this:

task unzip(type: Copy) {
  def zipFile = file('src/dists/dist.zip')
  def outputDir = file("${buildDir}/unpacked/dist")

  from zipTree(zipFile)
  into outputDir
}

http://mrhaki.blogspot.de/2012/06/gradle-goodness-unpacking-archive.html

edited Mar 20 '18 at 20:03

mkobit

43,979
12
156
150

answered Oct 30 '14 at 13:48

Matthias M

12,906
17
87
116

I tried moving the ```from zipTree() & into outputDir``` to the execution phase by moving them to ```doLast { }```. What I did not understand was that it never worked. Even the ```println``` code in ```doLast``` did not work. However, this code works in the gradle configuration phase. Why? – nashter Dec 20 '18 at 09:18
Because it configures the task. The execution still is done during execution phase. Changing the configuration during the execution phase should not be done actually and in your case the execution phase would not be executed at all, as the task thinks there is nothing to copy, so the task execution is skipped. But actually this answer is not related to the question anyway, as this does not work with an HTTP URL. – Vampire Oct 29 '19 at 15:26

score 5 · Answer 5 · answered Jul 30 '19 at 13:12

5

This works with Gradle 5 (tested with 5.5.1):

task download {
    doLast {
        def f = new File('file_path')
        new URL('url').withInputStream{ i -> f.withOutputStream{ it << i }}
    }
}

Calling gradle download downloads the file from url to file_path.

You can use the other methods from other answers to unzip the file if necessary.

answered Jul 30 '19 at 13:12

Johny

809
9
19

Also works in gradle 4.7. In case someone needs it. – camikiller Jul 07 '21 at 17:05

score 1 · Answer 6 · answered Jan 22 '19 at 17:23

I got @RaGe's answer working, but I had to adapt it since the ivy layout method has been depreciated see https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.repositories.IvyArtifactRepository.html#org.gradle.api.artifacts.repositories.IvyArtifactRepository:layout(java.lang.String,%20groovy.lang.Closure)

So to get it working I had to adjust it to this for a tomcat keycloak adapter:

ivy {
    url 'https://downloads.jboss.org/'
    patternLayout {
        artifact '/[organization]/[revision]/adapters/keycloak-oidc/[module]-[revision].[ext]'
    }
}

dependencies {
    // https://downloads.jboss.org/keycloak/4.8.3.Final/adapters/keycloak-oidc/keycloak-tomcat8-adapter-dist-4.8.3.Final.zip
    compile "keycloak:keycloak-tomcat8-adapter-dist:$project.ext.keycloakAdapterVersion@zip"
}

task unzipKeycloak(type: Copy) {

    def zipPath = project.configurations.compile.find {it.name.startsWith("keycloak") }
    println zipPath
    def zipFile = file(zipPath)
    def outputDir = file("${buildDir}/tomcat/lib")

    from zipTree(zipFile)
    into outputDir
}

Unfortunately, this solution also has Gradle looking for an Ivy XML file. — barfuin, Apr 17 '19 at 16:51
Doesn't work. Gradle 7.2 is looking for a normal maven dependency. — rios0rios0, Sep 07 '21 at 13:16

score 1 · Answer 7 · answered Feb 03 '20 at 16:37

"native gradle", only the download part (see other answers for unzipping)

task foo {
  def src = 'http://example.com/file.zip'
  def destdir = 'somedir'
  def destfile = "$destdir/file.zip"
  doLast {
    def url = new URL(src)
    def f = new File(destfile)
    if (f.exists()) {
      println "file $destfile already exists, skipping download"
    } else {
      mkdir "$destdir"
      println "Downloading $destfile from $url..."
      url.withInputStream { i -> f.withOutputStream { it << i } }
    }
  }
}

gradle - download and unzip file from url

7 Answers7

Linked