9

I am having issues with Githubs action runners while trying to compile the release version of my app. My app has 8 different flavors that we build and provide on the play store, and a few months ago I was able to build up to 3 of those flavors at the same time on one runner using Github Actions. If we tried to do more than 3 flavors at a time, after about 15 minutes, the build would fail saying

FAILURE: Build failed with an exception.

* What went wrong:

Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)

and the Gradle log does not really help, besides pointing to which task was running at the time (minify r8)

When I inherited this code base, I saw that we have the max heap size in the jvm args set to 6 gigs, and if I lower that at all, I end up with out of memory errors when r8 runs. It seems kinda crazy to me that r8 would be taking that much memory, and on top of that, the github hosted runners only have 7 Gb of ram available for usage anyway. So if we combine what r8 is using in the gradle daemon, with knowing that the kotlin daemon is also running, I think the runner is killing off the gradle daemon for its memory usage or it is dying from lack of memory.

Ever since we added the Jetpack Compose Libraries, we now can't build more than one flavor of the app at a time, and I am worried about losing the ability to build on github runners due to the memory footprint r8 is causing.

This is what my current gradle.properties looks like:

org.gradle.jvmargs=-Xms1024m -Xmx6144m -XX:-UseGCOverheadLimit -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=512m -XX:+UseParallelGC org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.daemon=true org.gradle.caching=true android.useAndroidX=true android.debug.obsoleteApi=true kapt.incremental.apt=true

I have tried a few things to lower the workload for r8:

  1. Removed all unused resources that had managed to clutter up our resource folder
  2. We weren't actually using databinding but had both the kotlin android extensions and databinding turned on, so I turned that off
  3. We had Jetifier turned on when we didn't actually need it anymore, so I have disabled that
  4. I changed to using the parallel garbage collector
  5. I have attempted to not actually used the gradle daemon and saw no difference

With all of these changes I have seen that the overall footprint during the r8 step actually did drop to just over 5 gigs during that step, but when running on actions, if I try to do more than one flavor at a time, the daemon will still vanish.

I have spent a lot of time trying to find more info on this online, and all I can seem to find is people saying to increase the max heap size, but that will not help when my memory is limited by the machine it is running on.

I am kinda running out of ideas of what could be causing this, any help is appreciated!

Phoenix
  • 93
  • 4
  • If you have a large app it is not uncommon for R8 to require a pretty large heap. As it does whole program optimizations it does keep large data structures in memory. Usually a heap of 8G (`-Xmx8G`) works well. – sgjesse Mar 28 '22 at 07:31
  • @sgjesse The only issue is that, this being a github hosted runner, we only have access to 7G of ram – Phoenix Mar 28 '22 at 16:10
  • Which Java version is used for gradle? And why do you use `UseParallelGC`? – Robert Mar 28 '22 at 17:16
  • Did you get a solution to this? I'm facing the exact same issue now... :( – user1898712 Apr 14 '22 at 15:10
  • sorry @user1898712 I am still struggling with this. – Phoenix May 06 '22 at 16:02
  • @Phoenix I managed to fix it by dropping the heap size to 2G. 3G was too much for our CI system and it was running out of memory. – user1898712 May 06 '22 at 17:36
  • Github Action allows to run on self hosted machine now, so it's possible to use an large EC2 machine – cheng yang Nov 28 '22 at 07:15

1 Answers1

1

The problem is the default size of ubuntus swapfile. It is only ~1GB in the case of a gitHub action (sudo swapon --show). When ubuntu is running out of memory (RAM + swapfile), it is going to freeze. (happened on my local machine after I enabled R8). We could fix this problem by increasing the swapfile as first step of the action:

  - name: Increase swapfile
    run: |
      sudo swapoff -a
      sudo fallocate -l 15G /swapfile
      sudo chmod 600 /swapfile
      sudo mkswap /swapfile
      sudo swapon /swapfile
      sudo swapon --show
user1185087
  • 4,468
  • 1
  • 30
  • 38
  • Is this safe, if standard github runner have only 14 GB storage on hdd/ssd? It looks for me, that we try to allocate more storage than we have available just for swap file... – mtrakal Aug 25 '23 at 10:36
  • You can run `df -h` before you increase the swapfile. In our case there is 22GB available on `/dev/root` : `84G (Size) 62G (Used) 22G (Avail)` – user1185087 Aug 30 '23 at 07:17
  • I just referred my comment on [GitHub Documentation](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources). I didn't check how much is really available. Thanks for info, that they provide more space :) – mtrakal Aug 30 '23 at 09:32