How to publish pip wheels using github actions

Question

Maybe a bit of a broad question, but I think this is relevant for any maintainer of python packages that uses github and could reduce their workload significantly, so hopefully the powers that be will let it stand.

Essentially, it seems to me that :

Building & publishing wheels for python is nice, but sometimes they have to be specific to the operating system for them to work properly (i.e. built on a similar OS/Env)
Github actions provides a wide range of OSes to run jobs on (for now, for free), so it could be used to build these wheels
Github actions can be easily used to publish sidst via twine

So my question is:

Is there a straight-forward way to build wheels for a bunch of environments (e.g. using the OS/env matrix in github actions) then publish them all via github actionse.g. by aggregating them in one place and then running some twine command? The issue seems to be that there's no way to communicated between the different envs/vms setup by github action.

I can think of various solutions involving e.g. an intermeidary s3 bucket, but I'm likely hugely mistaken in regard to how pypi and/or github actions work in this regard so there might be a very simple issue I'm glancing over.

You don't need to collect built wheels — you just push them with `twine` one by one. Every virtual machine builds one wheel for one specific OS and then publish one wheel. — phd, Feb 25 '21 at 15:07
Example: this is how I build wheels for [CheetahTemplates](https://cheetahtemplate.org/); source code at [GitHub](https://github.com/CheetahTemplate3) includes configs for Travis CI and AppVeyor CI. At Travis I [run](https://travis-ci.org/github/CheetahTemplate3/cheetah3/builds/759974689) virtual boxes with Linux and OSX; at AppVeyor [w32/w64](https://ci.appveyor.com/project/phdru/cheetah3/builds/37885947). The result is https://pypi.org/project/Cheetah3/3.2.6.post1/#files — phd, Feb 25 '21 at 15:07
For Github Actions it should be the same. Every job pushes one wheel. — phd, Feb 25 '21 at 15:08
Wouldn't this result in a "lul" between when the version is published to pypi and when all the wheels are pushed though? So you have 20-30 minutes where pip installs are acting wired, essentially — George, Feb 25 '21 at 15:38
Nobody reported any problem with this for me so I don't care. Perhaps `pip` recognizes that a wheel is not yet available at the latest version at PyPI and uses the previous one. — phd, Feb 25 '21 at 15:40
it's exactly that undefined behavior that I'd want to avoid, though. But I might have to go with this solution + a few bells and whistles to sync the releases. — George, Feb 25 '21 at 15:45
Like mentioned by @phd it does seem to make more sense to have a matrix of OS where each job build and push it's own artifact. You can however have a single job aggregating all the results of the matrix and push everything at ones. If you want I can show you the workflow structure but unfortunately I don't know the pip and twine specifics or commands. — ITChap, Feb 25 '21 at 16:32
@ITChap Use `build` and `upload` instead of `pip` and `twine`. :-) The OP will figure the real commands. — phd, Feb 25 '21 at 16:38
@ITChap seeing how to aggregate files from the action matrix would be enough if you can show me an example, that would be very helpful, googling around didn't help — George, Feb 25 '21 at 17:28

ITChap · Accepted Answer · 2021-02-26T06:50:30.043

As mentioned in my comment, here is one possible way to run parallel builds but a single upload:

name: 'Aggregation'

on: [push]

env:
  ARTIFACT: artifact.bin

jobs:
  build:
    runs-on: ${{ matrix.os }} 
    strategy:
      matrix:
        os:
          - windows-latest
          - ubuntu-latest
          - macos-latest
    steps:
      - uses: actions/checkout@v2

      - shell: bash
        run: |
          echo "Run your build command here"
          echo "This is a fake ${{ matrix.os }} build artifact" >$ARTIFACT

      - uses: actions/upload-artifact@v2
        with:
          name: build-${{ matrix.os }}-${{ github.sha }}
          path: ${{ env.ARTIFACT }}

  publish:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - uses: actions/download-artifact@v2
        with:
          path: artifacts

      - shell: bash
        working-directory: artifacts
        run: |
          for i in $( ls ); do
            cat $i/$ARTIFACT
          done

Each matrix job build and upload its own artifact to github. The publish job waits for all the previous jobs to complete, before downloading all the artifacts and in this case iterating over them. One added benefit is that if any matrix job fails the publishing will fail. Of course this is simple only if the build steps and commands are the same on all the OSes.

Here are some screenshots of the test run:

The resulting artifacts:

The output of the publish job:

How to publish pip wheels using github actions

1 Answers1