Use GitHub workflows to automatically publish to PyPI when new tags are created

General Show and Tell
1

I recently came across a pretty nice pattern for using GitHub Workflows to automatically publish packages to PyPI when new tags are created. This lets you generate a new release via the GitHub Draft a New Release UI. (e.g. here’s the page for the pydata sphinx theme). Writing it down here in case it’s useful for others and for me to re-discover in the future!

:warning: This will allow anybody who can create a repository tag to also create a release! That could either be a good or bad thing depending on your aversion to risk!

Here are the quick steps:

  1. Generate a release secret via PyPI for your account.

  2. Create a new secret for your repository via settings → secrets → actions → New repository secret:

    image
    image1920×769 67.2 KB

  3. Paste in the PyPI token that you just created, e.g.

    image
    image1207×686 17.2 KB

  4. You can now use this secret to authenticate PyPI in your GitHub Workflows

  5. In your tests workflow, add a workflow_call trigger, which will allow other workflows to call it on their own. E.g. you might have something like this in a tests.yml workflow:

     name: continuous-integration
 on:
   push:
     branches:
       - main
   pull_request:
   workflow_call:

 ...your tests steps here

  6. Create a publish.yml workflow to publish your package. Make it call, and need success for your tests workflow. Then make it publish your package upon success. Use a template like below (note the ${{ secrets.PYPI_KEY }}, which re-uses the secret you stored above).

    # Build the package and publish it to PyPI after tests pass.
name: Publish to PyPI
on:
  push:
    tags:
      - "*"

jobs:
  tests:
    uses: ./.github/workflows/tests.yml
  publish:
    name: publish
    needs: [tests] # require tests to pass before deploy runs
    runs-on: ubuntu-latest
    steps:
      - name: Checkout source
        uses: actions/checkout@v3
      - name: Set up Python 3.9
        uses: actions/setup-python@v4
        with:
          python-version: 3.9
      - name: Build package
        run: |
          python -m pip install -U pip build
          python -m build
      - name: Publish
        uses: pypa/gh-action-pypi-publish@v1.5.0
        with:
          user: __token__
          password: ${{ secrets.PYPI_KEY }}

And that should be it! Now to create a release you should simply do so via the GitHub repository UI, and when this happens it will trigger the publish.yml workflow, which will call tests.yml and wait for it to pass, and if so then it will publish to PyPI.

I hope this is useful!

3 Likes
2

Huge +1! I have a workflow to do similar for my main Python libs - here is the recipe that makes the pypi release on a GitHub release. Of course the only thing you need to be sure to do is that the version number is incremented.

name: Release Python Package

on:
  release:
    types: [created]

jobs:
  deploy:
    runs-on: ubuntu-20.04

    steps:
    - uses: actions/checkout@v2

    - name: Install
      run: conda create --quiet --name shpc twine

    - name: Install dependencies
      run: |
        export PATH="/usr/share/miniconda/bin:$PATH"
        source activate shpc
        pip install -e .
        pip install setuptools wheel twine
    - name: Build and publish
      env:
        TWINE_USERNAME: ${{ secrets.PYPI_USER }} 
        TWINE_PASSWORD: ${{ secrets.PYPI_PASS }}
      run: |
        export PATH="/usr/share/miniconda/bin:$PATH"
        source activate shpc
        python setup.py sdist bdist_wheel
        twine upload dist/*

And direct link to it: singularity-hpc/release.yaml at fb9a99a95d596f9ffbe1f2155ad91bfadf145bac · singularityhub/singularity-hpc · GitHub

1 Like
3

For reference there is also the Jupyter Releaser which is a set of GitHub Actions to make the release process consistent and easier to manage across repositories: Jupyter Releaser — Jupyter Releaser 0.22.5 documentation

With some improvements and simplifications on the roadmap: Simplified Workflow · Issue #272 · jupyter-server/jupyter_releaser · GitHub

1 Like