Use GitHub workflows to automatically publish to PyPI when new tags are created

I recently came across a pretty nice pattern for using GitHub Workflows to automatically publish packages to PyPI when new tags are created. This lets you generate a new release via the GitHub Draft a New Release UI. (e.g. here’s the page for the pydata sphinx theme). Writing it down here in case it’s useful for others and for me to re-discover in the future!

:warning: This will allow anybody who can create a repository tag to also create a release! That could either be a good or bad thing depending on your aversion to risk!

Here are the quick steps:

  1. Generate a release secret via PyPI for your account.

  2. Create a new secret for your repository via settingssecretsactionsNew repository secret:

  3. Paste in the PyPI token that you just created, e.g.

  4. You can now use this secret to authenticate PyPI in your GitHub Workflows

  5. In your tests workflow, add a workflow_call trigger, which will allow other workflows to call it on their own. E.g. you might have something like this in a tests.yml workflow:

     name: continuous-integration
           - main
     ...your tests steps here
  6. Create a publish.yml workflow to publish your package. Make it call, and need success for your tests workflow. Then make it publish your package upon success. Use a template like below (note the ${{ secrets.PYPI_KEY }}, which re-uses the secret you stored above).

    # Build the package and publish it to PyPI after tests pass.
    name: Publish to PyPI
          - "*"
        uses: ./.github/workflows/tests.yml
        name: publish
        needs: [tests] # require tests to pass before deploy runs
        runs-on: ubuntu-latest
          - name: Checkout source
            uses: actions/checkout@v3
          - name: Set up Python 3.9
            uses: actions/setup-python@v4
              python-version: 3.9
          - name: Build package
            run: |
              python -m pip install -U pip build
              python -m build
          - name: Publish
            uses: pypa/gh-action-pypi-publish@v1.5.0
              user: __token__
              password: ${{ secrets.PYPI_KEY }}

And that should be it! Now to create a release you should simply do so via the GitHub repository UI, and when this happens it will trigger the publish.yml workflow, which will call tests.yml and wait for it to pass, and if so then it will publish to PyPI.

I hope this is useful!


Huge +1! I have a workflow to do similar for my main Python libs - here is the recipe that makes the pypi release on a GitHub release. Of course the only thing you need to be sure to do is that the version number is incremented.

name: Release Python Package

    types: [created]

    runs-on: ubuntu-20.04

    - uses: actions/checkout@v2

    - name: Install
      run: conda create --quiet --name shpc twine

    - name: Install dependencies
      run: |
        export PATH="/usr/share/miniconda/bin:$PATH"
        source activate shpc
        pip install -e .
        pip install setuptools wheel twine
    - name: Build and publish
        TWINE_USERNAME: ${{ secrets.PYPI_USER }} 
        TWINE_PASSWORD: ${{ secrets.PYPI_PASS }}
      run: |
        export PATH="/usr/share/miniconda/bin:$PATH"
        source activate shpc
        python sdist bdist_wheel
        twine upload dist/*

And direct link to it: singularity-hpc/release.yaml at fb9a99a95d596f9ffbe1f2155ad91bfadf145bac · singularityhub/singularity-hpc · GitHub

1 Like

For reference there is also the Jupyter Releaser which is a set of GitHub Actions to make the release process consistent and easier to manage across repositories: Jupyter Releaser — Jupyter Releaser 0.22.5 documentation

With some improvements and simplifications on the roadmap: Simplified Workflow · Issue #272 · jupyter-server/jupyter_releaser · GitHub

1 Like