Distribution

How eigenoid distributes Python packages across environments using GCP Artifact Registry, automated tagging, and GitHub Actions.

Overview

Every Python package in the org follows the same distribution pipeline:

1. Push code to a branch (dev, qa, or main)
2. Auto-tag -- a workflow reads the version from pyproject.toml and creates a git tag with the environment suffix
3. Release -- the tag triggers a release workflow that builds, tests, and publishes the package to the correct Artifact Registry
4. Promote -- merge the branch forward (dev -> qa -> main) to promote through environments

No manual tagging, no manual publishing, no manual anything.

Key concepts

Concept	What it means
Artifact Registry (AR)	Google Cloud service that hosts Python packages. Each GCP project has its own AR repository.
Environment suffix	Tags ending in -dev or -qa target those environments. Clean tags (e.g., v1.0.0) target production.
Auto-tag	A GitHub Actions workflow that creates git tags automatically on push -- no manual git tag needed.
WIF (Workload Identity Federation)	Keyless authentication from GitHub Actions to GCP. No service account keys stored anywhere.

Environments

Each environment has its own Artifact Registry repository in its own GCP project:

Environment	Branch	Tag format	GCP Project	AR URL
Development	dev	v1.0.0-dev	eigenoid-dev	us-central1-python.pkg.dev/eigenoid-dev/eigenoid-python/simple/
QA	qa	v1.0.0-qa	eigenoid-qa	us-central1-python.pkg.dev/eigenoid-qa/eigenoid-python/simple/
Production	main	v1.0.0	eigenoid-prd	us-central1-python.pkg.dev/eigenoid-prd/eigenoid-python/simple/

How it works

Auto-tagging

When you push to dev, qa, or main, the auto-tag.yml workflow:

1. Reads the version field from pyproject.toml
2. Appends the environment suffix (e.g., 0.5.0 on dev becomes v0.5.0-dev)
3. Checks if that tag already exists
4. If not, creates an annotated tag via the GitHub API using the automation bot

The automation bot token is used instead of GITHUB_TOKEN because tags created with GITHUB_TOKEN do not trigger other workflows.

Release

When the auto-tag creates a new tag, release.yml triggers and:

1. Resolves the environment from the tag suffix (-dev -> dev, -qa -> qa, clean -> prd)
2. Runs tests across a Python version matrix
3. Builds the source distribution and wheel
4. Creates a GitHub Release (marked as pre-release for dev/qa) using the releaser bot
5. Publishes to Artifact Registry using WIF authentication with environment-specific credentials

Each GitHub Environment (dev, qa, prd) stores its own WIF provider, service account, and AR repository variables.

Promotion

To promote a version from dev to qa to production:

# After validating in dev, create PR from dev to qa
gh pr create --base qa --head dev --title "chore: promote v0.5.0 to qa"

# After validating in qa, create PR from qa to main
gh pr create --base main --head qa --title "chore: promote v0.5.0 to prd"

bash

Each merge triggers auto-tag -> release -> publish in the target environment.

Squash merge caveat

When using squash merge for promotion PRs (dev -> qa), the qa branch gets a new commit that does not share ancestry with main. This can make the subsequent qa -> main PR show as "not mergeable." Workaround: use merge commits for promotion PRs, or merge main into qa before creating the PR to main.

Installing packages

Prerequisites

pip install keyrings.google-artifactregistry-auth
gcloud auth application-default login

bash

From production

pip install --extra-index-url \
  https://us-central1-python.pkg.dev/eigenoid-prd/eigenoid-python/simple/ \
  eigenoid==0.5.0

bash

From dev (for testing)

pip install --extra-index-url \
  https://us-central1-python.pkg.dev/eigenoid-dev/eigenoid-python/simple/ \
  eigenoid-sample==0.5.0

bash

In requirements.txt

--extra-index-url https://us-central1-python.pkg.dev/eigenoid-prd/eigenoid-python/simple/
eigenoid==0.5.0

text

Infrastructure

The Artifact Registry repositories are managed by Terraform in iac-distribution:

Layer	What it creates
artifact-registry	AR repository for eigenoid/eigenoid + publisher service account + WIF binding
sample-artifact-registry	AR repository for eigenoid/eigenoid-sample + publisher service account + WIF binding

Each layer creates:

• A Python-format Artifact Registry repository
• A publisher service account (e.g., "Eigenoid PyPI Publisher")
• A WIF binding that allows the specific GitHub repo to authenticate as the publisher SA
• A cleanup policy that keeps the last 5 versions

See IaC docs for how Terraform deployment works.

GitHub Apps involved

Bot	Role	Token type
eigenoid-automation	Creates tags via GitHub API	vars.AUTOMATION_BOT_CLIENT_ID + secrets.AUTOMATION_BOT_PRIVATE_KEY
eigenoid-releaser	Authors GitHub Releases	vars.RELEASER_BOT_CLIENT_ID + secrets.RELEASER_BOT_PRIVATE_KEY

Both are org-level GitHub Apps. Their credentials are stored as organization variables/secrets with ALL repository visibility.

Reference: eigenoid-sample

eigenoid-sample is the reference implementation of this pipeline. It is a minimal Python package that exists solely to validate the distribution flow end-to-end. Use it as a template when setting up a new package.

Adding a new package

To set up distribution for a new Python package:

1. Create the AR infrastructure -- add a new layer in iac-distribution (copy sample-artifact-registry as a template)
2. Deploy to all environments -- merge to main, let Terraflow provision dev -> qa -> prd
3. Create GitHub Environments on the repo -- dev, qa, prd with the correct GCP_WIF_PROVIDER, GCP_SA_EMAIL, GCP_PROJECT_ID, AR_LOCATION, AR_REPOSITORY variables
4. Add workflows -- copy auto-tag.yml and release.yml from eigenoid-sample
5. Push to dev -- the pipeline takes over from here