Compare commits

..
28 Commits
Author SHA1 Message Date
Brian Bjarke JensenandCursor a8b7b59484 Migrate Python workflows to python-repositories-ci runner label.
PR Title Check / check-title (pull_request) Successful in 6s
Test Python Package / unit-tests (pull_request) Successful in 13s
Code Quality Pipeline / code-quality (pull_request) Successful in 23s
Test Python Package / integration-tests (pull_request) Successful in 26s
Test Python Package / coverage-report (pull_request) Successful in 16s
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-10 13:46:10 +02:00
brian 0e6941dccf Merge pull request 'Use lowercase CI image registry owner for Docker compatibility' (#45) from cursor/fix-ci-registry-owner-lowercase into main
Release on merge to main / release (push) Successful in 8s
Test Python Package / integration-tests (push) Successful in 29s
Code Quality Pipeline / code-quality (push) Successful in 48s
Test Python Package / unit-tests (push) Successful in 19s
Test Python Package / coverage-report (push) Successful in 15s
Reviewed-on: LilleVemmelund/python-repositories#45
2026-07-10 13:13:01 +02:00
Brian Bjarke JensenandCursor a6428ccd29 Use lowercase CI image registry owner for Docker compatibility.
PR Title Check / check-title (pull_request) Successful in 7s
Test Python Package / unit-tests (pull_request) Successful in 1m12s
Code Quality Pipeline / code-quality (pull_request) Successful in 1m31s
Test Python Package / integration-tests (pull_request) Successful in 2m25s
Test Python Package / coverage-report (pull_request) Successful in 22s
Docker requires lowercase repository names; Gitea maps lillevemmelund to the LilleVemmelund org.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-10 13:08:28 +02:00
brian 655fdc4ee8 Merge pull request 'Fix CI image registry owner to match Gitea org slug' (#44) from cursor/fix-ci-registry-owner-case into main
Release on merge to main / release (push) Successful in 8s
Test Python Package / unit-tests (push) Successful in 14s
Code Quality Pipeline / code-quality (push) Successful in 30s
Test Python Package / integration-tests (push) Successful in 59s
Test Python Package / coverage-report (push) Successful in 39s
Reviewed-on: LilleVemmelund/python-repositories#44
2026-07-10 13:01:33 +02:00
Brian Bjarke JensenandCursor e031c549e5 Fix CI image registry owner to match Gitea org slug.
PR Title Check / check-title (pull_request) Successful in 7s
Test Python Package / unit-tests (pull_request) Successful in 18s
Test Python Package / integration-tests (pull_request) Successful in 23s
Code Quality Pipeline / code-quality (pull_request) Successful in 34s
Test Python Package / coverage-report (pull_request) Successful in 12s
Use LilleVemmelund instead of lille-vemmelund so container registry pushes resolve the correct namespace.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-10 12:58:29 +02:00
brian 49b6238b7d Merge pull request 'Rename CI registry owner from brian to lille-vemmelund' (#43) from cursor/rename-ci-owner-to-lille-vemmelund into main
Release on merge to main / release (push) Successful in 7s
Code Quality Pipeline / code-quality (push) Successful in 31s
Test Python Package / unit-tests (push) Successful in 37s
Test Python Package / integration-tests (push) Successful in 1m14s
Test Python Package / coverage-report (push) Successful in 13s
Reviewed-on: LilleVemmelund/python-repositories#43
2026-07-10 12:51:03 +02:00
Brian Bjarke JensenandCursor 0549358fe1 Keep python-utils package index under brian owner.
PR Title Check / check-title (pull_request) Successful in 7s
Code Quality Pipeline / code-quality (pull_request) Successful in 29s
Test Python Package / unit-tests (pull_request) Successful in 1m28s
Test Python Package / integration-tests (pull_request) Successful in 1m48s
Test Python Package / coverage-report (pull_request) Successful in 17s
Only the CI image registry owner changes to lille-vemmelund; python-utils remains published under brian.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-10 12:48:32 +02:00
Brian Bjarke JensenandCursor 33efbd1005 Update CI registry owner from brian to lille-vemmelund.
PR Title Check / check-title (pull_request) Successful in 6s
Test Python Package / integration-tests (pull_request) Failing after 30s
Code Quality Pipeline / code-quality (pull_request) Failing after 1m12s
Test Python Package / unit-tests (pull_request) Failing after 1m28s
Test Python Package / coverage-report (pull_request) Has been skipped
Align CI image paths, package index URLs, and lockfile entries with the new Gitea owner name.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-10 09:32:39 +02:00
brian 24126e4044 Merge pull request 'Fix CI image build failures on runners with isolated DinD DNS' (#42) from fix/ci-image-build-dns into main
Release on merge to main / release (push) Successful in 10s
Test Python Package / unit-tests (push) Successful in 16s
Code Quality Pipeline / code-quality (push) Successful in 39s
Build CI Image / build-and-push (push) Failing after 55s
Test Python Package / integration-tests (push) Successful in 1m11s
Test Python Package / coverage-report (push) Successful in 20s
Reviewed-on: brian/python-repositories#42
2026-07-10 09:19:12 +02:00
Brian Bjarke JensenandCursor f3b6cdbc9e Fix CI image build failures on runners with isolated DinD DNS.
PR Title Check / check-title (pull_request) Successful in 8s
Test Python Package / unit-tests (pull_request) Successful in 17s
Code Quality Pipeline / code-quality (pull_request) Successful in 51s
Test Python Package / integration-tests (pull_request) Successful in 1m40s
Test Python Package / coverage-report (pull_request) Successful in 17s
Copy the Docker CLI from the official image instead of apt, and use
--network=host during docker build so apt and layer pulls can resolve
external registries on standalone runners.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-10 09:16:19 +02:00
brian fa2554ef5f Merge pull request 'Add nightly CI base image and migrate workflows to python-repositories-ci' (#40) from cursor/ci-base-image into main
Release on merge to main / release (push) Successful in 7s
Test Python Package / unit-tests (push) Successful in 19s
Code Quality Pipeline / code-quality (push) Successful in 31s
Test Python Package / integration-tests (push) Successful in 1m8s
Test Python Package / coverage-report (push) Successful in 39s
Build CI Image / build-and-push (push) Failing after 6m24s
Reviewed-on: brian/python-repositories#40
2026-07-09 22:51:31 +02:00
Brian Bjarke JensenandCursor b21fd247c5 Revert workflows to ubuntu-latest until CI image is seeded.
PR Title Check / check-title (pull_request) Successful in 6s
Test Python Package / unit-tests (pull_request) Successful in 1m4s
Code Quality Pipeline / code-quality (pull_request) Successful in 1m48s
Test Python Package / integration-tests (pull_request) Successful in 2m10s
Test Python Package / coverage-report (pull_request) Successful in 20s
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 22:47:52 +02:00
Brian Bjarke JensenandCursor 441a21204d Add local CI image build script and document bootstrap flow.
Code Quality Pipeline / code-quality (pull_request) Failing after 1s
Test Python Package / unit-tests (pull_request) Failing after 1s
Test Python Package / integration-tests (pull_request) Failing after 1s
Test Python Package / coverage-report (pull_request) Has been skipped
PR Title Check / check-title (pull_request) Successful in 34s
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 22:39:33 +02:00
Brian Bjarke JensenandCursor f2a8581de2 Read Python version from .python-version in CI image build.
PR Title Check / check-title (pull_request) Successful in 41s
Code Quality Pipeline / code-quality (pull_request) Failing after 1s
Test Python Package / integration-tests (pull_request) Failing after 34s
Test Python Package / unit-tests (pull_request) Failing after 40s
Test Python Package / coverage-report (pull_request) Has been skipped
Avoid hardcoding 3.12 in the Dockerfile so the CI image tracks the same source of truth as local tooling and uv sync.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 22:27:59 +02:00
Brian Bjarke JensenandCursor 3b05260323 Add nightly CI base image and migrate workflows to python-repositories-ci.
Code Quality Pipeline / code-quality (pull_request) Failing after 1s
Test Python Package / unit-tests (pull_request) Failing after 0s
Test Python Package / integration-tests (pull_request) Failing after 0s
PR Title Check / check-title (pull_request) Successful in 6s
Test Python Package / coverage-report (pull_request) Has been skipped
Pre-build Python, uv, and locked deps in a Gitea container image so daily jobs skip repeated bootstrap; document runner registry auth and ci-bot credentials.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 22:21:22 +02:00
CI Bot 369295b8ae chore: release v2.0.2 [skip ci] 2026-07-09 16:45:24 +00:00
brian 95c78596e7 Merge pull request '[patch] Use dict[str, Any] for JSON repository document types' (#38) from cursor/json-dict-str-any-typing into main
Test Python Package / integration-tests (push) Successful in 48s
Release on merge to main / release (push) Successful in 12s
Test Python Package / unit-tests (push) Successful in 22s
Code Quality Pipeline / code-quality (push) Successful in 29s
Test Python Package / coverage-report (push) Successful in 12s
Reviewed-on: brian/python-repositories#38
2026-07-09 18:45:10 +02:00
Brian Bjarke JensenandCursor 42e885edd5 Use dict[str, Any] for JSON repository document types.
PR Title Check / check-title (pull_request) Successful in 6s
Code Quality Pipeline / code-quality (pull_request) Successful in 1m28s
Test Python Package / unit-tests (pull_request) Successful in 1m29s
Test Python Package / coverage-report (pull_request) Successful in 16s
Test Python Package / integration-tests (pull_request) Successful in 48s
Clarifies that JSON CRUD operates on string-keyed objects without changing runtime behavior.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 18:41:23 +02:00
CI Bot bd4794d1c0 chore: release v2.0.1 [skip ci] 2026-07-09 14:38:25 +00:00
brian 3dc412cd15 Merge pull request '[patch] Align Redis and MinIO connect() idempotency' (#37) from cursor/connect-idempotency-alignment into main
Test Python Package / coverage-report (push) Successful in 17s
Release on merge to main / release (push) Successful in 21s
Test Python Package / integration-tests (push) Successful in 1m17s
Test Python Package / unit-tests (push) Successful in 1m26s
Code Quality Pipeline / code-quality (push) Successful in 1m36s
Reviewed-on: brian/python-repositories#37
2026-07-09 16:38:05 +02:00
Brian Bjarke JensenandCursor b8d88e6703 Cover injected-client connect when client was cleared by disconnect.
PR Title Check / check-title (pull_request) Successful in 5s
Code Quality Pipeline / code-quality (pull_request) Successful in 26s
Test Python Package / integration-tests (pull_request) Successful in 1m5s
Test Python Package / unit-tests (pull_request) Successful in 29s
Test Python Package / coverage-report (pull_request) Successful in 41s
Restore 100% coverage for the _validate_injected_client early-return paths in Redis and MinIO adapters.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 15:57:07 +02:00
Brian Bjarke JensenandCursor 67bb88fbb4 Align Redis and MinIO connect() idempotency in ConnectionAwareAdapter.
PR Title Check / check-title (pull_request) Successful in 5s
Test Python Package / unit-tests (pull_request) Successful in 13s
Test Python Package / integration-tests (pull_request) Successful in 30s
Test Python Package / coverage-report (pull_request) Failing after 15s
Code Quality Pipeline / code-quality (pull_request) Successful in 53s
Move shared connect orchestration into the base adapter so both backends short-circuit when already healthy and only reconnect after a failed probe.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 15:46:39 +02:00
CI Bot c2e6a96c5d chore: release v2.0.0 [skip ci] 2026-07-09 13:17:19 +00:00
brian 547a1b3a90 Merge pull request '[breaking] Make MinIO bucket creation opt-in with production-safe defaults' (#36) from cursor/minio-bucket-creation-opt-in into main
Code Quality Pipeline / code-quality (push) Successful in 41s
Release on merge to main / release (push) Successful in 25s
Test Python Package / unit-tests (push) Successful in 17s
Test Python Package / integration-tests (push) Successful in 1m44s
Test Python Package / coverage-report (push) Successful in 23s
Reviewed-on: brian/python-repositories#36
2026-07-09 15:16:54 +02:00
Brian Bjarke JensenandCursor 39baf9badc Fix examples integration test by provisioning MinIO bucket in fixture.
Test Python Package / unit-tests (pull_request) Successful in 18s
PR Title Check / check-title (pull_request) Successful in 34s
Test Python Package / integration-tests (pull_request) Successful in 33s
Code Quality Pipeline / code-quality (pull_request) Successful in 49s
Test Python Package / coverage-report (pull_request) Successful in 51s
Ensure set_example_env depends on raw_minio_client so ArtifactObjectRepository connects against an existing bucket under the new production-safe default.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 15:06:30 +02:00
Brian Bjarke JensenandCursor 4a5b3b631f Make MinIO bucket creation opt-in with production-safe defaults.
PR Title Check / check-title (pull_request) Successful in 6s
Test Python Package / unit-tests (pull_request) Successful in 18s
Code Quality Pipeline / code-quality (pull_request) Successful in 1m28s
Test Python Package / integration-tests (pull_request) Failing after 1m54s
Test Python Package / coverage-report (pull_request) Has been skipped
Require buckets to exist by default on connect, extract env_bool parsing, and document local dev overrides for secure and auto-creation settings.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 14:52:14 +02:00
brian 1431f455c1 Merge pull request 'Move Redis test container helpers into integration conftest' (#35) from cursor/move-redis-container-to-conftest into main
Release on merge to main / release (push) Successful in 6s
Test Python Package / unit-tests (push) Successful in 21s
Code Quality Pipeline / code-quality (push) Successful in 28s
Test Python Package / integration-tests (push) Successful in 50s
Test Python Package / coverage-report (push) Successful in 1m19s
Reviewed-on: brian/python-repositories#35
2026-07-08 21:23:48 +02:00
Brian Bjarke JensenandCursor eafc717045 Move Redis test container helpers into integration conftest.
PR Title Check / check-title (pull_request) Successful in 5s
Test Python Package / unit-tests (pull_request) Successful in 14s
Code Quality Pipeline / code-quality (pull_request) Successful in 31s
Test Python Package / integration-tests (pull_request) Successful in 1m9s
Test Python Package / coverage-report (pull_request) Successful in 29s
Colocate RedisTestContainer with the fixtures that use it and remove the misleading redis_container_test module plus collect_ignore workaround.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-08 21:17:33 +02:00
33 changed files with 751 additions and 234 deletions
+16
View File
@@ -0,0 +1,16 @@
.git
.github
.gitea
.mypy_cache
.pytest_cache
.ruff_cache
.venv
__pycache__
*.pyc
.coverage
coverage-unit
coverage-integration
coverage.txt
dist
build
*.egg-info
+1
View File
@@ -7,3 +7,4 @@ MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin
MINIO_BUCKET=my-bucket
MINIO_SECURE=false
MINIO_CREATE_BUCKET_IF_MISSING=true
+52
View File
@@ -0,0 +1,52 @@
# Build and publish the per-repo CI base image to the Gitea container registry.
name: Build CI Image
on:
schedule:
- cron: "0 2 * * *"
workflow_dispatch:
push:
branches:
- main
paths:
- pyproject.toml
- uv.lock
- docker/ci/**
env:
REGISTRY: gitea.lille-vemmelund.dk
REGISTRY_USER: ci-bot
IMAGE: gitea.lille-vemmelund.dk/lillevemmelund/python-repositories-ci
jobs:
build-and-push:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Log in to Gitea container registry
env:
CI_RUNNER_TOKEN: ${{ secrets.CI_RUNNER_TOKEN }}
run: |
echo "$CI_RUNNER_TOKEN" | docker login "$REGISTRY" -u "$REGISTRY_USER" --password-stdin
- name: Build CI image
env:
CI_RUNNER_TOKEN: ${{ secrets.CI_RUNNER_TOKEN }}
run: |
echo "$CI_RUNNER_TOKEN" > /tmp/uv_token
docker build --network=host -f docker/ci/Dockerfile \
--secret id=uv_token,src=/tmp/uv_token \
-t "${IMAGE}:latest" \
.
rm -f /tmp/uv_token
- name: Tag and push CI image
run: |
STAMP="$(date -u +%Y%m%d%H%M)"
docker tag "${IMAGE}:latest" "${IMAGE}:${STAMP}"
docker push "${IMAGE}:latest"
docker push "${IMAGE}:${STAMP}"
echo "Pushed ${IMAGE}:latest and ${IMAGE}:${STAMP}"
+5 -11
View File
@@ -8,23 +8,17 @@ on:
jobs:
code-quality:
runs-on: ubuntu-latest
runs-on: python-repositories-ci
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: .python-version
- name: Install uv
run: pip install uv
- name: Install dependencies
- name: Sync dependencies
env:
UV_LINK_MODE: copy
run: uv sync --all-extras
UV_INDEX_GITEA_USERNAME: ci-bot
UV_INDEX_GITEA_PASSWORD: ${{ secrets.CI_RUNNER_TOKEN }}
run: uv sync --all-extras --frozen
- name: Type check with mypy
run: uv run mypy .
+4 -9
View File
@@ -7,22 +7,17 @@ on:
jobs:
update-check:
runs-on: ubuntu-latest
runs-on: python-repositories-ci
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
token: ${{ secrets.CI_RUNNER_TOKEN }}
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: .python-version
- name: Install uv
run: pip install uv
- name: Upgrade dependencies
env:
UV_INDEX_GITEA_USERNAME: ci-bot
UV_INDEX_GITEA_PASSWORD: ${{ secrets.CI_RUNNER_TOKEN }}
run: uv lock --upgrade
- name: Commit and push changes
+9 -10
View File
@@ -9,25 +9,24 @@ on:
jobs:
build-and-publish:
runs-on: ubuntu-latest
runs-on: python-repositories-ci
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: .python-version
- name: Install uv
run: pip install uv
- name: Update version in pyproject.toml to match tag
env:
UV_INDEX_GITEA_USERNAME: ci-bot
UV_INDEX_GITEA_PASSWORD: ${{ secrets.CI_RUNNER_TOKEN }}
run: scripts/ci/bump-version.sh --from-tag "${GITHUB_REF##*/}"
- name: Build package
env:
UV_LINK_MODE: copy
UV_INDEX_GITEA_USERNAME: ci-bot
UV_INDEX_GITEA_PASSWORD: ${{ secrets.CI_RUNNER_TOKEN }}
run: |
uv sync --no-dev
uv sync --no-dev --frozen
uv build
- name: Publish to Gitea Package Registry
+4 -11
View File
@@ -8,7 +8,7 @@ on:
jobs:
release:
if: ${{ !contains(github.event.head_commit.message, '[skip ci]') }}
runs-on: ubuntu-latest
runs-on: python-repositories-ci
steps:
- name: Checkout code
uses: actions/checkout@v4
@@ -22,19 +22,12 @@ jobs:
COMMIT_MSG: ${{ github.event.head_commit.message }}
run: scripts/ci/parse-merge-commit.sh "$COMMIT_MSG"
- name: Set up Python
if: steps.meta.outputs.bump != 'skip'
uses: actions/setup-python@v5
with:
python-version-file: .python-version
- name: Install uv
if: steps.meta.outputs.bump != 'skip'
run: pip install uv
- name: Bump version
if: steps.meta.outputs.bump != 'skip'
id: bump
env:
UV_INDEX_GITEA_USERNAME: ci-bot
UV_INDEX_GITEA_PASSWORD: ${{ secrets.CI_RUNNER_TOKEN }}
run: scripts/ci/bump-version.sh "${{ steps.meta.outputs.bump }}"
- name: Generate release notes
+5 -11
View File
@@ -7,23 +7,17 @@ on:
jobs:
safety:
runs-on: ubuntu-latest
runs-on: python-repositories-ci
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: .python-version
- name: Install uv
run: pip install uv
- name: Install dependencies
- name: Sync dependencies
env:
UV_LINK_MODE: copy
run: uv sync
UV_INDEX_GITEA_USERNAME: ci-bot
UV_INDEX_GITEA_PASSWORD: ${{ secrets.CI_RUNNER_TOKEN }}
run: uv sync --frozen
- name: Run safety check
run: uv run safety check
+15 -33
View File
@@ -8,23 +8,17 @@ on:
jobs:
unit-tests:
runs-on: ubuntu-latest
runs-on: python-repositories-ci
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: .python-version
- name: Install uv
run: pip install uv
- name: Install dependencies
- name: Sync dependencies
env:
UV_LINK_MODE: copy
run: uv sync --all-extras
UV_INDEX_GITEA_USERNAME: ci-bot
UV_INDEX_GITEA_PASSWORD: ${{ secrets.CI_RUNNER_TOKEN }}
run: uv sync --all-extras --frozen
- name: Run unit tests
run: |
@@ -43,23 +37,17 @@ jobs:
compression-level: 0
integration-tests:
runs-on: ubuntu-latest
runs-on: python-repositories-ci
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: .python-version
- name: Install uv
run: pip install uv
- name: Install dependencies
- name: Sync dependencies
env:
UV_LINK_MODE: copy
run: uv sync --all-extras
UV_INDEX_GITEA_USERNAME: ci-bot
UV_INDEX_GITEA_PASSWORD: ${{ secrets.CI_RUNNER_TOKEN }}
run: uv sync --all-extras --frozen
- name: Verify Docker
run: docker info
@@ -83,24 +71,18 @@ jobs:
coverage-report:
# Merges unit + integration coverage and enforces fail_under from pyproject.toml.
needs: [unit-tests, integration-tests]
runs-on: ubuntu-latest
runs-on: python-repositories-ci
if: github.event_name == 'pull_request' || github.event_name == 'push'
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: .python-version
- name: Install uv
run: pip install uv
- name: Install dependencies
- name: Sync dependencies
env:
UV_LINK_MODE: copy
run: uv sync --all-extras
UV_INDEX_GITEA_USERNAME: ci-bot
UV_INDEX_GITEA_PASSWORD: ${{ secrets.CI_RUNNER_TOKEN }}
run: uv sync --all-extras --frozen
- name: Download unit coverage
uses: https://github.com/christopherHX/gitea-download-artifact@v4
+24 -11
View File
@@ -12,7 +12,7 @@ Subclass an adapter in your own repository to add domain-specific methods while
| **Adapters** | Technology-specific base classes (`RedisAdapter`, `MinioAdapter`) |
| **Your project** | Subclass an adapter and add domain methods |
Connection adapters expose `connect()`, `disconnect()`, and `is_connected()`. The latter verifies backend reachability with a cached health probe (default TTL: 1 second). Subclasses may override `health_check_ttl_seconds`.
Connection adapters expose `connect()`, `disconnect()`, and `is_connected()`. The latter verifies backend reachability with a cached health probe (default TTL: 1 second). Subclasses may override `health_check_ttl_seconds`. `connect()` is idempotent: calling it while already connected and healthy is a no-op.
## Optional dependencies
@@ -48,16 +48,19 @@ for key in repo.scan_keys("user:*"):
### MinIO (`ObjectRepositoryInterface`)
| Environment variable | Description |
| -------------------- | ------------------------------------------- |
| `MINIO_ENDPOINT` | MinIO server endpoint |
| `MINIO_ACCESS_KEY` | Access key |
| `MINIO_SECRET_KEY` | Secret key |
| `MINIO_BUCKET` | Bucket name (created on connect if missing) |
| `MINIO_SECURE` | Use HTTPS (`true`/`false`; default: `true`) |
| Environment variable | Description |
| -------------------------------- | ------------------------------------------------------------------------ |
| `MINIO_ENDPOINT` | MinIO server endpoint |
| `MINIO_ACCESS_KEY` | Access key |
| `MINIO_SECRET_KEY` | Secret key |
| `MINIO_BUCKET` | Bucket name |
| `MINIO_SECURE` | Use HTTPS (`true`/`false`; default: `true`) |
| `MINIO_CREATE_BUCKET_IF_MISSING` | Auto-create `MINIO_BUCKET` on connect (`true`/`false`; default: `false`) |
Copy [`.env.example`](.env.example) to `.env` for local development. `RedisConfig.from_env()` and `MinioConfig.from_env()` load `.env` automatically when resolving configuration from the environment.
For production, it is recommended to leave `MINIO_CREATE_BUCKET_IF_MISSING` unset so that `connect()` fails fast if the expected bucket is missing. For local development, you will often want `MINIO_SECURE=false` and `MINIO_CREATE_BUCKET_IF_MISSING=true`.
## Configuration injection
Adapters accept optional `config` and `client` keyword arguments for explicit setup and testing:
@@ -73,6 +76,7 @@ minio = MinioAdapter(
secret_key="minioadmin",
bucket="my-bucket",
secure=False,
# create_bucket_if_missing=True, # convenient for local dev
)
)
```
@@ -119,16 +123,18 @@ with ArtifactObjectRepository() as repo:
### Subclassing in your own project
```python
from typing import Any
from python_repositories import RedisAdapter
class UserRepository(RedisAdapter):
def _key(self, user_id: str) -> str:
return f"user:{user_id}"
def get_user(self, user_id: str) -> dict | None:
def get_user(self, user_id: str) -> dict[str, Any] | None:
return self.get(self._key(user_id))
def save_user(self, user_id: str, user: dict) -> None:
def save_user(self, user_id: str, user: dict[str, Any]) -> None:
self.set(self._key(user_id), user)
```
@@ -160,7 +166,14 @@ uv run pytest -v # full suite (requires Doc
Integration tests are marked with `@pytest.mark.integration` and require Docker (testcontainers). Run unit tests alone for quick local feedback.
CI runs unit and integration tests in parallel with coverage, then merges `.coverage` artifacts in a follow-up job (via [christopherhx/gitea-\*-artifact@v4](https://github.com/christopherHX/gitea-upload-artifact) for Gitea 1.26 compatibility). Combined coverage must be at least **90%**; the floor is set by [`fail_under` in `pyproject.toml`](pyproject.toml#L45-L48) and enforced after merging unit and integration coverage, not on unit-only runs.
### CI base image
Gitea Actions jobs use a pre-built image (`python-repositories-ci`) with Python 3.12,
`uv`, and locked dependencies baked in. The image is rebuilt nightly and when
`uv.lock` changes; see [`docs/ci-image.md`](docs/ci-image.md) for runner setup and
bootstrap order.
CI runs unit and integration tests in parallel with coverage, then merges `.coverage` artifacts in a follow-up job (via [christopherhx/gitea-\*-artifact@v4](https://github.com/christopherHX/gitea-upload-artifact) for Gitea 1.26 compatibility). Combined coverage must meet the floor in [`fail_under` in `pyproject.toml`](pyproject.toml#L45-L48); enforcement happens after merging unit and integration coverage, not on unit-only runs.
To check coverage locally (requires Docker for the full suite):
+29
View File
@@ -0,0 +1,29 @@
# CI job image for Gitea Actions (act_runner requires Node.js in job containers).
FROM node:20-bookworm
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
git \
&& rm -rf /var/lib/apt/lists/*
COPY --from=docker:27-cli /usr/local/bin/docker /usr/local/bin/docker
COPY --from=ghcr.io/astral-sh/uv:0.7.0 /uv /usr/local/bin/uv
ENV UV_PYTHON_INSTALL_DIR=/opt/uv-python \
UV_LINK_MODE=copy
COPY .python-version /tmp/.python-version
RUN uv python install "$(cat /tmp/.python-version)"
WORKDIR /app
COPY pyproject.toml uv.lock .python-version ./
ENV PATH="/app/.venv/bin:${PATH}"
RUN --mount=type=secret,id=uv_token \
UV_INDEX_GITEA_USERNAME=ci-bot \
UV_INDEX_GITEA_PASSWORD="$(cat /run/secrets/uv_token)" \
uv sync --all-extras --frozen --no-install-project
+169
View File
@@ -0,0 +1,169 @@
# CI base image
This repository uses a **per-repo Docker image** for Gitea Actions jobs instead of
installing Python and `uv` on every run. Infrastructure images (redis, minio, etc.)
are cached cluster-wide via Harbor (see homelab-platform
[`docs/harbor-registry-mirror.md`](https://gitea.lille-vemmelund.dk/LilleVemmelund/homelab-platform/src/branch/main/docs/harbor-registry-mirror.md)).
## Image contents
Built from [`docker/ci/Dockerfile`](../docker/ci/Dockerfile):
- Node.js 20 (required by act_runner job containers)
- Python 3.12 (installed via `uv python install` from [`.python-version`](../.python-version)) and pinned `uv` 0.7.0
- Docker CLI (integration tests via testcontainers)
- Dev dependencies from `uv.lock` (`uv sync --all-extras --no-install-project`)
Published to the Gitea container registry (owner segment must be lowercase for Docker):
- `gitea.lille-vemmelund.dk/lillevemmelund/python-repositories-ci:latest`
- `gitea.lille-vemmelund.dk/lillevemmelund/python-repositories-ci:YYYYMMDDHHmm` (timestamped rollback tag)
## Rebuild triggers
[`ci-image.yml`](../.gitea/workflows/ci-image.yml) runs on:
- Nightly cron (`0 2 * * *` UTC)
- Manual `workflow_dispatch`
- Push to `main` when `pyproject.toml`, `uv.lock`, or `docker/ci/**` change
## Workflow usage
Python jobs use `runs-on: python-repositories-ci` and a fast incremental sync:
```yaml
runs-on: python-repositories-ci
steps:
- uses: actions/checkout@v4
- name: Sync dependencies
env:
UV_LINK_MODE: copy
UV_INDEX_GITEA_USERNAME: ci-bot
UV_INDEX_GITEA_PASSWORD: ${{ secrets.CI_RUNNER_TOKEN }}
run: uv sync --all-extras --frozen
```
`ci-image.yml` uses `runs-on: ubuntu-latest` so it can bootstrap before the custom
image exists.
## Registry authentication
act_runner pulls the job image **before any workflow step runs**, so a `docker
login` step inside a job cannot authenticate that pull. Authentication must be
configured on the runner (or via a job-level `container.credentials` block — not
used here).
### k8s Gitea runners (automatic)
The homelab-platform runners mount Gitea registry credentials automatically:
- Secret: `gitea-runners/gitea-registry-dockerconfig` (created by
[`scripts/create-gitea-registry-secret.sh`](https://gitea.lille-vemmelund.dk/LilleVemmelund/homelab-platform/src/branch/main/scripts/create-gitea-registry-secret.sh))
- Mounted at `/root/.docker/config.json` on the **runner** container (not DinD)
- Configured in
[`platform/gitea-runners/values.yaml`](https://gitea.lille-vemmelund.dk/LilleVemmelund/homelab-platform/src/branch/main/platform/gitea-runners/values.yaml)
No manual `docker login` is required on the in-cluster runners once the secret
exists. To create or rotate credentials:
```bash
export KUBECONFIG=/path/to/homelab-cluster/talos/_out/kubeconfig
export GITEA_REGISTRY_USERNAME='ci-bot'
export GITEA_REGISTRY_PASSWORD='personal-access-token-with-read-package'
bash scripts/create-gitea-registry-secret.sh
```
Then restart runner pods so they pick up the updated secret. See homelab-platform
[`docs/gitea-actions-runners.md`](https://gitea.lille-vemmelund.dk/LilleVemmelund/homelab-platform/src/branch/main/docs/gitea-actions-runners.md)
for full runner setup.
Verify on a running pod:
```bash
kubectl -n gitea-runners exec homelab-cluster-gitea-runner-0 -c runner -- \
test -f /root/.docker/config.json && echo "registry auth mounted"
```
### Standalone runners (e.g. Unraid `homelab`)
The 4th runner is outside the k8s cluster and does **not** get the automatic
mount. Configure registry auth manually on that host:
```bash
docker login gitea.lille-vemmelund.dk -u ci-bot -p <token>
```
Also add the `python-repositories-ci` label to that runner's act_runner config.
### `ci-image.yml` push login
The build workflow still runs `docker login` before `docker push`. That step
authenticates **DinD inside the job** for pushing the image to Gitea — a different
code path from act_runner pulling the job container.
## Runner label
Jobs use the `python-repositories-ci` act_runner label, configured in
homelab-platform
[`platform/gitea-runners/values.yaml`](https://gitea.lille-vemmelund.dk/LilleVemmelund/homelab-platform/src/branch/main/platform/gitea-runners/values.yaml):
```yaml
python-repositories-ci:docker://gitea.lille-vemmelund.dk/lillevemmelund/python-repositories-ci:latest
```
After label changes, roll runner pods so they re-register with Gitea.
## Bootstrap order
1. Ensure homelab-platform runners have `gitea-registry-dockerconfig` and the
`python-repositories-ci` label (see homelab-platform docs).
2. Merge `ci-image.yml`, `docker/ci/Dockerfile`, and workflow migrations to `main`.
3. Seed the registry with a first image (see below).
4. Confirm a test workflow job starts on `python-repositories-ci`.
Until step 3 completes, jobs targeting `python-repositories-ci` will fail because
the image does not exist in the Gitea registry yet.
### Seed the image
**After merge to `main`:** Actions → **Build CI Image****Run workflow**.
**Before merge (e.g. PR branch):** the workflow file is not on `main` yet — build
and push locally with [`scripts/ci/build-ci-image.sh`](../scripts/ci/build-ci-image.sh):
```bash
export CI_RUNNER_TOKEN='ci-bot-personal-access-token'
bash scripts/ci/build-ci-image.sh --push
```
Run from the repo root on the branch you want to test. Runners pull
`gitea.lille-vemmelund.dk/lillevemmelund/python-repositories-ci:latest` from the registry;
they do not care which git branch built it.
Override registry settings if needed:
```bash
export REGISTRY=gitea.lille-vemmelund.dk
export REGISTRY_USER=ci-bot
export CI_RUNNER_TOKEN='...'
bash scripts/ci/build-ci-image.sh --push
```
## Local build
Build only (no registry login or push):
```bash
export CI_RUNNER_TOKEN='ci-bot-personal-access-token'
bash scripts/ci/build-ci-image.sh --local
```
Or manually:
```bash
echo "$CI_RUNNER_TOKEN" > /tmp/uv_token
docker build -f docker/ci/Dockerfile \
--secret id=uv_token,src=/tmp/uv_token \
-t python-repositories-ci:local .
rm -f /tmp/uv_token
```
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "python-repositories"
version = "1.1.0"
version = "2.0.2"
description = "Various python repository interfaces exposed as a python package."
authors = [
{ name = "Brian Bjarke Jensen", email = "schnitzelen@gmail.com" }
@@ -22,6 +22,7 @@ class ConnectionAwareAdapter(ConnectionAwareInterface, ContextAwareInterface):
def __init__(self) -> None:
self.logger = structlog.get_logger(self.__class__.__name__)
self._client_injected = False
self._health_check_at: float | None = None
self._health_check_ok: bool = False
@@ -57,6 +58,27 @@ class ConnectionAwareAdapter(ConnectionAwareInterface, ContextAwareInterface):
def _probe_connection(self) -> bool:
"""Backend-specific liveness check; called only when client is ready."""
@abstractmethod
def _validate_injected_client(self) -> None:
"""Verify an injected client is reachable; raise ConnectionError on failure."""
@abstractmethod
def _establish_connection(self) -> None:
"""Create a backend client and set internal connection state."""
def connect(self) -> None:
"""Connect to the backend; idempotent when already connected and healthy."""
if self._client_injected:
self._validate_injected_client()
self._invalidate_health_cache()
return
if self._is_client_ready() and self.is_connected():
self.logger.info(f"Already connected to {self.connection_name}")
return
self.disconnect()
self._establish_connection()
self._invalidate_health_cache()
def is_connected(self) -> bool:
"""Check if connected to the backend."""
if not self._is_client_ready():
+18 -19
View File
@@ -26,6 +26,7 @@ class MinioAdapter(ObjectRepositoryInterface, ConnectionAwareAdapter):
secret_key_env_var_name: str = "MINIO_SECRET_KEY"
bucket_env_var_name: str = "MINIO_BUCKET"
secure_env_var_name: str = "MINIO_SECURE"
create_bucket_if_missing_env_var_name: str = "MINIO_CREATE_BUCKET_IF_MISSING"
chunk_size: int = 5 * 2**20 # 5 MiB
connection_name: str = "Minio"
@@ -36,15 +37,16 @@ class MinioAdapter(ObjectRepositoryInterface, ConnectionAwareAdapter):
client: minio.Minio | None = None,
) -> None:
super().__init__()
if client is not None and config is None:
raise ValueError("config is required when client is provided")
if config is None:
if client is not None:
raise ValueError("config is required when client is provided")
config = MinioConfig.from_env(
self.endpoint_env_var_name,
self.access_key_env_var_name,
self.secret_key_env_var_name,
self.bucket_env_var_name,
self.secure_env_var_name,
self.create_bucket_if_missing_env_var_name,
)
self._config = config
self._client_injected = client is not None
@@ -56,23 +58,17 @@ class MinioAdapter(ObjectRepositoryInterface, ConnectionAwareAdapter):
def _is_client_ready(self) -> bool:
return self._client is not None and self._bucket_name is not None
def connect(self) -> None:
"""Connect to the Minio server."""
if self._client_injected:
if self._client is not None:
try:
_ = self._client.list_buckets()
except Exception as exc: # pylint: disable=broad-except
raise ConnectionError(
f"Could not connect to Minio at {self._config.endpoint}"
) from exc
self._invalidate_health_cache()
def _validate_injected_client(self) -> None:
if self._client is None:
return
if self._client is not None and self.is_connected():
self.logger.info("Already connected to Minio")
return
if self._client is not None:
self.disconnect()
try:
_ = self._client.list_buckets()
except Exception as exc: # pylint: disable=broad-except
raise ConnectionError(
f"Could not connect to Minio at {self._config.endpoint}"
) from exc
def _establish_connection(self) -> None:
endpoint = self._config.endpoint
access_key = self._config.access_key
secret_key = self._config.secret_key
@@ -88,11 +84,14 @@ class MinioAdapter(ObjectRepositoryInterface, ConnectionAwareAdapter):
except Exception as exc: # pylint: disable=broad-except
raise ConnectionError(f"Could not connect to Minio at {endpoint}") from exc
if not client.bucket_exists(bucket):
if not self._config.create_bucket_if_missing:
raise ConnectionError(
f"Bucket '{bucket}' does not exist on Minio at {endpoint}"
)
self.logger.info(f"Creating bucket '{bucket}'")
client.make_bucket(bucket)
self._client = client
self._bucket_name = bucket
self._invalidate_health_cache()
def disconnect(self) -> None:
"""Disconnect from the Minio server."""
+19 -25
View File
@@ -2,7 +2,7 @@
from __future__ import annotations
from collections.abc import Iterator
from typing import cast
from typing import Any, cast
from python_repositories.adapters.connection_aware_adapter import (
ConnectionAwareAdapter,
@@ -35,9 +35,9 @@ class RedisAdapter(JsonRepositoryInterface, ConnectionAwareAdapter):
client: redis.Redis | None = None,
) -> None:
super().__init__()
if client is not None and config is None:
raise ValueError("config is required when client is provided")
if config is None:
if client is not None:
raise ValueError("config is required when client is provided")
config = RedisConfig.from_env(self.uri_env_var_name)
self._config = config
self._client_injected = client is not None
@@ -49,25 +49,20 @@ class RedisAdapter(JsonRepositoryInterface, ConnectionAwareAdapter):
def _is_client_ready(self) -> bool:
return self._client is not None
def connect(self) -> None:
"""Connect to the Redis server."""
if self._client_injected:
if self._client is not None:
try:
if not self._client.ping():
raise ConnectionError(
f"Could not connect to Redis at {self._config.uri}"
)
except (redis.ConnectionError, redis.TimeoutError) as exc:
raise ConnectionError(
f"Could not connect to Redis at {self._config.uri}"
) from exc
self._invalidate_health_cache()
def _validate_injected_client(self) -> None:
if self._client is None:
return
if self._client is not None:
self._client.close()
self._client = None
self._invalidate_health_cache()
try:
if not self._client.ping():
raise ConnectionError(
f"Could not connect to Redis at {self._config.uri}"
)
except (redis.ConnectionError, redis.TimeoutError) as exc:
raise ConnectionError(
f"Could not connect to Redis at {self._config.uri}"
) from exc
def _establish_connection(self) -> None:
uri = self._config.uri
try:
client = redis.Redis.from_url(
@@ -79,7 +74,6 @@ class RedisAdapter(JsonRepositoryInterface, ConnectionAwareAdapter):
except (redis.ConnectionError, redis.TimeoutError) as exc:
raise ConnectionError(f"Could not connect to Redis at {uri}") from exc
self._client = client
self._invalidate_health_cache()
def disconnect(self) -> None:
"""Disconnect from the Redis server."""
@@ -95,7 +89,7 @@ class RedisAdapter(JsonRepositoryInterface, ConnectionAwareAdapter):
except (redis.ConnectionError, redis.TimeoutError):
return False
def set(self, key: str, data: dict) -> None:
def set(self, key: str, data: dict[str, Any]) -> None:
"""Set a JSON object in Redis."""
# Check input
if not isinstance(key, str) or len(key) == 0:
@@ -109,7 +103,7 @@ class RedisAdapter(JsonRepositoryInterface, ConnectionAwareAdapter):
self._client.json().set(key, self.path, data)
self.logger.debug(f"Set {key} to {data}")
def get(self, key: str) -> dict | None:
def get(self, key: str) -> dict[str, Any] | None:
"""Get a JSON object from Redis."""
# Check input
if not isinstance(key, str) or len(key) == 0:
@@ -119,7 +113,7 @@ class RedisAdapter(JsonRepositoryInterface, ConnectionAwareAdapter):
assert self._client is not None
# Get data
data = cast(
dict | None,
dict[str, Any] | None,
self._client.json().get(key),
)
self.logger.debug(f"Got {data} from {key}")
+21
View File
@@ -0,0 +1,21 @@
"""Parse boolean values from environment variables."""
from __future__ import annotations
import os
_TRUTHY = frozenset({"1", "true", "yes", "on"})
_FALSY = frozenset({"0", "false", "no", "off"})
def env_bool(name: str, default: bool) -> bool:
"""Parse an environment variable as a boolean value."""
raw = os.getenv(name)
if raw is None:
return default
normalized = raw.strip().lower()
if normalized in _TRUTHY:
return True
if normalized in _FALSY:
return False
raise ValueError(f"Invalid boolean value for {name}: {raw!r}")
+8 -16
View File
@@ -8,21 +8,7 @@ from dataclasses import dataclass
from python_utils import check_env
from python_repositories.config.dotenv_loader import load_dotenv
_TRUTHY = frozenset({"1", "true", "yes", "on"})
_FALSY = frozenset({"0", "false", "no", "off"})
def _env_bool(name: str, default: bool) -> bool:
raw = os.getenv(name)
if raw is None:
return default
normalized = raw.strip().lower()
if normalized in _TRUTHY:
return True
if normalized in _FALSY:
return False
raise ValueError(f"Invalid boolean value for {name}: {raw!r}")
from python_repositories.config.env_bool import env_bool
@dataclass(frozen=True)
@@ -34,6 +20,7 @@ class MinioConfig:
secret_key: str
bucket: str
secure: bool = True
create_bucket_if_missing: bool = False
@classmethod
def from_env(
@@ -43,6 +30,7 @@ class MinioConfig:
secret_key_env_var_name: str = "MINIO_SECRET_KEY",
bucket_env_var_name: str = "MINIO_BUCKET",
secure_env_var_name: str = "MINIO_SECURE",
create_bucket_if_missing_env_var_name: str = ("MINIO_CREATE_BUCKET_IF_MISSING"),
*,
use_dotenv: bool = True,
) -> MinioConfig:
@@ -61,5 +49,9 @@ class MinioConfig:
access_key=str(os.getenv(access_key_env_var_name)),
secret_key=str(os.getenv(secret_key_env_var_name)),
bucket=str(os.getenv(bucket_env_var_name)),
secure=_env_bool(secure_env_var_name, default=True),
secure=env_bool(secure_env_var_name, default=True),
create_bucket_if_missing=env_bool(
create_bucket_if_missing_env_var_name,
default=False,
),
)
@@ -1,5 +1,7 @@
"""Example domain repository backed by Redis JSON."""
from typing import Any
from python_repositories.adapters.redis_adapter import RedisAdapter
@@ -9,10 +11,10 @@ class UserJsonRepository(RedisAdapter):
def _key(self, user_id: str) -> str:
return f"user:{user_id}"
def get_user(self, user_id: str) -> dict | None:
def get_user(self, user_id: str) -> dict[str, Any] | None:
return self.get(self._key(user_id))
def save_user(self, user_id: str, user: dict) -> None:
def save_user(self, user_id: str, user: dict[str, Any]) -> None:
self.set(self._key(user_id), user)
def delete_user(self, user_id: str) -> None:
@@ -8,7 +8,11 @@ class ConnectionAwareInterface(ABC):
@abstractmethod
def connect(self) -> None:
"""Connect to resource."""
"""Connect to resource.
Implementations should be idempotent: calling connect while already
connected and healthy is a no-op.
"""
...
@abstractmethod
@@ -2,18 +2,19 @@
from collections.abc import Iterator
from abc import ABC, abstractmethod
from typing import Any
class JsonRepositoryInterface(ABC):
"""Interface that defines JSON document CRUD methods."""
@abstractmethod
def get(self, key: str) -> dict | None:
def get(self, key: str) -> dict[str, Any] | None:
"""Get a JSON object by key."""
...
@abstractmethod
def set(self, key: str, data: dict) -> None:
def set(self, key: str, data: dict[str, Any]) -> None:
"""Set a JSON object by key."""
...
+78
View File
@@ -0,0 +1,78 @@
#!/usr/bin/env bash
# Build and optionally push the Gitea Actions CI base image locally.
#
# Use when ci-image.yml is not yet on main (e.g. bootstrapping a PR branch) or
# when you need to rebuild without waiting for the nightly workflow.
#
# Build only (smoke test):
# export CI_RUNNER_TOKEN='ci-bot-personal-access-token'
# bash scripts/ci/build-ci-image.sh --local
#
# Build and push to Gitea (unblocks python-repositories-ci jobs):
# export CI_RUNNER_TOKEN='ci-bot-personal-access-token'
# bash scripts/ci/build-ci-image.sh --push
#
# Optional overrides:
# REGISTRY=gitea.lille-vemmelund.dk
# REGISTRY_USER=ci-bot
# IMAGE_OWNER=lillevemmelund
# IMAGE_NAME=python-repositories-ci
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
REGISTRY="${REGISTRY:-gitea.lille-vemmelund.dk}"
REGISTRY_USER="${REGISTRY_USER:-ci-bot}"
IMAGE_OWNER="${IMAGE_OWNER:-lillevemmelund}"
IMAGE_NAME="${IMAGE_NAME:-python-repositories-ci}"
IMAGE="${IMAGE:-${REGISTRY}/${IMAGE_OWNER}/${IMAGE_NAME}}"
MODE="push"
if [[ "${1:-}" == "--local" ]]; then
MODE="local"
elif [[ "${1:-}" == "--push" || -z "${1:-}" ]]; then
MODE="push"
elif [[ -n "${1:-}" ]]; then
echo "Usage: build-ci-image.sh [--local | --push]" >&2
exit 1
fi
: "${CI_RUNNER_TOKEN:?CI_RUNNER_TOKEN is required (ci-bot token with read/write package access)}"
TOKEN_FILE="$(mktemp)"
cleanup() {
rm -f "$TOKEN_FILE"
}
trap cleanup EXIT
printf '%s' "$CI_RUNNER_TOKEN" >"$TOKEN_FILE"
cd "$REPO_ROOT"
if [[ "$MODE" == "local" ]]; then
echo "=== Building local CI image (no push): python-repositories-ci:local ==="
docker build --network=host -f docker/ci/Dockerfile \
--secret "id=uv_token,src=${TOKEN_FILE}" \
-t python-repositories-ci:local \
.
echo "Built python-repositories-ci:local"
exit 0
fi
echo "=== Logging in to ${REGISTRY} as ${REGISTRY_USER} ==="
echo "$CI_RUNNER_TOKEN" | docker login "$REGISTRY" -u "$REGISTRY_USER" --password-stdin
echo "=== Building ${IMAGE}:latest ==="
docker build --network=host -f docker/ci/Dockerfile \
--secret "id=uv_token,src=${TOKEN_FILE}" \
-t "${IMAGE}:latest" \
.
STAMP="$(date -u +%Y%m%d%H%M)"
echo "=== Pushing ${IMAGE}:latest and ${IMAGE}:${STAMP} ==="
docker tag "${IMAGE}:latest" "${IMAGE}:${STAMP}"
docker push "${IMAGE}:latest"
docker push "${IMAGE}:${STAMP}"
echo "Done. Re-run failing CI jobs — runners pull ${IMAGE}:latest"
+31 -2
View File
@@ -2,23 +2,52 @@
import logging
from collections.abc import Generator
from typing import Any, cast
import pytest
import redis
import structlog
from minio import Minio
from testcontainers.core.container import DockerContainer
from testcontainers.core.waiting_utils import WaitStrategy, WaitStrategyTarget
from testcontainers.minio import MinioContainer
from python_repositories.config import MinioConfig, RedisConfig
from tests.integration.redis_container_test import REDIS_PORT, RedisTestContainer
collect_ignore = ["redis_container_test.py"]
REDIS_PORT = 6379
MINIO_ACCESS_KEY = "minioadmin"
MINIO_SECRET_KEY = "minioadmin"
MINIO_BUCKET = "test-bucket"
class _RedisPingWaitStrategy(WaitStrategy):
def __init__(self) -> None:
super().__init__()
self.with_transient_exceptions(redis.exceptions.ConnectionError)
def wait_until_ready(self, container: WaitStrategyTarget) -> None:
redis_container = cast("RedisTestContainer", container)
if not self._poll(lambda: redis_container.get_client().ping()):
raise redis.exceptions.ConnectionError("Could not connect to Redis")
class RedisTestContainer(DockerContainer):
"""Redis container using wait strategies instead of the deprecated decorator."""
def __init__(self, image: str, port: int = REDIS_PORT) -> None:
super().__init__(image, _wait_strategy=_RedisPingWaitStrategy())
self.port = port
self.with_exposed_ports(self.port)
def get_client(self, **kwargs: Any) -> redis.Redis:
return redis.Redis(
host=self.get_container_host_ip(),
port=self.get_exposed_port(self.port),
**kwargs,
)
@pytest.fixture(scope="session", autouse=True)
def configure_logging() -> None:
"""Configure logging for the test session."""
+3
View File
@@ -6,6 +6,7 @@ import os
import random
import pytest
from minio import Minio
from python_repositories.examples.artifact_object_repository import (
ArtifactObjectRepository,
@@ -19,8 +20,10 @@ pytestmark = pytest.mark.integration
def set_example_env(
redis_container: str,
minio_container: dict[str, str],
raw_minio_client: Minio,
) -> Generator[None, None, None]:
"""Set env vars so example repositories can use from_env() defaults."""
_ = raw_minio_client
env_vars = {"REDIS_URI": redis_container, **minio_container}
for key, value in env_vars.items():
os.environ[key] = value
+24 -3
View File
@@ -1,6 +1,7 @@
"""Integration tests for the MinioAdapter."""
from collections.abc import Generator
from dataclasses import replace
import logging
import random
from io import BytesIO
@@ -118,17 +119,37 @@ def test_should_raise_connection_error_when_unable_to_connect() -> None:
assert not adapter.is_connected()
def test_should_log_info_when_creating_expected_bucket(
def test_connect_raises_when_bucket_missing(
raw_minio_client: Minio,
minio_config: MinioConfig,
) -> None:
"""Test that connect fails when the configured bucket is missing."""
bucket_name = minio_config.bucket
raw_minio_client.remove_bucket(bucket_name)
adapter = MinioAdapter(config=minio_config)
try:
with pytest.raises(ConnectionError, match="does not exist"):
adapter.connect()
assert not adapter.is_connected()
finally:
raw_minio_client.make_bucket(bucket_name)
def test_connect_creates_bucket_when_create_bucket_if_missing_enabled(
raw_minio_client: Minio,
minio_config: MinioConfig,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test that the MinioAdapter logs info when creating the expected bucket."""
"""Test that connect can create the configured bucket when enabled."""
bucket_name = minio_config.bucket
raw_minio_client.remove_bucket(bucket_name)
adapter = MinioAdapter(config=minio_config)
config = replace(minio_config, create_bucket_if_missing=True)
adapter = MinioAdapter(config=config)
with caplog.at_level(logging.INFO):
adapter.connect()
assert f"Creating bucket '{bucket_name}'" in caplog.text
+11
View File
@@ -1,6 +1,7 @@
"""Integration tests for the RedisAdapter."""
from collections.abc import Generator
import logging
import pytest
import redis
@@ -48,6 +49,16 @@ def clear_redis(raw_redis_client: redis.Redis) -> None:
raw_redis_client.flushall()
def test_should_log_info_when_already_connected(
redis_adapter: RedisAdapter,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test that the RedisAdapter logs info when connect is called while already connected."""
with caplog.at_level(logging.INFO):
redis_adapter.connect()
assert "Already connected to Redis" in caplog.text
def test_should_raise_connection_error_when_unable_to_connect() -> None:
"""Test that the RedisAdapter raises ConnectionError when unable to connect."""
adapter = RedisAdapter(config=RedisConfig(uri="redis://invalid:6379"))
-36
View File
@@ -1,36 +0,0 @@
"""Redis test container without testcontainers' deprecated wait decorator."""
from typing import Any, cast
import redis
from testcontainers.core.container import DockerContainer
from testcontainers.core.waiting_utils import WaitStrategy, WaitStrategyTarget
REDIS_PORT = 6379
class _RedisPingWaitStrategy(WaitStrategy):
def __init__(self) -> None:
super().__init__()
self.with_transient_exceptions(redis.exceptions.ConnectionError)
def wait_until_ready(self, container: WaitStrategyTarget) -> None:
redis_container = cast("RedisTestContainer", container)
if not self._poll(lambda: redis_container.get_client().ping()):
raise redis.exceptions.ConnectionError("Could not connect to Redis")
class RedisTestContainer(DockerContainer):
"""Redis container using wait strategies instead of the deprecated decorator."""
def __init__(self, image: str, port: int = REDIS_PORT) -> None:
super().__init__(image, _wait_strategy=_RedisPingWaitStrategy())
self.port = port
self.with_exposed_ports(self.port)
def get_client(self, **kwargs: Any) -> redis.Redis:
return redis.Redis(
host=self.get_container_host_ip(),
port=self.get_exposed_port(self.port),
**kwargs,
)
+43
View File
@@ -0,0 +1,43 @@
"""Unit tests for env_bool."""
from __future__ import annotations
import pytest
from python_repositories.config.env_bool import env_bool
@pytest.mark.parametrize(
("value", "expected"),
[
("true", True),
("1", True),
("yes", True),
("on", True),
("false", False),
("0", False),
("no", False),
("off", False),
],
)
def test_env_bool_parses_truthy_and_falsy(
monkeypatch: pytest.MonkeyPatch,
value: str,
expected: bool,
) -> None:
monkeypatch.setenv("TEST_BOOL", value)
assert env_bool("TEST_BOOL", default=not expected) is expected
def test_env_bool_returns_default_when_unset(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("TEST_BOOL", raising=False)
assert env_bool("TEST_BOOL", default=True) is True
assert env_bool("TEST_BOOL", default=False) is False
def test_env_bool_raises_for_invalid_value(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv("TEST_BOOL", "not-a-bool")
with pytest.raises(ValueError, match="TEST_BOOL"):
env_bool("TEST_BOOL", default=False)
+10 -8
View File
@@ -1,5 +1,7 @@
"""Unit tests for JsonRepositoryInterface."""
from typing import Any
import pytest
from python_repositories.interfaces.json_repository_interface import (
JsonRepositoryInterface,
@@ -12,7 +14,7 @@ def test_instantiation_fails_when_get_not_implemented() -> None:
class Incomplete(JsonRepositoryInterface):
"""A class that does not implement get."""
def set(self, key: str, data: dict) -> None:
def set(self, key: str, data: dict[str, Any]) -> None:
pass
def delete(self, key: str) -> None:
@@ -31,7 +33,7 @@ def test_instantiation_fails_when_set_not_implemented() -> None:
class Incomplete(JsonRepositoryInterface):
"""A class that does not implement set."""
def get(self, key: str) -> dict | None:
def get(self, key: str) -> dict[str, Any] | None:
return None
def delete(self, key: str) -> None:
@@ -50,10 +52,10 @@ def test_instantiation_fails_when_delete_not_implemented() -> None:
class Incomplete(JsonRepositoryInterface):
"""A class that does not implement delete."""
def get(self, key: str) -> dict | None:
def get(self, key: str) -> dict[str, Any] | None:
return None
def set(self, key: str, data: dict) -> None:
def set(self, key: str, data: dict[str, Any]) -> None:
pass
def list_keys(self, pattern: str) -> list[str]:
@@ -69,10 +71,10 @@ def test_instantiation_fails_when_list_keys_not_implemented() -> None:
class Incomplete(JsonRepositoryInterface):
"""A class that does not implement list_keys."""
def get(self, key: str) -> dict | None:
def get(self, key: str) -> dict[str, Any] | None:
return None
def set(self, key: str, data: dict) -> None:
def set(self, key: str, data: dict[str, Any]) -> None:
pass
def delete(self, key: str) -> None:
@@ -86,10 +88,10 @@ def test_scan_keys_defaults_to_list_keys() -> None:
"""Test that the default scan_keys implementation delegates to list_keys."""
class Complete(JsonRepositoryInterface):
def get(self, key: str) -> dict | None:
def get(self, key: str) -> dict[str, Any] | None:
return None
def set(self, key: str, data: dict) -> None:
def set(self, key: str, data: dict[str, Any]) -> None:
pass
def delete(self, key: str) -> None:
+70
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
from dataclasses import replace
from unittest.mock import MagicMock
import pytest
@@ -74,6 +75,16 @@ def test_connect_with_injected_client_raises_on_failure() -> None:
adapter.connect()
def test_connect_with_injected_client_skips_validation_when_client_cleared() -> None:
mock_client = MagicMock(spec=Minio)
adapter = MinioAdapter(config=TEST_MINIO_CONFIG, client=mock_client)
adapter.disconnect()
adapter.connect()
mock_client.list_buckets.assert_not_called()
def test_connect_disconnects_before_reconnect(
monkeypatch: pytest.MonkeyPatch,
) -> None:
@@ -94,6 +105,65 @@ def test_connect_disconnects_before_reconnect(
new_client.list_buckets.assert_called_once()
def test_connect_skips_reconnect_when_already_connected(
monkeypatch: pytest.MonkeyPatch,
) -> None:
stale_client = MagicMock(spec=Minio)
stale_client.bucket_exists.return_value = True
adapter = MinioAdapter(config=TEST_MINIO_CONFIG)
adapter._client = stale_client
adapter._bucket_name = TEST_MINIO_CONFIG.bucket
minio_ctor = MagicMock()
monkeypatch.setattr(
"python_repositories.adapters.minio_adapter.minio.Minio",
minio_ctor,
)
adapter.connect()
minio_ctor.assert_not_called()
assert adapter._client is stale_client
def test_connect_raises_when_bucket_missing(
monkeypatch: pytest.MonkeyPatch,
) -> None:
mock_client = MagicMock(spec=Minio)
mock_client.bucket_exists.return_value = False
monkeypatch.setattr(
"python_repositories.adapters.minio_adapter.minio.Minio",
lambda *args, **kwargs: mock_client,
)
adapter = MinioAdapter(config=TEST_MINIO_CONFIG)
with pytest.raises(ConnectionError, match="does not exist"):
adapter.connect()
mock_client.make_bucket.assert_not_called()
def test_connect_creates_bucket_when_create_bucket_if_missing_enabled(
monkeypatch: pytest.MonkeyPatch,
) -> None:
mock_client = MagicMock(spec=Minio)
mock_client.bucket_exists.return_value = False
config = replace(TEST_MINIO_CONFIG, create_bucket_if_missing=True)
monkeypatch.setattr(
"python_repositories.adapters.minio_adapter.minio.Minio",
lambda *args, **kwargs: mock_client,
)
adapter = MinioAdapter(config=config)
adapter.connect()
mock_client.bucket_exists.assert_called_once_with(config.bucket)
mock_client.make_bucket.assert_called_once_with(config.bucket)
def test_get_closes_response_on_success() -> None:
"""get() must close and release the get_object HTTP response."""
mock_client = MagicMock(spec=Minio)
+16 -21
View File
@@ -22,6 +22,7 @@ def test_from_env_loads_all_fields(monkeypatch: pytest.MonkeyPatch) -> None:
assert config.secret_key == "secret"
assert config.bucket == "my-bucket"
assert config.secure is True
assert config.create_bucket_if_missing is False
def test_from_env_defaults_secure_to_true_when_unset(
@@ -33,33 +34,27 @@ def test_from_env_defaults_secure_to_true_when_unset(
assert config.secure is True
@pytest.mark.parametrize(
("value", "expected"),
[
("true", True),
("1", True),
("yes", True),
("false", False),
("0", False),
("no", False),
],
)
def test_from_env_parses_secure(
def test_from_env_reads_secure_from_env(monkeypatch: pytest.MonkeyPatch) -> None:
_set_required_minio_env(monkeypatch)
monkeypatch.setenv("MINIO_SECURE", "false")
assert MinioConfig.from_env(use_dotenv=False).secure is False
def test_from_env_reads_create_bucket_if_missing_from_env(
monkeypatch: pytest.MonkeyPatch,
value: str,
expected: bool,
) -> None:
_set_required_minio_env(monkeypatch)
monkeypatch.setenv("MINIO_SECURE", value)
config = MinioConfig.from_env(use_dotenv=False)
assert config.secure is expected
monkeypatch.setenv("MINIO_CREATE_BUCKET_IF_MISSING", "true")
assert MinioConfig.from_env(use_dotenv=False).create_bucket_if_missing is True
def test_from_env_raises_for_invalid_secure(monkeypatch: pytest.MonkeyPatch) -> None:
def test_from_env_defaults_create_bucket_if_missing_to_false_when_unset(
monkeypatch: pytest.MonkeyPatch,
) -> None:
_set_required_minio_env(monkeypatch)
monkeypatch.setenv("MINIO_SECURE", "not-a-bool")
with pytest.raises(ValueError, match="MINIO_SECURE"):
MinioConfig.from_env(use_dotenv=False)
monkeypatch.delenv("MINIO_CREATE_BUCKET_IF_MISSING", raising=False)
config = MinioConfig.from_env(use_dotenv=False)
assert config.create_bucket_if_missing is False
def test_from_env_raises_when_endpoint_missing(
+30 -1
View File
@@ -99,10 +99,21 @@ def test_connect_with_injected_client_raises_on_redis_error() -> None:
adapter.connect()
def test_connect_closes_existing_non_injected_client(
def test_connect_with_injected_client_skips_validation_when_client_cleared() -> None:
mock_client = MagicMock(spec=redis.Redis)
adapter = RedisAdapter(config=TEST_REDIS_CONFIG, client=mock_client)
adapter.disconnect()
adapter.connect()
mock_client.ping.assert_not_called()
def test_connect_reconnects_when_existing_client_unhealthy(
monkeypatch: pytest.MonkeyPatch,
) -> None:
stale_client = MagicMock(spec=redis.Redis)
stale_client.ping.side_effect = redis.ConnectionError("connection lost")
new_client = MagicMock(spec=redis.Redis)
new_client.ping.return_value = True
adapter = RedisAdapter(config=TEST_REDIS_CONFIG)
@@ -116,6 +127,24 @@ def test_connect_closes_existing_non_injected_client(
assert adapter._client is new_client
def test_connect_skips_reconnect_when_already_connected(
monkeypatch: pytest.MonkeyPatch,
) -> None:
stale_client = MagicMock(spec=redis.Redis)
stale_client.ping.return_value = True
adapter = RedisAdapter(config=TEST_REDIS_CONFIG)
adapter._client = stale_client
from_url = MagicMock()
monkeypatch.setattr("redis.Redis.from_url", from_url)
adapter.connect()
stale_client.close.assert_not_called()
from_url.assert_not_called()
assert adapter._client is stale_client
def test_scan_keys_yields_decoded_keys() -> None:
mock_client = MagicMock(spec=redis.Redis)
mock_client.scan_iter.return_value = iter([b"key1", b"key2"])
Generated
+1 -1
View File
@@ -1056,7 +1056,7 @@ wheels = [
[[package]]
name = "python-repositories"
version = "1.1.0"
version = "2.0.2"
source = { editable = "." }
dependencies = [
{ name = "python-dotenv" },