Compare commits

..
Author SHA1 Message Date
CI Bot 43c5e53c7f chore(deps): update dependencies [automated]
PR Title Check / check-title (pull_request) Successful in 5s
Test Python Package / unit-tests (pull_request) Successful in 23s
Code Quality Pipeline / code-quality (pull_request) Successful in 57s
Test Python Package / integration-tests (pull_request) Successful in 1m48s
Test Python Package / coverage-report (pull_request) Successful in 18s
2026-07-20 03:01:36 +00:00
brian ac82f71347 Merge pull request '[minor] Add QueueRepositoryInterface with memory and file-backed adapters' (#62) from cursor/queue-adapters into main
Release on merge to main / release (push) Failing after 15s
Test Python Package / unit-tests (push) Successful in 14s
Code Quality Pipeline / code-quality (push) Successful in 28s
Test Python Package / integration-tests (push) Successful in 23s
Test Python Package / coverage-report (push) Successful in 8s
Reviewed-on: #62
2026-07-16 21:27:49 +02:00
Brian Bjarke JensenandCursor 24d5ea14d6 Apply Prettier formatting to README and CHANGELOG.
PR Title Check / check-title (pull_request) Successful in 8s
Test Python Package / unit-tests (pull_request) Successful in 15s
Test Python Package / integration-tests (pull_request) Successful in 24s
Code Quality Pipeline / code-quality (pull_request) Successful in 31s
Test Python Package / coverage-report (pull_request) Successful in 12s
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-16 21:25:49 +02:00
Brian Bjarke JensenandCursor b8d8176877 Align changelog section headings with Prettier markdown spacing.
Test Python Package / unit-tests (pull_request) Successful in 13s
Test Python Package / coverage-report (pull_request) Successful in 13s
PR Title Check / check-title (pull_request) Successful in 7s
Code Quality Pipeline / code-quality (pull_request) Failing after 29s
Test Python Package / integration-tests (pull_request) Successful in 1m15s
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-16 21:24:06 +02:00
Brian Bjarke JensenandCursor a78320b434 Move queue helper functions to staticmethods on their adapters.
PR Title Check / check-title (pull_request) Successful in 8s
Test Python Package / unit-tests (pull_request) Successful in 20s
Code Quality Pipeline / code-quality (pull_request) Failing after 32s
Test Python Package / integration-tests (pull_request) Successful in 1m33s
Test Python Package / coverage-report (pull_request) Successful in 16s
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-16 21:07:35 +02:00
Brian Bjarke JensenandCursor 1822d886b1 Apply ruff formatting to queue adapter and tests.
PR Title Check / check-title (pull_request) Successful in 11s
Test Python Package / unit-tests (pull_request) Successful in 16s
Code Quality Pipeline / code-quality (pull_request) Failing after 30s
Test Python Package / integration-tests (pull_request) Successful in 34s
Test Python Package / coverage-report (pull_request) Successful in 16s
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-16 20:24:58 +02:00
Brian Bjarke JensenandCursor 4f58c32dd6 Add QueueRepositoryInterface with memory and file-backed adapters.
PR Title Check / check-title (pull_request) Successful in 9s
Code Quality Pipeline / code-quality (pull_request) Failing after 53s
Test Python Package / unit-tests (pull_request) Successful in 1m1s
Test Python Package / integration-tests (pull_request) Successful in 1m44s
Test Python Package / coverage-report (pull_request) Successful in 13s
Provide a generic disk-backed FIFO queue with configurable path, retention, and dedup keys so consumers can buffer items across restarts without optional extras.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-16 20:20:52 +02:00
brian 88ea7f06ab Merge pull request 'chore(deps): update dependencies' (#61) from renovate/auto-deps-update into main
Release on merge to main / release (push) Failing after 5s
Test Python Package / integration-tests (push) Failing after 1s
Test Python Package / unit-tests (push) Failing after 2s
Test Python Package / coverage-report (push) Has been skipped
Code Quality Pipeline / code-quality (push) Failing after 1m10s
Build CI Image / build-and-push (push) Successful in 1m40s
Reviewed-on: #61
2026-07-16 20:19:13 +02:00
CI Bot 2b5718ba41 chore(deps): update dependencies [automated]
PR Title Check / check-title (pull_request) Successful in 7s
Code Quality Pipeline / code-quality (pull_request) Successful in 31s
Test Python Package / unit-tests (pull_request) Successful in 15s
Test Python Package / integration-tests (pull_request) Successful in 41s
Test Python Package / coverage-report (pull_request) Successful in 13s
2026-07-13 03:00:38 +00:00
CI Bot b6538f9e91 chore: release v2.2.0 [skip ci] 2026-07-12 16:52:22 +00:00
brian b37b2e6e50 Merge pull request '[minor] Add Postgres table adapter with TableRepositoryInterface' (#58) from cursor/postgres-table-adapter into main
Release on merge to main / release (push) Failing after 9s
Code Quality Pipeline / code-quality (push) Successful in 20s
Test Python Package / unit-tests (push) Successful in 13s
Build CI Image / build-and-push (push) Successful in 38s
Test Python Package / integration-tests (push) Successful in 1m9s
Test Python Package / coverage-report (push) Successful in 13s
Reviewed-on: LilleVemmelund/python-repositories#58
2026-07-12 18:51:59 +02:00
Brian Bjarke JensenandCursor 1babb52d09 Apply ruff formatting to postgres adapter and tests.
PR Title Check / check-title (pull_request) Successful in 8s
Test Python Package / unit-tests (pull_request) Successful in 13s
Code Quality Pipeline / code-quality (pull_request) Successful in 24s
Test Python Package / integration-tests (pull_request) Successful in 29s
Test Python Package / coverage-report (pull_request) Successful in 10s
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-11 15:01:58 +02:00
Brian Bjarke JensenandCursor 63ee17d544 Apply prettier formatting to README and CHANGELOG.
PR Title Check / check-title (pull_request) Successful in 8s
Test Python Package / unit-tests (pull_request) Successful in 10s
Code Quality Pipeline / code-quality (pull_request) Failing after 18s
Test Python Package / integration-tests (pull_request) Successful in 56s
Test Python Package / coverage-report (pull_request) Successful in 13s
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-11 15:01:08 +02:00
Brian Bjarke JensenandCursor 2e38bd2406 Fix ruff import ordering in postgres test files.
PR Title Check / check-title (pull_request) Successful in 8s
Test Python Package / unit-tests (pull_request) Successful in 10s
Code Quality Pipeline / code-quality (pull_request) Failing after 17s
Test Python Package / integration-tests (pull_request) Successful in 47s
Test Python Package / coverage-report (pull_request) Successful in 12s
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-11 15:00:01 +02:00
Brian Bjarke JensenandCursor dc6f8a3e89 Add Postgres table adapter with TableRepositoryInterface.
PR Title Check / check-title (pull_request) Successful in 7s
Code Quality Pipeline / code-quality (pull_request) Failing after 19s
Test Python Package / unit-tests (pull_request) Successful in 47s
Test Python Package / integration-tests (pull_request) Successful in 44s
Test Python Package / coverage-report (pull_request) Successful in 11s
Introduce PostgresAdapter for dict-based row CRUD via psycopg3, including config, lazy exports, unit/integration tests with testcontainers, and an example UserTableRepository.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-11 14:57:34 +02:00
CI Bot f28d3c4844 chore: release v2.1.0 [skip ci] 2026-07-11 08:56:19 +00:00
brian 1cbcc1c19b Merge pull request '[minor] Add runtime-checkable Protocol typing to public interfaces' (#57) from cursor/protocol-abc-interfaces into main
Test Python Package / unit-tests (push) Successful in 10s
Release on merge to main / release (push) Successful in 15s
Code Quality Pipeline / code-quality (push) Successful in 23s
Test Python Package / integration-tests (push) Successful in 20s
Test Python Package / coverage-report (push) Successful in 7s
Reviewed-on: LilleVemmelund/python-repositories#57
2026-07-11 10:56:01 +02:00
Brian Bjarke JensenandCursor a0e5c9dcca Document test organization and add Cursor workflow rules.
PR Title Check / check-title (pull_request) Successful in 8s
Test Python Package / unit-tests (pull_request) Successful in 13s
Code Quality Pipeline / code-quality (pull_request) Successful in 23s
Test Python Package / integration-tests (pull_request) Successful in 1m14s
Test Python Package / coverage-report (pull_request) Successful in 12s
Record the one-to-one unit test file convention in README and add rules for test placement and copy-ready PR text after first push.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-11 10:50:51 +02:00
Brian Bjarke JensenandCursor 44b15cb21a Move structural typing tests into interface unit test files.
Code Quality Pipeline / code-quality (pull_request) Successful in 24s
PR Title Check / check-title (pull_request) Successful in 35s
Test Python Package / coverage-report (pull_request) Successful in 11s
Test Python Package / unit-tests (pull_request) Successful in 12s
Test Python Package / integration-tests (pull_request) Successful in 26s
Align test layout with the one-test-file-per-interface convention by removing structural_typing_test.py.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-11 10:32:17 +02:00
Brian Bjarke JensenandCursor f6ee9a3724 Add runtime-checkable Protocol typing to all public interfaces.
PR Title Check / check-title (pull_request) Successful in 7s
Test Python Package / unit-tests (pull_request) Successful in 9s
Code Quality Pipeline / code-quality (pull_request) Successful in 22s
Test Python Package / integration-tests (pull_request) Successful in 24s
Test Python Package / coverage-report (pull_request) Successful in 8s
Enables structural subtyping for consumers while preserving nominal adapter inheritance, instantiation guards, and scan_keys defaults.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-11 09:45:42 +02:00
brian 0829354cc4 Merge pull request 'Add async future note and remove redundant pytest pythonpath' (#56) from cursor/readme-async-and-test-path-cleanup into main
Release on merge to main / release (push) Successful in 7s
Test Python Package / unit-tests (push) Successful in 12s
Code Quality Pipeline / code-quality (push) Successful in 23s
Test Python Package / integration-tests (push) Successful in 24s
Build CI Image / build-and-push (push) Successful in 49s
Test Python Package / coverage-report (push) Successful in 9s
Reviewed-on: LilleVemmelund/python-repositories#56
2026-07-11 09:28:01 +02:00
Brian Bjarke JensenandCursor a0d509a2a3 Format CHANGELOG.md with Prettier.
PR Title Check / check-title (pull_request) Successful in 7s
Test Python Package / unit-tests (pull_request) Successful in 12s
Code Quality Pipeline / code-quality (pull_request) Successful in 23s
Test Python Package / integration-tests (pull_request) Successful in 1m14s
Test Python Package / coverage-report (pull_request) Successful in 13s
Fixes pre-commit prettier check in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-11 09:25:16 +02:00
Brian Bjarke JensenandCursor 9b7b15a67e Document possible async support and drop redundant pytest pythonpath.
Test Python Package / unit-tests (pull_request) Successful in 12s
Code Quality Pipeline / code-quality (pull_request) Failing after 23s
PR Title Check / check-title (pull_request) Successful in 32s
Test Python Package / integration-tests (pull_request) Successful in 25s
Test Python Package / coverage-report (pull_request) Successful in 11s
The package is already installed via uv sync, so pytest no longer needs a manual repo-root path.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-11 09:22:37 +02:00
47 changed files with 3374 additions and 438 deletions
+58
View File
@@ -0,0 +1,58 @@
---
description: After first branch push, provide copy-ready PR title and body
alwaysApply: true
---
# PR Title and Comment After First Push
The user creates pull requests manually. After the **first push** of a feature branch to remote in a session, always end your response with a copy-ready PR title and PR comment.
## When to provide it
- After the first successful `git push` (or `git push -u origin <branch>`) for a branch in the current task
- Do **not** create the PR with `gh pr create` unless explicitly asked
- On later pushes to the same branch, only repeat if the user asks or the change set meaningfully updates what the PR should say
## PR title
- Use a fitting version-bump prefix when changes touch `python_repositories/`:
- `[patch]` or `[fix]` — bug fix
- `[minor]` or `[feat]` — new feature
- `[major]` or `[breaking]` — breaking change
- Docs-, CI-, or test-only changes: no prefix required
- Keep the title concise and descriptive
## PR comment format
Provide two separate fenced code blocks the user can copy directly:
1. **PR title** — single line in a code block
2. **PR comment** — body with `## Summary` and `## Test plan` sections, checklist items reflecting what was actually run
Example structure:
````
PR title:
```
[minor] Short description of the change
```
PR comment:
```
## Summary
- Bullet points of what changed and why
## Test plan
- [x] Commands or checks that were run
```
````
## Notes
- Match the repo PR template ([`.gitea/PULL_REQUEST_TEMPLATE.md`](.gitea/PULL_REQUEST_TEMPLATE.md))
- Be accurate: only mark test-plan items done if they were run
- Keep prose clear; the user copies this verbatim into Gitea
+31
View File
@@ -0,0 +1,31 @@
---
description: Unit test file naming and placement conventions
globs: tests/**/*.py
alwaysApply: false
---
# Test Organization
Unit tests use a one-to-one mapping between source modules and test files.
## Naming
- `python_repositories/<path>/<module>.py` → `tests/unit/<module>_test.py`
- Test files must end with `_test.py` (enforced by pre-commit `name-tests-test`)
## Placement
- Add tests to the existing `*_test.py` for the module under test
- Do not create cross-cutting test files (e.g. `structural_typing_test.py`); colocate with the relevant interface/adapter/config test file
- Shared fixtures → `tests/conftest.py`
- Module-specific helper classes → the matching test file
## Examples
```
json_repository_interface.py → tests/unit/json_repository_interface_test.py
redis_adapter.py → tests/unit/redis_adapter_test.py
redis_config.py → tests/unit/redis_config_test.py
```
Integration tests live under `tests/integration/<backend>/` with the same `_test.py` suffix.
+9
View File
@@ -8,3 +8,12 @@ MINIO_SECRET_KEY=minioadmin
MINIO_BUCKET=my-bucket
MINIO_SECURE=false
MINIO_CREATE_BUCKET_IF_MISSING=true
# Postgres (requires postgres extra)
POSTGRES_URI=postgresql://localhost/mydb
POSTGRES_TABLE=users
POSTGRES_PRIMARY_KEY=id
# File-backed queue (no optional extra)
FILE_QUEUE_PATH=/tmp/python-repositories-queue.jsonl
# FILE_QUEUE_MAX_AGE_HOURS=24
+2 -2
View File
@@ -1,7 +1,7 @@
repos:
# General repository hygiene hooks
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
rev: v6.0.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
@@ -38,7 +38,7 @@ repos:
# Formatting for Markdown, JSON, and YAML with Prettier
- repo: https://github.com/pre-commit/mirrors-prettier
rev: v3.1.0
rev: v4.0.0-alpha.8
hooks:
- id: prettier
files: "\\.(md|json|yaml|yml)$"
+32
View File
@@ -7,12 +7,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.2.0] - 2026-07-12
### Summary
Add Postgres table adapter with TableRepositoryInterface
### Changed
- b37b2e6 Merge pull request '[minor] Add Postgres table adapter with TableRepositoryInterface' (#58) from cursor/postgres-table-adapter into main
- 1babb52 Apply ruff formatting to postgres adapter and tests.
- 63ee17d Apply prettier formatting to README and CHANGELOG.
- 2e38bd2 Fix ruff import ordering in postgres test files.
- dc6f8a3 Add Postgres table adapter with TableRepositoryInterface.
## [2.1.0] - 2026-07-11
### Summary
Add runtime-checkable Protocol typing to public interfaces
### Changed
- 1cbcc1c Merge pull request '[minor] Add runtime-checkable Protocol typing to public interfaces' (#57) from cursor/protocol-abc-interfaces into main
- a0e5c9d Document test organization and add Cursor workflow rules.
- 44b15cb Move structural typing tests into interface unit test files.
- f6ee9a3 Add runtime-checkable Protocol typing to all public interfaces.
- 0829354 Merge pull request 'Add async future note and remove redundant pytest pythonpath' (#56) from cursor/readme-async-and-test-path-cleanup into main
- a0d509a Format CHANGELOG.md with Prettier.
- 9b7b15a Document possible async support and drop redundant pytest pythonpath.
## [2.0.5] - 2026-07-10
### Summary
Allow empty payloads in Redis and MinIO adapters
### Changed
- 6d02f32 Merge pull request '[patch] Allow empty payloads in Redis and MinIO adapters' (#55) from cursor/allow-empty-payloads into main
- 57b396b Allow empty payloads in Redis and MinIO adapters.
- 093e538 Merge pull request '[chore] Scope integration fixtures per backend subdirectory' (#54) from cursor/scoped-integration-fixtures into main
+115 -13
View File
@@ -6,24 +6,31 @@ Subclass an adapter in your own repository to add domain-specific methods while
## Architecture
| Layer | Responsibility |
| ---------------- | ----------------------------------------------------------------- |
| **Interfaces** | Abstract contracts for connection, context, and CRUD |
| **Adapters** | Technology-specific base classes (`RedisAdapter`, `MinioAdapter`) |
| **Your project** | Subclass an adapter and add domain methods |
| Layer | Responsibility |
| ---------------- | ---------------------------------------------------------------------------------------------------- |
| **Interfaces** | Abstract contracts for connection, context, CRUD, and queues |
| **Adapters** | Technology-specific base classes (`RedisAdapter`, `MinioAdapter`, `PostgresAdapter`, queue adapters) |
| **Your project** | Subclass an adapter and add domain methods |
Each public interface is a `@runtime_checkable` `Protocol` with `@abstractmethod` members. **Subclass an adapter** when you need connection management and shared behavior — explicit subclasses get runtime instantiation guards and inherited default methods (e.g. `scan_keys`). **Type-annotate against an interface** when you want loose coupling — any object with the right methods satisfies the contract for mypy and `isinstance()` checks, without inheriting from this package.
Connection adapters expose `connect()`, `disconnect()`, and `is_connected()`. The latter verifies backend reachability with a cached health probe (default TTL: 1 second). Subclasses may override `health_check_ttl_seconds`. `connect()` is idempotent: calling it while already connected and healthy is a no-op.
## Future direction
The current API is synchronous. Async repository interfaces and adapters may be added in a future release; existing sync usage would remain supported.
## Optional dependencies
Repository **interfaces** import with the base package. **Adapters** require the matching extra; importing an adapter without its extra raises `ImportError` with install instructions.
Repository **interfaces** import with the base package. Networked **adapters** (`RedisAdapter`, `MinioAdapter`, `PostgresAdapter`) require the matching extra; importing one without its extra raises `ImportError` with install instructions. Queue adapters (`MemoryQueueAdapter`, `FileBackedQueueAdapter`) need no extra.
Install with the extras you need:
```bash
uv add python-repositories[redis]
uv add python-repositories[minio]
uv add python-repositories[redis,minio]
uv add python-repositories[postgres]
uv add python-repositories[redis,minio,postgres]
```
### Redis (`JsonRepositoryInterface`)
@@ -57,16 +64,69 @@ for key in repo.scan_keys("user:*"):
| `MINIO_SECURE` | Use HTTPS (`true`/`false`; default: `true`) |
| `MINIO_CREATE_BUCKET_IF_MISSING` | Auto-create `MINIO_BUCKET` on connect (`true`/`false`; default: `false`) |
Copy [`.env.example`](.env.example) to `.env` for local development. `RedisConfig.from_env()` and `MinioConfig.from_env()` load `.env` automatically when resolving configuration from the environment.
For production, it is recommended to leave `MINIO_CREATE_BUCKET_IF_MISSING` unset so that `connect()` fails fast if the expected bucket is missing. For local development, you will often want `MINIO_SECURE=false` and `MINIO_CREATE_BUCKET_IF_MISSING=true`.
### Postgres (`TableRepositoryInterface`)
Requires PostgreSQL 10 or later; tested against PostgreSQL 16 in CI. Tables are owned by your migrations — the adapter verifies the configured table exists on connect.
| Environment variable | Description |
| ---------------------- | --------------------------------------- |
| `POSTGRES_URI` | PostgreSQL connection URL |
| `POSTGRES_TABLE` | Table name for CRUD operations |
| `POSTGRES_PRIMARY_KEY` | Primary key column name (default: `id`) |
### File-backed queue (`QueueRepositoryInterface`)
In-process and disk-backed FIFO queues for buffering dict items. No optional extra required.
| Adapter | Role |
| ------------------------ | --------------------------------------------------------------------------------- |
| `MemoryQueueAdapter` | Thread-safe in-memory buffer with optional dedup and age eviction |
| `FileBackedQueueAdapter` | Mirrors memory to a JSONL file; replays on `connect()`, compacts on dequeue/evict |
| Environment variable | Description |
| -------------------------- | ----------------------------------------- |
| `FILE_QUEUE_PATH` | Path to the JSONL queue file |
| `FILE_QUEUE_MAX_AGE_HOURS` | Retention window in hours (default: `24`) |
Dedup keys and the age field name are domain-specific: pass them when constructing `FileQueueConfig` (or as kwargs to `FileQueueConfig.from_env(...)`). Callers choose the file path and item schema.
```python
from pathlib import Path
from python_repositories import FileBackedQueueAdapter, FileQueueConfig
config = FileQueueConfig(
path=Path("/var/lib/my-service/queue/items.jsonl"),
max_age_hours=24,
dedup_keys=("id",),
age_key="created_at",
)
with FileBackedQueueAdapter(config=config) as queue:
queue.enqueue([{"id": 1, "created_at": "2024-01-01T00:00:00+00:00"}])
batch = queue.dequeue_batch(max_items=100)
```
Copy [`.env.example`](.env.example) to `.env` for local development. `RedisConfig.from_env()`, `MinioConfig.from_env()`, `PostgresConfig.from_env()`, and `FileQueueConfig.from_env()` load `.env` automatically when resolving configuration from the environment.
## Configuration injection
Adapters accept optional `config` and `client` keyword arguments for explicit setup and testing:
Adapters accept optional `config` and `client` keyword arguments for explicit setup and testing. Queue adapters accept `config` (and an optional injected `memory` buffer for `FileBackedQueueAdapter`):
```python
from python_repositories import RedisAdapter, RedisConfig, MinioAdapter, MinioConfig
from pathlib import Path
from python_repositories import (
FileBackedQueueAdapter,
FileQueueConfig,
MinioAdapter,
MinioConfig,
PostgresAdapter,
PostgresConfig,
RedisAdapter,
RedisConfig,
)
redis = RedisAdapter(config=RedisConfig(uri="redis://localhost:6379"))
minio = MinioAdapter(
@@ -79,6 +139,20 @@ minio = MinioAdapter(
# create_bucket_if_missing=True, # convenient for local dev
)
)
postgres = PostgresAdapter(
config=PostgresConfig(
uri="postgresql://localhost/mydb",
table="users",
primary_key="user_id",
)
)
queue = FileBackedQueueAdapter(
config=FileQueueConfig(
path=Path("/var/lib/my-service/queue/items.jsonl"),
dedup_keys=("id",),
age_key="created_at",
)
)
```
When both `config` and `client` are provided, `connect()` skips client creation (the caller owns the client lifecycle). `config` is required whenever `client` is injected.
@@ -91,7 +165,7 @@ from python_repositories import load_dotenv
load_dotenv() # optional — from_env() also loads .env by default
```
Calling `RedisAdapter()` or `MinioAdapter()` with no arguments still loads configuration from environment variables (and `.env` if present).
Calling `RedisAdapter()`, `MinioAdapter()`, or `PostgresAdapter()` with no arguments still loads configuration from environment variables (and `.env` if present).
## Quick start
@@ -120,6 +194,17 @@ with ArtifactObjectRepository() as repo:
data = repo.get_artifact("report-1")
```
### Relational rows with Postgres
```python
from python_repositories.examples.user_table_repository import UserTableRepository
with UserTableRepository() as repo:
repo.save_user("alice", {"name": "Alice", "email": "alice@example.com"})
user = repo.get_user("alice")
repo.delete_user("alice")
```
### Subclassing in your own project
```python
@@ -144,12 +229,19 @@ class UserRepository(RedisAdapter):
from python_repositories import (
ConnectionAwareInterface,
ContextAwareInterface,
FileBackedQueueAdapter,
FileQueueConfig,
JsonRepositoryInterface,
MemoryQueueAdapter,
MinioAdapter,
MinioConfig,
ObjectRepositoryInterface,
PostgresAdapter,
PostgresConfig,
QueueRepositoryInterface,
RedisAdapter,
RedisConfig,
TableRepositoryInterface,
load_dotenv,
)
```
@@ -163,10 +255,20 @@ uv run pytest tests/unit/ -v # fast, no Docker
uv run pytest -m "not integration" -v # all non-Docker tests
uv run pytest tests/integration/redis/ -v # Redis container only
uv run pytest tests/integration/minio/ -v # MinIO container only
uv run pytest tests/integration/postgres/ -v # Postgres container only
uv run pytest -v # full suite (requires Docker)
```
Integration tests are marked with `@pytest.mark.integration` and require Docker (testcontainers). Backend-specific markers (`needs_redis`, `needs_minio`) let you run only the containers a test module needs. Run unit tests alone for quick local feedback.
Integration tests are marked with `@pytest.mark.integration` and require Docker (testcontainers). Backend-specific markers (`needs_redis`, `needs_minio`, `needs_postgres`) let you run only the containers a test module needs. Run unit tests alone for quick local feedback.
### Test organization
Unit tests live in `tests/unit/` and follow a one-to-one naming convention: `<module>_test.py` tests `python_repositories/<module path>.py`. Examples:
- `json_repository_interface.py``tests/unit/json_repository_interface_test.py`
- `redis_adapter.py``tests/unit/redis_adapter_test.py`
Add new tests to the existing file for that module rather than creating cross-cutting test files. Shared fixtures belong in `tests/conftest.py`; module-specific helpers may live in the matching test file.
### CI base image
+5 -2
View File
@@ -1,6 +1,6 @@
[project]
name = "python-repositories"
version = "2.0.5"
version = "2.2.0"
description = "Various python repository interfaces exposed as a python package."
authors = [
{ name = "Brian Bjarke Jensen", email = "schnitzelen@gmail.com" }
@@ -32,6 +32,9 @@ redis = [
minio = [
"minio>=7.2.16",
]
postgres = [
"psycopg[binary]>=3.2.0",
]
[build-system]
requires = ["hatchling"]
@@ -39,12 +42,12 @@ build-backend = "hatchling.build"
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
addopts = "--import-mode=importlib"
markers = [
"integration: tests requiring Docker containers (deselect with '-m \"not integration\"')",
"needs_redis: integration test requiring a Redis container",
"needs_minio: integration test requiring a MinIO container",
"needs_postgres: integration test requiring a Postgres container",
]
[tool.coverage.run]
+18 -1
View File
@@ -6,26 +6,43 @@ from typing import TYPE_CHECKING
# Interfaces are always available; they have no optional backend dependencies.
from . import adapters
from .config import MinioConfig, RedisConfig, load_dotenv
from .config import (
FileQueueConfig,
MinioConfig,
PostgresConfig,
RedisConfig,
load_dotenv,
)
from .interfaces import (
ConnectionAwareInterface,
ContextAwareInterface,
JsonRepositoryInterface,
ObjectRepositoryInterface,
QueueRepositoryInterface,
TableRepositoryInterface,
)
# Adapters are imported only for static type checkers; runtime loading is delegated below.
if TYPE_CHECKING:
from .adapters.file_backed_queue_adapter import (
FileBackedQueueAdapter as FileBackedQueueAdapter,
)
from .adapters.memory_queue_adapter import MemoryQueueAdapter as MemoryQueueAdapter
from .adapters.minio_adapter import MinioAdapter as MinioAdapter
from .adapters.postgres_adapter import PostgresAdapter as PostgresAdapter
from .adapters.redis_adapter import RedisAdapter as RedisAdapter
__all__ = [
"ConnectionAwareInterface",
"ContextAwareInterface",
"FileQueueConfig",
"JsonRepositoryInterface",
"MinioConfig",
"ObjectRepositoryInterface",
"PostgresConfig",
"QueueRepositoryInterface",
"RedisConfig",
"TableRepositoryInterface",
"load_dotenv",
*adapters.__all__,
]
+12
View File
@@ -11,7 +11,10 @@ from typing import TYPE_CHECKING
# Adapters are imported only for static type checkers; runtime loading is deferred below.
if TYPE_CHECKING:
from .file_backed_queue_adapter import FileBackedQueueAdapter
from .memory_queue_adapter import MemoryQueueAdapter
from .minio_adapter import MinioAdapter
from .postgres_adapter import PostgresAdapter
from .redis_adapter import RedisAdapter
# Map public adapter names to their defining module and class.
@@ -20,11 +23,20 @@ if TYPE_CHECKING:
_LAZY_EXPORTS = {
"RedisAdapter": (".redis_adapter", "RedisAdapter"),
"MinioAdapter": (".minio_adapter", "MinioAdapter"),
"PostgresAdapter": (".postgres_adapter", "PostgresAdapter"),
"MemoryQueueAdapter": (".memory_queue_adapter", "MemoryQueueAdapter"),
"FileBackedQueueAdapter": (
".file_backed_queue_adapter",
"FileBackedQueueAdapter",
),
}
__all__ = [
"RedisAdapter",
"MinioAdapter",
"PostgresAdapter",
"MemoryQueueAdapter",
"FileBackedQueueAdapter",
]
@@ -0,0 +1,215 @@
"""File-backed queue adapter: in-memory hot buffer mirrored to JSONL on disk."""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
import json
import os
from pathlib import Path
import threading
from typing import Any, Self, TextIO
import structlog
from python_repositories.adapters.memory_queue_adapter import MemoryQueueAdapter
from python_repositories.config.file_queue_config import FileQueueConfig
from python_repositories.interfaces.queue_repository_interface import (
QueueRepositoryInterface,
)
class FileBackedQueueAdapter(QueueRepositoryInterface):
"""Queue that mirrors an in-memory buffer to an append-friendly JSONL file.
Does not extend ``ConnectionAwareAdapter``. ``connect()`` replays the JSONL
file into memory; ``disconnect()`` flushes any open handle.
"""
def __init__(
self,
*,
config: FileQueueConfig | None = None,
memory: MemoryQueueAdapter | None = None,
) -> None:
if config is None:
config = FileQueueConfig.from_env()
if memory is None:
memory = MemoryQueueAdapter(
dedup_keys=config.dedup_keys,
age_key=config.age_key,
)
self._config = config
self._memory = memory
self._lock = threading.RLock()
self._connected = False
self._append_handle: TextIO | None = None
self.logger = structlog.get_logger(self.__class__.__name__)
@staticmethod
def _json_default(value: object) -> str:
if isinstance(value, datetime):
return value.isoformat()
raise TypeError(f"Object of type {type(value)!r} is not JSON serializable")
def __enter__(self) -> Self:
self.connect()
return self
def __exit__(
self, exc_type: type | None, exc_val: object | None, exc_tb: object | None
) -> None:
del exc_type, exc_val, exc_tb
self.disconnect()
def connect(self) -> None:
"""Create parent directories and replay the JSONL file into memory."""
with self._lock:
if self._connected:
return
path = self._config.path
path.parent.mkdir(parents=True, exist_ok=True)
self._memory.clear()
if path.is_file():
self._replay_file(path)
self._append_handle = path.open("a", encoding="utf-8")
self._connected = True
def disconnect(self) -> None:
"""Flush and close the append handle if open."""
with self._lock:
if self._append_handle is not None:
self._append_handle.flush()
try:
os.fsync(self._append_handle.fileno())
except OSError:
pass
self._append_handle.close()
self._append_handle = None
self._connected = False
def is_connected(self) -> bool:
"""Return whether ``connect()`` has completed successfully."""
return self._connected
def enqueue(self, items: list[dict[str, Any]]) -> None:
"""Append items to memory and the JSONL file; optionally age-evict."""
self._require_connected()
with self._lock:
added = self._memory.enqueue_and_return_added(items)
if added:
self._append_items(added)
if self._config.age_key is not None:
cutoff = datetime.now(UTC) - timedelta(hours=self._config.max_age_hours)
if self._memory.evict_older_than(cutoff):
self._compact()
def dequeue_batch(self, *, max_items: int = 1000) -> list[dict[str, Any]]:
"""Dequeue from memory and compact the JSONL file to match."""
self._require_connected()
with self._lock:
batch = self._memory.dequeue_batch(max_items=max_items)
if batch:
self._compact()
return batch
def size(self) -> int:
"""Return the number of items currently in the in-memory buffer."""
self._require_connected()
return self._memory.size()
def evict_older_than(self, cutoff: datetime) -> int:
"""Evict aged items from memory and compact the JSONL file."""
self._require_connected()
with self._lock:
removed = self._memory.evict_older_than(cutoff)
if removed:
self._compact()
return removed
def _require_connected(self) -> None:
if not self._connected:
raise RuntimeError("FileBackedQueueAdapter is not connected")
def _replay_file(self, path: Path) -> None:
cutoff: datetime | None = None
if self._config.age_key is not None:
cutoff = datetime.now(UTC) - timedelta(hours=self._config.max_age_hours)
with path.open(encoding="utf-8") as handle:
for line_number, line in enumerate(handle, start=1):
stripped = line.strip()
if not stripped:
continue
try:
payload = json.loads(stripped)
except json.JSONDecodeError:
self.logger.warning(
"Skipping corrupt JSONL line",
path=str(path),
line_number=line_number,
)
continue
if not isinstance(payload, dict):
self.logger.warning(
"Skipping non-object JSONL line",
path=str(path),
line_number=line_number,
)
continue
item: dict[str, Any] = payload
if cutoff is not None and self._config.age_key is not None:
age_value = item.get(self._config.age_key)
if age_value is None:
self.logger.warning(
"Skipping item missing age key on replay",
path=str(path),
line_number=line_number,
age_key=self._config.age_key,
)
continue
try:
if MemoryQueueAdapter._parse_age(age_value) < cutoff:
continue
except ValueError:
self.logger.warning(
"Skipping item with invalid age on replay",
path=str(path),
line_number=line_number,
)
continue
try:
self._memory.enqueue_and_return_added([item])
except ValueError as exc:
self.logger.warning(
"Skipping invalid item on replay",
path=str(path),
line_number=line_number,
error=str(exc),
)
def _append_items(self, items: list[dict[str, Any]]) -> None:
if self._append_handle is None:
raise RuntimeError("append handle is not open")
for item in items:
self._append_handle.write(
json.dumps(item, default=self._json_default, separators=(",", ":"))
)
self._append_handle.write("\n")
self._append_handle.flush()
def _compact(self) -> None:
"""Rewrite the JSONL file from the current in-memory snapshot."""
path = self._config.path
tmp_path = path.with_suffix(path.suffix + ".tmp")
if self._append_handle is not None:
self._append_handle.flush()
self._append_handle.close()
self._append_handle = None
with tmp_path.open("w", encoding="utf-8") as handle:
for item in self._memory.snapshot():
handle.write(
json.dumps(item, default=self._json_default, separators=(",", ":"))
)
handle.write("\n")
handle.flush()
tmp_path.replace(path)
self._append_handle = path.open("a", encoding="utf-8")
@@ -0,0 +1,128 @@
"""In-memory queue adapter with optional dedup and age-based eviction."""
from __future__ import annotations
from collections import OrderedDict
from collections.abc import Hashable
from datetime import UTC, datetime
import threading
from typing import Any
from python_repositories.interfaces.queue_repository_interface import (
QueueRepositoryInterface,
)
class MemoryQueueAdapter(QueueRepositoryInterface):
"""Thread-safe in-process FIFO queue with optional deduplication."""
def __init__(
self,
*,
dedup_keys: tuple[str, ...] = (),
age_key: str | None = None,
) -> None:
self._dedup_keys = dedup_keys
self._age_key = age_key
self._lock = threading.RLock()
self._items: OrderedDict[Hashable, dict[str, Any]] = OrderedDict()
self._seq = 0
@staticmethod
def _parse_age(value: object) -> datetime:
"""Normalize an age field to a timezone-aware UTC datetime."""
if isinstance(value, datetime):
if value.tzinfo is None:
return value.replace(tzinfo=UTC)
return value.astimezone(UTC)
if isinstance(value, str):
normalized = value.replace("Z", "+00:00")
parsed = datetime.fromisoformat(normalized)
if parsed.tzinfo is None:
return parsed.replace(tzinfo=UTC)
return parsed.astimezone(UTC)
raise ValueError(
f"age value must be datetime or ISO-8601 str, got {type(value)!r}"
)
def enqueue(self, items: list[dict[str, Any]]) -> None:
"""Append items; skip duplicates when ``dedup_keys`` is configured."""
self.enqueue_and_return_added(items)
def enqueue_and_return_added(
self, items: list[dict[str, Any]]
) -> list[dict[str, Any]]:
"""Enqueue items and return the subset that was newly stored."""
if not items:
return []
added: list[dict[str, Any]] = []
with self._lock:
for raw in items:
item = self._normalize_item(raw)
key = self._make_key(item)
if self._dedup_keys and key in self._items:
continue
self._items[key] = item
added.append(item)
return added
def dequeue_batch(self, *, max_items: int = 1000) -> list[dict[str, Any]]:
"""Remove and return up to ``max_items`` items in FIFO order."""
if max_items < 0:
raise ValueError("max_items must be >= 0")
with self._lock:
batch: list[dict[str, Any]] = []
for _ in range(min(max_items, len(self._items))):
_key, item = self._items.popitem(last=False)
batch.append(item)
return batch
def size(self) -> int:
"""Return the number of items currently in the queue."""
with self._lock:
return len(self._items)
def evict_older_than(self, cutoff: datetime) -> int:
"""Remove items whose age field is strictly older than ``cutoff``."""
if self._age_key is None:
return 0
cutoff_utc = self._parse_age(cutoff)
removed = 0
with self._lock:
to_remove = [
key
for key, item in self._items.items()
if self._parse_age(item[self._age_key]) < cutoff_utc
]
for key in to_remove:
del self._items[key]
removed += 1
return removed
def clear(self) -> None:
"""Remove all items from the queue."""
with self._lock:
self._items.clear()
def snapshot(self) -> list[dict[str, Any]]:
"""Return a shallow copy of queued items in FIFO order."""
with self._lock:
return [dict(item) for item in self._items.values()]
def _normalize_item(self, raw: dict[str, Any]) -> dict[str, Any]:
item = dict(raw)
if self._dedup_keys:
missing = [key for key in self._dedup_keys if key not in item]
if missing:
raise ValueError(f"item missing dedup key(s): {missing}")
if self._age_key is not None:
if self._age_key not in item:
raise ValueError(f"item missing age key: {self._age_key!r}")
item[self._age_key] = self._parse_age(item[self._age_key])
return item
def _make_key(self, item: dict[str, Any]) -> Hashable:
if not self._dedup_keys:
self._seq += 1
return self._seq
return tuple(item[key] for key in self._dedup_keys)
@@ -19,7 +19,7 @@ except ImportError as exc:
) from exc
class MinioAdapter(ObjectRepositoryInterface, ConnectionAwareAdapter):
class MinioAdapter(ConnectionAwareAdapter, ObjectRepositoryInterface):
"""Minio adapter exposing basic CRUD functionality."""
endpoint_env_var_name: str = "MINIO_ENDPOINT"
@@ -0,0 +1,230 @@
"""Definition of PostgresAdapter class."""
from __future__ import annotations
from typing import Any
from python_repositories.adapters.connection_aware_adapter import (
ConnectionAwareAdapter,
)
from python_repositories.config import PostgresConfig
from python_repositories.interfaces import TableRepositoryInterface
try:
import psycopg
from psycopg import sql
from psycopg.errors import UndefinedTable
from psycopg.rows import dict_row
except ImportError as exc:
raise ImportError(
"Postgres support requires the postgres extra. "
"Install with: pip install python-repositories[postgres]"
) from exc
class PostgresAdapter(ConnectionAwareAdapter, TableRepositoryInterface):
"""Postgres adapter exposing basic table CRUD functionality."""
uri_env_var_name: str = "POSTGRES_URI"
table_env_var_name: str = "POSTGRES_TABLE"
primary_key_env_var_name: str = "POSTGRES_PRIMARY_KEY"
connection_name: str = "Postgres"
def __init__(
self,
*,
config: PostgresConfig | None = None,
client: psycopg.Connection[Any] | None = None,
) -> None:
super().__init__()
if client is not None and config is None:
raise ValueError("config is required when client is provided")
if config is None:
config = PostgresConfig.from_env(
self.uri_env_var_name,
self.table_env_var_name,
self.primary_key_env_var_name,
)
self._config = config
self._client_injected = client is not None
self._client: psycopg.Connection[Any] | None = client
if self._client_injected:
self._invalidate_health_cache()
def _is_client_ready(self) -> bool:
return self._client is not None
def _table_identifier(self) -> sql.Identifier:
return sql.Identifier(self._config.table)
def _primary_key_identifier(self) -> sql.Identifier:
return sql.Identifier(self._config.primary_key)
def _validate_injected_client(self) -> None:
if self._client is None:
return
try:
self._run_connection_probe()
except (psycopg.Error, ConnectionError) as exc:
raise ConnectionError(
f"Could not connect to Postgres at {self._config.uri}"
) from exc
def _establish_connection(self) -> None:
uri = self._config.uri
try:
client = psycopg.connect(
uri,
row_factory=dict_row,
connect_timeout=10,
autocommit=True,
)
self._client = client
self._run_connection_probe()
except ConnectionError:
self._client = None
raise
except psycopg.Error as exc:
self._client = None
raise ConnectionError(f"Could not connect to Postgres at {uri}") from exc
def _run_connection_probe(self) -> None:
assert self._client is not None
with self._client.cursor() as cursor:
cursor.execute("SELECT 1")
try:
cursor.execute(
sql.SQL("SELECT 1 FROM {} LIMIT 0").format(self._table_identifier())
)
except UndefinedTable as exc:
raise ConnectionError(
f"Table '{self._config.table}' does not exist on Postgres at "
f"{self._config.uri}"
) from exc
def disconnect(self) -> None:
"""Disconnect from the Postgres server."""
if self._client is not None and not self._client_injected:
self._client.close()
self._client = None
self._invalidate_health_cache()
def _probe_connection(self) -> bool:
assert self._client is not None
try:
self._run_connection_probe()
except (psycopg.Error, ConnectionError):
return False
return True
def _validate_pk(self, pk: Any) -> None:
if pk is None:
raise ValueError("Primary key must not be None")
def _validate_row(self, row: dict[str, Any]) -> None:
if not isinstance(row, dict):
raise ValueError("Row must be a dictionary")
if self._config.primary_key not in row:
raise ValueError(
f"Row must include primary key column '{self._config.primary_key}'"
)
def _validate_limit(self, limit: int | None) -> None:
if limit is not None and (not isinstance(limit, int) or limit < 0):
raise ValueError("Limit must be a non-negative integer or None")
def fetch_one(self, pk: Any) -> dict[str, Any] | None:
"""Fetch a single row by primary key."""
self._validate_pk(pk)
self._require_connected()
assert self._client is not None
query = sql.SQL("SELECT * FROM {} WHERE {} = %s").format(
self._table_identifier(),
self._primary_key_identifier(),
)
with self._client.cursor() as cursor:
cursor.execute(query, (pk,))
row = cursor.fetchone()
self.logger.debug("Fetched row", pk=pk, found=row is not None)
return row
def fetch_all(self, *, limit: int | None = None) -> list[dict[str, Any]]:
"""Fetch all rows from the configured table."""
self._validate_limit(limit)
self._require_connected()
assert self._client is not None
query = sql.SQL("SELECT * FROM {}").format(self._table_identifier())
params: tuple[Any, ...] = ()
if limit is not None:
query = sql.Composed([query, sql.SQL(" LIMIT %s")])
params = (limit,)
with self._client.cursor() as cursor:
cursor.execute(query, params)
rows = cursor.fetchall()
self.logger.debug("Fetched rows", count=len(rows), limit=limit)
return list(rows)
def upsert(self, row: dict[str, Any]) -> None:
"""Insert or update a row by primary key."""
self._validate_row(row)
self._require_connected()
assert self._client is not None
pk_col = self._config.primary_key
columns = list(row.keys())
identifiers = [sql.Identifier(column) for column in columns]
placeholders = sql.SQL(", ").join(sql.Placeholder() * len(columns))
column_list = sql.SQL(", ").join(identifiers)
update_columns = [column for column in columns if column != pk_col]
on_conflict: sql.Composed | sql.SQL
if update_columns:
update_assignments = sql.SQL(", ").join(
sql.SQL("{} = EXCLUDED.{}").format(
sql.Identifier(column),
sql.Identifier(column),
)
for column in update_columns
)
on_conflict = sql.SQL("DO UPDATE SET {}").format(update_assignments)
else:
on_conflict = sql.SQL("DO NOTHING")
query = sql.SQL("INSERT INTO {} ({}) VALUES ({}) ON CONFLICT ({}) {}").format(
self._table_identifier(),
column_list,
placeholders,
self._primary_key_identifier(),
on_conflict,
)
with self._client.cursor() as cursor:
cursor.execute(query, tuple(row[column] for column in columns))
self.logger.debug("Upserted row", pk=row[pk_col], columns=columns)
def delete(self, pk: Any) -> None:
"""Delete a row by primary key."""
self._validate_pk(pk)
self._require_connected()
assert self._client is not None
query = sql.SQL("DELETE FROM {} WHERE {} = %s").format(
self._table_identifier(),
self._primary_key_identifier(),
)
with self._client.cursor() as cursor:
cursor.execute(query, (pk,))
self.logger.debug("Deleted row", pk=pk)
def execute(
self,
sql_text: str,
params: tuple[Any, ...] = (),
) -> list[dict[str, Any]]:
"""Run a read SQL statement and return rows as dicts."""
if not isinstance(sql_text, str) or len(sql_text) == 0:
raise ValueError("SQL must be a non-empty string")
if not isinstance(params, tuple):
raise ValueError("Params must be a tuple")
self._require_connected()
assert self._client is not None
with self._client.cursor() as cursor:
cursor.execute(sql_text, params)
rows = cursor.fetchall()
self.logger.debug("Executed SQL", row_count=len(rows))
return list(rows)
@@ -21,7 +21,7 @@ except ImportError as exc:
) from exc
class RedisAdapter(JsonRepositoryInterface, ConnectionAwareAdapter):
class RedisAdapter(ConnectionAwareAdapter, JsonRepositoryInterface):
"""Redis adapter exposing basic CRUD functionality."""
uri_env_var_name: str = "REDIS_URI"
+4
View File
@@ -1,9 +1,13 @@
from .dotenv_loader import load_dotenv as load_dotenv
from .file_queue_config import FileQueueConfig as FileQueueConfig
from .minio_config import MinioConfig as MinioConfig
from .postgres_config import PostgresConfig as PostgresConfig
from .redis_config import RedisConfig as RedisConfig
__all__ = [
"FileQueueConfig",
"MinioConfig",
"PostgresConfig",
"RedisConfig",
"load_dotenv",
]
@@ -0,0 +1,56 @@
"""File-backed queue configuration."""
from __future__ import annotations
from dataclasses import dataclass
import os
from pathlib import Path
from python_utils import check_env
from python_repositories.config.dotenv_loader import load_dotenv
_DEFAULT_MAX_AGE_HOURS = 24
@dataclass(frozen=True)
class FileQueueConfig:
"""Configuration for a JSONL file-backed queue.
Path and key schema are owned by the caller. This package does not assume
any particular directory layout or item field names.
"""
path: Path
max_age_hours: int = _DEFAULT_MAX_AGE_HOURS
dedup_keys: tuple[str, ...] = ()
age_key: str | None = None
@classmethod
def from_env(
cls,
path_env_var_name: str = "FILE_QUEUE_PATH",
*,
max_age_hours_env_var_name: str = "FILE_QUEUE_MAX_AGE_HOURS",
dedup_keys: tuple[str, ...] = (),
age_key: str | None = None,
use_dotenv: bool = True,
) -> FileQueueConfig:
"""Load path and optional max age from environment variables.
``dedup_keys`` and ``age_key`` are domain-specific and must be passed
explicitly; they are not read from the environment.
"""
if use_dotenv:
load_dotenv()
check_env(path_env_var_name)
raw_max_age = os.getenv(max_age_hours_env_var_name)
max_age_hours = (
_DEFAULT_MAX_AGE_HOURS if raw_max_age is None else int(raw_max_age)
)
return cls(
path=Path(str(os.getenv(path_env_var_name))),
max_age_hours=max_age_hours,
dedup_keys=dedup_keys,
age_key=age_key,
)
@@ -0,0 +1,42 @@
"""PostgreSQL connection configuration."""
from __future__ import annotations
from dataclasses import dataclass
import os
from python_utils import check_env
from python_repositories.config.dotenv_loader import load_dotenv
@dataclass(frozen=True)
class PostgresConfig:
"""Configuration for connecting to PostgreSQL."""
uri: str
table: str
primary_key: str = "id"
@classmethod
def from_env(
cls,
uri_env_var_name: str = "POSTGRES_URI",
table_env_var_name: str = "POSTGRES_TABLE",
primary_key_env_var_name: str = "POSTGRES_PRIMARY_KEY",
*,
use_dotenv: bool = True,
) -> PostgresConfig:
"""Load configuration from environment variables."""
if use_dotenv:
load_dotenv()
env_var_names = {uri_env_var_name, table_env_var_name}
check_env(env_var_names)
primary_key = os.getenv(primary_key_env_var_name)
if primary_key is None or primary_key == "":
primary_key = "id"
return cls(
uri=str(os.getenv(uri_env_var_name)),
table=str(os.getenv(table_env_var_name)),
primary_key=primary_key,
)
@@ -0,0 +1,19 @@
"""Example domain repository backed by Postgres tables."""
from typing import Any
from python_repositories.adapters.postgres_adapter import PostgresAdapter
class UserTableRepository(PostgresAdapter):
"""Example: domain repository backed by a Postgres table."""
def get_user(self, user_id: str) -> dict[str, Any] | None:
return self.fetch_one(user_id)
def save_user(self, user_id: str, user: dict[str, Any]) -> None:
row = {self._config.primary_key: user_id, **user}
self.upsert(row)
def delete_user(self, user_id: str) -> None:
self.delete(user_id)
@@ -8,10 +8,18 @@ from .json_repository_interface import (
from .object_repository_interface import (
ObjectRepositoryInterface as ObjectRepositoryInterface,
)
from .queue_repository_interface import (
QueueRepositoryInterface as QueueRepositoryInterface,
)
from .table_repository_interface import (
TableRepositoryInterface as TableRepositoryInterface,
)
__all__ = [
"ConnectionAwareInterface",
"ContextAwareInterface",
"JsonRepositoryInterface",
"ObjectRepositoryInterface",
"QueueRepositoryInterface",
"TableRepositoryInterface",
]
@@ -1,9 +1,11 @@
"""Definition of ConnectionAwareInterface abstract base class."""
"""Definition of ConnectionAwareInterface protocol and abstract base class."""
from abc import ABC, abstractmethod
from abc import abstractmethod
from typing import Protocol, runtime_checkable
class ConnectionAwareInterface(ABC):
@runtime_checkable
class ConnectionAwareInterface(Protocol):
"""Interface that defines connection-related methods."""
@abstractmethod
@@ -1,12 +1,13 @@
"""Definition of ContextAwareInterface abstract base class."""
"""Definition of ContextAwareInterface protocol and abstract base class."""
from __future__ import annotations
from abc import ABC, abstractmethod
from typing import Self
from abc import abstractmethod
from typing import Protocol, Self, runtime_checkable
class ContextAwareInterface(ABC):
@runtime_checkable
class ContextAwareInterface(Protocol):
"""Interface that defines context-related methods."""
@abstractmethod
@@ -1,11 +1,12 @@
"""Definition of JsonRepositoryInterface abstract base class."""
"""Definition of JsonRepositoryInterface protocol and abstract base class."""
from abc import ABC, abstractmethod
from abc import abstractmethod
from collections.abc import Iterator
from typing import Any
from typing import Any, Protocol, runtime_checkable
class JsonRepositoryInterface(ABC):
@runtime_checkable
class JsonRepositoryInterface(Protocol):
"""Interface that defines JSON document CRUD methods."""
@abstractmethod
@@ -1,10 +1,12 @@
"""Definition of ObjectRepositoryInterface abstract base class."""
"""Definition of ObjectRepositoryInterface protocol and abstract base class."""
from abc import ABC, abstractmethod
from abc import abstractmethod
from io import BytesIO
from typing import Protocol, runtime_checkable
class ObjectRepositoryInterface(ABC):
@runtime_checkable
class ObjectRepositoryInterface(Protocol):
"""Interface that defines binary object CRUD methods."""
@abstractmethod
@@ -0,0 +1,41 @@
"""Definition of QueueRepositoryInterface protocol."""
from abc import abstractmethod
from datetime import datetime
from typing import Any, Protocol, runtime_checkable
@runtime_checkable
class QueueRepositoryInterface(Protocol):
"""Interface that defines buffered queue enqueue/dequeue methods."""
@abstractmethod
def enqueue(self, items: list[dict[str, Any]]) -> None:
"""Append items to the queue.
Duplicates may be skipped when the implementation is configured with
dedup keys. An empty list is a no-op.
"""
...
@abstractmethod
def dequeue_batch(self, *, max_items: int = 1000) -> list[dict[str, Any]]:
"""Remove and return up to ``max_items`` items in FIFO order.
Returns an empty list when the queue is empty.
"""
...
@abstractmethod
def size(self) -> int:
"""Return the number of items currently in the queue."""
...
@abstractmethod
def evict_older_than(self, cutoff: datetime) -> int:
"""Remove items older than ``cutoff`` and return how many were removed.
Age is determined by an implementation-specific item field. When no age
field is configured, this is a no-op that returns ``0``.
"""
...
@@ -0,0 +1,55 @@
"""Definition of TableRepositoryInterface protocol."""
from abc import abstractmethod
from typing import Any, Protocol, runtime_checkable
@runtime_checkable
class TableRepositoryInterface(Protocol):
"""Interface that defines relational table CRUD methods."""
@abstractmethod
def fetch_one(self, pk: Any) -> dict[str, Any] | None:
"""Fetch a single row by primary key.
Returns ``None`` when no row matches. Use ``value is not None`` to test
existence; avoid truthiness checks.
"""
...
@abstractmethod
def fetch_all(self, *, limit: int | None = None) -> list[dict[str, Any]]:
"""Fetch all rows from the configured table.
Returns an empty list when the table has no rows.
"""
...
@abstractmethod
def upsert(self, row: dict[str, Any]) -> None:
"""Insert or update a row by primary key.
``row`` must include the configured primary key column. Raises
``ValueError`` for a non-dict row or a missing primary key.
"""
...
@abstractmethod
def delete(self, pk: Any) -> None:
"""Delete a row by primary key.
Idempotent: no error when the row is already absent.
"""
...
@abstractmethod
def execute(
self,
sql: str,
params: tuple[Any, ...] = (),
) -> list[dict[str, Any]]:
"""Run a read SQL statement and return rows as dicts.
Escape hatch for joins, filters, and other queries in subclasses.
"""
...
+4
View File
@@ -103,13 +103,17 @@ changelog_section() {
local summary="$3"
local commits="$4"
# Blank lines after ATX headings match Prettier v4 markdown formatting
# (see mirrors-prettier in .pre-commit-config.yaml).
cat <<EOF
## [${version}] - ${date}
### Summary
${summary}
### Changed
${commits:-- (no commits recorded)}
EOF
}
+6 -1
View File
@@ -1,6 +1,6 @@
"""Shared test configuration constants."""
from python_repositories.config import MinioConfig, RedisConfig
from python_repositories.config import MinioConfig, PostgresConfig, RedisConfig
TEST_REDIS_CONFIG = RedisConfig(uri="redis://localhost:6379")
TEST_MINIO_CONFIG = MinioConfig(
@@ -10,3 +10,8 @@ TEST_MINIO_CONFIG = MinioConfig(
bucket="test-bucket",
secure=False,
)
TEST_POSTGRES_CONFIG = PostgresConfig(
uri="postgresql://localhost/mydb",
table="test_items",
primary_key="id",
)
+26 -1
View File
@@ -6,12 +6,17 @@ from minio import Minio
import pytest
import redis
from python_repositories.config import MinioConfig, RedisConfig
from python_repositories.config import MinioConfig, PostgresConfig, RedisConfig
from tests.integration.minio._containers import (
minio_config_from_env,
minio_env,
raw_minio_client_from_env,
)
from tests.integration.postgres._containers import (
postgres_config_from_container,
postgres_uri,
raw_postgres_client_from_container,
)
from tests.integration.redis._containers import (
raw_redis_client_from_container,
redis_config_from_container,
@@ -53,3 +58,23 @@ def minio_config(minio_container: dict[str, str]) -> MinioConfig:
def raw_minio_client(minio_container: dict[str, str]) -> Generator[Minio, None, None]:
"""Provide a raw Minio client connected to the test Minio container."""
yield from raw_minio_client_from_env(minio_container)
@pytest.fixture(scope="session")
def postgres_container() -> Generator[str, None, None]:
"""Set up a Postgres container for testing and yield the Postgres URI."""
yield from postgres_uri()
@pytest.fixture(scope="session")
def postgres_config(postgres_container: str) -> PostgresConfig:
"""Provide PostgresConfig built from the test container."""
return postgres_config_from_container(postgres_container)
@pytest.fixture(scope="session")
def raw_postgres_client(
postgres_container: str,
) -> Generator[object, None, None]:
"""Provide a raw Postgres client connected to the test Postgres container."""
yield from raw_postgres_client_from_container(postgres_container)
@@ -6,12 +6,15 @@ import os
import random
from minio import Minio
import psycopg
import pytest
from python_repositories.examples.artifact_object_repository import (
ArtifactObjectRepository,
)
from python_repositories.examples.user_json_repository import UserJsonRepository
from python_repositories.examples.user_table_repository import UserTableRepository
from tests.integration.postgres._containers import TEST_TABLE
pytestmark = [
pytest.mark.integration,
@@ -36,6 +39,24 @@ def set_example_env(
_ = os.environ.pop(key, default=None)
@pytest.fixture(scope="module", autouse=True)
def set_postgres_example_env(
postgres_container: str,
raw_postgres_client: psycopg.Connection,
) -> Generator[None, None, None]:
"""Set Postgres env vars for example repositories using from_env() defaults."""
env_vars = {
"POSTGRES_URI": postgres_container,
"POSTGRES_TABLE": TEST_TABLE,
"POSTGRES_PRIMARY_KEY": "id",
}
for key, value in env_vars.items():
os.environ[key] = value
yield
for key in env_vars:
_ = os.environ.pop(key, default=None)
@pytest.fixture(scope="module")
def user_data() -> Generator[dict[str, str], None, None]:
"""Provide sample user data for tests."""
@@ -77,3 +98,27 @@ def test_artifact_object_repository_store_and_get(
assert received is not None
artifact_data.seek(0)
assert received.read() == artifact_data.read()
@pytest.mark.needs_postgres
def test_user_table_repository_save_and_get(
user_data: dict[str, str],
) -> None:
"""Test that UserTableRepository can save and retrieve a user."""
with UserTableRepository() as repo:
repo.save_user("alice", user_data)
user = repo.get_user("alice")
assert user is not None
assert user["name"] == user_data["name"]
assert user["email"] == user_data["email"]
@pytest.mark.needs_postgres
def test_user_table_repository_delete(
user_data: dict[str, str],
) -> None:
"""Test that UserTableRepository can delete a user."""
with UserTableRepository() as repo:
repo.save_user("alice", user_data)
repo.delete_user("alice")
assert repo.get_user("alice") is None
+76
View File
@@ -0,0 +1,76 @@
"""Postgres container session helpers for integration tests."""
from collections.abc import Generator
import psycopg
from psycopg.rows import dict_row
from testcontainers.postgres import PostgresContainer
from python_repositories.config import PostgresConfig
POSTGRES_IMAGE = "postgres:16"
TEST_TABLE = "test_items"
_postgres_uri: str | None = None
_postgres_container: PostgresContainer | None = None
_raw_postgres_client: psycopg.Connection | None = None
_postgres_uri_refs = 0
_raw_postgres_client_refs = 0
def postgres_uri() -> Generator[str, None, None]:
"""Yield a session-scoped Postgres URI, starting the container once."""
global _postgres_uri, _postgres_container, _postgres_uri_refs
if _postgres_uri is None:
_postgres_container = PostgresContainer(POSTGRES_IMAGE, driver=None)
_postgres_container.start()
_postgres_uri = _postgres_container.get_connection_url()
_postgres_uri_refs += 1
yield _postgres_uri
_postgres_uri_refs -= 1
if _postgres_uri_refs == 0 and _postgres_container is not None:
_postgres_container.stop()
_postgres_container = None
_postgres_uri = None
def postgres_config_from_container(uri: str) -> PostgresConfig:
"""Build PostgresConfig from a container URI."""
return PostgresConfig(uri=uri, table=TEST_TABLE, primary_key="id")
def raw_postgres_client_from_container(
uri: str,
) -> Generator[psycopg.Connection, None, None]:
"""Yield a session-scoped raw Postgres client, reusing one client per session."""
global _raw_postgres_client, _raw_postgres_client_refs
if _raw_postgres_client is None:
_raw_postgres_client = psycopg.connect(
uri,
row_factory=dict_row, # type: ignore[arg-type]
autocommit=True,
)
with _raw_postgres_client.cursor() as cursor:
cursor.execute(
f"""
CREATE TABLE IF NOT EXISTS {TEST_TABLE} (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
value INTEGER,
email TEXT
)
"""
)
_raw_postgres_client_refs += 1
yield _raw_postgres_client
_raw_postgres_client_refs -= 1
if _raw_postgres_client_refs == 0 and _raw_postgres_client is not None:
with _raw_postgres_client.cursor() as cursor:
cursor.execute(f"TRUNCATE TABLE {TEST_TABLE}")
_raw_postgres_client.close()
_raw_postgres_client = None
+33
View File
@@ -0,0 +1,33 @@
"""Postgres integration test fixtures."""
from collections.abc import Generator
import psycopg
import pytest
from python_repositories.config import PostgresConfig
from tests.integration.postgres._containers import (
postgres_config_from_container,
postgres_uri,
raw_postgres_client_from_container,
)
@pytest.fixture(scope="session")
def postgres_container() -> Generator[str, None, None]:
"""Set up a Postgres container for testing and yield the Postgres URI."""
yield from postgres_uri()
@pytest.fixture(scope="session")
def postgres_config(postgres_container: str) -> PostgresConfig:
"""Provide PostgresConfig built from the test container."""
return postgres_config_from_container(postgres_container)
@pytest.fixture(scope="session")
def raw_postgres_client(
postgres_container: str,
) -> Generator[psycopg.Connection, None, None]:
"""Provide a raw Postgres client connected to the test container."""
yield from raw_postgres_client_from_container(postgres_container)
@@ -0,0 +1,318 @@
"""Integration tests for the PostgresAdapter."""
from collections.abc import Generator
import logging
import psycopg
import pytest
from python_repositories.adapters.postgres_adapter import PostgresAdapter
from python_repositories.config import PostgresConfig
from tests.integration.postgres._containers import TEST_TABLE
pytestmark = [pytest.mark.integration, pytest.mark.needs_postgres]
@pytest.fixture(scope="module")
def row() -> Generator[dict[str, object], None, None]:
"""Provide a sample row for tests."""
yield {"id": "test-id", "name": "Alice", "value": 42}
@pytest.fixture(scope="function")
def row_in_postgres(
raw_postgres_client: psycopg.Connection,
row: dict[str, object],
) -> Generator[tuple[str, dict[str, object]], None, None]:
"""Fixture to set up a known row in Postgres before each test."""
with raw_postgres_client.cursor() as cursor:
cursor.execute(
f"INSERT INTO {TEST_TABLE} (id, name, value) VALUES (%s, %s, %s)",
(row["id"], row["name"], row["value"]),
)
yield str(row["id"]), row
with raw_postgres_client.cursor() as cursor:
cursor.execute(f"DELETE FROM {TEST_TABLE} WHERE id = %s", (row["id"],))
@pytest.fixture(scope="module")
def postgres_adapter(
postgres_config: PostgresConfig,
raw_postgres_client: psycopg.Connection,
) -> Generator[PostgresAdapter, None, None]:
"""Fixture to provide a connected PostgresAdapter instance."""
_ = raw_postgres_client # ensure test table exists before connect probe
adapter = PostgresAdapter(config=postgres_config)
adapter.connect()
yield adapter
adapter.disconnect()
@pytest.fixture(scope="function", autouse=True)
def clear_postgres(raw_postgres_client: psycopg.Connection) -> None:
"""Fixture to clear all rows before each test."""
with raw_postgres_client.cursor() as cursor:
cursor.execute(f"TRUNCATE TABLE {TEST_TABLE}")
def test_should_log_info_when_already_connected(
postgres_adapter: PostgresAdapter,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test that PostgresAdapter logs info when connect is called while connected."""
with caplog.at_level(logging.INFO):
postgres_adapter.connect()
assert "Already connected" in caplog.text
assert "Postgres" in caplog.text
def test_should_raise_connection_error_when_unable_to_connect() -> None:
"""Test that PostgresAdapter raises ConnectionError when unable to connect."""
adapter = PostgresAdapter(
config=PostgresConfig(
uri="postgresql://invalid:5432/nodb",
table=TEST_TABLE,
)
)
with pytest.raises(ConnectionError):
adapter.connect()
assert adapter._client is None
assert not adapter.is_connected()
def test_should_raise_connection_error_when_table_missing(
postgres_config: PostgresConfig,
) -> None:
"""Test that PostgresAdapter raises ConnectionError when the table is missing."""
config = PostgresConfig(
uri=postgres_config.uri,
table="missing_table",
primary_key="id",
)
adapter = PostgresAdapter(config=config)
with pytest.raises(ConnectionError, match="does not exist"):
adapter.connect()
def test_should_log_error_on_exception_during_exit(
postgres_config: PostgresConfig,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test that PostgresAdapter logs an error when an exception occurs during exit."""
try:
with PostgresAdapter(config=postgres_config) as adapter:
assert adapter.is_connected()
raise ValueError("Simulated error")
except ValueError:
pass
assert "Error while exiting context" in caplog.text
def test_should_have_context_manager(postgres_config: PostgresConfig) -> None:
"""Test that PostgresAdapter can be used as a context manager."""
with PostgresAdapter(config=postgres_config) as adapter:
assert adapter._client is not None
assert adapter._client is None
def test_should_fetch_one(
row_in_postgres: tuple[str, dict[str, object]],
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that PostgresAdapter can fetch a row by primary key."""
pk, row = row_in_postgres
fetched = postgres_adapter.fetch_one(pk)
assert fetched is not None
assert fetched["id"] == row["id"]
assert fetched["name"] == row["name"]
assert fetched["value"] == row["value"]
def test_should_fetch_none_for_missing_row(
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that fetching a non-existent row returns None."""
assert postgres_adapter.fetch_one("missing-id") is None
def test_should_raise_value_error_on_invalid_fetch_one_pk(
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that PostgresAdapter raises ValueError for an invalid primary key."""
with pytest.raises(ValueError):
postgres_adapter.fetch_one(None)
def test_should_raise_connection_error_on_fetch_one_when_not_connected(
postgres_config: PostgresConfig,
) -> None:
"""Test that fetch_one raises ConnectionError when not connected."""
adapter = PostgresAdapter(config=postgres_config)
with pytest.raises(ConnectionError):
adapter.fetch_one("some-id")
def test_should_upsert_row(
row: dict[str, object],
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that PostgresAdapter can insert a row."""
assert postgres_adapter.fetch_one(row["id"]) is None
postgres_adapter.upsert(row)
fetched = postgres_adapter.fetch_one(row["id"])
assert fetched is not None
assert fetched["name"] == row["name"]
def test_should_update_row(
row_in_postgres: tuple[str, dict[str, object]],
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that PostgresAdapter can update an existing row."""
pk, _ = row_in_postgres
new_row = {"id": pk, "name": "Bob", "value": 99}
postgres_adapter.upsert(new_row)
fetched = postgres_adapter.fetch_one(pk)
assert fetched is not None
assert fetched["name"] == "Bob"
assert fetched["value"] == 99
def test_should_raise_value_error_on_invalid_upsert_row(
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that PostgresAdapter raises ValueError for invalid upsert data."""
with pytest.raises(ValueError):
postgres_adapter.upsert({"name": "Alice"}) # missing primary key
def test_should_raise_value_error_on_non_dict_upsert_row(
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that PostgresAdapter raises ValueError when upsert row is not a dict."""
with pytest.raises(ValueError):
postgres_adapter.upsert("not-a-dict") # type: ignore[arg-type]
def test_should_raise_connection_error_on_upsert_when_not_connected(
postgres_config: PostgresConfig,
row: dict[str, object],
) -> None:
"""Test that upsert raises ConnectionError when not connected."""
adapter = PostgresAdapter(config=postgres_config)
with pytest.raises(ConnectionError):
adapter.upsert(row)
def test_should_fetch_all_rows(
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that PostgresAdapter can fetch all rows."""
postgres_adapter.upsert({"id": "a", "name": "Alice", "value": 1})
postgres_adapter.upsert({"id": "b", "name": "Bob", "value": 2})
rows = postgres_adapter.fetch_all()
assert len(rows) == 2
ids = {row["id"] for row in rows}
assert ids == {"a", "b"}
def test_should_fetch_all_with_limit(
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that PostgresAdapter respects fetch_all limit."""
postgres_adapter.upsert({"id": "a", "name": "Alice", "value": 1})
postgres_adapter.upsert({"id": "b", "name": "Bob", "value": 2})
rows = postgres_adapter.fetch_all(limit=1)
assert len(rows) == 1
def test_should_return_empty_list_for_empty_table(
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that fetch_all returns an empty list for an empty table."""
assert postgres_adapter.fetch_all() == []
def test_should_raise_value_error_on_invalid_fetch_all_limit(
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that PostgresAdapter raises ValueError for an invalid limit."""
with pytest.raises(ValueError):
postgres_adapter.fetch_all(limit=-1)
def test_should_raise_connection_error_on_fetch_all_when_not_connected(
postgres_config: PostgresConfig,
) -> None:
"""Test that fetch_all raises ConnectionError when not connected."""
adapter = PostgresAdapter(config=postgres_config)
with pytest.raises(ConnectionError):
adapter.fetch_all()
def test_should_delete_row(
row_in_postgres: tuple[str, dict[str, object]],
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that deleting a row removes it from the table."""
pk, _ = row_in_postgres
assert postgres_adapter.fetch_one(pk) is not None
postgres_adapter.delete(pk)
assert postgres_adapter.fetch_one(pk) is None
def test_should_delete_idempotently(
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that deleting a missing row does not raise."""
postgres_adapter.delete("missing-id")
def test_should_raise_value_error_on_invalid_delete_pk(
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that PostgresAdapter raises ValueError for an invalid delete pk."""
with pytest.raises(ValueError):
postgres_adapter.delete(None)
def test_should_raise_connection_error_on_delete_when_not_connected(
postgres_config: PostgresConfig,
) -> None:
"""Test that delete raises ConnectionError when not connected."""
adapter = PostgresAdapter(config=postgres_config)
with pytest.raises(ConnectionError):
adapter.delete("some-id")
def test_should_execute_sql(
row_in_postgres: tuple[str, dict[str, object]],
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that PostgresAdapter can execute read SQL."""
pk, _ = row_in_postgres
rows = postgres_adapter.execute(
f"SELECT * FROM {TEST_TABLE} WHERE id = %s",
(pk,),
)
assert len(rows) == 1
assert rows[0]["id"] == pk
def test_should_raise_value_error_on_invalid_execute_sql(
postgres_adapter: PostgresAdapter,
) -> None:
"""Test that PostgresAdapter raises ValueError for invalid SQL."""
with pytest.raises(ValueError):
postgres_adapter.execute("")
def test_should_raise_connection_error_on_execute_when_not_connected(
postgres_config: PostgresConfig,
) -> None:
"""Test that execute raises ConnectionError when not connected."""
adapter = PostgresAdapter(config=postgres_config)
with pytest.raises(ConnectionError):
adapter.execute("SELECT 1")
+47 -2
View File
@@ -45,6 +45,9 @@ from python_repositories import JsonRepositoryInterface
assert JsonRepositoryInterface is not None
assert "python_repositories.adapters.redis_adapter" not in sys.modules
assert "python_repositories.adapters.minio_adapter" not in sys.modules
assert "python_repositories.adapters.postgres_adapter" not in sys.modules
assert "python_repositories.adapters.memory_queue_adapter" not in sys.modules
assert "python_repositories.adapters.file_backed_queue_adapter" not in sys.modules
"""
result = subprocess.run(
[sys.executable, "-c", script],
@@ -58,10 +61,19 @@ assert "python_repositories.adapters.minio_adapter" not in sys.modules
def test_lazy_adapter_load_succeeds_when_extra_present() -> None:
"""Adapters load when their optional dependencies are installed."""
from python_repositories import MinioAdapter, RedisAdapter
from python_repositories import (
FileBackedQueueAdapter,
MemoryQueueAdapter,
MinioAdapter,
PostgresAdapter,
RedisAdapter,
)
assert RedisAdapter.__name__ == "RedisAdapter"
assert MinioAdapter.__name__ == "MinioAdapter"
assert PostgresAdapter.__name__ == "PostgresAdapter"
assert MemoryQueueAdapter.__name__ == "MemoryQueueAdapter"
assert FileBackedQueueAdapter.__name__ == "FileBackedQueueAdapter"
def test_redis_adapter_import_error_without_extra() -> None:
@@ -86,6 +98,17 @@ def test_minio_adapter_import_error_without_extra() -> None:
importlib.reload(minio_adapter_module)
def test_postgres_adapter_import_error_without_extra() -> None:
"""Missing postgres extra raises ImportError with install hint."""
import python_repositories.adapters.postgres_adapter as postgres_adapter_module
with patch.object(builtins, "__import__", new=_block_backend_import("psycopg")):
with pytest.raises(ImportError, match=r"python-repositories\[postgres\]"):
importlib.reload(postgres_adapter_module)
importlib.reload(postgres_adapter_module)
def test_top_level_lazy_import_propagates_redis_import_error() -> None:
"""Top-level RedisAdapter access surfaces adapter import errors."""
with patch(
@@ -112,6 +135,19 @@ def test_top_level_lazy_import_propagates_minio_import_error() -> None:
_ = python_repositories.MinioAdapter
def test_top_level_lazy_import_propagates_postgres_import_error() -> None:
"""Top-level PostgresAdapter access surfaces adapter import errors."""
with patch(
"importlib.import_module",
side_effect=ImportError(
"Postgres support requires the postgres extra. "
"Install with: pip install python-repositories[postgres]"
),
):
with pytest.raises(ImportError, match=r"python-repositories\[postgres\]"):
_ = python_repositories.PostgresAdapter
def test_adapters_subpackage_lazy_import_succeeds() -> None:
"""Adapter subpackage imports delegate to the same lazy loader."""
from python_repositories.adapters import RedisAdapter
@@ -123,7 +159,13 @@ def test_adapters_dir_exposes_lazy_exports() -> None:
"""dir(adapters) includes lazy adapter names for tab completion."""
import python_repositories.adapters as adapters
assert {"RedisAdapter", "MinioAdapter"}.issubset(set(dir(adapters)))
assert {
"RedisAdapter",
"MinioAdapter",
"PostgresAdapter",
"MemoryQueueAdapter",
"FileBackedQueueAdapter",
}.issubset(set(dir(adapters)))
def test_adapters_getattr_raises_for_unknown() -> None:
@@ -138,3 +180,6 @@ def test_top_level_dir_exposes_lazy_exports() -> None:
"""dir(python_repositories) includes lazy adapter names for tab completion."""
assert "RedisAdapter" in dir(python_repositories)
assert "MinioAdapter" in dir(python_repositories)
assert "PostgresAdapter" in dir(python_repositories)
assert "MemoryQueueAdapter" in dir(python_repositories)
assert "FileBackedQueueAdapter" in dir(python_repositories)
+14 -1
View File
@@ -5,12 +5,14 @@ from __future__ import annotations
from unittest.mock import MagicMock
from minio import Minio
import psycopg
import pytest
import redis
from python_repositories.adapters.minio_adapter import MinioAdapter
from python_repositories.adapters.postgres_adapter import PostgresAdapter
from python_repositories.adapters.redis_adapter import RedisAdapter
from tests.conftest import TEST_MINIO_CONFIG, TEST_REDIS_CONFIG
from tests.conftest import TEST_MINIO_CONFIG, TEST_POSTGRES_CONFIG, TEST_REDIS_CONFIG
@pytest.fixture
@@ -25,3 +27,14 @@ def minio_adapter() -> MinioAdapter:
"""Provide a MinioAdapter with an injected mock client."""
mock_client = MagicMock(spec=Minio)
return MinioAdapter(config=TEST_MINIO_CONFIG, client=mock_client)
@pytest.fixture
def postgres_adapter() -> PostgresAdapter:
"""Provide a PostgresAdapter with an injected mock client."""
mock_client = MagicMock(spec=psycopg.Connection)
mock_cursor = MagicMock()
mock_cursor.__enter__ = MagicMock(return_value=mock_cursor)
mock_cursor.__exit__ = MagicMock(return_value=False)
mock_client.cursor.return_value = mock_cursor
return PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
+74 -1
View File
@@ -5,11 +5,13 @@ from __future__ import annotations
from typing import cast
from unittest.mock import MagicMock, patch
import psycopg
import redis
from python_repositories.adapters.minio_adapter import MinioAdapter
from python_repositories.adapters.postgres_adapter import PostgresAdapter
from python_repositories.adapters.redis_adapter import RedisAdapter
from tests.conftest import TEST_MINIO_CONFIG, TEST_REDIS_CONFIG
from tests.conftest import TEST_MINIO_CONFIG, TEST_POSTGRES_CONFIG, TEST_REDIS_CONFIG
class TestRedisConnectionHealth:
@@ -161,3 +163,74 @@ class TestMinioConnectionHealth:
assert reinjected.is_connected()
assert mock_client.bucket_exists.call_count == 2
class TestPostgresConnectionHealth:
def test_not_connected_when_no_client(self) -> None:
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG)
assert not adapter.is_connected()
def test_connected_when_probe_succeeds(
self, postgres_adapter: PostgresAdapter
) -> None:
assert postgres_adapter.is_connected()
cast(MagicMock, postgres_adapter._client).cursor.assert_called()
def test_stale_connection_when_probe_fails(
self, postgres_adapter: PostgresAdapter
) -> None:
mock_cursor = cast(MagicMock, postgres_adapter._client).cursor.return_value
mock_cursor.execute.side_effect = psycopg.OperationalError("connection lost")
assert not postgres_adapter.is_connected()
def test_cache_hit_avoids_second_probe(
self, postgres_adapter: PostgresAdapter
) -> None:
mock_client = cast(MagicMock, postgres_adapter._client)
with patch(
"python_repositories.adapters.connection_aware_adapter.time.monotonic",
return_value=100.0,
):
assert postgres_adapter.is_connected()
assert postgres_adapter.is_connected()
assert mock_client.cursor.call_count == 1
def test_cache_miss_runs_probe_again(
self, postgres_adapter: PostgresAdapter
) -> None:
mock_client = cast(MagicMock, postgres_adapter._client)
with patch(
"python_repositories.adapters.connection_aware_adapter.time.monotonic",
side_effect=[100.0, 102.0],
):
assert postgres_adapter.is_connected()
assert postgres_adapter.is_connected()
assert mock_client.cursor.call_count == 2
def test_disconnect_clears_cache(self, postgres_adapter: PostgresAdapter) -> None:
mock_client = cast(MagicMock, postgres_adapter._client)
with patch(
"python_repositories.adapters.connection_aware_adapter.time.monotonic",
return_value=100.0,
):
assert postgres_adapter.is_connected()
postgres_adapter.disconnect()
reinjected = PostgresAdapter(
config=postgres_adapter._config,
client=mock_client,
)
with patch(
"python_repositories.adapters.connection_aware_adapter.time.monotonic",
return_value=100.0,
):
assert reinjected.is_connected()
assert mock_client.cursor.call_count == 2
@@ -7,6 +7,24 @@ from python_repositories.interfaces.connection_aware_interface import (
)
class FakeConnection:
"""Plain class that satisfies ConnectionAwareInterface without inheritance."""
def connect(self) -> None:
pass
def disconnect(self) -> None:
pass
def is_connected(self) -> bool:
return True
def accepts_connection_aware(connection: ConnectionAwareInterface) -> None:
"""Type-checking hook for ConnectionAwareInterface structural subtyping."""
connection.is_connected()
def test_instantiation_fails_when_connect_not_implemented() -> None:
"""Test that instantiation fails if connect is not implemented."""
@@ -53,3 +71,10 @@ def test_instantiation_fails_when_is_connected_not_implemented() -> None:
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_structural_subtyping() -> None:
"""Test that a plain class satisfies ConnectionAwareInterface structurally."""
connection: ConnectionAwareInterface = FakeConnection()
accepts_connection_aware(connection)
assert isinstance(connection, ConnectionAwareInterface)
@@ -7,6 +7,24 @@ import pytest
from python_repositories.interfaces.context_aware_interface import ContextAwareInterface
class FakeContextManager:
"""Plain class that satisfies ContextAwareInterface without inheritance."""
def __enter__(self) -> FakeContextManager:
return self
def __exit__(
self, exc_type: type | None, exc_val: object | None, exc_tb: object | None
) -> None:
pass
def accepts_context_aware(context: ContextAwareInterface) -> None:
"""Type-checking hook for ContextAwareInterface structural subtyping."""
with context:
pass
def test_instantiation_fails_when_enter_not_implemented() -> None:
"""Test that instantiation fails if __enter__ is not implemented."""
@@ -33,3 +51,10 @@ def test_instantiation_fails_when_exit_not_implemented() -> None:
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_structural_subtyping() -> None:
"""Test that a plain class satisfies ContextAwareInterface structurally."""
context: ContextAwareInterface = FakeContextManager()
accepts_context_aware(context)
assert isinstance(context, ContextAwareInterface)
@@ -0,0 +1,239 @@
"""Unit tests for FileBackedQueueAdapter."""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from pathlib import Path
import pytest
from python_repositories.adapters.file_backed_queue_adapter import (
FileBackedQueueAdapter,
)
from python_repositories.adapters.memory_queue_adapter import MemoryQueueAdapter
from python_repositories.config.file_queue_config import FileQueueConfig
from python_repositories.interfaces.queue_repository_interface import (
QueueRepositoryInterface,
)
def _config(path: Path, **kwargs: object) -> FileQueueConfig:
return FileQueueConfig(path=path, **kwargs) # type: ignore[arg-type]
def test_implements_interface() -> None:
assert issubclass(FileBackedQueueAdapter, QueueRepositoryInterface)
def test_ops_require_connect(tmp_path: Path) -> None:
queue = FileBackedQueueAdapter(config=_config(tmp_path / "q.jsonl"))
with pytest.raises(RuntimeError, match="not connected"):
queue.enqueue([{"id": 1}])
with pytest.raises(RuntimeError, match="not connected"):
queue.dequeue_batch()
with pytest.raises(RuntimeError, match="not connected"):
queue.size()
with pytest.raises(RuntimeError, match="not connected"):
queue.evict_older_than(datetime.now(UTC))
def test_enqueue_dequeue_persists_and_compacts(tmp_path: Path) -> None:
path = tmp_path / "items.jsonl"
with FileBackedQueueAdapter(config=_config(path, dedup_keys=("id",))) as queue:
queue.enqueue([{"id": 1}, {"id": 2}, {"id": 1}])
assert queue.size() == 2
assert path.is_file()
lines = path.read_text(encoding="utf-8").strip().splitlines()
assert len(lines) == 2
batch = queue.dequeue_batch(max_items=1)
assert batch == [{"id": 1}]
remaining = path.read_text(encoding="utf-8").strip().splitlines()
assert len(remaining) == 1
assert queue.size() == 1
def test_replay_on_connect(tmp_path: Path) -> None:
path = tmp_path / "items.jsonl"
config = _config(path, dedup_keys=("id",), age_key="created_at", max_age_hours=24)
now = datetime.now(UTC)
with FileBackedQueueAdapter(config=config) as queue:
queue.enqueue(
[
{"id": 1, "created_at": now - timedelta(hours=1)},
{"id": 2, "created_at": now - timedelta(minutes=10)},
]
)
with FileBackedQueueAdapter(config=config) as queue:
assert queue.size() == 2
assert [item["id"] for item in queue.dequeue_batch(max_items=10)] == [1, 2]
def test_replay_filters_by_max_age(tmp_path: Path) -> None:
path = tmp_path / "items.jsonl"
now = datetime.now(UTC)
path.write_text(
"\n".join(
[
f'{{"id": "old", "created_at": "{(now - timedelta(hours=30)).isoformat()}"}}',
f'{{"id": "new", "created_at": "{(now - timedelta(hours=1)).isoformat()}"}}',
]
)
+ "\n",
encoding="utf-8",
)
config = _config(path, dedup_keys=("id",), age_key="created_at", max_age_hours=24)
with FileBackedQueueAdapter(config=config) as queue:
assert queue.size() == 1
assert queue.dequeue_batch(max_items=10)[0]["id"] == "new"
def test_connect_skips_corrupt_and_invalid_lines(tmp_path: Path) -> None:
path = tmp_path / "items.jsonl"
path.write_text(
"\n".join(
[
"",
"not-json",
"[1, 2]",
'{"id": 1}',
'{"name": "missing-id"}',
]
)
+ "\n",
encoding="utf-8",
)
config = _config(path, dedup_keys=("id",))
with FileBackedQueueAdapter(config=config) as queue:
assert queue.size() == 1
assert queue.dequeue_batch(max_items=10) == [{"id": 1}]
def test_connect_skips_missing_or_invalid_age_on_replay(tmp_path: Path) -> None:
path = tmp_path / "items.jsonl"
now = datetime.now(UTC)
path.write_text(
"\n".join(
[
'{"id": 1}',
'{"id": 2, "created_at": "not-a-date"}',
f'{{"id": 3, "created_at": "{now.isoformat()}"}}',
]
)
+ "\n",
encoding="utf-8",
)
config = _config(path, dedup_keys=("id",), age_key="created_at")
with FileBackedQueueAdapter(config=config) as queue:
assert queue.size() == 1
assert queue.dequeue_batch(max_items=10)[0]["id"] == 3
def test_connect_is_idempotent(tmp_path: Path) -> None:
path = tmp_path / "items.jsonl"
queue = FileBackedQueueAdapter(config=_config(path))
queue.connect()
queue.enqueue([{"id": 1}])
queue.connect()
assert queue.size() == 1
queue.disconnect()
assert not queue.is_connected()
def test_evict_compacts_file(tmp_path: Path) -> None:
path = tmp_path / "items.jsonl"
config = _config(path, dedup_keys=("id",), age_key="created_at", max_age_hours=24)
now = datetime.now(UTC)
with FileBackedQueueAdapter(config=config) as queue:
queue.enqueue(
[
{"id": 1, "created_at": now - timedelta(hours=2)},
{"id": 2, "created_at": now - timedelta(minutes=5)},
]
)
removed = queue.evict_older_than(now - timedelta(hours=1))
assert removed == 1
assert queue.size() == 1
lines = [line for line in path.read_text(encoding="utf-8").splitlines() if line]
assert len(lines) == 1
def test_auto_evict_on_enqueue(tmp_path: Path) -> None:
path = tmp_path / "items.jsonl"
config = _config(path, dedup_keys=("id",), age_key="created_at", max_age_hours=1)
now = datetime.now(UTC)
with FileBackedQueueAdapter(config=config) as queue:
queue.enqueue([{"id": 1, "created_at": now - timedelta(hours=2)}])
# Item is enqueued then immediately age-evicted.
assert queue.size() == 0
def test_from_env_when_config_omitted(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
path = tmp_path / "from-env.jsonl"
monkeypatch.setenv("FILE_QUEUE_PATH", str(path))
with FileBackedQueueAdapter() as queue:
queue.enqueue([{"id": 1}])
assert queue.size() == 1
def test_injected_memory(tmp_path: Path) -> None:
path = tmp_path / "items.jsonl"
memory = MemoryQueueAdapter(dedup_keys=("id",))
config = _config(path, dedup_keys=("id",))
with FileBackedQueueAdapter(config=config, memory=memory) as queue:
queue.enqueue([{"id": 1}])
assert memory.size() == 1
def test_empty_enqueue_and_dequeue_skip_compact(tmp_path: Path) -> None:
path = tmp_path / "items.jsonl"
with FileBackedQueueAdapter(config=_config(path)) as queue:
queue.enqueue([])
assert queue.dequeue_batch(max_items=10) == []
assert queue.evict_older_than(datetime.now(UTC)) == 0
assert not path.exists() or path.read_text(encoding="utf-8") == ""
def test_reconnect_replays_after_disconnect(tmp_path: Path) -> None:
path = tmp_path / "items.jsonl"
queue = FileBackedQueueAdapter(config=_config(path, dedup_keys=("id",)))
queue.connect()
queue.enqueue([{"id": 1}])
queue.disconnect()
queue.connect()
assert queue.size() == 1
queue.disconnect()
def test_json_default_rejects_unsupported() -> None:
with pytest.raises(TypeError):
FileBackedQueueAdapter._json_default(object())
def test_disconnect_ignores_fsync_oserror(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
path = tmp_path / "items.jsonl"
queue = FileBackedQueueAdapter(config=_config(path))
queue.connect()
def boom(_fd: int) -> None:
raise OSError("fsync failed")
monkeypatch.setattr(
"python_repositories.adapters.file_backed_queue_adapter.os.fsync",
boom,
)
queue.disconnect()
assert not queue.is_connected()
def test_append_items_requires_open_handle(tmp_path: Path) -> None:
path = tmp_path / "items.jsonl"
queue = FileBackedQueueAdapter(config=_config(path))
queue.connect()
queue._append_handle = None # noqa: SLF001
with pytest.raises(RuntimeError, match="append handle is not open"):
queue._append_items([{"id": 1}]) # noqa: SLF001
queue._connected = False # noqa: SLF001
+56
View File
@@ -0,0 +1,56 @@
"""Unit tests for FileQueueConfig."""
from __future__ import annotations
from pathlib import Path
import pytest
from python_repositories.config import FileQueueConfig
def test_defaults() -> None:
config = FileQueueConfig(path=Path("/tmp/queue.jsonl"))
assert config.path == Path("/tmp/queue.jsonl")
assert config.max_age_hours == 24
assert config.dedup_keys == ()
assert config.age_key is None
def test_from_env_loads_path(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("FILE_QUEUE_PATH", "/data/items.jsonl")
monkeypatch.delenv("FILE_QUEUE_MAX_AGE_HOURS", raising=False)
config = FileQueueConfig.from_env(use_dotenv=False)
assert config.path == Path("/data/items.jsonl")
assert config.max_age_hours == 24
def test_from_env_loads_max_age_hours(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("FILE_QUEUE_PATH", "/data/items.jsonl")
monkeypatch.setenv("FILE_QUEUE_MAX_AGE_HOURS", "48")
config = FileQueueConfig.from_env(use_dotenv=False)
assert config.max_age_hours == 48
def test_from_env_raises_when_path_missing(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("FILE_QUEUE_PATH", raising=False)
with pytest.raises(Exception):
FileQueueConfig.from_env(use_dotenv=False)
def test_from_env_respects_custom_env_var_names(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv("CUSTOM_QUEUE_PATH", "/custom/q.jsonl")
monkeypatch.setenv("CUSTOM_MAX_AGE", "12")
config = FileQueueConfig.from_env(
"CUSTOM_QUEUE_PATH",
max_age_hours_env_var_name="CUSTOM_MAX_AGE",
dedup_keys=("id",),
age_key="created_at",
use_dotenv=False,
)
assert config.path == Path("/custom/q.jsonl")
assert config.max_age_hours == 12
assert config.dedup_keys == ("id",)
assert config.age_key == "created_at"
@@ -1,5 +1,6 @@
"""Unit tests for JsonRepositoryInterface."""
from collections.abc import Iterator
from typing import Any
import pytest
@@ -9,6 +10,36 @@ from python_repositories.interfaces.json_repository_interface import (
)
class InMemoryJsonRepo:
"""Plain class that satisfies JsonRepositoryInterface without inheritance."""
def get(self, key: str) -> dict[str, Any] | None:
return None
def set(self, key: str, data: dict[str, Any]) -> None:
pass
def delete(self, key: str) -> None:
pass
def list_keys(self, pattern: str) -> list[str]:
return []
def scan_keys(
self,
pattern: str,
*,
count: int | None = None,
) -> Iterator[str]:
del count
yield from self.list_keys(pattern)
def accepts_json_repo(repo: JsonRepositoryInterface) -> None:
"""Type-checking hook for JsonRepositoryInterface structural subtyping."""
repo.get("key")
def test_instantiation_fails_when_get_not_implemented() -> None:
"""Test that instantiation fails if get is not implemented."""
@@ -104,3 +135,10 @@ def test_scan_keys_defaults_to_list_keys() -> None:
repository = Complete()
assert list(repository.scan_keys("user")) == ["user-1", "user-2"]
def test_structural_subtyping() -> None:
"""Test that a plain class satisfies JsonRepositoryInterface structurally."""
repo: JsonRepositoryInterface = InMemoryJsonRepo()
accepts_json_repo(repo)
assert isinstance(repo, JsonRepositoryInterface)
+149
View File
@@ -0,0 +1,149 @@
"""Unit tests for MemoryQueueAdapter."""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
import threading
import pytest
from python_repositories.adapters.memory_queue_adapter import MemoryQueueAdapter
from python_repositories.interfaces.queue_repository_interface import (
QueueRepositoryInterface,
)
def test_implements_interface() -> None:
assert issubclass(MemoryQueueAdapter, QueueRepositoryInterface)
def test_fifo_without_dedup() -> None:
queue = MemoryQueueAdapter()
queue.enqueue([{"id": 1}, {"id": 2}, {"id": 3}])
assert queue.size() == 3
assert queue.dequeue_batch(max_items=2) == [{"id": 1}, {"id": 2}]
assert queue.dequeue_batch(max_items=10) == [{"id": 3}]
assert queue.dequeue_batch() == []
assert queue.size() == 0
def test_empty_enqueue_is_noop() -> None:
queue = MemoryQueueAdapter()
queue.enqueue([])
assert queue.size() == 0
assert queue.enqueue_and_return_added([]) == []
def test_dedup_keeps_first() -> None:
queue = MemoryQueueAdapter(dedup_keys=("id",))
queue.enqueue([{"id": 1, "v": "a"}, {"id": 1, "v": "b"}, {"id": 2, "v": "c"}])
assert queue.size() == 2
assert queue.dequeue_batch(max_items=10) == [
{"id": 1, "v": "a"},
{"id": 2, "v": "c"},
]
def test_missing_dedup_key_raises() -> None:
queue = MemoryQueueAdapter(dedup_keys=("id",))
with pytest.raises(ValueError, match="missing dedup key"):
queue.enqueue([{"name": "x"}])
def test_missing_age_key_raises() -> None:
queue = MemoryQueueAdapter(age_key="created_at")
with pytest.raises(ValueError, match="missing age key"):
queue.enqueue([{"id": 1}])
def test_evict_older_than() -> None:
queue = MemoryQueueAdapter(age_key="created_at", dedup_keys=("id",))
now = datetime.now(UTC)
queue.enqueue(
[
{"id": 1, "created_at": now - timedelta(hours=2)},
{"id": 2, "created_at": now - timedelta(minutes=30)},
{"id": 3, "created_at": (now - timedelta(hours=3)).isoformat()},
]
)
removed = queue.evict_older_than(now - timedelta(hours=1))
assert removed == 2
assert queue.size() == 1
remaining = queue.dequeue_batch(max_items=10)
assert remaining[0]["id"] == 2
def test_evict_without_age_key_is_noop() -> None:
queue = MemoryQueueAdapter()
queue.enqueue([{"id": 1}])
assert queue.evict_older_than(datetime.now(UTC)) == 0
assert queue.size() == 1
def test_parse_age_rejects_unsupported_type() -> None:
with pytest.raises(ValueError, match="age value must be"):
MemoryQueueAdapter._parse_age(123)
def test_parse_age_aware_datetime() -> None:
from datetime import timezone
eastern = timezone(timedelta(hours=-5))
aware = datetime(2024, 1, 1, 12, 0, 0, tzinfo=eastern)
assert MemoryQueueAdapter._parse_age(aware) == datetime(
2024, 1, 1, 17, 0, 0, tzinfo=UTC
)
def test_parse_age_naive_datetime() -> None:
naive = datetime(2024, 1, 1, 12, 0, 0)
assert MemoryQueueAdapter._parse_age(naive) == datetime(
2024, 1, 1, 12, 0, 0, tzinfo=UTC
)
def test_parse_age_zulu_string() -> None:
assert MemoryQueueAdapter._parse_age("2024-01-01T00:00:00Z") == datetime(
2024, 1, 1, 0, 0, 0, tzinfo=UTC
)
def test_parse_age_naive_string() -> None:
assert MemoryQueueAdapter._parse_age("2024-01-01T00:00:00") == datetime(
2024, 1, 1, 0, 0, 0, tzinfo=UTC
)
def test_dequeue_rejects_negative_max_items() -> None:
queue = MemoryQueueAdapter()
with pytest.raises(ValueError, match="max_items"):
queue.dequeue_batch(max_items=-1)
def test_clear_and_snapshot() -> None:
queue = MemoryQueueAdapter()
queue.enqueue([{"id": 1}, {"id": 2}])
assert queue.snapshot() == [{"id": 1}, {"id": 2}]
queue.clear()
assert queue.size() == 0
assert queue.snapshot() == []
def test_thread_safety_smoke() -> None:
queue = MemoryQueueAdapter(dedup_keys=("id",))
errors: list[BaseException] = []
def worker(start: int) -> None:
try:
for i in range(start, start + 50):
queue.enqueue([{"id": i}])
except BaseException as exc: # noqa: BLE001
errors.append(exc)
threads = [threading.Thread(target=worker, args=(i * 50,)) for i in range(4)]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
assert errors == []
assert queue.size() == 200
@@ -9,6 +9,32 @@ from python_repositories.interfaces.object_repository_interface import (
)
class InMemoryObjectRepo:
"""Plain class that satisfies ObjectRepositoryInterface without inheritance."""
def get(self, object_name: str) -> BytesIO | None:
return None
def put(
self,
object_name: str,
data: BytesIO,
content_type: str = "application/octet-stream",
) -> None:
pass
def delete(self, object_name: str) -> None:
pass
def list_objects(self, prefix: str = "") -> list[str]:
return []
def accepts_object_repo(repo: ObjectRepositoryInterface) -> None:
"""Type-checking hook for ObjectRepositoryInterface structural subtyping."""
repo.get("object")
def test_instantiation_fails_when_get_not_implemented() -> None:
"""Test that instantiation fails if get is not implemented."""
@@ -98,3 +124,10 @@ def test_instantiation_fails_when_list_objects_not_implemented() -> None:
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_structural_subtyping() -> None:
"""Test that a plain class satisfies ObjectRepositoryInterface structurally."""
repo: ObjectRepositoryInterface = InMemoryObjectRepo()
accepts_object_repo(repo)
assert isinstance(repo, ObjectRepositoryInterface)
+237
View File
@@ -0,0 +1,237 @@
"""Unit tests for PostgresAdapter instantiation and injection."""
from __future__ import annotations
from unittest.mock import MagicMock
import psycopg
import pytest
from python_repositories.adapters.postgres_adapter import PostgresAdapter
from python_repositories.interfaces import TableRepositoryInterface
from tests.conftest import TEST_POSTGRES_CONFIG
def _mock_cursor() -> MagicMock:
mock_cursor = MagicMock()
mock_cursor.__enter__ = MagicMock(return_value=mock_cursor)
mock_cursor.__exit__ = MagicMock(return_value=False)
return mock_cursor
def _mock_client(*, probe_raises: Exception | None = None) -> MagicMock:
mock_client = MagicMock(spec=psycopg.Connection)
mock_cursor = _mock_cursor()
if probe_raises is not None:
mock_cursor.execute.side_effect = probe_raises
mock_client.cursor.return_value = mock_cursor
return mock_client
def test_should_adhere_to_interface() -> None:
assert issubclass(PostgresAdapter, TableRepositoryInterface)
_ = PostgresAdapter(config=TEST_POSTGRES_CONFIG)
def test_should_have_logger_when_instantiated() -> None:
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG)
assert hasattr(adapter, "logger")
assert adapter.logger is not None
def test_should_not_be_connected_when_instantiated() -> None:
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG)
assert adapter._client is None
assert not adapter.is_connected()
def test_constructs_with_injected_config_without_env(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.delenv("POSTGRES_URI", raising=False)
monkeypatch.delenv("POSTGRES_TABLE", raising=False)
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG)
assert adapter._config == TEST_POSTGRES_CONFIG
def test_constructs_with_injected_client_without_env(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.delenv("POSTGRES_URI", raising=False)
monkeypatch.delenv("POSTGRES_TABLE", raising=False)
mock_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
assert adapter._client is mock_client
assert adapter._client_injected is True
def test_raises_when_client_provided_without_config() -> None:
mock_client = MagicMock(spec=psycopg.Connection)
with pytest.raises(ValueError, match="config is required"):
PostgresAdapter(client=mock_client)
def test_disconnect_does_not_close_injected_client() -> None:
mock_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
adapter.disconnect()
mock_client.close.assert_not_called()
assert adapter._client is None
class CustomEnvPostgresAdapter(PostgresAdapter):
uri_env_var_name = "CUSTOM_POSTGRES_URI"
table_env_var_name = "CUSTOM_POSTGRES_TABLE"
primary_key_env_var_name = "CUSTOM_POSTGRES_PRIMARY_KEY"
def test_subclass_custom_env_var_names(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("CUSTOM_POSTGRES_URI", "postgresql://custom/mydb")
monkeypatch.setenv("CUSTOM_POSTGRES_TABLE", "items")
monkeypatch.setenv("CUSTOM_POSTGRES_PRIMARY_KEY", "item_id")
adapter = CustomEnvPostgresAdapter()
assert adapter._config.uri == "postgresql://custom/mydb"
assert adapter._config.table == "items"
assert adapter._config.primary_key == "item_id"
def test_connect_with_injected_client_succeeds_when_probe_ok() -> None:
mock_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
adapter.connect()
assert mock_client.cursor.return_value.execute.call_count == 2
def test_connect_with_injected_client_raises_on_probe_failure() -> None:
mock_client = _mock_client(probe_raises=psycopg.OperationalError("connection lost"))
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
with pytest.raises(ConnectionError, match="Could not connect to Postgres"):
adapter.connect()
def test_connect_with_injected_client_skips_validation_when_client_cleared() -> None:
mock_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
adapter.disconnect()
adapter.connect()
mock_client.cursor.assert_not_called()
def test_connect_reconnects_when_existing_client_unhealthy(
monkeypatch: pytest.MonkeyPatch,
) -> None:
stale_client = _mock_client(
probe_raises=psycopg.OperationalError("connection lost")
)
new_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG)
adapter._client = stale_client
monkeypatch.setattr("psycopg.connect", lambda *args, **kwargs: new_client)
adapter.connect()
stale_client.close.assert_called_once()
assert adapter._client is new_client
def test_connect_skips_reconnect_when_already_connected(
monkeypatch: pytest.MonkeyPatch,
) -> None:
healthy_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG)
adapter._client = healthy_client
connect = MagicMock()
monkeypatch.setattr("psycopg.connect", connect)
adapter.connect()
healthy_client.close.assert_not_called()
connect.assert_not_called()
assert adapter._client is healthy_client
def test_fetch_one_raises_value_error_on_invalid_pk() -> None:
mock_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
with pytest.raises(ValueError, match="Primary key must not be None"):
adapter.fetch_one(None)
def test_fetch_one_raises_connection_error_when_not_connected() -> None:
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG)
with pytest.raises(ConnectionError):
adapter.fetch_one("some-id")
def test_upsert_raises_value_error_on_invalid_row() -> None:
mock_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
with pytest.raises(ValueError, match="Row must be a dictionary"):
adapter.upsert("not-a-dict") # type: ignore[arg-type]
def test_upsert_raises_value_error_when_primary_key_missing() -> None:
mock_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
with pytest.raises(ValueError, match="Row must include primary key column 'id'"):
adapter.upsert({"name": "Alice"})
def test_upsert_raises_connection_error_when_not_connected() -> None:
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG)
with pytest.raises(ConnectionError):
adapter.upsert({"id": "alice", "name": "Alice"})
def test_fetch_all_raises_value_error_on_invalid_limit() -> None:
mock_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
with pytest.raises(ValueError, match="Limit must be a non-negative integer"):
adapter.fetch_all(limit=-1)
def test_execute_raises_value_error_on_invalid_sql() -> None:
mock_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
with pytest.raises(ValueError, match="SQL must be a non-empty string"):
adapter.execute("")
def test_execute_raises_value_error_on_invalid_params() -> None:
mock_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
with pytest.raises(ValueError, match="Params must be a tuple"):
adapter.execute("SELECT 1", []) # type: ignore[arg-type]
def test_upsert_with_primary_key_only_uses_do_nothing() -> None:
mock_client = _mock_client()
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG, client=mock_client)
adapter.upsert({"id": "pk-only"})
mock_cursor = mock_client.cursor.return_value.__enter__.return_value
insert_call = mock_cursor.execute.call_args_list[-1]
assert insert_call[0][1] == ("pk-only",)
def test_execute_raises_connection_error_when_not_connected() -> None:
adapter = PostgresAdapter(config=TEST_POSTGRES_CONFIG)
with pytest.raises(ConnectionError):
adapter.execute("SELECT 1")
+69
View File
@@ -0,0 +1,69 @@
"""Unit tests for PostgresConfig."""
from __future__ import annotations
import pytest
from python_repositories.config import PostgresConfig
def _set_required_postgres_env(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("POSTGRES_URI", "postgresql://localhost/mydb")
monkeypatch.setenv("POSTGRES_TABLE", "users")
def test_from_env_loads_all_fields(monkeypatch: pytest.MonkeyPatch) -> None:
_set_required_postgres_env(monkeypatch)
config = PostgresConfig.from_env(use_dotenv=False)
assert config.uri == "postgresql://localhost/mydb"
assert config.table == "users"
assert config.primary_key == "id"
def test_from_env_reads_primary_key_from_env(
monkeypatch: pytest.MonkeyPatch,
) -> None:
_set_required_postgres_env(monkeypatch)
monkeypatch.setenv("POSTGRES_PRIMARY_KEY", "user_id")
config = PostgresConfig.from_env(use_dotenv=False)
assert config.primary_key == "user_id"
def test_from_env_defaults_primary_key_to_id_when_unset(
monkeypatch: pytest.MonkeyPatch,
) -> None:
_set_required_postgres_env(monkeypatch)
monkeypatch.delenv("POSTGRES_PRIMARY_KEY", raising=False)
config = PostgresConfig.from_env(use_dotenv=False)
assert config.primary_key == "id"
def test_from_env_raises_when_uri_missing(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("POSTGRES_URI", raising=False)
monkeypatch.setenv("POSTGRES_TABLE", "users")
with pytest.raises(Exception):
PostgresConfig.from_env(use_dotenv=False)
def test_from_env_raises_when_table_missing(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("POSTGRES_URI", "postgresql://localhost/mydb")
monkeypatch.delenv("POSTGRES_TABLE", raising=False)
with pytest.raises(Exception):
PostgresConfig.from_env(use_dotenv=False)
def test_from_env_respects_custom_env_var_names(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv("CUSTOM_POSTGRES_URI", "postgresql://custom/mydb")
monkeypatch.setenv("CUSTOM_POSTGRES_TABLE", "items")
monkeypatch.setenv("CUSTOM_POSTGRES_PRIMARY_KEY", "item_id")
config = PostgresConfig.from_env(
"CUSTOM_POSTGRES_URI",
"CUSTOM_POSTGRES_TABLE",
"CUSTOM_POSTGRES_PRIMARY_KEY",
use_dotenv=False,
)
assert config.uri == "postgresql://custom/mydb"
assert config.table == "items"
assert config.primary_key == "item_id"
@@ -0,0 +1,100 @@
"""Unit tests for QueueRepositoryInterface."""
from datetime import UTC, datetime
from typing import Any
import pytest
from python_repositories.interfaces.queue_repository_interface import (
QueueRepositoryInterface,
)
class InMemoryQueueRepo:
"""Plain class that satisfies QueueRepositoryInterface without inheritance."""
def enqueue(self, items: list[dict[str, Any]]) -> None:
pass
def dequeue_batch(self, *, max_items: int = 1000) -> list[dict[str, Any]]:
del max_items
return []
def size(self) -> int:
return 0
def evict_older_than(self, cutoff: datetime) -> int:
del cutoff
return 0
def accepts_queue_repo(repo: QueueRepositoryInterface) -> None:
"""Type-checking hook for QueueRepositoryInterface structural subtyping."""
repo.size()
def test_instantiation_fails_when_enqueue_not_implemented() -> None:
class Incomplete(QueueRepositoryInterface):
def dequeue_batch(self, *, max_items: int = 1000) -> list[dict[str, Any]]:
return []
def size(self) -> int:
return 0
def evict_older_than(self, cutoff: datetime) -> int:
return 0
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_dequeue_batch_not_implemented() -> None:
class Incomplete(QueueRepositoryInterface):
def enqueue(self, items: list[dict[str, Any]]) -> None:
pass
def size(self) -> int:
return 0
def evict_older_than(self, cutoff: datetime) -> int:
return 0
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_size_not_implemented() -> None:
class Incomplete(QueueRepositoryInterface):
def enqueue(self, items: list[dict[str, Any]]) -> None:
pass
def dequeue_batch(self, *, max_items: int = 1000) -> list[dict[str, Any]]:
return []
def evict_older_than(self, cutoff: datetime) -> int:
return 0
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_evict_older_than_not_implemented() -> None:
class Incomplete(QueueRepositoryInterface):
def enqueue(self, items: list[dict[str, Any]]) -> None:
pass
def dequeue_batch(self, *, max_items: int = 1000) -> list[dict[str, Any]]:
return []
def size(self) -> int:
return 0
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_structural_subtyping() -> None:
repo: QueueRepositoryInterface = InMemoryQueueRepo()
accepts_queue_repo(repo)
assert isinstance(repo, QueueRepositoryInterface)
assert repo.evict_older_than(datetime.now(UTC)) == 0
@@ -0,0 +1,162 @@
"""Unit tests for TableRepositoryInterface."""
from typing import Any
import pytest
from python_repositories.interfaces.table_repository_interface import (
TableRepositoryInterface,
)
class InMemoryTableRepo:
"""Plain class that satisfies TableRepositoryInterface without inheritance."""
def fetch_one(self, pk: Any) -> dict[str, Any] | None:
return None
def fetch_all(self, *, limit: int | None = None) -> list[dict[str, Any]]:
del limit
return []
def upsert(self, row: dict[str, Any]) -> None:
pass
def delete(self, pk: Any) -> None:
pass
def execute(
self,
sql: str,
params: tuple[Any, ...] = (),
) -> list[dict[str, Any]]:
del sql, params
return []
def accepts_table_repo(repo: TableRepositoryInterface) -> None:
"""Type-checking hook for TableRepositoryInterface structural subtyping."""
repo.fetch_one("pk")
def test_instantiation_fails_when_fetch_one_not_implemented() -> None:
"""Test that instantiation fails if fetch_one is not implemented."""
class Incomplete(TableRepositoryInterface):
def fetch_all(self, *, limit: int | None = None) -> list[dict[str, Any]]:
return []
def upsert(self, row: dict[str, Any]) -> None:
pass
def delete(self, pk: Any) -> None:
pass
def execute(
self,
sql: str,
params: tuple[Any, ...] = (),
) -> list[dict[str, Any]]:
return []
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_fetch_all_not_implemented() -> None:
"""Test that instantiation fails if fetch_all is not implemented."""
class Incomplete(TableRepositoryInterface):
def fetch_one(self, pk: Any) -> dict[str, Any] | None:
return None
def upsert(self, row: dict[str, Any]) -> None:
pass
def delete(self, pk: Any) -> None:
pass
def execute(
self,
sql: str,
params: tuple[Any, ...] = (),
) -> list[dict[str, Any]]:
return []
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_upsert_not_implemented() -> None:
"""Test that instantiation fails if upsert is not implemented."""
class Incomplete(TableRepositoryInterface):
def fetch_one(self, pk: Any) -> dict[str, Any] | None:
return None
def fetch_all(self, *, limit: int | None = None) -> list[dict[str, Any]]:
return []
def delete(self, pk: Any) -> None:
pass
def execute(
self,
sql: str,
params: tuple[Any, ...] = (),
) -> list[dict[str, Any]]:
return []
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_delete_not_implemented() -> None:
"""Test that instantiation fails if delete is not implemented."""
class Incomplete(TableRepositoryInterface):
def fetch_one(self, pk: Any) -> dict[str, Any] | None:
return None
def fetch_all(self, *, limit: int | None = None) -> list[dict[str, Any]]:
return []
def upsert(self, row: dict[str, Any]) -> None:
pass
def execute(
self,
sql: str,
params: tuple[Any, ...] = (),
) -> list[dict[str, Any]]:
return []
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_execute_not_implemented() -> None:
"""Test that instantiation fails if execute is not implemented."""
class Incomplete(TableRepositoryInterface):
def fetch_one(self, pk: Any) -> dict[str, Any] | None:
return None
def fetch_all(self, *, limit: int | None = None) -> list[dict[str, Any]]:
return []
def upsert(self, row: dict[str, Any]) -> None:
pass
def delete(self, pk: Any) -> None:
pass
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_structural_subtyping() -> None:
"""Test that a plain class satisfies TableRepositoryInterface structurally."""
repo: TableRepositoryInterface = InMemoryTableRepo()
accepts_table_repo(repo)
assert isinstance(repo, TableRepositoryInterface)
Generated
+496 -398
View File
File diff suppressed because it is too large Load Diff