Compare commits

...
33 Commits
Author SHA1 Message Date
brian 2bb361e3a3 Merge pull request 'Add public repository interfaces and subclassable adapter CRUD API.' (#11) from cursor/public-repository-api into main
Code Quality Pipeline / code-quality (push) Successful in 30s
Test Python Package / test (push) Successful in 52s
Publish Python Package / build-and-publish (push) Successful in 22s
Reviewed-on: brian/python-repositories#11
2026-06-28 20:05:14 +02:00
Brian Bjarke JensenandCursor c5169df10d Use pre-commit for Prettier in CI and document local hook setup.
Test Python Package / test (pull_request) Successful in 50s
Code Quality Pipeline / code-quality (pull_request) Successful in 31s
Run prettier via pre-commit in code-quality workflow and add README instructions for installing and running hooks during development.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-28 19:56:25 +02:00
Brian Bjarke JensenandCursor 497533f0e3 Fix mypy context manager typing for adapter subclasses.
Code Quality Pipeline / code-quality (pull_request) Failing after 26s
Test Python Package / test (pull_request) Successful in 1m7s
Return Self from __enter__ so with-blocks preserve subclass types, and align mypy python_version with the 3.12 runtime target.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-28 19:42:15 +02:00
Brian Bjarke JensenandCursor e98fd3b90d Add public repository interfaces and subclassable adapter CRUD API.
Code Quality Pipeline / code-quality (pull_request) Failing after 19s
Test Python Package / test (pull_request) Successful in 50s
Define split JSON and object repository ABCs, promote adapter methods to public CRUD, add example domain repositories, modernize interface stubs, and make the package installable for tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-28 10:58:33 +02:00
brian f4280f15c1 Merge pull request 'Remove non-existent Mongo adapter from README.' (#10) from cleanup/remove-mongo-adapter-mentions into main
Test Python Package / test (push) Successful in 1m12s
Code Quality Pipeline / code-quality (push) Successful in 24s
Reviewed-on: brian/python-repositories#10
2026-06-21 23:05:47 +02:00
Brian Bjarke Jensen d9b9fa49a2 updated package settings
Code Quality Pipeline / code-quality (pull_request) Successful in 41s
Test Python Package / test (pull_request) Successful in 1m18s
2026-06-21 22:45:29 +02:00
Brian Bjarke Jensen 9d25f7c722 updated package settings
Code Quality Pipeline / code-quality (pull_request) Failing after 39s
Test Python Package / test (pull_request) Successful in 1m23s
2026-06-21 22:43:14 +02:00
Brian Bjarke Jensen 163581526d updated package settings
Code Quality Pipeline / code-quality (pull_request) Failing after 21s
Test Python Package / test (pull_request) Failing after 12s
2026-06-21 22:39:50 +02:00
Brian Bjarke Jensen 933c4ee7c5 updated package settings
Test Python Package / test (pull_request) Successful in 1m27s
Code Quality Pipeline / code-quality (pull_request) Successful in 41s
2026-06-21 22:34:50 +02:00
Brian Bjarke Jensen 2b0d98f84c updated package settings
Code Quality Pipeline / code-quality (pull_request) Failing after 51s
Test Python Package / test (pull_request) Failing after 2m1s
2026-06-21 22:26:41 +02:00
Brian Bjarke JensenandCursor 729fa48505 Remove non-existent Mongo adapter from README.
Code Quality Pipeline / code-quality (pull_request) Failing after 2m9s
Test Python Package / test (pull_request) Failing after 2m10s
The package only ships Redis and MinIO adapters, so the docs should not reference a mongo extra.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-21 22:17:44 +02:00
Brian Bjarke Jensen fa011be862 Merge pull request 'added support for content type when putting object in minio' (#9) from add-support-for-content-type-in-minio into main
Code Quality Pipeline / code-quality (push) Successful in 26s
Test Python Package / test (push) Successful in 33s
Publish Python Package / build-and-publish (push) Successful in 15s
Dependency Update Bot / update-check (push) Failing after 10s
Security Audit / safety (push) Failing after 2m9s
Reviewed-on: brian/python-repositories#9
2025-09-18 19:40:27 +02:00
Brian Bjarke Jensen 2e9330949c ruff format fix
Code Quality Pipeline / code-quality (pull_request) Successful in 29s
Test Python Package / test (pull_request) Successful in 35s
2025-09-18 19:38:22 +02:00
Brian Bjarke Jensen 3ab8da92fb added support for content type when putting object in minio
Code Quality Pipeline / code-quality (pull_request) Failing after 24s
Test Python Package / test (pull_request) Successful in 1m9s
2025-09-18 19:34:05 +02:00
Brian Bjarke Jensen 1dd604dc2a Merge pull request 'moved import of optional packages into classes' (#6) from make-optional-parts-independent into main
Test Python Package / test (push) Successful in 34s
Publish Python Package / build-and-publish (push) Successful in 15s
Code Quality Pipeline / code-quality (push) Successful in 27s
Reviewed-on: brian/python-repositories#6
2025-09-17 12:38:58 +02:00
Brian Bjarke Jensen e5c4025aeb added graceful handling of missing optional packages
Code Quality Pipeline / code-quality (pull_request) Successful in 26s
Test Python Package / test (pull_request) Successful in 34s
2025-09-17 12:36:22 +02:00
Brian Bjarke Jensen 396cf83a05 code quality fixes
Code Quality Pipeline / code-quality (pull_request) Successful in 30s
Test Python Package / test (pull_request) Successful in 1m12s
2025-09-17 12:14:37 +02:00
Brian Bjarke Jensen 538b26b62f mypy fixed 2025-09-17 12:04:45 +02:00
Brian Bjarke Jensen ee6acc6791 moved import of optional packages into classes
Code Quality Pipeline / code-quality (pull_request) Failing after 1m53s
Test Python Package / test (pull_request) Failing after 22s
2025-09-17 11:44:47 +02:00
Brian Bjarke Jensen 0ca07c02ac Merge pull request 'exposed minio adapter' (#5) from expose-minio-adapter into main
Code Quality Pipeline / code-quality (push) Successful in 36s
Test Python Package / test (push) Successful in 35s
Publish Python Package / build-and-publish (push) Successful in 15s
Reviewed-on: brian/python-repositories#5
2025-09-15 14:44:00 +02:00
Brian Bjarke Jensen 7446e88850 added py.typed mypy config file
Code Quality Pipeline / code-quality (pull_request) Successful in 34s
Test Python Package / test (pull_request) Successful in 40s
2025-09-15 14:15:39 +02:00
Brian Bjarke Jensen d5f437f77c ruff format fix
Code Quality Pipeline / code-quality (pull_request) Successful in 1m21s
Test Python Package / test (pull_request) Successful in 1m30s
2025-09-15 13:59:03 +02:00
Brian Bjarke Jensen 7beca7c398 exposed minio adapter
Code Quality Pipeline / code-quality (pull_request) Failing after 1m9s
Test Python Package / test (pull_request) Successful in 1m29s
2025-09-15 13:38:06 +02:00
Brian Bjarke Jensen 65d8051c57 Merge pull request 'add-minio-adapter' (#4) from add-minio-adapter into main
Test Python Package / test (push) Successful in 25s
Publish Python Package / build-and-publish (push) Successful in 11s
Dependency Update Bot / update-check (push) Failing after 58s
Code Quality Pipeline / code-quality (push) Successful in 17s
Reviewed-on: brian/python-repositories#4
2025-09-15 01:05:17 +02:00
Brian Bjarke Jensen f5af30328d auto install all extra packages
Code Quality Pipeline / code-quality (pull_request) Successful in 18s
Test Python Package / test (pull_request) Successful in 59s
2025-09-15 01:03:34 +02:00
Brian Bjarke Jensen 3260a3ffb1 added new optional package to ci
Code Quality Pipeline / code-quality (pull_request) Failing after 6s
Test Python Package / test (pull_request) Failing after 6s
2025-09-15 00:59:40 +02:00
Brian Bjarke Jensen c7bced4254 ruff lint fixes
Code Quality Pipeline / code-quality (pull_request) Successful in 17s
Test Python Package / test (pull_request) Failing after 12s
2025-09-15 00:56:21 +02:00
Brian Bjarke Jensen 800d702d7a pyupgrade
Test Python Package / test (pull_request) Failing after 11s
Code Quality Pipeline / code-quality (pull_request) Failing after 43s
2025-09-15 00:54:01 +02:00
Brian Bjarke Jensen 5a72507775 ruff format 2025-09-15 00:53:38 +02:00
Brian Bjarke Jensen c45ba3c5d5 mypy fixes 2025-09-15 00:53:26 +02:00
Brian Bjarke Jensen a3b5443e0d added minio adapter and tests
Code Quality Pipeline / code-quality (pull_request) Failing after 46s
Test Python Package / test (pull_request) Failing after 11s
2025-09-15 00:43:43 +02:00
Brian Bjarke Jensen cc99fe5a2f minor corrections 2025-09-15 00:43:32 +02:00
Brian Bjarke Jensen 93fffe14e2 added optional group minio 2025-09-15 00:43:20 +02:00
26 changed files with 2042 additions and 769 deletions
+2 -4
View File
@@ -24,7 +24,7 @@ jobs:
- name: Install dependencies
env:
UV_LINK_MODE: copy
run: uv sync --extra redis
run: uv sync --all-extras
- name: Type check with mypy
run: uv run mypy .
@@ -39,6 +39,4 @@ jobs:
run: uv run pyupgrade --py313-plus $(git ls-files '*.py') && git diff --exit-code
- name: Prettier format
run: |
npm install --save-dev --save-exact prettier
npx prettier --check .
run: uv run pre-commit run prettier --all-files
+1 -1
View File
@@ -24,7 +24,7 @@ jobs:
- name: Install dependencies
env:
UV_LINK_MODE: copy
run: uv sync --extra redis
run: uv sync --all-extras
- name: Run pytest
env:
+106 -9
View File
@@ -1,17 +1,114 @@
# python-repositories
Various python repository interfaces exposed as a python package.
Unified repository interfaces and technology-specific adapters for Python projects.
Subclass an adapter in your own repository to add domain-specific methods while reusing connection management and CRUD operations.
## Architecture
| Layer | Responsibility |
| ---------------- | ----------------------------------------------------------------- |
| **Interfaces** | Abstract contracts for connection, context, and CRUD |
| **Adapters** | Technology-specific base classes (`RedisAdapter`, `MinioAdapter`) |
| **Your project** | Subclass an adapter and add domain methods |
## Optional dependencies
This package supports interacting with multiple different backends:
- Redis
- Mongo
- MinIO
To add support for a specific backend install this package with one or more of these optional packages:
Install with the extras you need:
```bash
uv add python-repositories[redis, mongo, minio]
uv add python-repositories[redis]
uv add python-repositories[minio]
uv add python-repositories[redis,minio]
```
### Redis (`JsonRepositoryInterface`)
Requires Redis with the RedisJSON module (e.g. redis-stack).
| Environment variable | Description |
| -------------------- | ---------------------------------------------------- |
| `REDIS_URI` | Redis connection URL (e.g. `redis://localhost:6379`) |
### MinIO (`ObjectRepositoryInterface`)
| Environment variable | Description |
| -------------------- | ------------------------------------------- |
| `MINIO_ENDPOINT` | MinIO server endpoint |
| `MINIO_ACCESS_KEY` | Access key |
| `MINIO_SECRET_KEY` | Secret key |
| `MINIO_BUCKET` | Bucket name (created on connect if missing) |
## Quick start
### JSON documents with Redis
```python
from python_repositories.examples.user_json_repository import UserJsonRepository
with UserJsonRepository() as repo:
repo.save_user("alice", {"name": "Alice", "email": "alice@example.com"})
user = repo.get_user("alice")
repo.delete_user("alice")
```
### Binary objects with MinIO
```python
from io import BytesIO
from python_repositories.examples.artifact_object_repository import (
ArtifactObjectRepository,
)
with ArtifactObjectRepository() as repo:
repo.store_artifact("report-1", BytesIO(b"pdf bytes here"))
data = repo.get_artifact("report-1")
```
### Subclassing in your own project
```python
from python_repositories import RedisAdapter
class UserRepository(RedisAdapter):
def _key(self, user_id: str) -> str:
return f"user:{user_id}"
def get_user(self, user_id: str) -> dict | None:
return self.get(self._key(user_id))
def save_user(self, user_id: str, user: dict) -> None:
self.set(self._key(user_id), user)
```
## Public API
```python
from python_repositories import (
ConnectionAwareInterface,
ContextAwareInterface,
JsonRepositoryInterface,
ObjectRepositoryInterface,
RedisAdapter,
MinioAdapter,
)
```
## Development
```bash
uv sync --all-extras
uv run pre-commit install # once per clone — runs hooks on git commit
uv run pytest tests/integration/ -v
```
`pre-commit` is included in the dev dependency group. `uv sync` installs the CLI, but git does not run hooks until you install them with `pre-commit install` (one time per clone). After that, commits run the checks defined in [`.pre-commit-config.yaml`](.pre-commit-config.yaml) (ruff, mypy, pyupgrade, prettier, and general file hygiene).
To run all hooks manually without committing:
```bash
uv run pre-commit run --all-files
```
Integration tests require Docker (testcontainers).
+18 -3
View File
@@ -22,22 +22,33 @@ dependencies = [
redis = [
"redis>=6.4.0",
]
minio = [
"minio>=7.2.16",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
[tool.uv.sources]
python-utils = { index = "gitea" }
[[tool.uv.index]]
name = "threadripper-proxpi-cache"
url = "http://10.0.0.2:5001/index/"
url = "https://proxpi.lille-vemmelund.dk/index/"
default = true
[[tool.uv.index]]
name = "gitea"
url = "https://gitea.gt-proj.com/api/packages/brian/pypi/simple/"
url = "https://gitea.lille-vemmelund.dk/api/packages/brian/pypi/simple/"
explicit = true
[tool.mypy]
python_version = "3.10"
python_version = "3.12"
warn_return_any = true # nudge to use stricter types
warn_unused_configs = true # nudge to remove unused configs
disallow_untyped_defs = true # disallow untyped function definitions
@@ -57,6 +68,10 @@ namespace_packages = true # enable namespace packages
module = "testcontainers.*"
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = "minio.*"
ignore_missing_imports = true
[dependency-groups]
dev = [
"mypy>=1.17.1",
+15 -3
View File
@@ -1,6 +1,18 @@
"""python_repositories: Unified repository interfaces and adapters."""
from . import adapters
from . import interfaces
from .adapters import MinioAdapter, RedisAdapter
from .interfaces import (
ConnectionAwareInterface,
ContextAwareInterface,
JsonRepositoryInterface,
ObjectRepositoryInterface,
)
__all__ = ["adapters", "interfaces"]
__all__ = [
"ConnectionAwareInterface",
"ContextAwareInterface",
"JsonRepositoryInterface",
"ObjectRepositoryInterface",
"RedisAdapter",
"MinioAdapter",
]
+9 -1
View File
@@ -1,5 +1,13 @@
from .redis_adapter import RedisAdapter as RedisAdapter
"""
Adapters for various backend repositories (e.g., Redis, Minio).
This module exposes concrete implementations for repository interfaces.
"""
from .redis_adapter import RedisAdapter
from .minio_adapter import MinioAdapter
__all__ = [
"RedisAdapter",
"MinioAdapter",
]
@@ -0,0 +1,227 @@
"""Definition of MinioAdapter class."""
from __future__ import annotations
import os
from io import BytesIO
from importlib.util import find_spec
from typing import Self
import structlog
from python_utils import check_env
from python_repositories.interfaces import (
ConnectionAwareInterface,
ContextAwareInterface,
ObjectRepositoryInterface,
)
# Handle optional dependencies
if find_spec("minio") is not None:
import minio
class MinioAdapter(
ObjectRepositoryInterface,
ContextAwareInterface,
ConnectionAwareInterface,
):
"""Minio adapter exposing basic CRUD functionality."""
endpoint_env_var_name: str = "MINIO_ENDPOINT"
access_key_env_var_name: str = "MINIO_ACCESS_KEY"
secret_key_env_var_name: str = "MINIO_SECRET_KEY"
bucket_env_var_name: str = "MINIO_BUCKET"
chunk_size: int = 5 * 2**20 # 5 MiB
def __init__(self) -> None:
# Setup logger
self.logger = structlog.get_logger(
self.__class__.__name__,
)
# Check environment variables
check_env(
{
self.endpoint_env_var_name,
self.access_key_env_var_name,
self.secret_key_env_var_name,
self.bucket_env_var_name,
},
)
# Prepare internal variables
self._client: minio.Minio | None = None
self._bucket_name: str | None = None
def __enter__(self) -> Self:
"""Enter the context."""
self.connect()
return self
def __exit__(
self, exc_type: type | None, exc_val: object | None, exc_tb: object | None
) -> None:
"""Exit the context."""
ctx_info = {"exc_type": exc_type, "exc_val": exc_val, "exc_tb": exc_tb}
if any(
(
exc_type is not None,
exc_val is not None,
exc_tb is not None,
),
):
self.logger.error("Error while exiting context", **ctx_info)
self.disconnect()
def connect(self) -> None:
"""Connect to the Minio server."""
# Stop if already connected
if self.is_connected:
self.logger.info("Already connected to Minio")
return
# Prepare arguments
endpoint = str(os.getenv(self.endpoint_env_var_name))
access_key = str(os.getenv(self.access_key_env_var_name))
secret_key = str(os.getenv(self.secret_key_env_var_name))
bucket = str(os.getenv(self.bucket_env_var_name))
# Connect client
client = minio.Minio(
endpoint=endpoint,
access_key=access_key,
secret_key=secret_key,
secure=False,
)
# Test the connection by listing buckets (will raise if connection fails)
try:
_ = client.list_buckets()
except Exception as exc: # pylint: disable=broad-except
raise ConnectionError(f"Could not connect to Minio at {endpoint}") from exc
# Ensure bucket exists
if not client.bucket_exists(bucket):
self.logger.info(f"Creating bucket '{bucket}'")
client.make_bucket(bucket)
# Persist information
self._client = client
self._bucket_name = bucket
def disconnect(self) -> None:
"""Disconnect from the Minio server."""
# Close connection
# N.B. Minio client does not have a close method, but we include this for symmetry with other adapters
# Reset client
self._client = None
self._bucket_name = None
@property
def is_connected(self) -> bool:
"""Check if connected to Minio server."""
res = bool(isinstance(self._client, minio.Minio))
self.logger.debug(res)
return res
def put(
self,
object_name: str,
data: BytesIO,
content_type: str = "application/octet-stream",
) -> None:
"""Put an object into the Minio bucket."""
# Check input
if not isinstance(object_name, str) or len(object_name) == 0:
raise ValueError("object_name must be a non-empty string")
if not isinstance(data, BytesIO) or data.getbuffer().nbytes == 0:
raise ValueError("data must be a non-empty BytesIO object")
if not isinstance(content_type, str) or len(content_type) == 0:
raise ValueError("content_type must be a non-empty string")
# Check connection
if self._client is None or not self.is_connected:
raise ConnectionError("Not connected to Minio")
# Prepare buffer for reading
num_bytes = data.getbuffer().nbytes
data.seek(0)
# Send data to bucket
# N.B. bucket name is set when connecting
self._client.put_object(
bucket_name=self._bucket_name,
object_name=object_name,
data=data,
length=num_bytes,
part_size=self.chunk_size,
content_type=content_type,
)
self.logger.debug(
f"Put object '{object_name}' into bucket '{self._bucket_name}'"
)
def get(self, object_name: str) -> BytesIO | None:
"""Get an object from the Minio bucket."""
# Check input
if not isinstance(object_name, str) or len(object_name) == 0:
raise ValueError("object_name must be a non-empty string")
# Check connection
if self._client is None or not self.is_connected:
raise ConnectionError("Not connected to Minio")
# Get data from bucket
# N.B. bucket name is set when connecting
try:
response = self._client.get_object(
bucket_name=self._bucket_name,
object_name=object_name,
)
# Get buffered data
buffer = BytesIO()
while chunk := response.read(self.chunk_size):
buffer.write(chunk)
buffer.seek(0)
self.logger.debug(
f"Got object '{object_name}' from bucket '{self._bucket_name}'"
)
return buffer
except minio.S3Error as exc:
if exc.code == "NoSuchKey":
self.logger.warning(
f"Object '{object_name}' not found in bucket '{self._bucket_name}'"
)
else:
self.logger.error(repr(exc))
except Exception as exc: # pylint: disable=broad-except
self.logger.error(repr(exc))
return None
def delete(self, object_name: str) -> None:
"""Delete an object from the Minio bucket."""
# Check input
if not isinstance(object_name, str) or len(object_name) == 0:
raise ValueError("object_name must be a non-empty string")
# Check connection
if self._client is None or not self.is_connected:
raise ConnectionError("Not connected to Minio")
# Delete object from bucket
# N.B. bucket name is set when connecting
self._client.remove_object(
bucket_name=self._bucket_name,
object_name=object_name,
)
self.logger.debug(
f"Deleted object '{object_name}' from bucket '{self._bucket_name}'"
)
def list_objects(self, prefix: str = "") -> list[str]:
"""List objects in the Minio bucket with an optional prefix."""
# Check input
if not isinstance(prefix, str):
raise ValueError("prefix must be a string")
# Check connection
# N.B. bucket name is set when connecting
if self._client is None or not self.is_connected:
raise ConnectionError("Not connected to Minio")
# List objects in bucket
objects = self._client.list_objects(
bucket_name=self._bucket_name,
prefix=prefix,
recursive=True,
)
object_names = [
obj.object_name for obj in objects if obj.object_name is not None
]
self.logger.debug(
f"Listed {len(object_names)} object(s) in bucket '{self._bucket_name}' with prefix '{prefix}'"
)
return object_names
+17 -12
View File
@@ -1,30 +1,34 @@
"""Definition of RedisAdapter class."""
from __future__ import annotations
from typing import cast
from typing import Self, cast
from importlib.util import find_spec
import os
import structlog
import redis
from redis.commands.json.path import Path as RedisPath
from python_utils import check_env
from python_repositories.interfaces import (
ContextAwareInterface,
ConnectionAwareInterface,
ContextAwareInterface,
JsonRepositoryInterface,
)
# Handle optional dependencies
if find_spec("redis") is not None:
import redis
from redis.commands.json.path import Path as RedisPath
class RedisAdapter(
JsonRepositoryInterface,
ContextAwareInterface,
ConnectionAwareInterface,
):
"""Redis adapter exposing basic CRUD functionality."""
uri_env_var_name: str = "REDIS_URI"
path: str = RedisPath.root_path()
path: str = "." # JSON root path, updated in __init__
encoding: str = "UTF-8"
def __init__(self) -> None:
@@ -36,8 +40,9 @@ class RedisAdapter(
check_env(self.uri_env_var_name)
# Prepare internal variables
self._client: redis.Redis | None = None
self.path: str = RedisPath.root_path()
def __enter__(self) -> RedisAdapter:
def __enter__(self) -> Self:
"""Enter the context."""
self.connect()
return self
@@ -89,7 +94,7 @@ class RedisAdapter(
self.logger.debug(res)
return res
def _set(self, key: str, data: dict) -> None:
def set(self, key: str, data: dict) -> None:
"""Set a JSON object in Redis."""
# Check input
if not isinstance(key, str) or len(key) == 0:
@@ -103,7 +108,7 @@ class RedisAdapter(
self._client.json().set(key, self.path, data)
self.logger.debug(f"Set {key} to {data}")
def _get(self, key: str) -> dict | None:
def get(self, key: str) -> dict | None:
"""Get a JSON object from Redis."""
# Check input
if not isinstance(key, str) or len(key) == 0:
@@ -119,7 +124,7 @@ class RedisAdapter(
self.logger.debug(f"Got {data} from {key}")
return data
def _delete(self, key: str) -> None:
def delete(self, key: str) -> None:
"""Delete data from Redis."""
# Check input
if not isinstance(key, str) or len(key) == 0:
@@ -131,7 +136,7 @@ class RedisAdapter(
self._client.json().delete(key)
self.logger.debug(f"Deleted {key}")
def _list_keys(self, pattern: str) -> list[str]:
def list_keys(self, pattern: str) -> list[str]:
"""List keys in Redis matching a pattern."""
# Check input
if not isinstance(pattern, str) or len(pattern) == 0:
+1
View File
@@ -0,0 +1 @@
"""Example domain repositories built on technology adapters."""
@@ -0,0 +1,23 @@
"""Example domain repository backed by MinIO objects."""
from io import BytesIO
from python_repositories.adapters.minio_adapter import MinioAdapter
class ArtifactObjectRepository(MinioAdapter):
"""Example: domain repository backed by MinIO objects."""
def _object_name(self, artifact_id: str) -> str:
return f"artifacts/{artifact_id}"
def get_artifact(self, artifact_id: str) -> BytesIO | None:
return self.get(self._object_name(artifact_id))
def store_artifact(
self,
artifact_id: str,
data: BytesIO,
content_type: str = "application/octet-stream",
) -> None:
self.put(self._object_name(artifact_id), data, content_type)
@@ -0,0 +1,19 @@
"""Example domain repository backed by Redis JSON."""
from python_repositories.adapters.redis_adapter import RedisAdapter
class UserJsonRepository(RedisAdapter):
"""Example: domain repository backed by Redis JSON."""
def _key(self, user_id: str) -> str:
return f"user:{user_id}"
def get_user(self, user_id: str) -> dict | None:
return self.get(self._key(user_id))
def save_user(self, user_id: str, user: dict) -> None:
self.set(self._key(user_id), user)
def delete_user(self, user_id: str) -> None:
self.delete(self._key(user_id))
@@ -2,8 +2,16 @@ from .connection_aware_interface import (
ConnectionAwareInterface as ConnectionAwareInterface,
)
from .context_aware_interface import ContextAwareInterface as ContextAwareInterface
from .json_repository_interface import (
JsonRepositoryInterface as JsonRepositoryInterface,
)
from .object_repository_interface import (
ObjectRepositoryInterface as ObjectRepositoryInterface,
)
__all__ = [
"ConnectionAwareInterface",
"ContextAwareInterface",
"JsonRepositoryInterface",
"ObjectRepositoryInterface",
]
@@ -9,15 +9,15 @@ class ConnectionAwareInterface(ABC):
@abstractmethod
def connect(self) -> None:
"""Connect to resource."""
raise NotImplementedError()
...
@abstractmethod
def disconnect(self) -> None:
"""Disconnect from resource."""
raise NotImplementedError()
...
@property
@abstractmethod
def is_connected(self) -> bool:
"""Check if connected to resource."""
raise NotImplementedError()
...
@@ -1,21 +1,22 @@
"""Definition of ConnectionAwareInterface abstract base class."""
"""Definition of ContextAwareInterface abstract base class."""
from __future__ import annotations
from abc import ABC, abstractmethod
from typing import Self
class ContextAwareInterface(ABC):
"""Interface that defined context-related methods."""
"""Interface that defines context-related methods."""
@abstractmethod
def __enter__(self) -> ContextAwareInterface:
def __enter__(self) -> Self:
"""Enter the context."""
raise NotImplementedError()
...
@abstractmethod
def __exit__(
self, exc_type: type | None, exc_val: object | None, exc_tb: object | None
) -> None:
"""Exit the context."""
raise NotImplementedError()
...
@@ -0,0 +1,27 @@
"""Definition of JsonRepositoryInterface abstract base class."""
from abc import ABC, abstractmethod
class JsonRepositoryInterface(ABC):
"""Interface that defines JSON document CRUD methods."""
@abstractmethod
def get(self, key: str) -> dict | None:
"""Get a JSON object by key."""
...
@abstractmethod
def set(self, key: str, data: dict) -> None:
"""Set a JSON object by key."""
...
@abstractmethod
def delete(self, key: str) -> None:
"""Delete a JSON object by key."""
...
@abstractmethod
def list_keys(self, pattern: str) -> list[str]:
"""List keys matching a glob pattern."""
...
@@ -0,0 +1,33 @@
"""Definition of ObjectRepositoryInterface abstract base class."""
from abc import ABC, abstractmethod
from io import BytesIO
class ObjectRepositoryInterface(ABC):
"""Interface that defines binary object CRUD methods."""
@abstractmethod
def get(self, object_name: str) -> BytesIO | None:
"""Get an object by name."""
...
@abstractmethod
def put(
self,
object_name: str,
data: BytesIO,
content_type: str = "application/octet-stream",
) -> None:
"""Put an object by name."""
...
@abstractmethod
def delete(self, object_name: str) -> None:
"""Delete an object by name."""
...
@abstractmethod
def list_objects(self, prefix: str = "") -> list[str]:
"""List object names with an optional prefix."""
...
View File
+62 -1
View File
@@ -6,8 +6,14 @@ import pytest
import redis
import structlog
import logging
from minio import Minio
from testcontainers.redis import RedisContainer
from testcontainers.minio import MinioContainer
MINIO_ACCESS_KEY = "minioadmin"
MINIO_SECRET_KEY = "minioadmin"
MINIO_BUCKET = "test-bucket"
@pytest.fixture(scope="session", autouse=True)
@@ -16,6 +22,10 @@ def configure_logging() -> None:
# Configure structlog
structlog.configure(
processors=[
structlog.stdlib.filter_by_level,
structlog.stdlib.add_logger_name,
structlog.stdlib.add_log_level,
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.JSONRenderer(),
],
logger_factory=structlog.stdlib.LoggerFactory(),
@@ -32,7 +42,6 @@ def redis_container() -> Generator[str]:
# Start container
container = RedisContainer(
image="redis/redis-stack:7.2.0-v0",
port=6379,
)
container.start()
# Set environment variable for Redis URI
@@ -46,13 +55,42 @@ def redis_container() -> Generator[str]:
container.stop()
@pytest.fixture(scope="session")
def minio_container() -> Generator[dict[str, str]]:
"""Set up a Minio container for testing and yield the Minio URI."""
# Start container
container = MinioContainer(
image="minio/minio:latest",
access_key=MINIO_ACCESS_KEY,
secret_key=MINIO_SECRET_KEY,
)
container.start()
# Build environment variables dictionary
minio_host = container.get_container_host_ip()
minio_port = container.get_exposed_port(9000)
minio_endpoint = f"{minio_host}:{minio_port}"
env_vars = {
"MINIO_ENDPOINT": minio_endpoint,
"MINIO_ACCESS_KEY": MINIO_ACCESS_KEY,
"MINIO_SECRET_KEY": MINIO_SECRET_KEY,
"MINIO_BUCKET": MINIO_BUCKET,
}
yield env_vars
# Stop container
container.stop()
@pytest.fixture(scope="session", autouse=True)
def set_environment_variables(
redis_container: str,
minio_container: dict[str, str],
) -> Generator[dict[str, str]]:
"""Set environment variables needed for tests."""
# Build environment variables dictionary
env_vars = {"REDIS_URI": redis_container}
env_vars.update(minio_container)
# Set environment variables
for key, value in env_vars.items():
os.environ[key] = value
@@ -78,3 +116,26 @@ def raw_redis_client(redis_container: str) -> Generator[redis.Redis]:
# Cleanup
client.flushall()
client.close()
@pytest.fixture(scope="session")
def raw_minio_client(minio_container: dict[str, str]) -> Generator[Minio]:
"""Provide a raw Minio client connected to the test Minio container."""
# Connect client
client = Minio(
endpoint=minio_container["MINIO_ENDPOINT"],
access_key=minio_container["MINIO_ACCESS_KEY"],
secret_key=minio_container["MINIO_SECRET_KEY"],
secure=False,
)
# Ensure bucket exists
bucket_name = minio_container["MINIO_BUCKET"]
if not client.bucket_exists(bucket_name):
client.make_bucket(bucket_name)
yield client
# Cleanup
objects = client.list_objects(bucket_name, recursive=True)
for obj in objects:
client.remove_object(bucket_name, obj.object_name)
@@ -54,66 +54,3 @@ def test_instantiation_fails_when_is_connected_not_implemented() -> None:
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_connect_raises_not_implemented_if_not_overwritten() -> None:
"""Test that connect raises NotImplementedError if not implemented."""
class Incomplete(ConnectionAwareInterface):
"""A class that does not implement connect."""
def connect(self) -> None:
super().connect() # type: ignore
def disconnect(self) -> None:
pass
@property
def is_connected(self) -> bool:
return False
instance = Incomplete()
with pytest.raises(NotImplementedError):
instance.connect()
def test_disconnect_raises_not_implemented_if_not_overwritten() -> None:
"""Test that disconnect raises NotImplementedError if not implemented."""
class Incomplete(ConnectionAwareInterface):
"""A class that does not implement disconnect."""
def connect(self) -> None:
pass
def disconnect(self) -> None:
super().disconnect() # type: ignore
@property
def is_connected(self) -> bool:
return False
instance = Incomplete()
with pytest.raises(NotImplementedError):
instance.disconnect()
def test_is_connected_raises_not_implemented_if_not_overwritten() -> None:
"""Test that is_connected raises NotImplementedError if not implemented."""
class Incomplete(ConnectionAwareInterface):
"""A class that does not implement is_connected."""
def connect(self) -> None:
pass
def disconnect(self) -> None:
pass
@property
def is_connected(self) -> bool:
return super().is_connected # type: ignore
instance = Incomplete()
with pytest.raises(NotImplementedError):
_ = instance.is_connected
@@ -1,6 +1,7 @@
"""Integration tests for ContextAwareInterface."""
from __future__ import annotations
import pytest
from python_repositories.interfaces.context_aware_interface import ContextAwareInterface
@@ -31,42 +32,3 @@ def test_instantiation_fails_when_exit_not_implemented() -> None:
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_enter_raises_not_implemented_if_not_overwritten() -> None:
"""Test that __enter__ raises NotImplementedError if not implemented."""
class Incomplete(ContextAwareInterface):
"""A class that does not implement __enter__."""
def __enter__(self) -> Incomplete:
super().__enter__() # type: ignore
return self
def __exit__(
self, exc_type: type | None, exc_val: object | None, exc_tb: object | None
) -> None:
pass
instance = Incomplete()
with pytest.raises(NotImplementedError):
instance.__enter__()
def test_exit_raises_not_implemented_if_not_overwritten() -> None:
"""Test that __exit__ raises NotImplementedError if not implemented."""
class Incomplete(ContextAwareInterface):
"""A class that does not implement __exit__."""
def __enter__(self) -> ContextAwareInterface:
return self
def __exit__(
self, exc_type: type | None, exc_val: object | None, exc_tb: object | None
) -> None:
super().__exit__(exc_type, exc_val, exc_tb) # type: ignore
instance = Incomplete()
with pytest.raises(NotImplementedError):
instance.__exit__(None, None, None)
+57
View File
@@ -0,0 +1,57 @@
"""Integration tests for example domain repositories."""
from collections.abc import Generator
from io import BytesIO
import random
import pytest
from python_repositories.examples.artifact_object_repository import (
ArtifactObjectRepository,
)
from python_repositories.examples.user_json_repository import UserJsonRepository
@pytest.fixture(scope="module")
def user_data() -> Generator[dict[str, str]]:
"""Provide sample user data for tests."""
yield {"name": "Alice", "email": "alice@example.com"}
@pytest.fixture(scope="module")
def artifact_data() -> Generator[BytesIO]:
"""Provide sample artifact data for tests."""
yield BytesIO(random.randbytes(2**20))
def test_user_json_repository_save_and_get(
redis_container: str,
user_data: dict[str, str],
) -> None:
"""Test that UserJsonRepository can save and retrieve a user."""
with UserJsonRepository() as repo:
repo.save_user("alice", user_data)
assert repo.get_user("alice") == user_data
def test_user_json_repository_delete(
redis_container: str,
user_data: dict[str, str],
) -> None:
"""Test that UserJsonRepository can delete a user."""
with UserJsonRepository() as repo:
repo.save_user("alice", user_data)
repo.delete_user("alice")
assert repo.get_user("alice") is None
def test_artifact_object_repository_store_and_get(
minio_container: dict[str, str],
artifact_data: BytesIO,
) -> None:
"""Test that ArtifactObjectRepository can store and retrieve an artifact."""
with ArtifactObjectRepository() as repo:
repo.store_artifact("report-1", artifact_data)
received = repo.get_artifact("report-1")
assert received is not None
artifact_data.seek(0)
assert received.read() == artifact_data.read()
@@ -0,0 +1,82 @@
"""Integration tests for JsonRepositoryInterface."""
import pytest
from python_repositories.interfaces.json_repository_interface import (
JsonRepositoryInterface,
)
def test_instantiation_fails_when_get_not_implemented() -> None:
"""Test that instantiation fails if get is not implemented."""
class Incomplete(JsonRepositoryInterface):
"""A class that does not implement get."""
def set(self, key: str, data: dict) -> None:
pass
def delete(self, key: str) -> None:
pass
def list_keys(self, pattern: str) -> list[str]:
return []
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_set_not_implemented() -> None:
"""Test that instantiation fails if set is not implemented."""
class Incomplete(JsonRepositoryInterface):
"""A class that does not implement set."""
def get(self, key: str) -> dict | None:
return None
def delete(self, key: str) -> None:
pass
def list_keys(self, pattern: str) -> list[str]:
return []
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_delete_not_implemented() -> None:
"""Test that instantiation fails if delete is not implemented."""
class Incomplete(JsonRepositoryInterface):
"""A class that does not implement delete."""
def get(self, key: str) -> dict | None:
return None
def set(self, key: str, data: dict) -> None:
pass
def list_keys(self, pattern: str) -> list[str]:
return []
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_list_keys_not_implemented() -> None:
"""Test that instantiation fails if list_keys is not implemented."""
class Incomplete(JsonRepositoryInterface):
"""A class that does not implement list_keys."""
def get(self, key: str) -> dict | None:
return None
def set(self, key: str, data: dict) -> None:
pass
def delete(self, key: str) -> None:
pass
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
+500
View File
@@ -0,0 +1,500 @@
"""Integration tests for the MinioAdapter."""
from collections.abc import Generator
import pytest
from unittest.mock import MagicMock
from minio import Minio
from io import BytesIO
import random
import os
import logging
from minio import S3Error
from python_repositories.adapters.minio_adapter import MinioAdapter
from python_repositories.interfaces import ObjectRepositoryInterface
def same_data(
data_a: BytesIO,
data_b: BytesIO,
) -> bool:
"""Check if two BytesIO-objects contain the same data."""
assert isinstance(data_a, BytesIO)
assert isinstance(data_b, BytesIO)
# prepare for being read
data_a.seek(0)
data_b.seek(0)
# convert to bytes
data_a_bytes = data_a.read()
data_b_bytes = data_b.read()
# compare size
if len(data_a_bytes) != len(data_b_bytes):
logging.error(
"data has different length: %s and %s",
len(data_a_bytes),
len(data_b_bytes),
)
return False
# compare content
if data_a_bytes != data_b_bytes:
logging.error("data has different bytes")
return False
return True
@pytest.fixture(scope="module")
def data() -> Generator[BytesIO]:
"""Provide a sample data bytes for tests."""
# Generate random bytes
random_bytes = random.randbytes(2**21) # 2 MiB
yield BytesIO(random_bytes)
@pytest.fixture(scope="function")
def data_in_minio(
raw_minio_client: Minio,
data: BytesIO,
) -> Generator[tuple[str, BytesIO]]:
"""Fixture to set up a known value in Minio before each test."""
object_name = "test_object"
bucket_name = str(os.getenv("MINIO_BUCKET"))
# Upload object
num_bytes = data.getbuffer().nbytes
data.seek(0)
raw_minio_client.put_object(
bucket_name,
object_name,
data,
length=num_bytes,
part_size=MinioAdapter.chunk_size,
)
# Reset data for reading in tests
data.seek(0)
yield object_name, data
# Cleanup
raw_minio_client.remove_object(bucket_name, object_name)
@pytest.fixture(scope="module")
def minio_adapter() -> Generator[MinioAdapter]:
"""Fixture to provide a connected MinioAdapter instance."""
adapter = MinioAdapter()
adapter.connect()
yield adapter
adapter.disconnect()
@pytest.fixture(scope="function", autouse=True)
def clear_minio(
raw_minio_client: Minio,
) -> None:
"""Fixture to clear all Minio objects before each test."""
bucket_name = str(os.getenv("MINIO_BUCKET"))
# Clear all objects before each test
objects = raw_minio_client.list_objects(bucket_name, recursive=True)
for obj in objects:
if not obj.object_name:
continue
raw_minio_client.remove_object(bucket_name, obj.object_name)
def test_should_adhere_to_interface() -> None:
"""Test that the MinioAdapter adheres to the expected interface."""
assert issubclass(MinioAdapter, ObjectRepositoryInterface)
_ = MinioAdapter()
def test_should_have_logger_when_instantiated() -> None:
"""Test that the MinioAdapter has a logger when instantiated."""
adapter = MinioAdapter()
assert hasattr(adapter, "logger")
def test_should_not_be_connected_when_instantiated() -> None:
"""Test that the MinioAdapter is not connected when instantiated."""
adapter = MinioAdapter()
assert not adapter.is_connected
def test_should_log_info_when_already_connected(
minio_adapter: MinioAdapter,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test that the MinioAdapter logs info when connect is called while already connected."""
with caplog.at_level(logging.INFO):
minio_adapter.connect()
assert "Already connected to Minio" in caplog.text
def test_should_raise_connection_error_when_unable_to_connect(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Test that the MinioAdapter raises a ConnectionError when unable to connect."""
# Arrange
monkeypatch.setenv("MINIO_ENDPOINT", "invalid_uri")
adapter = MinioAdapter()
with pytest.raises(ConnectionError):
adapter.connect()
assert not adapter.is_connected
def test_should_log_info_when_creating_expected_bucket(
raw_minio_client: Minio,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test that the MinioAdapter logs info when creating the expected bucket."""
# Arrange
bucket_name = str(os.getenv("MINIO_BUCKET"))
raw_minio_client.remove_bucket(bucket_name)
adapter = MinioAdapter()
# Act
with caplog.at_level(logging.INFO):
adapter.connect()
# Assert
assert f"Creating bucket '{bucket_name}'" in caplog.text
def test_should_log_error_on_exception_during_exit(
minio_container: dict[str, str],
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test that the MinioAdapter logs an error if an exception occurs during __exit__."""
try:
with MinioAdapter() as adapter:
assert adapter.is_connected
raise ValueError("Simulated error")
except ValueError:
pass # Expected
# Assert error was logged
assert "Error while exiting context" in caplog.text
def test_should_have_context_manager() -> None:
"""Test that the MinioAdapter can be used as a context manager."""
with MinioAdapter() as adapter:
assert adapter._client is not None
assert adapter._client is None
def test_should_get_data(
data_in_minio: tuple[str, BytesIO],
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter can get data from a bucket."""
# Arrange
object_name, expected_data = data_in_minio
# Act
received_data = minio_adapter.get(object_name)
# Assert
assert received_data is not None
assert same_data(expected_data, received_data)
def test_should_get_none_for_nonexistent_object(
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter returns None for a nonexistent object."""
# Arrange
object_name = "nonexistent_object"
# Act
received_data = minio_adapter.get(object_name)
# Assert
assert received_data is None
def test_should_raise_value_error_on_invalid_get_object_name(
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter raises ValueError when getting with an invalid object name."""
# Arrange
invalid_object_names = ["", 123, None]
# Act & Assert
for object_name in invalid_object_names:
with pytest.raises(ValueError):
minio_adapter.get(object_name) # type: ignore
def test_should_raise_connection_error_on_get_when_not_connected(
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter raises ConnectionError when getting while not connected."""
# Arrange
adapter = MinioAdapter() # not connected
# Act & Assert
with pytest.raises(ConnectionError):
adapter.get("some_object")
def test_should_log_warning_when_getting_nonexistent_object(
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test that the MinioAdapter logs a warning when getting a nonexistent object."""
# Arrange
adapter = MinioAdapter()
adapter._client = MagicMock(spec=Minio)
adapter._client.get_object.side_effect = S3Error(
code="NoSuchKey",
message="",
resource="",
request_id="",
host_id="",
response="",
bucket_name="test-bucket",
object_name="missing-object",
)
adapter._bucket_name = "test-bucket"
object_name = "missing-object"
# Act
with caplog.at_level("WARNING"):
result = adapter.get(object_name)
# Assert
assert result is None
assert (
f"Object '{object_name}' not found in bucket '{adapter._bucket_name}'"
in caplog.text
)
def test_should_log_error_when_getting_with_s3error_other_than_no_such_key(
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test that the MinioAdapter logs an error when getting a nonexistent object."""
# Arrange
adapter = MinioAdapter()
adapter._client = MagicMock(spec=Minio)
other_s3error = S3Error(
code="UnhandledError",
message="",
resource="",
request_id="",
host_id="",
response="",
bucket_name="test-bucket",
object_name="missing-object",
)
adapter._client.get_object.side_effect = other_s3error
adapter._bucket_name = "test-bucket"
object_name = "missing-object"
# Act
with caplog.at_level("ERROR"):
result = adapter.get(object_name)
# Assert
assert result is None
assert repr(other_s3error) in caplog.text
def test_should_log_error_when_getting_with_general_exception(
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test that the MinioAdapter logs an error when getting a nonexistent object."""
# Arrange
adapter = MinioAdapter()
adapter._client = MagicMock(spec=Minio)
general_exception = Exception("General failure")
adapter._client.get_object.side_effect = general_exception
adapter._bucket_name = "test-bucket"
object_name = "missing-object"
# Act
with caplog.at_level("ERROR"):
result = adapter.get(object_name)
# Assert
assert result is None
assert repr(general_exception) in caplog.text
def test_should_put_data(
data: BytesIO,
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter can put data into a bucket."""
# Arrange
object_name = "new_test_object"
received_data = minio_adapter.get(object_name)
assert received_data is None # ensure object does not exist yet
# Act
minio_adapter.put(object_name, data)
# Assert
received_data = minio_adapter.get(object_name)
assert received_data is not None
assert same_data(data, received_data)
# Cleanup
bucket_name = str(os.getenv("MINIO_BUCKET"))
minio_adapter._client.remove_object(bucket_name, object_name) # type: ignore
def test_should_update_data(
data_in_minio: tuple[str, BytesIO],
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter can update data in a bucket."""
# Arrange
object_name, _ = data_in_minio
new_data = BytesIO(random.randbytes(2**21)) # 2 MiB
received_data = minio_adapter.get(object_name)
assert received_data is not None
assert not same_data(received_data, new_data)
# Act
minio_adapter.put(object_name, new_data)
# Assert
received_data = minio_adapter.get(object_name)
assert received_data is not None
assert same_data(new_data, received_data)
def test_should_raise_value_error_on_invalid_put_object_name(
data: BytesIO,
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter raises ValueError when putting with an invalid object name."""
# Arrange
invalid_object_names = ["", 123, None]
# Act & Assert
for object_name in invalid_object_names:
with pytest.raises(ValueError):
minio_adapter.put(object_name, data) # type: ignore
def test_should_raise_value_error_on_invalid_put_data(
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter raises ValueError when putting with invalid data."""
# Arrange
object_name = "valid_object_name"
invalid_data = ["not_bytesio", 123, None]
# Act & Assert
for data in invalid_data:
with pytest.raises(ValueError):
minio_adapter.put(object_name, data) # type: ignore
def test_should_raise_value_error_on_invalid_put_content_type(
data: BytesIO,
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter raises ValueError when putting with an invalid content type."""
# Arrange
object_name = "valid_object_name"
invalid_content_types = ["", 123, None]
# Act & Assert
for content_type in invalid_content_types:
with pytest.raises(ValueError):
minio_adapter.put(object_name, data, content_type) # type: ignore
def test_should_raise_connection_error_on_put_when_not_connected(
data: BytesIO,
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter raises ConnectionError when putting while not connected."""
# Arrange
adapter = MinioAdapter() # not connected
# Act & Assert
with pytest.raises(ConnectionError):
adapter.put("some_object", data)
def test_should_delete_object(
data_in_minio: tuple[str, BytesIO],
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter can delete an object from a bucket."""
# Arrange
object_name, _ = data_in_minio
received_data = minio_adapter.get(object_name)
assert received_data is not None # ensure object exists
# Act
minio_adapter.delete(object_name)
# Assert
received_data = minio_adapter.get(object_name)
assert received_data is None
def test_should_raise_value_error_on_invalid_delete_object_name(
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter raises ValueError when deleting with an invalid object name."""
# Arrange
invalid_object_names = ["", 123, None]
# Act & Assert
for object_name in invalid_object_names:
with pytest.raises(ValueError):
minio_adapter.delete(object_name) # type: ignore
def test_should_raise_connection_error_on_delete_when_not_connected(
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter raises ConnectionError when deleting while not connected."""
# Arrange
adapter = MinioAdapter() # not connected
# Act & Assert
with pytest.raises(ConnectionError):
adapter.delete("some_object")
def test_should_list_objects(
data_in_minio: tuple[str, BytesIO],
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter can list objects in a bucket."""
# Arrange
object_name, _ = data_in_minio
new_data = BytesIO(random.randbytes(2**21)) # 2 MiB
new_data_name = "another_test_object"
minio_adapter.put(new_data_name, new_data)
# Act
objects = minio_adapter.list_objects()
# Assert
assert isinstance(objects, list)
assert len(objects) == 2
assert object_name in objects
assert new_data_name in objects
def test_should_list_objects_with_prefix(
data_in_minio: tuple[str, BytesIO],
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter can list objects in a bucket with a prefix."""
# Arrange
object_name, _ = data_in_minio
new_data = BytesIO(random.randbytes(2**21)) # 2 MiB
new_data_name = "prefix_test_object"
minio_adapter.put(new_data_name, new_data)
prefix = "prefix_"
# Act
objects = minio_adapter.list_objects(prefix)
# Assert
assert isinstance(objects, list)
assert len(objects) == 1
assert new_data_name in objects
assert object_name not in objects
def test_should_raise_value_error_on_invalid_list_objects_prefix(
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter raises ValueError when listing with an invalid prefix."""
# Arrange
invalid_prefixes = [123, None]
# Act & Assert
for prefix in invalid_prefixes:
with pytest.raises(ValueError):
minio_adapter.list_objects(prefix) # type: ignore
def test_should_raise_connection_error_on_list_objects_when_not_connected(
minio_adapter: MinioAdapter,
) -> None:
"""Test that the MinioAdapter raises ConnectionError when listing while not connected."""
# Arrange
adapter = MinioAdapter() # not connected
# Act & Assert
with pytest.raises(ConnectionError):
adapter.list_objects()
# allows local debugging by running file as script
if __name__ == "__main__":
pytest.main(["-s", "-v", __file__])
@@ -0,0 +1,99 @@
"""Integration tests for ObjectRepositoryInterface."""
from io import BytesIO
import pytest
from python_repositories.interfaces.object_repository_interface import (
ObjectRepositoryInterface,
)
def test_instantiation_fails_when_get_not_implemented() -> None:
"""Test that instantiation fails if get is not implemented."""
class Incomplete(ObjectRepositoryInterface):
"""A class that does not implement get."""
def put(
self,
object_name: str,
data: BytesIO,
content_type: str = "application/octet-stream",
) -> None:
pass
def delete(self, object_name: str) -> None:
pass
def list_objects(self, prefix: str = "") -> list[str]:
return []
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_put_not_implemented() -> None:
"""Test that instantiation fails if put is not implemented."""
class Incomplete(ObjectRepositoryInterface):
"""A class that does not implement put."""
def get(self, object_name: str) -> BytesIO | None:
return None
def delete(self, object_name: str) -> None:
pass
def list_objects(self, prefix: str = "") -> list[str]:
return []
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_delete_not_implemented() -> None:
"""Test that instantiation fails if delete is not implemented."""
class Incomplete(ObjectRepositoryInterface):
"""A class that does not implement delete."""
def get(self, object_name: str) -> BytesIO | None:
return None
def put(
self,
object_name: str,
data: BytesIO,
content_type: str = "application/octet-stream",
) -> None:
pass
def list_objects(self, prefix: str = "") -> list[str]:
return []
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
def test_instantiation_fails_when_list_objects_not_implemented() -> None:
"""Test that instantiation fails if list_objects is not implemented."""
class Incomplete(ObjectRepositoryInterface):
"""A class that does not implement list_objects."""
def get(self, object_name: str) -> BytesIO | None:
return None
def put(
self,
object_name: str,
data: BytesIO,
content_type: str = "application/octet-stream",
) -> None:
pass
def delete(self, object_name: str) -> None:
pass
with pytest.raises(TypeError):
_ = Incomplete() # type: ignore
+34 -27
View File
@@ -1,6 +1,3 @@
# pylint: disable=protected-access
# The above line disables pylint's protected member access warnings for this file,
# allowing tests to access RedisAdapter's internal methods as needed for integration testing.
"""Integration tests for the RedisAdapter."""
from collections.abc import Generator
@@ -8,9 +5,10 @@ import pytest
import redis
from redis.commands.json.path import Path as RedisPath
from python_repositories.adapters.redis_adapter import RedisAdapter
from python_repositories.interfaces import JsonRepositoryInterface
@pytest.fixture(scope="session")
@pytest.fixture(scope="module")
def data() -> Generator[dict[str, str]]:
"""Provide a sample data dictionary for tests."""
yield {"foo": "bar"}
@@ -50,7 +48,7 @@ def clear_redis(raw_redis_client: redis.Redis) -> None:
def test_should_adhere_to_interface(redis_container: str) -> None:
"""Test that the RedisAdapter adheres to the expected interface."""
# Instantiation fails if interface not adhered to
assert issubclass(RedisAdapter, JsonRepositoryInterface)
_ = RedisAdapter()
@@ -115,7 +113,6 @@ def test_should_log_error_on_exception_during_exit(
raise ValueError("Simulated error")
except ValueError:
pass # Expected
# Assert error was logged
assert "Error while exiting context" in caplog.text
@@ -135,8 +132,9 @@ def test_should_get_value(
# Arrange
key, data = data_in_redis
# Act
value = redis_adapter._get(key)
value = redis_adapter.get(key)
# Assert
assert value is not None
assert value == data
@@ -145,7 +143,7 @@ def test_should_get_none_for_missing_key(
) -> None:
"""Test that getting a non-existent key returns None."""
# Act
value = redis_adapter._get("nonexistent_key")
value = redis_adapter.get("nonexistent_key")
# Assert
assert value is None
@@ -159,7 +157,7 @@ def test_should_raise_value_error_on_invalid_get_key(
# Act & Assert
for key in invalid_keys:
with pytest.raises(ValueError):
redis_adapter._get(key) # type: ignore
redis_adapter.get(key) # type: ignore
def test_should_raise_connection_error_on_get_when_not_connected(
@@ -167,10 +165,10 @@ def test_should_raise_connection_error_on_get_when_not_connected(
) -> None:
"""Test that the RedisAdapter raises ConnectionError when getting while not connected."""
# Arrange
adapter = RedisAdapter()
adapter = RedisAdapter() # not connected
# Act & Assert
with pytest.raises(ConnectionError):
adapter._get("some_key")
adapter.get("some_key")
def test_should_set_value(
@@ -180,10 +178,14 @@ def test_should_set_value(
"""Test that the RedisAdapter can set a value."""
# Arrange
key = "test_key"
received_data = redis_adapter.get(key)
assert received_data is None # Ensure key does not exist
# Act
redis_adapter._set(key, data)
redis_adapter.set(key, data)
# Assert
assert redis_adapter._get(key) == data
received_data = redis_adapter.get(key)
assert received_data is not None
assert received_data == data
def test_should_update_value(
@@ -194,10 +196,13 @@ def test_should_update_value(
# Arrange
key, _ = data_in_redis
new_data = {"new_key": "new_value"}
received_data = redis_adapter.get(key)
assert received_data is not None
assert received_data != new_data
# Act
redis_adapter._set(key, new_data)
redis_adapter.set(key, new_data)
# Assert
assert redis_adapter._get(key) == new_data
assert redis_adapter.get(key) == new_data
def test_should_raise_value_error_on_invalid_set_key(
@@ -210,7 +215,7 @@ def test_should_raise_value_error_on_invalid_set_key(
# Act & Assert
for key in invalid_keys:
with pytest.raises(ValueError):
redis_adapter._set(key, data) # type: ignore
redis_adapter.set(key, data) # type: ignore
def test_should_raise_value_error_on_invalid_set_data(
@@ -223,7 +228,7 @@ def test_should_raise_value_error_on_invalid_set_data(
# Act & Assert
for data in invalid_data:
with pytest.raises(ValueError):
redis_adapter._set(key, data) # type: ignore
redis_adapter.set(key, data) # type: ignore
def test_should_raise_connection_error_on_set_when_not_connected(
@@ -236,7 +241,7 @@ def test_should_raise_connection_error_on_set_when_not_connected(
key = "test_key"
# Act & Assert
with pytest.raises(ConnectionError):
adapter._set(key, data)
adapter.set(key, data)
def test_should_delete_key(
@@ -246,10 +251,12 @@ def test_should_delete_key(
"""Test that deleting a key removes it from Redis."""
# Arrange
key, _ = data_in_redis
received_data = redis_adapter.get(key)
assert received_data is not None # Ensure key exists
# Act
redis_adapter._delete(key)
redis_adapter.delete(key)
# Assert
assert redis_adapter._get(key) is None
assert redis_adapter.get(key) is None
def test_should_raise_value_error_on_invalid_delete_key(
@@ -261,7 +268,7 @@ def test_should_raise_value_error_on_invalid_delete_key(
# Act & Assert
for key in invalid_keys:
with pytest.raises(ValueError):
redis_adapter._delete(key) # type: ignore
redis_adapter.delete(key) # type: ignore
def test_should_raise_connection_error_on_delete_when_not_connected(
@@ -272,7 +279,7 @@ def test_should_raise_connection_error_on_delete_when_not_connected(
adapter = RedisAdapter()
# Act & Assert
with pytest.raises(ConnectionError):
adapter._delete("some_key")
adapter.delete("some_key")
def test_should_list_keys(
@@ -280,10 +287,10 @@ def test_should_list_keys(
) -> None:
"""Test listing keys matching a pattern returns correct keys."""
# Arrange
redis_adapter._set("key1", {"a": 1})
redis_adapter._set("key2", {"b": 2})
redis_adapter.set("key1", {"a": 1})
redis_adapter.set("key2", {"b": 2})
# Act
keys = redis_adapter._list_keys("key*")
keys = redis_adapter.list_keys("key*")
# Assert
assert set(keys) == {"key1", "key2"}
@@ -297,7 +304,7 @@ def test_should_raise_value_error_on_invalid_list_keys_pattern(
# Act & Assert
for pattern in invalid_patterns:
with pytest.raises(ValueError):
redis_adapter._list_keys(pattern) # type: ignore
redis_adapter.list_keys(pattern) # type: ignore
def test_should_raise_connection_error_on_list_keys_when_not_connected(
@@ -308,7 +315,7 @@ def test_should_raise_connection_error_on_list_keys_when_not_connected(
adapter = RedisAdapter()
# Act & Assert
with pytest.raises(ConnectionError):
adapter._list_keys("some_pattern")
adapter.list_keys("some_pattern")
# allows local debugging by running file as script
Generated
+692 -598
View File
File diff suppressed because it is too large Load Diff