Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 36 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@ AGENTS.md — Instructions for AI coding assistants (Claude, Cursor, Copilot, Co
pre-commit install # Required: install git hooks
uv sync --all-extras --all-groups # Install all deps (required for tests)
ollama serve # Start Ollama (required for most tests)
uv run pytest -m "not qualitative" # Skips LLM quality tests (~2 min)
uv run pytest # Full suite (includes LLM quality tests)
uv run pytest # Default: qualitative tests, skip slow tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#369 merge conflict note: the ruff/mypy steps are edited here as well

uv run pytest -m "not qualitative" # Fast tests only (~2 min)
uv run pytest -m slow # Run only slow tests (>5 min)
pytest # Run ALL tests including slow (no config)
uv run ruff format . && uv run ruff check . # Lint & format
```
**Branches**: `feat/topic`, `fix/issue-id`, `docs/topic`
Expand All @@ -28,10 +30,41 @@ uv run ruff format . && uv run ruff check . # Lint & format
| `scratchpad/` | Experiments (git-ignored) |

## 3. Test Markers
All tests and examples use markers to indicate requirements. The test infrastructure automatically skips tests based on system capabilities.

**Backend Markers:**
- `@pytest.mark.ollama` — Requires Ollama running (local, lightweight)
- `@pytest.mark.huggingface` — Requires HuggingFace backend (local, heavy)
- `@pytest.mark.vllm` — Requires vLLM backend (local, GPU required)
- `@pytest.mark.openai` — Requires OpenAI API (requires API key)
- `@pytest.mark.watsonx` — Requires Watsonx API (requires API key)
- `@pytest.mark.litellm` — Requires LiteLLM backend

**Capability Markers:**
- `@pytest.mark.requires_gpu` — Requires GPU
- `@pytest.mark.requires_heavy_ram` — Requires 48GB+ RAM
- `@pytest.mark.requires_api_key` — Requires external API keys
- `@pytest.mark.qualitative` — LLM output quality tests (skipped in CI via `CICD=1`)
- **Unmarked** — Unit tests (may still require Ollama running locally)
- `@pytest.mark.llm` — Makes LLM calls (needs at least Ollama)
- `@pytest.mark.slow` — Tests taking >5 minutes (skipped via `SKIP_SLOW=1`)

**Examples in `docs/examples/`** use comment-based markers for clean code:
```python
# pytest: ollama, llm, requires_heavy_ram
"""Example description..."""

# Your clean example code here
```

Tests/examples automatically skip if system lacks required resources. Heavy examples (e.g., HuggingFace) are skipped during collection to prevent memory issues.

**Default behavior:**
- `uv run pytest` skips slow tests (>5 min) but runs qualitative tests
- Use `pytest -m "not qualitative"` for fast tests only (~2 min)
- Use `pytest -m slow` or `pytest` (without config) to include slow tests

⚠️ Don't add `qualitative` to trivial tests—keep the fast loop fast.
⚠️ Mark tests taking >5 minutes with `slow` (e.g., dataset loading, extensive evaluations).

## 4. Coding Standards
- **Types required** on all core functions
Expand Down
22 changes: 19 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,9 +264,25 @@ uv pip install -e ".[all]" --group dev
pre-commit install
```

You can then run all tests by running `pytest`, or only the CI/CD tests by
running `CICD=1 pytest`. See [test/MARKERS_GUIDE.md](test/MARKERS_GUIDE.md) for
details on running specific test categories (e.g., by backend, resource requirements).
You can then run tests:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#369 merge conflict note: This section is moved to CONTRIBUTING.md

```bash
# Default: qualitative tests, skip slow tests
uv run pytest

# Fast tests only (no qualitative, no slow)
uv run pytest -m "not qualitative"

# Run only slow tests
uv run pytest -m slow

# Run ALL tests including slow (bypass config)
pytest --co -q

# CI/CD mode (skips qualitative tests)
CICD=1 uv run pytest
```

See [test/MARKERS_GUIDE.md](test/MARKERS_GUIDE.md) for details on running specific test categories (e.g., by backend, resource requirements).

Tip: you can bypass the hooks by passing the `-n` flag to `git commit`.
This is sometimes helpful for intermediate commits that you intend to later
Expand Down
2 changes: 2 additions & 0 deletions docs/examples/aLora/101_example.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# pytest: huggingface, requires_heavy_ram, llm

import time

from mellea import MelleaSession
Expand Down
2 changes: 2 additions & 0 deletions docs/examples/agents/react.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# pytest: ollama, llm

import datetime
import inspect
import json
Expand Down
2 changes: 2 additions & 0 deletions docs/examples/agents/react_instruct.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# pytest: ollama, llm

import datetime
import inspect
import json
Expand Down
Loading