Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/_check_docstrings.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ jobs:
python-version: ${{ env.PYTHON_VERSION }}

- name: Install dependencies
run: make install-dev
run: uv run poe install-dev

- name: Async docstrings check
run: make check-async-docstrings
run: uv run poe check-async-docstrings
9 changes: 3 additions & 6 deletions .github/workflows/_release_docs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -68,13 +68,10 @@ jobs:
python-version: ${{ env.PYTHON_VERSION }}

- name: Install Python dependencies
run: make install-dev
run: uv run poe install-dev

- name: Build generated API reference
run: make build-api-reference

- name: Build Docusaurus docs
run: make build-docs
- name: Build docs
run: uv run poe build-docs
env:
APIFY_SIGNING_TOKEN: ${{ secrets.APIFY_SIGNING_TOKEN }}

Expand Down
6 changes: 3 additions & 3 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,18 @@ repos:
hooks:
- id: lint-check
name: Lint check
entry: make lint
entry: uv run poe lint
language: system
pass_filenames: false

- id: type-check
name: Type check
entry: make type-check
entry: uv run poe type-check
language: system
pass_filenames: false

- id: docstrings-check
name: Docstrings check
entry: make check-async-docstrings
entry: uv run poe check-async-docstrings
language: system
pass_filenames: false
103 changes: 48 additions & 55 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,43 @@ For local development, it is required to have Python 3.10 (or a later version) i

We use [uv](https://docs.astral.sh/uv/) for project management. Install it and set up your IDE accordingly.

We use [Poe the Poet](https://poethepoet.natn.io/) as a task runner, similar to npm scripts in `package.json`.
All tasks are defined in `pyproject.toml` under `[tool.poe.tasks]` and can be run with `uv run poe <task>`.

### Available tasks

| Task | Description |
| ---- | ----------- |
| `install-dev` | Install development dependencies |
| `check-code` | Run lint, type-check, unit-tests, and docstring check |
| `lint` | Run linter |
| `format` | Fix lint issues and format code |
| `type-check` | Run type checker |
| `unit-tests` | Run unit tests |
| `unit-tests-cov` | Run unit tests with coverage |
| `integration-tests` | Run integration tests |
| `integration-tests-cov` | Run integration tests with coverage |
| `check-async-docstrings` | Check async client docstrings |
| `fix-async-docstrings` | Fix async client docstrings |
| `build-docs` | Build documentation website |
| `run-docs` | Run documentation website locally |
| `build` | Build package |
| `clean` | Remove build artifacts and clean caches |

## Dependencies

To install this package and its development dependencies, run:

```sh
make install-dev
uv run poe install-dev
```

## Code checking

To execute all code checking tools together, run:

```sh
make check-code
uv run poe check-code
```

### Linting
Expand All @@ -31,7 +54,7 @@ We utilize [ruff](https://docs.astral.sh/ruff/) for linting, which analyzes code
To run linting:

```sh
make lint
uv run poe lint
```

### Formatting
Expand All @@ -41,7 +64,7 @@ Our automated code formatting also leverages [ruff](https://docs.astral.sh/ruff/
To run formatting:

```sh
make format
uv run poe format
```

### Type checking
Expand All @@ -51,81 +74,51 @@ Type checking is handled by [ty](https://docs.astral.sh/ty/), verifying code aga
To run type checking:

```sh
make type-check
uv run poe type-check
```

### Unit tests

We employ pytest as our testing framework, equipped with various plugins. Check pyproject.toml for configuration details and installed plugins.

We use [pytest](https://docs.pytest.org/) as a testing framework with many plugins. Check `pyproject.toml` for configuration details and installed plugins.

To run unit tests:

```sh
make unit-tests
uv run poe unit-tests
```

To run unit tests with HTML coverage report:
To run unit tests with XML coverage report:

```sh
make unit-tests-cov
uv run poe unit-tests-cov
```

## Integration tests

We have integration tests which build and run Actors using the Python SDK on the Apify Platform. To run these tests,
you need to set the `APIFY_TEST_USER_API_TOKEN` environment variable to the API token of the Apify user you want to
use for the tests, and then start them with `make integration-tests`.

For subset of integration tests another token is needed `APIFY_TEST_USER_2_API_TOKEN`. Such tests are testing
the storage restricted access and thus need two user accounts.

If you want to run the integration tests on a different environment than the main Apify Platform, you need to set
the `APIFY_INTEGRATION_TESTS_API_URL` environment variable to the right URL to the Apify API you want to use.

## Documentation

We adhere to the [Google docstring format](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) for documenting our codebase. Every user-facing class or method is documented. Documentation standards are enforced using [Ruff](https://docs.astral.sh/ruff/).
We have integration tests which interact with the Apify platform.

Our API documentation is generated from these docstrings using [pydoc-markdown](https://pypi.org/project/pydoc-markdown/) with additional post-processing. Markdown files in the `docs/` folder complement the autogenerated content. Final documentation is rendered using [Docusaurus](https://docusaurus.io/) and published to GitHub Pages.
Prerequisites:

To run the documentation locally, you need to have Node.js version 20 or higher installed. Once you have the correct version of Node.js, follow these steps:
- Set `APIFY_TEST_USER_API_TOKEN` to your Apify API token
- Set `APIFY_TEST_USER_2_API_TOKEN` for storage access tests (requires two accounts)
- Optionally set `APIFY_INTEGRATION_TESTS_API_URL` to use a different Apify API environment

Navigate to the `website/` directory:
To run integration tests:

```sh
cd website/
uv run poe integration-tests
```

Enable Corepack, which installs Yarn automatically:

```sh
corepack enable
```

Build the API reference:

```sh
./build_api_reference.sh
```

Install the necessary dependencies:

```sh
yarn
```
## Documentation

Start the project in development mode with Hot Module Replacement (HMR):
We adhere to the [Google docstring format](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) for documenting our codebase. Every user-facing class or method is documented. Documentation standards are enforced using [Ruff](https://docs.astral.sh/ruff/).

```sh
yarn start
```
Our API documentation is generated from these docstrings using [pydoc-markdown](https://pypi.org/project/pydoc-markdown/) with additional post-processing. Markdown files in the `docs/` folder complement the autogenerated content. Final documentation is rendered using [Docusaurus](https://docusaurus.io/) and published to GitHub Pages.

Or using `make`:
To run the documentation locally (requires Node.js):

```sh
make run-doc
uv run poe run-docs
```

## Release process
Expand All @@ -150,14 +143,14 @@ name = "apify_client"
version = "x.z.y"
```

4. Generate the distribution archives for the package:
4. Build the package:

```shell
uv build
```sh
uv run poe build
```

5. Set up the PyPI API token for authentication and upload the package to PyPI:
5. Upload to PyPI:

```shell
```sh
uv publish --token YOUR_API_TOKEN
```
78 changes: 0 additions & 78 deletions Makefile

This file was deleted.

33 changes: 33 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ dependencies = [
dev = [
"dycw-pytest-only<3.0.0",
"griffe",
"poethepoet<1.0.0",
"pre-commit<5.0.0",
"pydoc-markdown<5.0.0",
"pytest-asyncio<2.0.0",
Expand Down Expand Up @@ -178,3 +179,35 @@ exclude_lines = ["pragma: no cover", "if TYPE_CHECKING:", "assert_never()"]

[tool.ipdb]
context = 7

# Run tasks with: uv run poe <task>
[tool.poe.tasks]
clean = "rm -rf .coverage .pytest_cache .ruff_cache .ty_cache build dist htmlcov"
install-sync = "uv sync --all-extras"
build = "uv build --verbose"
publish-to-pypi = "uv publish --verbose --token ${APIFY_PYPI_TOKEN_CRAWLEE}"
type-check = "uv run ty check"
unit-tests = "uv run pytest --numprocesses=auto --verbose tests/unit"
unit-tests-cov = "uv run pytest --numprocesses=auto --verbose --cov=src/apify_client --cov-report=xml:coverage-unit.xml tests/unit"
integration-tests = "uv run pytest --numprocesses=${INTEGRATION_TESTS_CONCURRENCY:-1} --verbose tests/integration"
integration-tests-cov = "uv run pytest --numprocesses=${INTEGRATION_TESTS_CONCURRENCY:-1} --verbose --cov=src/apify_client --cov-report=xml:coverage-integration.xml tests/integration"
check-async-docstrings = "uv run python scripts/check_async_docstrings.py"
fix-async-docstrings = "uv run python scripts/fix_async_docstrings.py"
check-code = ["lint", "type-check", "unit-tests", "check-async-docstrings"]

[tool.poe.tasks.install-dev]
shell = "uv sync --all-extras && uv run pre-commit install"

[tool.poe.tasks.lint]
shell = "uv run ruff format --check && uv run ruff check"

[tool.poe.tasks.format]
shell = "uv run ruff check --fix && uv run ruff format"

[tool.poe.tasks.build-docs]
shell = "./build_api_reference.sh && npm ci && npm run build"
cwd = "website"

[tool.poe.tasks.run-docs]
shell = "./build_api_reference.sh && npm ci && npm run start"
cwd = "website"
4 changes: 3 additions & 1 deletion scripts/check_async_docstrings.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,9 @@

if found_issues:
print()
print('Issues with async docstrings found. Please fix them manually or by running `make fix-async-docstrings`.')
print(
'Issues with async docstrings found. Please fix them manually or by running `uv run poe fix-async-docstrings`.'
)
sys.exit(1)
else:
print('Success: async method docstrings are in sync with sync method docstrings.')
Loading
Loading