Skip to content

Commit

Permalink
Hotfix: Added celery bug report snippet to CI (#260)
Browse files Browse the repository at this point in the history
* Hotfix: Improved celery bug report documentation

* Hotfix: Added celery bug report to CI
  • Loading branch information
Nusnus authored Mar 24, 2024
1 parent a5d633e commit f20cd46
Show file tree
Hide file tree
Showing 5 changed files with 164 additions and 114 deletions.
33 changes: 33 additions & 0 deletions .github/workflows/examples.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,39 @@ permissions:
contents: read # to fetch code (actions/checkout)

jobs:
celery_bug_report:
runs-on: ${{ matrix.os }}

strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["ubuntu-latest"]

steps:
- name: Install apt packages
if: startsWith(matrix.os, 'ubuntu-')
run: |
sudo apt update
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install dependencies
working-directory: examples
run: |
python -m pip install --upgrade pip pytest-cov
pip install -U "pytest-celery[all]@git+https://github.com/celery/pytest-celery.git"
- name: Run tests
working-directory: examples
timeout-minutes: 5
run: |
pytest -xsv celery_bug_report.py --no-cov
myworker:
runs-on: ${{ matrix.os }}

Expand Down
122 changes: 11 additions & 111 deletions docs/userguide/celery-bug-report.rst
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,12 @@ Inline handlers can be used like this:
Injecting signal handlers is using a similar pattern to adding tasks and can be done according
to the :ref:`signal-handlers-modules-injection` section.

Tasks
~~~~~

The :ref:`default-tasks` can be used out-of-the-box, but you can also add new tasks to the worker by creating a new module
and injecting it into the environment. See :ref:`injecting-tasks` for more information.

Templates
=========

Expand All @@ -283,6 +289,8 @@ Standalone Test Snippet
The following snippet can be used as a starting point for a bug report script. To use it, just copy and paste it into a
new file and run it with pytest.

The snippet is also part of the `CI system <https://github.com/celery/pytest-celery/actions/workflows/examples.yml>`_.

RabbitMQ Management Broker
--------------------------

Expand All @@ -298,117 +306,9 @@ Built-in Worker

We'll use the :ref:`built-in-worker` to use a specific Celery release.

.. code-block:: python
# flake8: noqa
from __future__ import annotations
import pytest
from celery import Celery
from celery.canvas import Signature
from celery.result import AsyncResult
from pytest_celery import RABBITMQ_PORTS
from pytest_celery import CeleryBackendCluster
from pytest_celery import CeleryBrokerCluster
from pytest_celery import CeleryTestSetup
from pytest_celery import RabbitMQContainer
from pytest_celery import RabbitMQTestBroker
from pytest_celery import RedisTestBackend
from pytest_celery import ping
###############################################################################
# RabbitMQ Management Broker
###############################################################################
class RabbitMQManagementTestBroker(RabbitMQTestBroker):
def get_management_url(self) -> str:
"""Opening this link during debugging allows you to see the
RabbitMQ management UI in your browser.
"""
ports = self.container.attrs["NetworkSettings"]["Ports"]
ip = ports["15672/tcp"][0]["HostIp"]
port = ports["15672/tcp"][0]["HostPort"]
return f"http://{ip}:{port}"
@pytest.fixture
def default_rabbitmq_broker_image() -> str:
return "rabbitmq:management"
@pytest.fixture
def default_rabbitmq_broker_ports() -> dict:
# Expose the management UI port
ports = RABBITMQ_PORTS.copy()
ports.update({"15672/tcp": None})
return ports
@pytest.fixture
def celery_rabbitmq_broker(default_rabbitmq_broker: RabbitMQContainer) -> RabbitMQTestBroker:
broker = RabbitMQManagementTestBroker(default_rabbitmq_broker)
yield broker
broker.teardown()
@pytest.fixture
def celery_broker_cluster(celery_rabbitmq_broker: RabbitMQTestBroker) -> CeleryBrokerCluster:
cluster = CeleryBrokerCluster(celery_rabbitmq_broker)
yield cluster
cluster.teardown()
###############################################################################
# Redis Result Backend
###############################################################################
@pytest.fixture
def celery_backend_cluster(celery_redis_backend: RedisTestBackend) -> CeleryBackendCluster:
cluster = CeleryBackendCluster(celery_redis_backend)
yield cluster
cluster.teardown()
@pytest.fixture
def default_redis_backend_image() -> str:
return "redis:latest"
###############################################################################
# Worker Configuration
###############################################################################
@pytest.fixture(scope="session")
def default_worker_celery_log_level() -> str:
return "INFO"
@pytest.fixture(scope="session")
def default_worker_celery_version() -> str:
return "5.2.7"
@pytest.fixture
def default_worker_app(default_worker_app: Celery) -> Celery:
app = default_worker_app
# app.conf... # Add any additional configuration here
return app
###############################################################################
# Bug Reproduction
###############################################################################
def test_issue_1234(celery_setup: CeleryTestSetup):
sig: Signature = ping.s()
res: AsyncResult = sig.delay()
assert res.get() == "pong"
.. literalinclude:: ../../examples/celery_bug_report.py
:language: python
:caption: examples.celery_bug_report.py

Execute with Pytest
###################
Expand Down
14 changes: 13 additions & 1 deletion docs/userguide/default-tasks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,21 @@
The plugin provides a list of built-in celery tasks that can be used out of the box. This page will
list all the available tasks.

To import the tasks, you can use the following code:

.. code-block:: python
from pytest_celery import the, tasks, you, want
or

.. code-block:: python
from pytest_celery.vendors.worker import tasks
.. tip::

The tasks injected into the workers that use the default volume with:
The tasks are injected into the workers that use the default volume with:

.. code-block:: python
Expand Down
106 changes: 106 additions & 0 deletions examples/celery_bug_report.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
from __future__ import annotations

import pytest
from celery import Celery
from celery.canvas import Signature
from celery.result import AsyncResult

from pytest_celery import RABBITMQ_PORTS
from pytest_celery import CeleryBackendCluster
from pytest_celery import CeleryBrokerCluster
from pytest_celery import CeleryTestSetup
from pytest_celery import RabbitMQContainer
from pytest_celery import RabbitMQTestBroker
from pytest_celery import RedisTestBackend
from pytest_celery import ping

###############################################################################
# RabbitMQ Management Broker
###############################################################################


class RabbitMQManagementTestBroker(RabbitMQTestBroker):
def get_management_url(self) -> str:
"""Opening this link during debugging allows you to see the RabbitMQ
management UI in your browser."""
ports = self.container.attrs["NetworkSettings"]["Ports"]
ip = ports["15672/tcp"][0]["HostIp"]
port = ports["15672/tcp"][0]["HostPort"]
return f"http://{ip}:{port}"


@pytest.fixture
def default_rabbitmq_broker_image() -> str:
return "rabbitmq:management"


@pytest.fixture
def default_rabbitmq_broker_ports() -> dict:
# Expose the management UI port
ports = RABBITMQ_PORTS.copy()
ports.update({"15672/tcp": None})
return ports


@pytest.fixture
def celery_rabbitmq_broker(default_rabbitmq_broker: RabbitMQContainer) -> RabbitMQTestBroker:
broker = RabbitMQManagementTestBroker(default_rabbitmq_broker)
yield broker
broker.teardown()


@pytest.fixture
def celery_broker_cluster(celery_rabbitmq_broker: RabbitMQTestBroker) -> CeleryBrokerCluster:
cluster = CeleryBrokerCluster(celery_rabbitmq_broker)
yield cluster
cluster.teardown()


###############################################################################
# Redis Result Backend
###############################################################################


@pytest.fixture
def celery_backend_cluster(celery_redis_backend: RedisTestBackend) -> CeleryBackendCluster:
cluster = CeleryBackendCluster(celery_redis_backend)
yield cluster
cluster.teardown()


@pytest.fixture
def default_redis_backend_image() -> str:
return "redis:latest"


###############################################################################
# Worker Configuration
###############################################################################


@pytest.fixture(scope="session")
def default_worker_celery_log_level() -> str:
return "INFO"


@pytest.fixture(scope="session")
def default_worker_celery_version() -> str:
return "5.2.7"


@pytest.fixture
def default_worker_app(default_worker_app: Celery) -> Celery:
app = default_worker_app
# app.conf... # Add any additional configuration here
return app


###############################################################################
# Bug Reproduction
###############################################################################


def test_issue_1234(celery_setup: CeleryTestSetup):
sig: Signature = ping.s()
res: AsyncResult = sig.delay()
assert res.get() == "pong"
3 changes: 1 addition & 2 deletions src/pytest_celery/vendors/worker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,7 @@ ENV PYTHONDONTWRITEBYTECODE=1
RUN pip install --no-cache-dir --upgrade \
pip \
celery[redis,pymemcache]${WORKER_VERSION:+==$WORKER_VERSION} \
pytest-celery@git+https://github.com/Katz-Consulting-Group/pytest-celery.git@hotfix
# pytest-celery@git+https://github.com/celery/pytest-celery.git@hotfix
pytest-celery@git+https://github.com/celery/pytest-celery.git

# The workdir must be /app
WORKDIR /app
Expand Down

0 comments on commit f20cd46

Please sign in to comment.