Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make cloud providers dynamic #15537

Merged
merged 3 commits into from
Oct 23, 2024

Conversation

djyasin
Copy link
Member

@djyasin djyasin commented Sep 17, 2024

SUMMARY
ISSUE TYPE
  • New or Enhanced Feature
COMPONENT NAME
  • API
AWX VERSION
awx: 24.6.2
ADDITIONAL INFORMATION

#####Testing Steps:

  • Can view dynamically generated cloud providers here: /api/v2/inventory_sources/
  • If you want to test to see where they are being pulled from, can comment out compute_cloud_inventory_sources() and feed a static dict of constants there instead.

@djyasin djyasin marked this pull request as draft September 17, 2024 13:29
@djyasin djyasin changed the title Make cloud providers dynamic 28722 [WIP]Make cloud providers dynamic Sep 17, 2024
awx/main/constants.py Outdated Show resolved Hide resolved
@webknjaz
Copy link
Member

Let's untangle this ball of yarn...

PYTHONDONTWRITEBYTECODE=1 py.test -p no:cacheprovider --create-db --cov --cov-report=xml --junitxml=reports/junit.xml awx/main/tests/unit awx/main/tests/functional awx/conf/tests awx/sso/tests

So this is what's failing ^

And below is a traceback chain:

Traceback (most recent call last):
  File "/usr/lib64/python3.11/logging/config.py", line 400, in resolve
    found = getattr(found, frag)
            ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'awx.main' has no attribute 'utils'

This one is not fully clear where it originates. Perhaps, it's pytest's collection stage.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.11/logging/config.py", line 552, in configure
    formatters[name] = self.configure_formatter(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/logging/config.py", line 664, in configure_formatter
    result = self.configure_custom(config)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/logging/config.py", line 479, in configure_custom
    c = self.resolve(c)
        ^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/logging/config.py", line 402, in resolve
    self.importer(used)
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/assertion/rewrite.py", line 184, in exec_module
    exec(co, module.__dict__)
  File "/awx_devel/awx/main/utils/__init__.py", line 14, in <module>
    from awx.main.utils.licensing import get_licenser  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/assertion/rewrite.py", line 184, in exec_module
    exec(co, module.__dict__)
  File "/awx_devel/awx/main/utils/licensing.py", line 38, in <module>
    from awx.main.constants import SUBSCRIPTION_USAGE_MODEL_UNIQUE_HOSTS
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/assertion/rewrite.py", line 184, in exec_module
    exec(co, module.__dict__)
  File "/awx_devel/awx/main/constants.py", line 8, in <module>
    from awx.main.models.inventory import InventorySourceOptions
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/assertion/rewrite.py", line 184, in exec_module
    exec(co, module.__dict__)
  File "/awx_devel/awx/main/models/__init__.py", line 12, in <module>
    from ansible_base.resource_registry.fields import AnsibleResourceField
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/assertion/rewrite.py", line 184, in exec_module
    exec(co, module.__dict__)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/resource_registry/fields.py", line 1, in <module>
    from django.contrib.contenttypes.models import ContentType
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/assertion/rewrite.py", line 184, in exec_module
    exec(co, module.__dict__)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/contrib/contenttypes/models.py", line 139, in <module>
    class ContentType(models.Model):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/base.py", line 129, in __new__
    app_config = apps.get_containing_app_config(module)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/apps/registry.py", line 260, in get_containing_app_config
    self.check_apps_ready()
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/apps/registry.py", line 138, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

This is an import happening before the django things got initialized ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/bin/py.test", line 8, in <module>
    sys.exit(console_main())
             ^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/config/__init__.py", line 201, in console_main
    code = main()
           ^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/config/__init__.py", line 156, in main
    config = _prepareconfig(args, plugins)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/config/__init__.py", line 341, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/helpconfig.py", line 105, in pytest_cmdline_parse
    config = yield
             ^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/config/__init__.py", line 1140, in pytest_cmdline_parse
    self.parse(args)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/config/__init__.py", line 1494, in parse
    self._preparse(args, addopts=addopts)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/config/__init__.py", line 1398, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/warnings.py", line 151, in pytest_load_initial_conftests
    return (yield)
            ^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/capture.py", line 154, in pytest_load_initial_conftests
    yield
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pytest_django/plugin.py", line 361, in pytest_load_initial_conftests
    _setup_django(early_config)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/pytest_django/plugin.py", line 237, in _setup_django
    django.setup()
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/log.py", line 76, in configure_logging
    logging_config_func(logging_settings)
  File "/usr/lib64/python3.11/logging/config.py", line 823, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/lib64/python3.11/logging/config.py", line 555, in configure
    raise ValueError('Unable to configure '
ValueError: Unable to configure formatter 'json'

And this is the last one that cascaded from the previous two ^

I see pytest-django in the path and that's what's likely attempting to load things early...

I think that before this new import from awx.main.models.inventory import InventorySourceOptions, the respective code path wasn't being hit and the imports happened to be in the right order in tests.

awx/main/apps.py Outdated Show resolved Hide resolved
awx/main/constants.py Outdated Show resolved Hide resolved
awx/main/constants.py Outdated Show resolved Hide resolved
awx/main/constants.py Outdated Show resolved Hide resolved
awx/main/apps.py Outdated Show resolved Hide resolved
awx/api/serializers.py Outdated Show resolved Hide resolved
awx/main/apps.py Outdated Show resolved Hide resolved
awx/main/utils/plugins.py Outdated Show resolved Hide resolved
awx/main/utils/plugins.py Outdated Show resolved Hide resolved
awx/main/utils/plugins.py Outdated Show resolved Hide resolved
awx/api/serializers.py Outdated Show resolved Hide resolved
awx/main/apps.py Outdated Show resolved Hide resolved
awx/main/models/base.py Outdated Show resolved Hide resolved
awx/main/utils/plugins.py Outdated Show resolved Hide resolved
awx/main/utils/plugins.py Outdated Show resolved Hide resolved
awx/main/utils/plugins.py Outdated Show resolved Hide resolved
@djyasin djyasin marked this pull request as ready for review October 16, 2024 22:52
@djyasin djyasin changed the title [WIP]Make cloud providers dynamic Make cloud providers dynamic Oct 16, 2024
@djyasin djyasin requested review from webknjaz, thedoubl3j and chrismeyersfsu and removed request for webknjaz October 16, 2024 22:53
@djyasin djyasin force-pushed the make_cloud_providers_dynamic_28722 branch from 44dbc32 to d6d35bb Compare October 17, 2024 21:04
Copy link
Member

@webknjaz webknjaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The failing test https://github.com/ansible/awx/actions/runs/11393017089/job/31700397825?pr=15537#step:4:427 is related to the changes in this PR: it expects a mapping of inventory plugins to their description as it is on devel, but the API responds with just a list of plugin names w/o descriptions.

@djyasin djyasin force-pushed the make_cloud_providers_dynamic_28722 branch from 329d817 to a81a8e1 Compare October 22, 2024 15:51
awx/main/utils/plugins.py Outdated Show resolved Hide resolved
awx/main/utils/plugins.py Outdated Show resolved Hide resolved
awx/api/serializers.py Outdated Show resolved Hide resolved
awx/main/utils/plugins.py Outdated Show resolved Hide resolved
@djyasin djyasin force-pushed the make_cloud_providers_dynamic_28722 branch from c45f079 to 0e30261 Compare October 22, 2024 17:45
djyasin and others added 3 commits October 22, 2024 15:48
… tests

Add corresponding migrations

Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>
@djyasin djyasin force-pushed the make_cloud_providers_dynamic_28722 branch from 0e30261 to 1aece92 Compare October 22, 2024 19:48
Copy link

sonarcloud bot commented Oct 22, 2024

@djyasin djyasin merged commit e21dd0a into ansible:devel Oct 23, 2024
20 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants