How to properly switch from a released dependency to a pre-release

[b-morgenthaler]

Here's a work flow that did not work out quite well for me. Maybe anybody can help with it.

BTW: I am still using pdm in version 2.16.1 currently

At our company, we store the applications and libraries developed in two company internal pypi repositories: one for the released versions and for the pre-released (aka "dev") versions.

The additional, company internal pypi index are configured in pyproject.toml:

[tool.pdm.resolution]
respect-source-order = true
allow-prereleases = false  # set this to true if prereleases like development versions shall be resolved

[[tool.pdm.source]]
name = "company_pypi_default"  # this is just a proxy to official pypi.org/simple index
url = "https://company_server/repository/company_pypi_default/simple"
verify_ssl = false

[[tool.pdm.source]]
name = "company_pypi_internal"
url = "https://company_server/repository/company_pypi_internal/simple"
verify_ssl = false

[[tool.pdm.source]]
# only pre-releases/development versions are stored in the dev pypi repo
name = "company_pypi_internal_dev"
url = "https://company_server/repository/company_pypi_internal_dev/simple"
verify_ssl = false

Given I am developing on an app that depends on an internal library (no prerelease), the dependency is configured as follows:

dependencies = [
    "InternalLibrary>=1.1.0",
]

This works well, InternalLibrary is resolved from company_pypi_internal without any hustle.

But when I want to install the latest "dev" version for InternalLibrary in version 1.1.0, I switched this configuration in pyproject.toml:

[tool.pdm.resolution]
allow-prereleases = true

In the company_pypi_internal_dev, there are pre-releases published like internallibrary==1.1.0.dev4455+g21c532cb.

After changing this, I ran pdm install but the InternalLibrary always got resolved from company_pypi_internal with the released versions, I also tried pdm sync -r or deleting the lock file all together: the prereleases couldn't be resolved.

Am I thinking it wrong? Am I missing a configuration or command to make this scenario work?

Thanks in advance.

Appendix:

Here's the term output from running pdm install after the allow-prereleases = true change:

WARNING: Lockfile hash doesn't match pyproject.toml, packages may be outdated
Updating the lock file...
🔒 Lock failed
ERROR: Unable to find a resolution for internallibrary
Please make sure the package name is correct.
See C:\Users\[user]\AppData\Local\pdm\pdm\Logs\pdm-lock-j1g8h8zz.log for detailed debug log.
[ResolutionImpossible]: Unable to find a resolution
WARNING: Add '-v' to see the detailed traceback

And here's the debug log:

pdm.termui: ======== Start resolving requirements ========
pdm.termui:   pytest>=8.2
pdm.termui:   pytest-cov
pdm.termui:   pytest-bdd
pdm.termui:   pytest-timeout
pdm.termui:   black
pdm.termui:   pylint
pdm.termui:   robotframework>=7.0
pdm.termui:   docutils
pdm.termui:   InternalLibrary>=1.1.0
pdm.termui:   python>=3.10.14
pdm.termui:   Adding requirement pytest>=8.2
pdm.termui:   Adding requirement pytest-cov
pdm.termui:   Adding requirement pytest-bdd
pdm.termui:   Adding requirement pytest-timeout
pdm.termui:   Adding requirement black
pdm.termui:   Adding requirement pylint
pdm.termui:   Adding requirement robotframework>=7.0
pdm.termui:   Adding requirement docutils
pdm.termui:   Adding requirement InternalLibrary>=1.1.0
unearth.collector: Collecting links from https://pypi.org/simple/internallibrary/
unearth.evaluator: Skipping link <Link https://pypi.org/simple/internallibrary/ (from None)>: Not a file: 
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Failed to collect links from https://pypi.org/simple/internallibrary/: Client Error(404): Not Found
unearth.collector: Collecting links from https://company_server/repository/company_pypi_default/simple/internallibrary/
unearth.evaluator: Skipping link <Link https://company_server/repository/company_pypi_default/simple/internallibrary/ (from None)>: Not a file: 
unearth.auth: Found index url https://company_server/repository/company_pypi_default/simple/
unearth.collector: Failed to collect links from https://company_server/repository/company_pypi_default/simple/internallibrary/: Client Error(404): Not Found
unearth.collector: Collecting links from https://company_server/repository/company_pypi_internal/simple/internallibrary/
unearth.evaluator: Skipping link <Link https://company_server/repository/company_pypi_internal/simple/internallibrary/ (from None)>: Not a file: 
unearth.auth: Found index url https://company_server/repository/company_pypi_internal/simple/
unearth.collector: Fetching HTML page https://company_server/repository/company_pypi_internal/simple/internallibrary/
unearth.collector: Collecting links from https://company_server/repository/company_pypi_internal_dev/simple/internallibrary/
unearth.evaluator: Skipping link <Link https://company_server/repository/company_pypi_internal_dev/simple/internallibrary/ (from None)>: Not a file: 
unearth.auth: Found index url https://company_server/repository/company_pypi_internal_dev/simple/
unearth.collector: Fetching HTML page https://company_server/repository/company_pypi_internal_dev/simple/internallibrary/
pdm.termui: 	Could not find any matching candidates.
pdm.termui: Error occurs
Traceback (most recent call last):
  File "C:\Users\[user]\micromamba\envs\rf-internallibrary\Lib\site-packages\pdm\termui.py", line 257, in logging
    yield logger
  File "C:\Users\[user]\micromamba\envs\rf-internallibrary\Lib\site-packages\pdm\cli\actions.py", line 124, in do_lock
    raise ResolutionImpossible("Unable to find a resolution") from None
resolvelib.resolvers.ResolutionImpossible: Unable to find a resolution

[b-morgenthaler]

To some extend, I can answer my question by myself:

Development versions are handled like pre-releases (see https://packaging.python.org/en/latest/discussions/versioning/): InternalLibrary in version 1.1.0.devN is a "pre" release to version 1.1.0 (not "post" releases) which means it is not "greater than" version 1.1.0. So for my example above to work, I need a development version like 1.1.1.devN. I did this, but still the aforementioned development version of InternalLibrary is not resolved when allowing prereleases.

It only worked when I changed my dependency from

dependencies = [
    "InternalLibrary>=1.1.0",
]

to

dependencies = [
    "InternalLibrary>1.1.0",
]

Please note the difference in >= compared to >. This deviates from what Python's packaging module is resolving:

from packaging.version import Version
>>> Version("1.1.0.dev0") >= Version("1.1.0")
False
>>> Version("1.1.1.dev0") >= Version("1.1.0")
True

Can anybody tell me if this is expected (to have to change the dependency even though Python resolves it differently?

[pawamoy]

It might be because PDM stops at the first index that satisfies the dependency specifier, in the order you configured them.

dependencies = [
    "InternalLibrary>=1.1.0",
]

• look into release index, find 1.1.0, satisfied

dependencies = [
    "InternalLibrary>1.1.0",
]

• look into release index, find 1.1.0, not satisfied
• look into dev index, find 1.1.1.dev0, satisfied

Try moving the dev index above the release one in pyproject.toml and see if you get the behavior you want 🤷

[b-morgenthaler]

Yes you are right, the order of source repository index makes the difference so I can leave the original dependency specifier >=. If that's expected behavior, that's fine for me because I now know how to handle/configure it properly but that's a lot of implicit knowledge there.

Thanks.

[frostming]

https://pdm-project.org/en/latest/usage/config/#respect-the-order-of-the-sources

[b-morgenthaler]

Thanks for the referencing. I knew this chapter but always thought the order preference was about equal packages, so if two pypi index repos (a public one and a private one) provide the same version than the first one in the order is preferred to download. But now that I know more about versioning and resolving, it makes more sense.

[b-morgenthaler]

Sorry that I have to dig into this again:

I tried to reproduce what I have learned from this discussion.

So I changed the order back to my original one (dev is added after release repo) and made sure the source order is not respected. Unfortunately, if respect-source-order = false and allow-prereleases = true the development version is not resolved by calling pdm install. I have to manually change the dependency specifier to '>' to get the development version specified.