REFACTOR: Unite Your Requirements!

[akmalsoliev]

With the arrival of a lightning-fast package manager, mighty UV, the requirements.txt is like yesterday's news. Instead of navigating through a jungle of dependencies, like a mismatched sock drawer, everything can now unite in one glorious spot—just like Italy in 1861!

This marvelous change even makes installing dependencies blazingly fast (except for cudf—polars is better anyways). Now it's as easy as pie with just one command: uv sync --all-extras. No more typing three commands when you can do it in one! 1 > 3 — gives you more time to enjoy that delicious Italian caffè.

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Checklist

• Code follows style guide (ruff)
• Tests added
• Documented the changes

If you have comments or can explain your changes, please do so below.

[akmalsoliev]

P.S. I've updated pyproject.toml and banished those pesky .txt files. More changes are coming; just testing the waters before diving in!

[FBruzzesi]

Hey there! Thanks for your PR. We definitly started to be opinionated to suggest uv in our contribution guide. However I have mixed feelings about this one. Let me try to mention a few reason (sorted by dev experience):

• If someone's workflow is not on uv, then how can he/she/they pick up dev-dependencies?
• uv sync is too strict for my (but I guess many) development settings and to avoid that it doesn't uninstall the rest of the libraries in the env, I need to remember to run uv sync --inexact instead
• For a few set up based on docker (e.g. devcontainers and codespaces), having a file allows to easily copy and install the dependencies before even copying the python package. This leads to easier caching (at least in my experience).

[FBruzzesi]

Maybe Marco recalls why that was the min version

[FBruzzesi]

Not sure why this was resolved, but 2024.7 is indeed the min version we can support, as we need dask_expr.

[akmalsoliev]

Buongiorno Francesco (@FBruzzesi),

Embracing change can indeed be challenging, and that's perfectly normal.

• If someone's workflow is not on uv, then how can he/she/they pick up dev-dependencies?

It's understandable that this might seem like a hurdle. However, I see it as an exciting chance to introduce the efficiency of uv. For those who prefer sticking with pip (those poor souls), I've ensured that the pyproject.toml is structured to allow all optional dependencies to be installed effortlessly with the command:
pip install .[docs,modin,pandas,polars,pyarrow,dask,dev]

• uv sync is too strict for my (but I guess many) development settings, and to avoid that it doesn't uninstall the rest of the libraries in the env, I need to remember to run uv sync --inexact instead.

I understand that might be inconvenient, but this rigidity also promotes consistency in workflows. uv sync --inexact is a helpful option to adapt it to your specific needs.

• For a few setups based on Docker (e.g., devcontainers and Codespaces), having a file allows for easier copying and installation of dependencies before even copying the Python package, which leads to better caching (at least in my experience).

From what I understand, Docker caching should remain intact whether using pyproject.toml or requirements.txt. If the current setup is causing issues, I'm keen to explore alternatives that work better for you. Here’s an example to illustrate:

FROM python:latest

COPY ./pyproject.toml /pyproject.toml

RUN pip install .

RUN echo installing dev && pip install .[dev]

RUN echo installing polars && pip install .[polars]

Please let me know if there's anything specific you find challenging, and I’d be happy to discuss further.

[FBruzzesi]

Embracing change can indeed be challenging, and that's perfectly normal.

Definitely, and we would like to let people use whatever workflow works for them (not only pip) as well. Thanks for adjusting the pyproject section.

From what I understand, Docker caching should remain intact whether using pyproject.toml or requirements.txt. If the current setup is causing issues, I'm keen to explore alternatives that work better for you. Here’s an example to illustrate:

FROM python:latest

COPY ./pyproject.toml /pyproject.toml

RUN pip install .

RUN echo installing dev && pip install .[dev]

RUN echo installing polars && pip install .[polars]

Please let me know if there's anything specific you find challenging, and I’d be happy to discuss further.

The above snippet would run from RUN pip install . each time something changes in the codebase.
What I usually do is something along the lines:

...
COPY ./requirements.txt /requirements.txt

RUN uv pip install -r requirements.txt # this will be cached until something changes in requirements.txt

COPY <rest of codebase> <rest of codebase>

RUN uv pip install .
...

There should be a way to link requirements.txt dynamically into pyproject, so personally that's how I would do it but happy to discuss

[MarcoGorelli]

thanks - you'll need to update the instructions for this to work (e.g. ci, contributing)

fancy making a PR first, so that github automatically runs ci for you, and then coming back to this?

[akmalsoliev]

Ciao @FBruzzesi,
Sorry for late reply, what you think about this setup?

FROM python:3.12-slim-bookworm
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

COPY ./pyproject.toml pyproject.toml

RUN uv venv

RUN uv pip compile pyproject.toml --extra dev > requirements.txt
RUN uv pip install -r requirements.txt

RUN echo hello 

RUN uv pip install -r pyproject.toml --extra docs

COPY . .

RUN uv sync --all-extras 

When trying on the local machine it felt like there was a caching of

RUN uv pip compile pyproject.toml --extra dev > requirements.txt
RUN uv pip install -r requirements.txt

[FBruzzesi]

Hey @akmalsoliev thanks again for the effort. The workflow you are proposing seems to work quite well and being able to cache layers properly in docker.

Given that is sorted out, I left some comments that probably require attention and, as @MarcoGorelli mentioned, we need this to be incorporated in the ci/cd workflows properly.
If possible including nox and the contributing documentation as well!

Let me know if I can help in any way forward 😉

[FBruzzesi]

Not sure why this was resolved, but 2024.7 is indeed the min version we can support, as we need dask_expr.

[FBruzzesi]

For running in ci/cd we should not pin any of these, as we want to have some old and random versions as well. Also dask requires python 3.9 or above.

[akmalsoliev]

not an issue, was more of demonstration, will be fixed

[FBruzzesi]

Any reason for commenting this one out? 🤔

[akmalsoliev]

Hi, sorry for the delayed response. If I recall correctly, @MarcoGorelli brought up this issue regarding installing cuDF with conda. The installation process is quite complex, and I'm not sure how to resolve it.