"File name too long" with HTTP presigned requests

[cgrass]

Hello,
I'm new to python and fsspec, so hopefully there is an obvious answer to my question. Thanks for the help!

Issue

I need to fetch a file from an s3 bucket using https and a pre-signed URL. The URL produced is quite long, commonly over 1500 characters (mostly query params). If I use the code below, I get an OS exception: File name too long.

def test_this():
    kwargs = {}
    spec = fsspec.filesystem("https", **kwargs)
    frompath = {longurl}
    topath = "/var/"  ### in prod this is a real dest
    response = spec.get(frompath, topath, recursive=False, **kwargs)
    print(response)

I walked through the code and can see that the rpath is typically appended to the lpath for storage in asyn.py:

lpaths = other_paths(
    rpaths,
    lpath,
    exists=exists,
    flatten=not source_is_str,
)

def other_paths(
    paths: list[str],
    path2: str | list[str],
    exists: bool = False,
    flatten: bool = False,
) -> list[str]:
    if isinstance(path2, str):
        path2 = path2.rstrip("/")

        if flatten:
            path2 = ["/".join((path2, p.split("/")[-1])) for p in paths]
        else:
            cp = common_prefix(paths)
            if exists:
                cp = cp.rsplit("/", 1)[0]
            if not cp and all(not s.startswith("/") for s in paths):
                path2 = ["/".join([path2, p]) for p in paths]
            else:
                path2 = [p.replace(cp, path2, 1) for p in paths]
    else:
        assert len(paths) == len(path2)
    return path2

Question

Am I misusing the lib or protocol? Is there a way to configure fsspec to scrub query params (or use some arbitrary string) for the appended destination string?

Setup

MacOS Ventura M1 Pro
fsspec-2023.12.2
Python 3.9, 3.10 (tried both)

[martindurant]

Because your topath is a directory (ends with "/"), the operation will be
longurl -> "/var/" + longurl

This may not be possible to achieve, as you have found.

Instead, make your topath a specific file.

[cgrass]

@martindurant i'm inclined to turn this into an issue, or maybe two: one for the long URL and one for changing behavior when calling get with an lpath that is a file and rpath is a URL.

thoughts?

[martindurant]

Lodging issues is fine, it might get more notice. For the latter issue, there are quite a lot of cross-implementation batch operation tests (fsspec/tests/abstract), and perhaps best would be to find the specific case that might be missed in those, defining how the operation ought to work.

[cgrass]

ok, i created #1504 and #1505

[martindurant]

Feel free to close this discussion. I can't promise anyone else will get involved... If you have a possible fix, of course I would like to see it.

[cgrass]

Ok, i'll close this discussion and leave a comment in both bug reports. I'm not a python dev, so it's hard for me to recommend a specific fix, but i have a couple ideas.