Why does url_to_fs return inconsisten paths for different filesystems?

[kadykov]

Why does the format of the path, returned by url_to_fs is so inconsistent depending on the file system and url format?

from fsspec.core import url_to_fs

urls = (
    "file://a.txt",
    "/a.txt",
    "D:/a.txt",
    "http://a.txt",
    "s3://domain.name/a.txt",
    "smb://share.domain.name/a.txt",
    "ftp://test.rebex.net/a.txt",
    "memory://a.txt",
)
for url in urls:
    fs, fs_path = url_to_fs(url)
    print(f"input {url} output {fs_path}")

And the output is:

input file://a.txt output c:/Users/kadykov/Documents/prog/git_repos/intake/a.txt
input /a.txt output c:/a.txt
input D:/a.txt output D:/a.txt
input http://a.txt output http://a.txt
input s3://domain.name/a.txt output domain.name/a.txt
input smb://share.domain.name/a.txt output /a.txt
input ftp://test.rebex.net/a.txt output /a.txt
input memory://a.txt output /a.txt

[martindurant]

There are two purposes of path handling in url_to_fs:

• expand out glob characters
• normalise the path(s) to the form that the given filesystem would refer to them

The latter point differs by filesystem. You notice, for instance, that localFS converts everything to posix-like, and may behave unexpectedly when the input is semi-posix on windows. HTTP-fs requires that the protocol be retained (because http and https are different), but most implementations drop the protocol. Also, if you has complex URLs like chains (simplecache::s3://bucket/file) or embedded parameters (ssh://user:pass@host:22/path/file) you would find only the "path" is returned, the rest having become arguments to the implementation class.