How to use azurite with fsspec.open?

[JoostvDoorn]

I have been testing with azurite in some situations, and now want to expand my azurite testing on functions that use fsspec. I and was wondering if anyone has experience using fsspec with azurite, and what would be their opinion on the best practice to test this with fsspec?

Since azurite uses http to reference files instead of the abfs/az protocol. Out of the box using fsspec.open will not work. Probably there is some way to monkey patch this to redirect the http protocol to azure.

[JoostvDoorn]

So what I did now is to make sure I pass storage_options to my functions and pass:

ACCOUNT_NAME = "devstoreaccount1"
KEY = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="  # NOQA
CONN_STR = f"DefaultEndpointsProtocol=http;AccountName={ACCOUNT_NAME};AccountKey={KEY};BlobEndpoint={URL}/{ACCOUNT_NAME};"  # NOQA
storage_options = {"protocol": "abfs", "connection_string": CONN_STR, "account_name": ACCOUNT_NAME}
# Add you container here.....
# ...
url = "az://container_name"
fs, path = fsspec.core.url_to_fs(url, **storage_options)

Similar to how it's done in ABFS lib:
https://github.com/fsspec/adlfs/blob/main/adlfs/tests/conftest.py#L9-L11

[martindurant]

The testing code in adlfs is definitely the place to go to see this kind of usage. Of course, azurite itself comes with various config options, so the complete situation may depend on what you want to do.

url = "az://container_name"
fs, path = fsspec.core.url_to_fs(url, **storage_options)

can be shorter

fs = fsspec.filesystem("abfs", **storage_options)

(and you don't need to separately include "protocol" in the options)

The usage would be the same for fsspec.open().