Update datasets to 2.19.0 and fsspec to 2024.3.1

[albertvillanova]

Update datasets to 2.19.0.

Fix #2739.

[albertvillanova]

Tests for worker are not passing: https://github.com/huggingface/dataset-viewer/actions/runs/8753796799/job/24024224560?pr=2740

I am fixing it.

[albertvillanova]

The issue was introduced by @lhoestq in PR:

• Add loading tags and code (pandas, dask, wds) #2454

if any(
    NON_WORD_GLOB_SEPARATOR not in pattern.format(keyword="train", sep=NON_WORDS_CHARS)
    for pattern in datasets.data_files.KEYWORDS_IN_PATH_NAME_BASE_PATTERNS
):
    raise ImportError(

Any reason why did you put an indirect constraint on the upper version of fsspec and how to overcome it now?

[albertvillanova]

As @mariosasko and @lhoestq suggested (see huggingface/datasets#6704 (comment) and huggingface/datasets#6704 (comment)), I am fixing the issue by creating a new variable by joining the 2 new ones).

[albertvillanova]

@lhoestq, I have just checked that the fix is not as direct as suggested.

Before the changes introduced by @mariosasko in huggingface/datasets#6704, the ImportError was not raised if fsspec version was < 2023.12.0, i.e. the pattern NON_WORD_GLOB_SEPARATOR is present in ALL KEYWORDS_IN_PATH_NAME_BASE_PATTERNS for both cases fsspec < 2023.9.0 or fsspec < 2023.12.0.
That means the ImportError was only raised if fsspec >= 2023.12.0.

But after the changes introduced by @mariosasko in huggingface/datasets#6704, if we join the variables KEYWORDS_IN_DIR_NAME_BASE_PATTERNS and KEYWORDS_IN_FILENAME_BASE_PATTERNS, the ImportError will always be raised: there is always at least one pattern that does not contain the NON_WORD_GLOB_SEPARATOR, for example:

• for fsspec < 2023.9.0:
  • filename = "{keyword}[{sep}]*"
  • firname = "{keyword}/**"
• for fsspec < 2023.12.0:
  • filename = "{keyword}[{sep}]*"
  • dirname = "{keyword}/**/*"
• for fsspec >= 2023.12.0: any of the filenames or dirnames

[albertvillanova]

@lhoestq if you are trying to put a constraint on the version of fsspec, wouldn't it be easier to check the installed version?

[lhoestq]

Then we need to update NON_WORD_GLOB_SEPARATOR, I can continue this PR

[lhoestq]

tests are all passing now, feel free to check if my changes look good to you @albertvillanova and merge

[albertvillanova]

@lhoestq I have updated fsspec in docs, e2e and admin_ui as well.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

thanks !