How to pickle/save memory filesystem?

[TWTom041]

I want to save a filesystem to a file, but it seems like that I can't do it with pickle, how to do that?

>>> import fsspec
>>> import pickle
>>> fs = fsspec.filesystem("memory")
>>> fs.touch("a") 
>>> fs.ls("/") 
[{'name': '/a', 'size': 0, 'type': 'file', 'created': 1721555345.246705}]
>>> with open("test.fs", "wb") as f:
...     pickle.dump(fs, f)
>>> with open("test.fs", "rb") as f:
...     newfs = pickle.load(f)
>>> newfs.ls("/")
[{'name': '/a', 'size': 0, 'type': 'file', 'created': 1721555345.246705}]
>>> exit()  # restart Python

>>> with open("test.fs", "rb") as f:
>>>     newfs = pickle.load(f)
>>> newfs.ls("/")
[]  # why?

[martindurant]

Filesystems reconstitute themselves from the arguments passed to them at init time. This is to ensure that they can be moved between distributed machines without holding any state. memoryFS is special, since "state" is essentially everything in the filesystem in this case. I still think that copying the internal storage at pickle time might be surprising behaviour in the distributed context (we don't copy the local filesystem, for instance). Instead, I recommend you implement a save() and load() pair for memoryFS, or a pickle_with_data() method which satisfies pickle. Are you interested?

[TWTom041]

I dig a little bit into the source code. I think the store variable inside MemoryFileSystem is the "internal storage", but it is a class variable instead of a instance variable, so using dill instead of pickle can do the job.

[martindurant]

Correct, there is only one global store in memoryFS

[TWTom041]

After a experiment, using dill won't do the job. I will create a PR for the save() and load() methods.