ZIPsFS.1

.\"t
.\" Automatically generated by Pandoc 2.5
.\"
.TH "" "" "" "" ""
.hy
.PP
% ZIPsFS(1)
.SH NAME
.PP
\f[B]ZIPsFS\f[R] \[em] FUSE\-based overlay union file system which
expands ZIP files
.SH SYNOPSIS
.PP
ZIPsFS [\f[I]ZIPsFS\-options\f[R]] \f[I]path\-of\-root1\f[R]
\f[I]path\-of\-root2\f[R] \f[I]path\-of\-root3\f[R] :
[\f[I]fuse\-options\f[R]] \f[I]mount\-point\f[R]
.SS Example 1
.PP
ZIPsFS \[ti]/tmp/ZIPsFS/writable \[ti]/local/file/tree //computer1/pub
//computer2/pub : \-f \-o allow_other \[ti]/mnt/ZIPsFS
.SH DESCRIPTION
.SS Summary
.PP
ZIPsFS acts as a union or overlay file system.
It combines multiple file structures into one, resulting in a single
directory structure that contains underlying files and sub\-directories
from the given sources.
Created or modified files are stored in the first file location.
The other file sources are read\-only and files will never be modified.
ZIPsFS expands ZIP files as folders.
Normally, the folder name is formed from the ZIP file name by appending
\[lq].Contents/\[rq].
This can be changed with rules based on file name patterns.
Extensive configuration is possible without interrupting the file
system.
Specific features and performance tweaks meet our needs for storing
large data from mass spectrometry experiments.
.SS Configuration
.PP
The default behavior can be modified with rules based on file names in
\f[I]ZIPsFS_configuration.h\f[R] and \f[I]ZIPsFS_configuration.c\f[R].
For changes to take effect, re\-compilation and restart is necessary.
Using the \f[I]\-s symlink\f[R] option, the configuration can be changed
without interrupting the file system.
Ongoing computations with file access to the ZIPsFS are not affected.
.SS Union / overlay file system
.PP
ZIPsFS is a union or overlay file system.
Several file locations are combined to one.
All files in the source file trees (in above example command three
sources) can be accessed via the mount point (in the example
\f[I]~/mnt/ZIPsFS\f[R]). When files are created or modified, they will
be stored in the first file tree (in the example
\f[I]~/tmp/ZIPsFS/writable\f[R]).
If files exist in two locations, the left most source file system takes
precedence.
.PP
New files are created in the first file tree, while the following file
trees are not modified.
If an empty string \[dq]\[dq] or \[cq]\[cq] is given for the first
source, no writable source is used.
.SS ZIP files
.PP
Let \f[I]file.zip\f[R] be a ZIP file in any of the source file systems.
It will appear in the virtual file system together with a folder
\f[I]file.zip.Content\f[R].
Normally, the folder name is formed by appending
\[lq]\f[I].Content\f[R]\[rq] to the zip file name.
This can be changed in \f[I]ZIPsFS_configuration.c\f[R].
.PP
For Sciex mass spectrometry files it is necessary that the containing
files are shown directly in the file listing rather than in a
sub\-folder.
The respective file extensions are specified in
\f[I]ZIPsFS_configuration.c\f[R].
.SS Cache
.PP
ZIPsFS can read certain ZIP entries entirely into RAM and provide the
data from the RAM copy at higher speed.
This may improve performance for compressed ZIP entries that are read
from varying positions in the file, so\-called file file\-seek.
This concerns file entries with file name and size rules given in in
\f[I]ZIPsFS_configuration.c\f[R].
Furthermore, compressed ZIP entries are cached in RAM if the reading
position ever jumps backward.
With the option \f[B]\-l\f[R] an upper limit of memory consumption for
the ZIP RAM cache is specified.
.PP
Further caches aim at faster file listing of large directories.
.SS Logs
.PP
Running ZIPsFS in the foreground with the option \f[I]\-f\f[R], allows
to observe logs continuously at the console.
It is convenient to run ZIPsFS in a terminal multiplexer like
\f[I]tmux\f[R].
Log files are found in \f[B]\[ti]/.ZIPsFS/\f[R].
An HTML file with status information is found in and are also accessible
from the generated folder \f[B]ZIPsFS\f[R] in the virtual ZIPsFS file
system.
.SS Autogeneration of files
.PP
ZIPsFS offers files which do not yet exist, and will be generated when
used.
This option is available if the first file source is provided.
Generally, the first file source is writable
.PP
It will not be available, if an empty string is passed as the first root
directory.
.PP
The typical use\-case are file conversions.
Auto\-generated files are displayed in the virtual file tree in
\f[B]ZIPsFS/a/\f[R] and physically stored in the first source.
.PP
With the current setting, Mascot mass spectrometry files and msML files
are generated.
As a more popular example, down\-scaled image files are offered for
jpeg, jpg, png and gif files.
The two scaling levels 50% and 25% demonstrate two different options:
For size reduction to 50%, a file is generated and saved in the first
file source in \f[I]ZIPsFS/a/\&...scale50%.jpg\f[R].
When requested for the first time there will be some delay.
When accessed a second time, the data comes without delay, because the
file is already there.
For down\-scaling to 25%, the generated image data is not saved.
Instead it is kept in RAM as long as the file handle is active.
After loading the image, the image data in the RAM is discarded.
The advantage is that no HD space is used.
The disadvantage is, that for a short time the entire data resides in
RAM and that the data is generated each time the file data is read.
.PP
Users can use these Imagemagick based examples as a template for their
own settings in \f[B]ZIPsFS_configuration_autogen.c\f[R].
.PP
Also see readme in the folder \f[B]ZIPsFS/a/\f[R].
.PP
Some exotic Windows executables do not work well within ZIPsFS.
As a work around, we developed the shell script
\f[B]ZIPsFS_autogen_queue.sh\f[R].
With each pass of an infinity loop one task is taken from a queue and
processed.
One file is converted at a time per script instance.
Several instances of this shell script can run in parallel.
In the settings, the symbol \f[B]PLACEHOLDER_EXTERNAL_QUEUE\f[R] is
given instead of an executable program.
.PP
Problems/Limitations: The system does not know the file size of
not\-yet\-generated files.
It will report an upper limit of the expected file size.
.SS ZIPsFS Options
.PP
\-h
.PP
Prints brief usage information.
.PP
\-l \f[I]Maximum memory for caching ZIP\-entries in the RAM\f[R]
.PP
Specifies a limit for the cache.
For example \f[I]\-l 8G\f[R] would limit the size of the cache to 8
Gigabyte.
.PP
\-c [NEVER,SEEK,RULE,COMPRESSED,ALWAYS]
.PP
Policy for ZIP entries cached in RAM.
.PP
.TS
tab(@);
cw(8.3n) lw(61.7n).
T{
NEVER
T}@T{
ZIP are never cached, even not in case of backward seek.
T}
T{
T}@T{
T}
T{
SEEK
T}@T{
ZIP entries are cached if the file position jumps backward.
This is the default
T}
T{
T}@T{
T}
T{
RULE
T}@T{
ZIP entries are cached according to rules in \f[B]configuration.c\f[R].
T}
T{
T}@T{
T}
T{
COMPRESSED
T}@T{
All compressed ZIP entries are cached.
T}
T{
T}@T{
T}
T{
ALWAYS
T}@T{
All ZIP entries are cached.
T}
T{
T}@T{
T}
.TE
.PP
\-s \f[I]path\-of\-symbolic\-link\f[R]
.PP
After initialization the specified symlink is created and points to the
mount point.
Previously existing links are overwritten.
This allows to restart ZIPsFS without affecting running programs which
access file in the virtual ZIPsFS file system.
For file paths in the virtual file system, the symlink is used rather
than the real mount\-point.
Consider a running ZIPsFS instance which needs to be replaced by a newer
one.
The new ZIPsFS instance is started with a different mount point.
Both instances work simultaneously.
The symlink which used to point to the mount point of the old instance
is now pointing to that of the new one.
The old instance should be let running for an hour or so until no file
handle is open any more.
.SS Debug Options
.PP
See ZIPsFS.compile.sh for activation of sanitizers.
.PP
\-T Checks the capability to print a backtrace.
This requires addr2line which is usually in /usr/bin/ of Linux and
FreeBSD.
For MacOSX, the tool atos is used.
.SS FUSE Options
.PP
\-f
.PP
Run in foreground and display some logs at stdout.
This mode is useful inside tmux.
.PP
\-s
.PP
Disable multi\-threaded operation to rescue ZIPsFS in case of threading
related bugs.
.PP
\-o \f[I]comma separated Options\f[R]
.PP
\-o allow_other
.PP
Other users can read the files
.SS Fault management
.PP
When source file structures are stored remotely, there is a risk that
they may be temporarily unavailable.
Overlay file systems typically freeze when calls to the file API block.
Conversely, ZIPsFS should continue to operate with the remaining file
roots.
This is implemented as follows: Paths starting with double slash (in the
example \f[I]//computer1/pub\f[R]) are regarded as remote paths and
treated specially.
ZIPsFS will periodically check file systems starting with a double
slash.
If the last responds was too long ago then the respective file system is
skipped.
Furthermore the stat() function to obtain the attributes for a file are
queued to be performed in extra threads.
.PP
For files which are located in ZIP files and which are first loaded
entirely into RAM, the system is also robust for interruptions and
blocks during loading.
The system will not freeze.
After some longer time it will try to load the same file from another
root or return ENOENT.
.PP
If loading of ZIP files fail, loading will be repeated after 1s.
.PP
For ZIP entries loaded entirely into the RAM, the CRC sum is validated
and possible errors are logged.
.SH FILES
.IP \[bu] 2
ZIPsFS_configuration.h and ZIPsFS_configuration.c and
ZIPsFS_configuration_autogen.c: Customizable rules.
Modification requires recompilation.
.IP \[bu] 2
\[ti]/.ZIPsFS: Contains the log file and cache and the folder a.
The later holds auto\-generated files.
.SH LIMITATIONS
.SS Hard\-links
.PP
Hard\-links are not implemented, while symlinks work.
.SS Deleting files
.PP
Files can only be deleted when their physical location is in the first
source.
Conversely, in the FUSE file systems unionfs\-fuse and fuse\-overlayfs,
files can be always deleted irrespectively of their physical location.
They are canceled out without actually deleting them from their physical
location.
If you need the same behaviour please drop a request\-for\-feature.
.SH BUGS
.PP
Current status: Testing and Bug fixing
.SH AUTHOR
.PP
Christoph Gille
.SH SEE ALSO
.IP \[bu] 2
https://github.com/openscopeproject/ZipROFS
.IP \[bu] 2
https://github.com/google/fuse\-archive
.IP \[bu] 2
https://bitbucket.org/agalanin/fuse\-zip/src
.IP \[bu] 2
https://github.com/google/mount\-zip
.IP \[bu] 2
https://github.com/cybernoid/archivemount
.IP \[bu] 2
https://github.com/mxmlnkn/ratarmount