Skip to content
/ law Public
forked from riga/law

High-level extension layer for Luigi analysis workflows.

License

Notifications You must be signed in to change notification settings

rfvc/law

 
 

Repository files navigation

law logo
Build status Docker images Documentation status Package version License DOI

Note: This project is currently under development. Version 1.0.0 will be the first, fully documented beta release, targetted for mid 2022.

Use law to build complex and large-scale task workflows. It is build on top of luigi and adds abstractions for run locations, storage locations and software environments. Law strictly disentangles these building blocks and ensures they remain interchangeable and resource-opportunistic.

Key features:

  • CLI with auto-completion and interactive status and dependency inspection.
  • Remote targets with automatic retries and local caching
    • WebDAV, HTTP, Dropbox, SFTP, all WLCG protocols (srm, xrootd, rfio, dcap, gsiftp, ...)
  • Automatic submission to batch systems from within tasks
    • HTCondor, LSF, gLite, ARC, Slurm
  • Environment sandboxing, configurable on task level
    • Docker, Singularity, Sub-Shells, Python-venv

First steps

Installation and Dependencies

Install via pip:

pip install law

This command also installs luigi and six.

The (default) remote target implementation also requires gfal2 and gfal2-python (optional, also via pip) to be installed.

Docker Images

To run and test law, there are three docker images available on the DockerHub, corresponding to Python versions 2.7, and 3.7 to 3.10. They are based on CentOS 7 and ship with the dependencies listed above, including gfal2.

docker run -ti riga/law:latest

Tags:

  • py310: Latest Python 3.10
  • py39, py3, latest: Latest Python 3.9
  • py38: Latest Python 3.8
  • py37: Latest Python 3.7
  • py2, py27: Python 2.7
  • example: Example runner, based on latest (see below)

Examples

All examples can be run either in a Jupyter notebook or a dedicated docker container. For the latter, do

docker run -ti riga/law:example <example_name>

Auto completion on the command-line

bash

source "$( law completion )"

zsh

zsh is able to load and evaluate bash completion scripts via bashcompinit. In order for bashcompinit to work, you should run compinstall to enable completion scripts:

autoload -Uz compinstall && compinstall

After following the instructions, these lines should be present in your ~/.zshrc:

# The following lines were added by compinstall
zstyle :compinstall filename '~/.zshrc'

autoload -Uz compinit
compinit
# End of lines added by compinstall

If this is the case, just source the law completion script (which internally enables bashcompinit) and you're good to go:

source "$( law completion )"

Development

About

High-level extension layer for Luigi analysis workflows.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 93.9%
  • Shell 6.1%