-
Notifications
You must be signed in to change notification settings - Fork 17
Home
This module is intended to accelerate development of Distributed Acoustic Sensing (DAS) analytics and interpretation. Often DAS comes with huge data volumes (typically 5km fibre will produce >450 Mb/s) and these data volumes can be overwhelming, particularly outside of geophysical applications.
This module is intended to lower the bar for researchers interested in developing their own DAS data analytics systems and workflows. Reproducibility of results is required to avoid "doodle bugs" and so the distpy module is designed to make signal processing flows readily transferrable as JSON files to enable researchers to have their findings tested and verified. This is not so rich as the Madagascar seismic project, which reasonably serves the geophysical community, but is aimed more at the community of flow assurance and well monitoring specialists who have less history of manipulating data at these volumes.
It is a lightweight layer designed to sit within existing Cloud, Edge, Windows and Linux systems and is released under a permissive MIT License to engage the widest possible community. distpy can also be easily containerized using Docker. The self documenting features support conversion of workflows between one deployment environment and another, they are also a useful aid to providing end-user documentation for your flows.
If you are new to distpy start by reading the Basic Principles, and then check out the discussion on Directed Graphs, the disection of CASE00.py or jump straight into one of the Tutorials.
For a complete end-to-end example on field data, try the Jupyter Notebook that uses data made available by UKGEOS Glasgow Observatory
Each of the other tutorials is a combination of an example deployment and an example workflow – all workflows can be executed on all deployment configurations, so hopefully what you need is there!
- Printing the SEGY data as thumbnails – mixed with deployment on Linux
- Creating the classic Noise Log (mixed with how to deploy on Azure Databricks
- Detecting events in the data (mixed with deployment on Windows)
- “Hello World” with Google Cloud Functions
- Writing your own controller (mixed with deployment on Docker and Kubernetes)
distpy is installed from packages at pypi.org using pip install
On Windows
python -m pip install distpy
On Linux
pip install distpy
The documentation in this Wiki has a prefix-based naming convention:
- Command Dictionary : is a dictionary of all the commands available for signal processing, this document is indexed by the sidebar to the right.
- User Tutorial : provides examples of installation on different platforms and some introductory signal-processing schemes to aid familiarisation.
- Reference : guides explain some of the what and how, they are background information aimed at both users and developers.
- Dev Tutorial : gives examples of extending distpy
- Dev Notes : are guides to architectural and implementation choices made in distpy development
The origin story is captured in Williams, M.J., J. Le Calvez, T. Cuny, 2019. A cloud approach to processing distributed vibration measurements, 81st EAGE Conference and Exhibition 2019
Examples of papers where distpy was used to support the DAS processing:
data abs angle add analytic_signal argmax approx_vlf bounded_select broaden butter clip conj convolve copy correlate count_peaks data_load deconvolve destripe diff downsample dip_filter down_wave extract fft from_gpu gather gaussian geometric_mean gradient harmonic_mean hard_threshold ifft keras kmeans kurtosis lin_transform macro median_filter mean multiply multiple_calcs peak_to_peak rms_from_fft real rescale roll running_mean sobel soft_threshold sum skewness sta_lta std_dev to_gpu unwrap up_wave velocity_map velocity_mask virtual_cmp wiener write_npy write_witsml