This repository contains a set of Bash scripts that make up a data pipeline, designed to automate the process of interacting with an SEL-735 meter. The pipeline is divided into two main executable scripts:
-
data_pipeline.sh
: Handles the first four steps:- Connecting to the meter via FTP
- Downloading new files
- Organizing and creating metadata
- Compressing data
-
archive_pipeline.sh
: Handles the final step:- Archiving and transferring event data to the Data Acquisition System (DAS)
Ensure you have the following before running the pipeline:
- Unix-like environment (Linux, macOS, or a Unix-like Windows terminal)
- FTP credentials for the meter
- Meter Configuration
- Must have installed:
lftp
yq
zip
rsync
jq
-
Clone the repository:
git clone git@github.com:acep-uaf/camio-meter-streams.git cd camio-meter-streams/cli_meter
Note: You can check your SSH connection with
ssh -T git@github.com
-
Navigate to the
config
directory and copy the example configuration file to a new file:cd config cp config.yml.example config.yml cp archive_config.yml.example archive_config.yml
-
Update the configuration files with the necessary details:
config.yml
: Add the FTP server credentials and meter configuration data.archive_config.yml
: Add the source and destination directories and other relevant details.
-
Secure the configuration files so that only the owner can read and write:
chmod 600 config.yml chmod 600 archive_config.yml
To run the data pipeline and then transfer data to the Data Acquisition System (DAS):
-
Run the Data Pipeline First
Execute the
data_pipeline
script from thecli_meter
directory. The script requires a configuration file specified via the-c/--config
flag. If this is your first time running the pipeline, the initial download may take a few hours. To pause the download safely, see: How to Stop the Pipeline./data_pipeline.sh -c config/config.yml
-
Run the Archive Pipeline
After the
data_pipeline
script completes, execute thearchive_pipeline
script from thecli_meter
directory. The script requires a configuration file specified via the-c/--config
flag../archive_pipeline.sh -c config/archive_config.yml
The rsync uses the
--exclude
flag to exclude theworking
directory to ensure only complete files are transfered. -
Run the Cleanup Process (Conditional)
If the
archive_pipeline
script completes successfully and theenable_cleanup
flag is set to true in the archive configuration file, thecleanup.sh
script will be executed automatically. This script removes outdated event files fromlevel0
based on the retention period specified in the configuration file.If the
enable_cleanup
flag is not enabled, you can run the cleanup manually by passing in the archive configuration file../cleanup.sh -c config/archive_config.yml
Ensure that the
archive_config.yml
file is properly configured with the retention periods for each directory in the cleanup process.
When you need to stop the pipeline:
- To Stop Safely/Pause Download:
- Use
Ctrl+C
to interrupt the process. - If interupting the proccess doesn't work try
Ctrl+\
to quit. - If you would like to resume the download, rerun the
data_pipeline
command.The download will resume from where it left off, provided the same config file (-c
)is used.
- Use
- Avoid Using
Ctrl+Z
:- Do not use
Ctrl+Z
to suspend the process, as it may cause the pipeline to end without properly closing the FTP connection.
- Do not use
This repository includes automated tests for the scripts using Bats (Bash Automated Testing System) along with helper libraries: bats-assert
, bats-mock
, and bats-support
. The tests are located in the test
directory.
Ensure you have cloned the repository with its required submodules, they should be located under the test
and test/test_helper
directories:
bats-core
bats-assert
bats-mock
bats-support
-
Clone the repository with submodules:
git clone --recurse-submodules git@github.com:acep-uaf/camio-meter-streams.git
If you have already cloned the repository without submodules, you can initialize and update them with:
git submodule update --init --recursive
-
Navigate to the project directory:
cd /path/to/camio-meter-streams/cli_meter
-
Run all the tests:
bats test