Skip to content
/ morumotto Public

Morumotto is a tool that automatically fills a seismic data archive, corrects overlaps and requests data from a pool of different sources to fill the gaps

License

View license
0 stars 1 fork Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

IPGP/morumotto

BranchesTags

Repository files navigation

MORUMOTTO

This development has been partly funded by the Eurovolc H2020 project.

  1. Installation
    1. With installer
    2. Manual installation
  2. Getting Started
  3. Read the docs

Installation

METHOD 1 : TL;DR

  • Step 1, in your database prompt (not required, but recommanded)

    • With MySQL / Mariadb :
    CREATE DATABASE MORUMOTTO;
CREATE USER 'morumotto_user'@'host_name' IDENTIFIED BY 'some password';
GRANT ALL ON MORUMOTTO.* TO 'morumotto_user'@'host_name';
ALTER DATABASE `MORUMOTTO` CHARACTER SET utf8;
FLUSH PRIVILEGES;
    • With PostgreSQL :
    CREATE DATABASE MORUMOTTO;
CREATE USER morumotto_user WITH ENCRYPTED PASSWORD 'some password';
ALTER ROLE morumotto_user SET client_encoding TO 'utf8';
ALTER ROLE morumotto_user SET default_transaction_isolation TO 'read committed';
ALTER ROLE morumotto_user SET timezone TO 'UTC';
GRANT ALL PRIVILEGES ON DATABASE MORUMOTTO to morumotto_user;

  • Step 2

    git clone https://github.com/IPGP/morumotto
cd morumotto  
./install.sh

You will need to confirm some things during installation, don't run away from your keyboard

  • Required for the install script to run correctly : mysql OR mariadb, python3 (Even if your python system is 2.7, you must be able to install libpython3.6-dev)
  • Installer tested on Ubuntu 16.10, 18.04, 18.10, 19.04 & Debian 9

METHOD 2 : Manual installation

Get source code
git clone https://github.com/IPGP/morumotto
cd morumotto

For the following steps, you must stay in this folder

Dependencies

  1. Before starting, make sure you have the following packages installed :

    • python
    • python-pip
    • mysql-server OR mariadb-dev libmariadb-dev-compat (depending on your linux distribution)
    • wget
    • libmysqlclient-dev OR libmariadb-dev (depending on your linux distribution)
    • wget
    • libpython3.6-dev
    • supervisor
    • cpulimit
    • java

    In Ubuntu:

    sudo apt-get install -y python wget make python-pip mysql-server libmysqlclient-dev python3-dev libpython3.6-dev python3-venv supervisor cpulimit default-jre

    In Debian:

    sudo apt-get install -y python wget make python-pip mariadb-dev libmariadb-dev-compat libmariadb-dev python3-dev libpython3.6-dev python3-venv supervisor cpulimit default-jre

  2. Install dataselect (version >= 3.20 ) in /morumotto/bin : (Visit the IRIS GitHub page to install)

  3. Install qmerge in /morumotto/bin : (Download tar file here). Then copy the qmerge executable to /morumotto/bin.

  4. Install msi in /morumotto/bin : (Visit the IRIS GitHub page to install)

  5. Install sdrsplit in /morumotto/bin : (Download tar file here) Then copy the sdrsplit executable to /morumotto/bin.

  6. Install IRIS stationxml validator in /morumotto/bin : (Download jar file here (v.1.5.9.5)) Then copy the jar file to /morumotto/bin. It would be better to create a symbolic link to that file (but not mandatory). Just run (inside the morumotto bin/ directory)

ln -s station-xml-validator-1.5.9.5.jar stationxml-validator.jar

Note that if you copy several versions of stationxml-validator (e.g. 1.5.9.5 and 1.6.0.2) you will be able to choose in the admin interface which version to use to validate your metadata.

  1. Install RabbitMQ (The parallel task are handled by the task queue manager Celery. Celery requires to install a broker, Morumotto uses RabbitMQ)

    In Ubuntu/Debian :

    sudo apt-get install rabbitmq-server
Initialise the environment

  1. You must know your system python version. For that, in a terminal window, type:

    python --version

  2. Create the virtual environment (inside the morumotto root directory)

    • If your python is < 3.3 :

      Install virtualenv, then create environment

      sudo apt install virtualenv
virtualenv morumotto-env -p python3.6

    • If your python is >= 3.4 :

      python3 -m venv morumotto-env

  3. Install requirements :

    source morumotto-env/bin/activate  
pip install numpy # needs to be installed first, doesn't work within requirements
pip install -r requirements.txt
Initialise LEAPSECONDS

  • Check if you have already initialised the leapsecond var in your shell :

    echo $LEAPSECONDS

  • If prompt returns an empty string, you may use the leapsecond list provided here ()

    echo "export LEAPSECONDS='${MORUMOTTO_ROOT_DIR}/leapsecond.list'" >> ${HOME}/.bashrc
Initialise your MySQL database (If you are using PostgreSQL, skip this paragraph)

Now, you will have to create a Morumotto database within MySQL. To do so:

  • connect to MySQL as root:

    mysql -u root -p

  • enter root password

  • in the mysql prompt, type:

    CREATE DATABASE MORUMOTTO;
CREATE USER 'morumotto_user'@'host' IDENTIFIED BY 'your_password';
GRANT ALL ON MORUMOTTO.* TO 'morumotto_user'@'host';
ALTER DATABASE `MORUMOTTO` CHARACTER SET utf8;
FLUSH PRIVILEGES;

Then add to your custom settings (morumotto/morumotto/custom_settings.py) :

DATABASE_ENGINE = 'django.db.backends.mysql'
DATABASE_NAME = 'MORUMOTTO'
DATABASE_USER_NAME = 'morumotto_user'
DATABASE_PASSWORD = 'your_password'
DATABASE_HOST = 'your_host' #defaults to 127.0.0.1 to use local database
CUSTOM_HOSTS = []

There is a template in the morumotto/morumotto folder. Just duplicate it and remove the .template at the end of the file name and put your own settings in it. For the custom hosts, see below ("Access from another computer in the same network")

Note: you can change the database, user and password at your convinience, but they must be the same in the database and in the custom settings file

Initialise your PostgreSQL database (If you are using mysql, skip this paragraph)

First:

  • connect to PostgreSQL as root:

    psql -U postgres

  • enter root password

  • in the postgresql prompt, type:

    CREATE DATABASE MORUMOTTO;
CREATE USER morumotto_user WITH ENCRYPTED PASSWORD 'some password';
ALTER ROLE morumotto_user SET client_encoding TO 'utf8';
ALTER ROLE morumotto_user SET default_transaction_isolation TO 'read committed';
ALTER ROLE morumotto_user SET timezone TO 'UTC';
GRANT ALL PRIVILEGES ON DATABASE MORUMOTTO to morumotto_user;

Then add to your custom settings (morumotto/morumotto/custom_settings.py) :

DATABASE_ENGINE = 'django.db.backends.postgresql'
DATABASE_NAME = 'MORUMOTTO'
DATABASE_USER_NAME = 'morumotto_user'
DATABASE_PASSWORD = 'your_password'
DATABASE_HOST = 'your_host' #defaults to 127.0.0.1 to use local database
CUSTOM_HOSTS = []

There is a template in the morumotto/morumotto folder. Just duplicate it and remove the .template at the end of the file name and put your own settings in it. For the custom hosts, see below ("Access from another computer in the same network")

If you want to use another RDBMS (NoSQL for example), please check the Django documentation

Bind database tables to Morumotto objects

  • Inside the morumotto root directory, execute

    python manage.py migrate  
python manage.py makemigrations archive monitoring qualitycontrol logdb
python manage.py migrate django_celery_results
python manage.py migrate
Create an admin user :

  • Inside the morumotto directory, execute

    python manage.py createsuperuser

Fill the required informations. Don't forget it, it will be your administrator user for the Morumotto software

All good, you're ready to go now !

Deamonize it :
  • Automatically :
Click to display script

In a terminal, paste the following code (or run is in a script) :

dir=$(pwd)
cat <<EOT >> morumotto.conf
[program:morumotto_celery]
command = ${dir}/morumotto-env/bin/celery -A morumotto worker -l info
user = ${USER}
directory = ${dir}
logfile = /var/log/supervisor/morumotto_celery.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
autostart = true
autorestart = true

[program:morumotto_flower]
command = ${dir}/morumotto-env/bin/celery flower -A morumotto --address=127.0.0.1 --port=5555
user = ${USER}
directory = ${dir}
logfile = /var/log/supervisor/morumotto_flower.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
autostart = true
autorestart = true

[program:morumotto_runserver]
command = ${dir}/morumotto-env/bin/python manage.py runserver 0.0.0.0:8000
user = ${USER}
directory = ${dir}
logfile = /var/log/supervisor/morumotto_runserver.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
autostart = true
autorestart = true

[group:morumotto]
programs=morumotto_celery,morumotto_flower,morumotto_runserver
priority=999

EOT
sudo mv morumotto.conf /etc/supervisor/conf.d/ || exit
sudo supervisorctl reread
sudo supervisorctl update
  • Manually :

Create a file named morumotto.conf, and paste the following text inside :

Click to display text

WARNING : you must change ${dir} with your morumotto directory and ${USER} with your user name

morumotto.conf

[program:morumotto_celery]
command = ${dir}/morumotto-env/bin/celery -A morumotto worker -l info
user = ${USER}
directory = ${dir}
logfile = /var/log/supervisor/morumotto_celery.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
autostart = true
autorestart = true

[program:morumotto_flower]
command = ${dir}/morumotto-env/bin/celery flower -A morumotto --address=127.0.0.1 --port=5555
user = ${USER}
directory = ${dir}
logfile = /var/log/supervisor/morumotto_flower.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
autostart = true
autorestart = true

[program:morumotto_runserver]
command = ${dir}/morumotto-env/bin/python manage.py runserver 0.0.0.0:8000
user = ${USER}
directory = ${dir}
logfile = /var/log/supervisor/morumotto_runserver.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
autostart = true
autorestart = true

[group:morumotto]
programs=morumotto_celery,morumotto_flower,morumotto_runserver
priority=999

Then move it to the /etc/supervisor/conf.d, and update supervisor

sudo cp morumotto.conf /etc/supervisor/conf.d/morumotto.conf
sudo supervisorctl reread
sudo supervisorctl update

Getting Started

Check that the deamon is running :
sudo supervisorctl status

This should display something like :

morumotto:morumotto_celery       RUNNING   pid 3939, uptime 0:38:21
morumotto:morumotto_flower       RUNNING   pid 3937, uptime 0:38:22
morumotto:morumotto_runserver    RUNNING   pid 3938, uptime 0:38:21
Open you browser and enter the following URL:
Visit the admin page:
Access from another computer in the same network :
  • add the server's IP (1.2.3.4 for example) where Morumotto is located to the ALLOWED_HOST in morumotto/morumotto/custom_settings.py

Then in a terminal, enter

source morumotto-env/bin/activate  
python manage.py runserver 0.0.0.0:8080

Note that you have changed the port used by Morumotto from 8000 (default) to 8080

Then from any computer in the network, visit the following URL:

To run the software without the daemon :

Note : this is only for development

In a terminal enter :

source morumotto-env/bin/activate  
python manage.py runserver

Read the docs

User and Dev guides

About

Morumotto is a tool that automatically fills a seismic data archive, corrects overlaps and requests data from a pool of different sources to fill the gaps

Resources

Readme

License

View license
Activity
Custom properties

Stars

0 stars

Watchers

4 watching

Forks

1 fork
Report repository

Releases

3 tags

Packages

No packages published

Contributors 5

Languages