(LINUX) Deploy & Maintain IMPatienT

How to deploy IMPatienT

Inspired and modified from: Flask MegaTutorial

/!\ In all command: Be sure to modifiy <YOUR_USER> to your login.

Setup Linux dependencies

# Install Dependencies 1
sudo apt update
sudo apt upgrade
sudo apt install supervisor nginx git poppler-utils
sudo apt install tesseract-ocr tesseract-ocr-osd tesseract-ocr-fra

# Clone Git Repo + Dependencies 2
git clone git@github.com:lambda-science/IMPatienT.git

Install Python with Miniconda (+Poetry & DVC)

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod +x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh
conda activate base
conda install -c conda-forge mamba 
mamba install -c conda-forge poetry dvc-ssh
# Optional bug w/ dvc fix  pip install sshfs[bcrypt]

Install your python Poetry Envrionnement & activate it

poetry config virtualenvs.in-project true
poetry install
poetry shell

echo "export FLASK_APP=impatient.py" >> ~/.profile
nano .env # SEE CONFIG FILES AT THE BOTTOM
flask db upgrade # Init DB Model

Setup Supervisor process for IMPatienT and NGINX

sudo nano /etc/supervisor/conf.d/impatient.conf # SEE CONFIG FILES AT THE BOTTOM
sudo supervisorctl reload
sudo rm /etc/nginx/sites-enabled/default
sudo nano /etc/nginx/sites-enabled/impatient # SEE CONFIG FILES AT THE BOTTOM
sudo nginx -t
sudo service nginx reload
sudo chown -R $USER /data/ # Get ownership on data folder

Setup HTTPS LetsEncrypt Certificates

sudo apt install software-properties-common
sudo add-apt-repository ppa:certbot/certbot
sudo apt update
sudo apt install certbot
mkdir /home/<YOUR_USER>/letsencrypt
sudo certbot certonly --webroot -w /home/<YOUR_USER>/letsencrypt -d <your_domaine.name.tld>

## To Renew Certificates (90 days)
sudo certbot renew

Installation should be done.

How to update IMPatienT

(impatient) $ git pull                              # download the new version
(impatient) $ sudo supervisorctl stop impatient     # stop the current server
(impatient) $ flask db upgrade                      # upgrade the database
(impatient) $ sudo supervisorctl start impatient    # start a new server

How to pull data using DVC

Using DVC you can pull the database and data with a simple dvc pull. You might need to modify .dvc/config to configure your access.

Register New users

You can register an user using flask shell:

flask shell
> user = User(username="demo", email="demo@demo.demo")
> user.set_password("demo")
> db.session.add(user)
> db.session.commit()

IMPatienT Log Files

/var/log/nginx/access.log
# All request in NGINX Format (IP, Date, Page)
/var/log/nginx/error.log
# NGINX Errors (often empty)

/var/log/impatient_access.log
# All Request in UNICORN Format (IP, Date, Page)
/var/log/impatient_error.log
# GUNICORN Errors (often empty)

/var/log/supervisor/impatient-stderr---supervisor-XXXXXXX.log
# Stdout GUNICORN + Python Errors (& worker print)

/var/log/supervisor/impatient-stdout---supervisor-XXXXXXX.log
# Print statement (stdout)

/home/<YOUR_USER>/impatient/logs/IMPatienT.log
# Python Errors
# Also the traceback sent by mail to the adress configures in IMPatienT

Automatic daily Backup of SQLITE DB and Ontology json.

mkdir /backup/
> crontab -e
0 0 * * * cd /backup/ && /bin/cp /home/<YOUR_USER>/impatient/data/ontology/ontology.json ontology.json && /bin/cp /home/<YOUR_USER>/impatient/data/database/app.db app.db && /bin/tar --remove-files -czf $(date +"%FT%H%M")-backup.tar.gz app.db ontology.json
> crontab -l

Config Files content

.env file

SECRET_KEY=<YOUR-SECRET-KEY>
MAIL_SERVER=<YOUR-MAIL-SERVER>
MAIL_PORT=<YOUR-MAIL-PORT>
MAIL_USE_TLS=1
MAIL_USERNAME=<YOUR-EMAIL>
MAIL_PASSWORD=<MDP-EMAIL>

Supervisor Process

[program:impatient]
command=/home/<YOUR_USER>/impatient/.venv/bin/gunicorn -b localhost:8000 --timeout 900 --workers 4 --threads 8 --preload impatient:app
directory=/home/<YOUR_USER>/impatient
user=<YOUR_USER>
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true

Nginx

server {
    # listen on port 80 (http)
    listen 80;
    server_name impatient.lbgi.fr;
        location ~ /.well-known {
        root /home/<YOUR_USER>/letsencrypt;
    }
    location / {
        # redirect any requests to the same URL but on https
        return 301 https://$host$request_uri;
    }
}
server {
    # listen on port 443 (https)
    listen 443 ssl;
    server_name impatient.lbgi.fr;

    # location of the self-signed SSL certificate
    ssl_certificate /etc/letsencrypt/live/impatient.lbgi.fr/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/impatient.lbgi.fr/privkey.pem;

    # write access and error logs to /var/log
    access_log /var/log/impatient_access.log;
    error_log /var/log/impatient_error.log;

    location / {
        # forward application requests to the gunicorn server
        proxy_pass http://localhost:8000;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

###############################################
# IN NGINX.CONF ; HTML SECTION
# Corentin Custom Config for large file (1gb Microscopy Image)
        keepalive_timeout 900s;
        #extra
        proxy_read_timeout 900s;
        uwsgi_read_timeout 900s;
        client_max_body_size 1000M;