escwaapi

A ViEWS2-based API for retrieving predictions, based on ESCWA-discussed specifications.

Pushing data to the API is a three-stage process:

(1) Fetch new forecasts from the prediction store and most recent predictors via viewser to your laptop using the get_views3.py script in escwatransfer.

(2) Upload the parquet files storing the forecasts and predictors to the API AWS box using scp (certificates required).

(3) Log in to the box (certificates required), and register the new forecasts and predictors in the API box’s database using the register_views3.py script.

If non-trivial modifications need to be made to the escwatransfer package (i.e. anything other than simple changes to the config files), ensure that the new code is present on your laptop and on the API box!

If a new run type (e.g. fatalities004) needs to be created, both lib_transfer.py (which runs on your laptop via get_views3.py and on the server via register_views3.py) and the low-level SQL tables on the box need to be modified. Take care that all the changes agree with each other!

Modifying lib_transfer.py to use a new run type

The structures of runs are specified by ‘specs’ in specs.py. Each spec is a function which returns a dictionary in the case of forecasts or a list of querysets in the case of predictors, and must of course have a unique name.

The forecast dictionary has four keys: cm_parent_model, cm_child_models, pgm_parent_model and pgm_child_models. The API was constructed to store ensembles, which are thought of as parents, and constituent models, thought of as children.

Each key in turn points to ether a single dictionary or a list of dictionaries, each of which specifies the charateristics of one column of data. Each has a model name obtained from config.fetch, a column (this is the name of the column in the data table fetched from the prediction store), an apiname (this is the name you want that same column to have in the API) and a transform, which must be one of the transforms specified in __get_transforms.

The predictors querysets are regular viewser querysets - the pgm data has been split into several querysets to reduce instantaneous memory consumption.

Once a new spec has been defined, it can be selected in config_fetch.py by choosing its spec_generator and the forecasts and predictors required should be fetched and stored on your laptop as temporary parquet files.

On the box, the new run type should be selected in config_store.py by specifying its name, which must correspond to a name in the API’s SQL tables - see below:

Modifying the API SQL tables to use a new run type

Forecasts and predictors are stored on the API box as SQL tables. There are further tables which are used to denote, for example, which run a given table column belongs to. The tables structure is as follows:

structure.generation

structure.generation - this is the most fundamental table. All types of run (e.g. fatalities002, fatalities003, etc) must be registered in this table before any other data relating to them can be stored. Each type of run has a name, a codebook name, and an id - the id identifies the type of run to the rest of the API. If you want to push a new type of run to the API, this table should be edited first and you need to choose a new unique id for the run. This can be done in psql using a command of the form

INSERT INTO structure.generation (id, name, codebook) VALUES (320, 'fatalities003', 'codebook_fatalities003_forecasts.json');

structure.model

structure.model - this table records what ‘models’ (i.e. columns from data tables of forecasts or predictions) belong to a run type (which must be specified already in the structure.generation table). These must correspond to the apinames you chose when creating a new spec in lib_transfer.py

Each model has a unique id, a generation_id (corresponsing to the structure.generation.id field), an loa (either pgm or cm), a parent (used for specifying whether a table is an ensemble table which has no parent, or a constituent table, in which case the parent is the ensemble it belongs to - predictor tables have no parents), a ‘node’ - this is the name of the column, e.g. main_mean_ln, ucdp_ged_os_best_sum, a ’dynasim’ value which is obsolescent and should be set to 1, and a ‘type_of_violence’ value, also obsolescent and should be set to ‘px’.

This table specifies, then, for a given run type, what columns should be in the dataframe pushed to the API, which is then mapped to an SQL table, and how those columns relate to each other at a given loa, i.e. an ensemble or a constituent model.

When a new run type needs to be created, this table needs to be edited, once it has been decided what columns will be made available in the API. This can be done in psql via a command of the form

INSERT INTO structure.model (id, generation_id, loa, parent, node, dynasim, type_of_violence) VALUES (223, 320, 'cm', '', 'main_mean_ln’, 1 , 'sb');

once for each column, taking care each time to use an id value which is not already in use, and that the generation_id matches that from structure.generation. In the example, the parent field is left blank, indicating that this is an ensemble column.

structure.register

structure.register - this table records which forecast and predictor columns have been pushed to the API. This table does not have to be edited manually except in emergencies - the register_views3.py script updates this table automatically, provided that the structure.generation and structure.model tables have been correctly updated.

The parameter ‘forecasts_run_prefix’ in config_store.py must match a run name in structure.generation. Assuming it does, the register_views3.py script infers from structure.generation and structure.model which columns it should expect to find in the dataframes specified in the forecasts and predictors filenames in config_store.py.

Works, including basic filtering, but code is throwaway quality, as it is built around the flat prediction files produced by ViEWS2. It's a terrible mess rightnow.

Supported ViEWS facilities:

• Multiple ViEWS runs simultaneously. Runs can be based on different ViEWS model definitions. Register runs using escwatransfer.
• Hierarchically nested ViEWS models (e.g. ensembles based on smaller ensembles based on components). Arbitrary depth can be specified, limited only by Python's recurssion. The tree-based metadata is delivered
• Delivery of both models (sc) and prediction steps for the models that have steps, if so desired.
• PG and C levels of analysis, with all categories of violence.

Supported paths and filters:

Paths:

• LOA (required)
• sb/ns/os/px (optional)
• individual model (optional)

Due to limitations in FastAPI, fuzzy paths (eg sb,ns) will not be possible at this time. Go one level above or run the API twice

Filters

• space (priogrid, countries (lists allowed, as well as GWNO et., BoundingBox, ISO))
• time (month, ISO dates)
• TODO : stored filtersets (escwa, africa etc.)

Won't do : Due to FastAPI limitations, country=AFG,ALG type filters are not available. You will have to work with country=AFG&country=ALG for multiple (Array-based) filters.

Paging

• works. Implemented next_url and prev_url URL scheme for user convenience.

Security and

• No SQL injection should be possible (dual layer of abstraction via FastAPI and SQLAlchemy) but no ORM (so no guarantees)
• TODO : API key (maybe?)

Requires :

• a Postgres Database (edit libdb/config.py).
• a ViEWS2 model hierarchy, manually built in the database in the structure. I did not write a dependency compiler from yamlor from Frederick's model dataclasses since that in itself is a huge amount of work, for something that has been very static until now. Ideally, start with a simple structure dump (in escwatransfer).
• escwatransfer to transfer data from views2 (janus or hermes) into ESCWA. Access to views2 or views2 dumps is required for transferring data.
• TODO : dockerize this whole mess and write an INSTALL file.