Tutorial_1_Quick_start.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "from typing import List, Optional, Union\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from tsururu.dataset import IndexSlicer, Pipeline, TSDataset\n",
    "from tsururu.models import CatBoostRegressor_CV\n",
    "from tsururu.strategies import (\n",
    "    DirectStrategy,\n",
    "    FlatWideMIMOStrategy,\n",
    "    MIMOStrategy,\n",
    "    RecursiveStrategy,\n",
    ")\n",
    "from tsururu.transformers import (\n",
    "    DateSeasonsGenerator,\n",
    "    DifferenceNormalizer,\n",
    "    LagTransformer,\n",
    "    LastKnownNormalizer,\n",
    "    SequentialTransformer,\n",
    "    StandardScalerTransformer,\n",
    "    TargetGenerator,\n",
    "    UnionTransformer,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_results(\n",
    "    cv: int,\n",
    "    regime: str,\n",
    "    y_true: Optional[List[np.ndarray]] = None,\n",
    "    y_pred: Optional[List[np.ndarray]] = None,\n",
    "    ids: Optional[List[Union[float, str]]] = None,\n",
    ") -> pd.DataFrame:\n",
    "    def _get_fold_value(\n",
    "        value: Optional[Union[float, np.ndarray]], idx: int\n",
    "    ) -> List[Optional[Union[float, np.ndarray]]]:\n",
    "        if value is None:\n",
    "            return [None]\n",
    "        if isinstance(value[idx], float):\n",
    "            return value[idx]\n",
    "        if isinstance(value[idx], np.ndarray):\n",
    "            return value[idx].reshape(-1)\n",
    "        raise TypeError(f\"Unexpected value type. Value: {value}\")\n",
    "\n",
    "    df_res_dict = {}\n",
    "\n",
    "    for idx_fold in range(cv):\n",
    "        # Fill df_res_dict\n",
    "        for name, value in [(\"y_true\", y_true), (\"y_pred\", y_pred)]:\n",
    "            df_res_dict[f\"{name}_{idx_fold+1}\"] = _get_fold_value(\n",
    "                value, idx_fold\n",
    "            )\n",
    "        if regime != \"local\":\n",
    "            df_res_dict[f\"id_{idx_fold+1}\"] = _get_fold_value(ids, idx_fold)\n",
    "\n",
    "    # Save datasets to specified directory\n",
    "    df_res = pd.DataFrame(df_res_dict)\n",
    "    return df_res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several main objects to look out for when working with the library:\n",
    "1) `TSDataset`.\n",
    "2) `Pipeline` and `Transformers`\n",
    "3) `Strategy`.\n",
    "4) `Model`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TSDataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This class is needed to store data and meta-information about it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To initialise it is necessary to submit the data in `pd.DataFrame` format and define some meta-information about roles that necessary for solving the task of time series forecasting: `id`, `date`, `target`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_path = \"datasets/global/simulated_data_to_check.csv\"\n",
    "\n",
    "dataset_params = {\n",
    "    \"target\": {\n",
    "        \"columns\": [\"value\"],\n",
    "        \"type\": \"continious\",\n",
    "    },\n",
    "    \"date\": {\n",
    "        \"columns\": [\"date\"],\n",
    "        \"type\": \"datetime\",\n",
    "    },\n",
    "    \"id\": {\n",
    "        \"columns\": [\"id\"],\n",
    "        \"type\": \"categorical\",\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "freq: Day; period: 1\n"
     ]
    }
   ],
   "source": [
    "dataset = TSDataset(\n",
    "    data=pd.read_csv(df_path),\n",
    "    columns_params=dataset_params,\n",
    "    print_freq_period_info=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipeline and Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What kind of transformers are there?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Special attention should be paid to the `Transformer` class: the elements of the pipeline that are responsible for transforming the values of a series and generating features. `Pipeline` class is a wrapper over transformers which is needed to provide some additional methods and functions above them.\n",
    "\n",
    "There are two types of transformers that are used to collect pipelines:\n",
    "- `Union` transformers;\n",
    "- `Sequential` transformers.\n",
    "\n",
    "Below is a list of available Transformers: \n",
    "- `StandardScalerTransformer` *(Series4Series)*.\n",
    "- `DifferenceNormalizer` *(Series4Series)*: subtract the previous value or divide by it.\n",
    "- `LastKnownNormalizer` *(Features4Features)*: normalize all lags by the last known one: divide by it or subtract.\n",
    "\n",
    "This three transformers provide flags `transform_features` / `transform_target`, that allow you to manipulate traits and targets separately and get different results from them.\n",
    "\n",
    "Besides, __DifferenceNormalizer__ and __LastKnownNormalizer__ can be applied in two regimes: `delta` and `ratio`: in the first case, normalisation means subtracting the target value from the current value, and in the second, dividing by it.\n",
    "\n",
    "- `LabelEncodingTransformer` and `OneHotEncodingTransformer` *(Series4Series)* - encoders for categorical features.\n",
    "- `TimeToNumGenerator` and `DateSeasonsGenerator` *(Series4Series)* - generator for seasonal features by dates.\n",
    "- `LagTransformer` *(Series4Features) - generator for lags. \n",
    "\n",
    "__!!!The lag transformer must necessarily be present in the sequential transformer, otherwise the features will not be generated.!!!__\n",
    "\n",
    "Finally, to generate targets, you need to use `TargetGenerator`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transformers must be assembled in order!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The __SeriesToSeries__ transformers should come first, followed by the LagTransformer and TargetGenerator (__SeriesToFeatures__), and then the __FeaturesToFeatures__ transformers.\n",
    "\n",
    "!!!Thus, `StandardScalerNormalizer` and `DifferenceNormalizer` should be before `LagTransformer` and `LastKnownNormalizer` after it!!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to build a Pipeline?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, there are two ways to build a pipline from transformers: initialise the transformers of interest by hand or use a config in the form of a dictionary. Let's look at both ways."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "standard_scaler = StandardScalerTransformer(\n",
    "    transform_features=True,\n",
    "    transform_target=True\n",
    ")\n",
    "\n",
    "lag = LagTransformer(lags=3)\n",
    "date_lag = LagTransformer(lags=3)\n",
    "id_lag = LagTransformer(lags=1)\n",
    "\n",
    "target_generator = TargetGenerator()\n",
    "\n",
    "date_seasons = DateSeasonsGenerator(\n",
    "    seasonalities=[\"doy\", \"m\", \"wd\"],\n",
    "    from_target_date=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "union_1 = UnionTransformer(transformers_list=[lag, target_generator])\n",
    "\n",
    "seq_1 = SequentialTransformer(transformers_list=[standard_scaler, union_1], input_features=[\"value\"])\n",
    "seq_2 = SequentialTransformer(transformers_list=[date_seasons, date_lag], input_features=[\"date\"])\n",
    "seq_3 = SequentialTransformer(transformers_list=[id_lag], input_features=[\"id\"])\n",
    "\n",
    "union = UnionTransformer(transformers_list=[seq_1, seq_2, seq_3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_1 = Pipeline(union, multivariate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'transformers': <tsururu.transformers.base.UnionTransformer at 0x7f49080b21c0>,\n",
       " 'multivariate': False,\n",
       " 'is_fitted': False,\n",
       " 'strategy_name': None,\n",
       " 'output_features': None,\n",
       " 'y_original_shape': None}"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_1.__dict__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_params = {\n",
    "    \"target\": {\n",
    "        \"columns\": [\"value\"],\n",
    "        \"features\": {\n",
    "            \"StandardScalerTransformer\":\n",
    "                {\n",
    "                    \"transform_target\": True, \n",
    "                    \"transform_features\": True\n",
    "                },\n",
    "            \"LagTransformer\": {\"lags\": 7},\n",
    "        },\n",
    "    },\n",
    "    \"date\": {\n",
    "        \"columns\": [\"date\"],\n",
    "        \"features\": {\n",
    "            \"DateSeasonsGenerator\": {\n",
    "                # Use seasonality features from the date column as \n",
    "                # features with datetime lags\n",
    "                # Possible values: [\n",
    "                #    \"y\": year, \"m\": month, \"d\": day, \n",
    "                #    \"wd\": weekday, \"doy\": dayofyear,\n",
    "                #    \"hour\": hour, \"min\": minute, \"sec\": second, \n",
    "                #    \"ms\": microsecond,  \"ns\": nanosecond\n",
    "                # ]\n",
    "                \"seasonalities\": ['doy', 'm', 'wd'], \n",
    "                # Use date from target point to make datetime features\n",
    "                \"from_target_date\": True,\n",
    "            },\n",
    "            \"LagTransformer\": {\"lags\": 3}\n",
    "        },\n",
    "    },\n",
    "    \"id\": {\n",
    "        \"columns\": [\"id\"],\n",
    "        \"features\": {\n",
    "            \"LagTransformer\": {\"lags\": 1},\n",
    "        },\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline.from_dict(pipeline_params, multivariate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_easy_params = {\n",
    "    # One from [\"none\", \"standard_scaler\", \"difference_normalizer\", \"last_known_normalizer\"]\n",
    "    \"target_normalizer\": \"standard_scaler\",\n",
    "\n",
    "    # One from [\"none\", \"delta\", \"ratio\"]  (MUST BE \"none\" for \"standard_scaler\" and NOT \"none\" for others)\n",
    "    \"normalizer_regime\": \"none\",\n",
    "\n",
    "    # One from [\"features\", \"target\", \"features_target\"]\n",
    "    \"normalizer_transform_regime\": \"features_target\",\n",
    "\n",
    "    \"target_lags\": 3,\n",
    "    \"date_lags\": 1,\n",
    "#    \"exog_lags\": 1,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline.easy_setup(dataset_params, pipeline_easy_params, multivariate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Can I use exogenous variables in the pipeline?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes! Exogenous variables can also be specified here. Just include them in your pipeline.\n",
    "\n",
    "However, their operation is currently tested only for the `MIMOStrategy` in global-modelling. For other strategies support of additional variables is under development."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pipeline_params[\"exog_group_1\"] = {\n",
    "#     \"columns\": [\"value\"],\n",
    "#     \"features\": {\n",
    "#         \"StandardScalerTransformer\":\n",
    "#             {\n",
    "#                 \"transform_target\": False, \n",
    "#                 \"transform_features\": True\n",
    "#             },\n",
    "#         \"LagTransformer\": {\"lags\": 7},\n",
    "#     },\n",
    "# }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Make sure you have the transform_target = False flag for exogenous features!__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is separate from the strategy. Any model can be run in any strategy if it supports this input and output format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the easiest options – is to use GBM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure the model parameters\n",
    "model_params = {\n",
    "    \"loss_function\": \"MultiRMSE\",\n",
    "    \"early_stopping_rounds\": 100,\n",
    "    \"verbose\": 500,\n",
    "}\n",
    "\n",
    "# Configure the validation parameters\n",
    "validation_params = {\n",
    "    \"type\": 'KFold',\n",
    "    \"n_splits\": 2,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = CatBoostRegressor_CV(validation_params, model_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Strategy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- _Recursive:_ \n",
    "    - one model for all points of the forecast horizon;\n",
    "    - *training*: the model is trained to predict one point ahead;\n",
    "    - *prediction*: a prediction is iteratively made one point ahead, and then this prediction is used to further shape the features in the test data. \n",
    "- _Recursive-reduced:_\n",
    "    - one model for all points in the prediction horizon;\n",
    "    - *training*: the model is trained to predict one point ahead;\n",
    "    - *prediction*: features are generated for all test observations at once, unavailable values are replaced by NaN.\n",
    "- _Direct:_ \n",
    "    - individual models for each point in the prediction horizon. \n",
    "- _MultiOutput (MIMO - Multi-input-multi-output):_\n",
    "    - one model that learns to predict the entire prediction horizon. \n",
    "    - __Also, this strategy supports the presence of `exogenous features` (only for local- or global-modelling).__\n",
    "- _FlatWideMIMO:_.\n",
    "    - mixture of Direct and MIMO, fit one model, but uses deployed over horizon Direct's features.\n",
    "    - __Number of `lags for datetime features` should be equal to `horizon` while using this strategy.__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "horizon = 3\n",
    "history = 7\n",
    "step = 1\n",
    "\n",
    "strategy = RecursiveStrategy(horizon, history, step, model, pipeline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.9603852\ttest: 0.9667066\tbest: 0.9667066 (0)\ttotal: 8.15ms\tremaining: 8.14s\n",
      "500:\tlearn: 0.0043385\ttest: 0.0045465\tbest: 0.0045465 (500)\ttotal: 1.19s\tremaining: 1.18s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "999:\tlearn: 0.0027344\ttest: 0.0030471\tbest: 0.0030471 (999)\ttotal: 2.4s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.003047062205\n",
      "bestIteration = 999\n",
      "\n",
      "Fold 0:\n",
      "MultiRMSE: 0.003047062205400275\n",
      "0:\tlearn: 0.9660362\ttest: 0.9613700\tbest: 0.9613700 (0)\ttotal: 4.77ms\tremaining: 4.77s\n",
      "500:\tlearn: 0.0043259\ttest: 0.0046427\tbest: 0.0046427 (500)\ttotal: 1.18s\tremaining: 1.18s\n",
      "999:\tlearn: 0.0026124\ttest: 0.0029169\tbest: 0.0029169 (999)\ttotal: 2.4s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.002916854444\n",
      "bestIteration = 999\n",
      "\n",
      "Fold 1:\n",
      "MultiRMSE: 0.002916854443892958\n",
      "Mean MultiRMSE: 0.003\n",
      "Std: 0.0001\n"
     ]
    }
   ],
   "source": [
    "fit_time, _ = strategy.fit(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast_time, current_pred = strategy.predict(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2022-09-27</td>\n",
       "      <td>1997.088714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>2022-09-28</td>\n",
       "      <td>1997.191979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>2022-09-29</td>\n",
       "      <td>1995.496356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-09-27</td>\n",
       "      <td>2997.475196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-09-28</td>\n",
       "      <td>2997.553282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-09-29</td>\n",
       "      <td>2995.826568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>2022-09-27</td>\n",
       "      <td>3997.392796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>2022-09-28</td>\n",
       "      <td>3997.46765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2</td>\n",
       "      <td>2022-09-29</td>\n",
       "      <td>3995.743526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>3</td>\n",
       "      <td>2022-09-27</td>\n",
       "      <td>4997.548721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>3</td>\n",
       "      <td>2022-09-28</td>\n",
       "      <td>4997.572455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>3</td>\n",
       "      <td>2022-09-29</td>\n",
       "      <td>4995.842271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>4</td>\n",
       "      <td>2022-09-27</td>\n",
       "      <td>5997.700033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>4</td>\n",
       "      <td>2022-09-28</td>\n",
       "      <td>5997.725905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>4</td>\n",
       "      <td>2022-09-29</td>\n",
       "      <td>5995.998695</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>5</td>\n",
       "      <td>2022-09-27</td>\n",
       "      <td>6997.694582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>5</td>\n",
       "      <td>2022-09-28</td>\n",
       "      <td>6997.722867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>5</td>\n",
       "      <td>2022-09-29</td>\n",
       "      <td>6995.997412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>6</td>\n",
       "      <td>2022-09-27</td>\n",
       "      <td>7997.620737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>6</td>\n",
       "      <td>2022-09-28</td>\n",
       "      <td>7997.648131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>6</td>\n",
       "      <td>2022-09-29</td>\n",
       "      <td>7995.923171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>7</td>\n",
       "      <td>2022-09-27</td>\n",
       "      <td>8997.53187</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>7</td>\n",
       "      <td>2022-09-28</td>\n",
       "      <td>8997.559264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>7</td>\n",
       "      <td>2022-09-29</td>\n",
       "      <td>8995.845535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>8</td>\n",
       "      <td>2022-09-27</td>\n",
       "      <td>9997.5219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>8</td>\n",
       "      <td>2022-09-28</td>\n",
       "      <td>9997.546915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>8</td>\n",
       "      <td>2022-09-29</td>\n",
       "      <td>9995.828639</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>9</td>\n",
       "      <td>2022-09-27</td>\n",
       "      <td>10997.380235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>9</td>\n",
       "      <td>2022-09-28</td>\n",
       "      <td>10997.403622</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>9</td>\n",
       "      <td>2022-09-29</td>\n",
       "      <td>10995.683994</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id       date         value\n",
       "0   0 2022-09-27   1997.088714\n",
       "1   0 2022-09-28   1997.191979\n",
       "2   0 2022-09-29   1995.496356\n",
       "3   1 2022-09-27   2997.475196\n",
       "4   1 2022-09-28   2997.553282\n",
       "5   1 2022-09-29   2995.826568\n",
       "6   2 2022-09-27   3997.392796\n",
       "7   2 2022-09-28    3997.46765\n",
       "8   2 2022-09-29   3995.743526\n",
       "9   3 2022-09-27   4997.548721\n",
       "10  3 2022-09-28   4997.572455\n",
       "11  3 2022-09-29   4995.842271\n",
       "12  4 2022-09-27   5997.700033\n",
       "13  4 2022-09-28   5997.725905\n",
       "14  4 2022-09-29   5995.998695\n",
       "15  5 2022-09-27   6997.694582\n",
       "16  5 2022-09-28   6997.722867\n",
       "17  5 2022-09-29   6995.997412\n",
       "18  6 2022-09-27   7997.620737\n",
       "19  6 2022-09-28   7997.648131\n",
       "20  6 2022-09-29   7995.923171\n",
       "21  7 2022-09-27    8997.53187\n",
       "22  7 2022-09-28   8997.559264\n",
       "23  7 2022-09-29   8995.845535\n",
       "24  8 2022-09-27     9997.5219\n",
       "25  8 2022-09-28   9997.546915\n",
       "26  8 2022-09-29   9995.828639\n",
       "27  9 2022-09-27  10997.380235\n",
       "28  9 2022-09-28  10997.403622\n",
       "29  9 2022-09-29  10995.683994"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "current_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backtest validation of pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.9616026\ttest: 0.9653712\tbest: 0.9653712 (0)\ttotal: 8.42ms\tremaining: 8.41s\n",
      "500:\tlearn: 0.0042762\ttest: 0.0045139\tbest: 0.0045139 (500)\ttotal: 1.2s\tremaining: 1.19s\n",
      "999:\tlearn: 0.0026729\ttest: 0.0029726\tbest: 0.0029726 (999)\ttotal: 2.44s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.002972644218\n",
      "bestIteration = 999\n",
      "\n",
      "Fold 0:\n",
      "MultiRMSE: 0.0029726442175072565\n",
      "0:\tlearn: 0.9646266\ttest: 0.9624752\tbest: 0.9624752 (0)\ttotal: 5.31ms\tremaining: 5.31s\n",
      "500:\tlearn: 0.0043855\ttest: 0.0045979\tbest: 0.0045979 (500)\ttotal: 1.2s\tremaining: 1.19s\n",
      "999:\tlearn: 0.0026660\ttest: 0.0029718\tbest: 0.0029718 (999)\ttotal: 2.44s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.002971764957\n",
      "bestIteration = 999\n",
      "\n",
      "Fold 1:\n",
      "MultiRMSE: 0.002971764957214351\n",
      "Mean MultiRMSE: 0.003\n",
      "Std: 0.0\n"
     ]
    }
   ],
   "source": [
    "ids, test, pred, fit_time, forecast_time = strategy.back_test(dataset, cv=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>y_true_1</th>\n",
       "      <th>y_pred_1</th>\n",
       "      <th>id_1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1997.0</td>\n",
       "      <td>1993.68704</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1998.0</td>\n",
       "      <td>1994.202883</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1999.0</td>\n",
       "      <td>1994.669445</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2997.0</td>\n",
       "      <td>2993.991901</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2998.0</td>\n",
       "      <td>2994.586288</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2999.0</td>\n",
       "      <td>2995.082926</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3997.0</td>\n",
       "      <td>3993.914717</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3998.0</td>\n",
       "      <td>3994.509697</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>3999.0</td>\n",
       "      <td>3995.015013</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>4997.0</td>\n",
       "      <td>4994.022114</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4998.0</td>\n",
       "      <td>4994.615917</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>4999.0</td>\n",
       "      <td>4995.170471</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>5997.0</td>\n",
       "      <td>5994.175996</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>5998.0</td>\n",
       "      <td>5994.779734</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>5999.0</td>\n",
       "      <td>5995.319329</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>6997.0</td>\n",
       "      <td>6994.179421</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>6998.0</td>\n",
       "      <td>6994.784167</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>6999.0</td>\n",
       "      <td>6995.312331</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>7997.0</td>\n",
       "      <td>7994.113358</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>7998.0</td>\n",
       "      <td>7994.705541</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>7999.0</td>\n",
       "      <td>7995.236341</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>8997.0</td>\n",
       "      <td>8993.839156</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>8998.0</td>\n",
       "      <td>8994.431338</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>8999.0</td>\n",
       "      <td>8995.147741</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>9997.0</td>\n",
       "      <td>9993.838821</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>9998.0</td>\n",
       "      <td>9994.4268</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>9999.0</td>\n",
       "      <td>9995.1378</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>10997.0</td>\n",
       "      <td>10993.687397</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>10998.0</td>\n",
       "      <td>10994.270971</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>10999.0</td>\n",
       "      <td>10994.990313</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    y_true_1      y_pred_1 id_1\n",
       "0     1997.0    1993.68704    0\n",
       "1     1998.0   1994.202883    0\n",
       "2     1999.0   1994.669445    0\n",
       "3     2997.0   2993.991901    1\n",
       "4     2998.0   2994.586288    1\n",
       "5     2999.0   2995.082926    1\n",
       "6     3997.0   3993.914717    2\n",
       "7     3998.0   3994.509697    2\n",
       "8     3999.0   3995.015013    2\n",
       "9     4997.0   4994.022114    3\n",
       "10    4998.0   4994.615917    3\n",
       "11    4999.0   4995.170471    3\n",
       "12    5997.0   5994.175996    4\n",
       "13    5998.0   5994.779734    4\n",
       "14    5999.0   5995.319329    4\n",
       "15    6997.0   6994.179421    5\n",
       "16    6998.0   6994.784167    5\n",
       "17    6999.0   6995.312331    5\n",
       "18    7997.0   7994.113358    6\n",
       "19    7998.0   7994.705541    6\n",
       "20    7999.0   7995.236341    6\n",
       "21    8997.0   8993.839156    7\n",
       "22    8998.0   8994.431338    7\n",
       "23    8999.0   8995.147741    7\n",
       "24    9997.0   9993.838821    8\n",
       "25    9998.0     9994.4268    8\n",
       "26    9999.0     9995.1378    8\n",
       "27   10997.0  10993.687397    9\n",
       "28   10998.0  10994.270971    9\n",
       "29   10999.0  10994.990313    9"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_results(cv=1, regime=\"global\", y_true=test, y_pred=pred, ids=ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with raw time series' granularity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time series come in different granularities, from hourly and daily time series to more complex ones such as the end of each quarter.\n",
    "\n",
    "If the rows do not contain segments that are too short (that are shorter than history + horizon), then `tsururu` will try to extract the row granularity on its own. We currently support the following types:\n",
    "\n",
    "- Yearly (and YearlyEnd)\n",
    "- Quarterly (and Quarterly)\n",
    "- Monthly (and MonthlyEnd)\n",
    "- Weekly\n",
    "- Daily\n",
    "- Hourly\n",
    "- Minlutely\n",
    "- Secondly\n",
    "- Microsecondly\n",
    "\n",
    "There is also support for compound granularities (10 days, 15 minutes, 32 seconds, etc.). The correctness of the selected granularity can be checked from the output after the `Dataset` class has been created.\n",
    "\n",
    "However, there are tricky situations (e.g. 28 days) where the monthly granularity may be guessed incorrectly. Therefore, it is possible to set your own granularity using the `pd.DateOffset` class or related classes from `pandas.tseries.offsets`, which must be fed as `delta` parameter into the `Dataset` class. Then the time column will be processed according to the user's settings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider a time series where each point is exactly __28 daily points away__ from each other"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_path_2 = \"datasets/global/simulated_data_to_check_28D.csv\"\n",
    "\n",
    "# Configure the features settings\n",
    "dataset_params_2 = {\n",
    "    \"target\": {\n",
    "        \"columns\": [\"value\"],\n",
    "        \"type\": \"continious\",\n",
    "    },\n",
    "    \"date\": {\n",
    "        \"columns\": [\"date\"],\n",
    "        \"type\": \"datetime\",\n",
    "    },\n",
    "    \"id\": {\n",
    "        \"columns\": [\"id\"],\n",
    "        \"type\": \"categorical\",\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "freq: Month; period: 1.0\n"
     ]
    }
   ],
   "source": [
    "dataset_2 = TSDataset(\n",
    "    data=pd.read_csv(df_path_2),\n",
    "    columns_params=dataset_params_2,\n",
    "    print_freq_period_info=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the frequency of the series is incorrectly defined as monthly. Let's try to pass the `delta` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Custom OffSet: <DateOffset: days=28>\n"
     ]
    }
   ],
   "source": [
    "dataset_2 = TSDataset(\n",
    "    data=pd.read_csv(df_path_2),\n",
    "    columns_params=dataset_params_2,\n",
    "    delta=pd.DateOffset(days=28),\n",
    "    print_freq_period_info=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's all detected correctly."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tsururu",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}