1-gentle-introductory-series/volume-3-analytics-model-building-deployment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Gentle Introduction to PyDAAL: Vol 3 of 3 Analytics Model Building and Deployment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Earlier in the Gentle Introduction Series ([Volume 1](volume-1-data-structures.ipynb) and [Volume 2](volume-2-basic-operations-on-numeric-tables.ipynb)), we covered fundamentals of the Intel® Data Analytics Acceleration Library (Intel® DAAL) custom Data Structure and basic operations that can be performed. Volume 3 will focus on the algorithm component of Intel® DAAL where the data management element is leveraged to drive analysis and build machine learning models<br>\n",
    "\n",
    "Intel® DAAL has classes available to construct a wide range of popular machine learning algorithms for analytics model building that include classification, regression, recommender systems and neural networks. Training and Prediction are separated into 2 pieces in Intel® DAAL model building. This separation allows the user to store and transfer only what’s needed for prediction when it comes time for model deployment. Typical Machine Learning workflow involves:<br><br>\n",
    "        • **Training stage** that includes identifying patterns in input data that maps behavior of data features in accordance with a target variable.<br>\n",
    "        • **Prediction stage ** that requires employing the trained model on a new data set.<br><br>\n",
    "Additionally, Intel® DAAL also contains on-board model scoring, in the form of separate classes to evaluate trained model performance and compute standard quality metrics. Various sets of quality metrics can be reported based on the type of analytics model built.<br>\n",
    "\n",
    "To accelerate the model building process, Intel® DAAL is reinforced with the distributed parallel processing mode for large data sets, including a programming model that makes it easy for users to implement a Master-Slave approach. Mpi4py can be easily interfaced with PyDAAL, as Intel® DAAL’s serialization and deserialization classes enable data exchange between nodes during parallel computation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Volumes in Gentle Introduction Series\n",
    "•\t**[Vol 1: Data Structures](volume-1-data-structures.ipynb)** - Covers introduction to  Data Management component of Intel® DAAL and available custom Data Structures(Numeric Table and Data Dictionary) with code examples.<br>\n",
    "•\t**[Vol 2: Basic Operations on Numeric Tables](volume-2-basic-operations-on-numeric-tables.ipynb)** - Covers introduction to possible operations that can be performed on Intel® DAAL's custom Data Structure (Numeric Table and Data Dictionary) with code examples.<br>\n",
    "•\t**Vol 3: Analytics Model Building and Deployment** – Covers introduction to analytics model building and evaluation in Intel® DAAL with serialized deployment and distributed model fitting on large datasets.<br>\n",
    "\n",
    "## Analytics Modelling:\n",
    "###  1.\tBatch Learning with PyDAAL:\n",
    "Intel DAAL includes classes that support the following stages in analytics model building and deployment process:\n",
    "\n",
    "1.\t[Training](#TrainPredict)\n",
    "2.\t[Prediction](#TrainPredict)\n",
    "3.\t[Model Evaluation and Quality Metrics](#QualityMet)\n",
    "4.\t[Trained Model Storage and Portability](#ModelPort)\n",
    "\n",
    "### 1.1\tAnalytics Modelling Training and Prediction Workflow: \n",
    "\n",
    "\n",
    "  <img style=\"float: center;\" src=\"_notebook-related-files/TrainPredictWFImage.png\">\n",
    "    \n",
    "    \n",
    "### <a id='TrainPredict'> 1.2 Build and Predict with PyDAAL Analytics Models: </a>\n",
    "   <img style=\"float: left;\" src=\"_notebook-related-files/TrainPredictRibbon.png\"> <br>As described earlier, Intel DAAL model building is separated into two different stages with two associated classes (“training”, “prediction”).<br>\n",
    "The training stage usually involves complex computations with possibly very large datasets, calling for extensive memory footprint. DAAL’s two separate classes allows users to perform training stage on a powerful machine, and optionally the subsequent prediction stage on a simpler machine. Furthermore, this facilitates the user to store and transmit only necessary training stage results that are required for prediction stage.<br><br><br><br>\n",
    "\n",
    "Four numeric tables are created at the beginning of model building process, two in each stage (training and prediction) as listed below:<br>\n",
    "\n",
    "|Stage|\tNumeric Tables| Description|\n",
    "| :-: |:-:|:-:|\n",
    "|<div style=\"text-align: left\">Training</div>\t|<div style=\"text-align: left\">trainData</div>|<div style=\"text-align: left\">\tThis includes the feature values/predictors</div>|\n",
    "|<div style=\"text-align: left\">Training</div>|<div style=\"text-align: left\">\ttrainDependentVariables</div>|<div style=\"text-align: left\">\tThis includes the target  values *(i.e., labels/responses)*</div>|\n",
    "|<div style=\"text-align: left\">Prediction</div>|<div style=\"text-align: left\">\ttestData</div>|<div style=\"text-align: left\">\tThis includes the feature values/predictors of test data</div>|\n",
    "|<div style=\"text-align: left\">Prediction</div>|<div style=\"text-align: left\">\ttestGroundTruth</div>|<div style=\"text-align: left\">\tThis includes the target  *(i.e., labels/responses)*</div>|\n",
    "\n",
    "\n",
    "**Note:** See [Volume 2](volume-2-basic-operations-on-numeric-tables.ipynb) for details on creating and working with numeric tables. <br>\n",
    " The table below illustrates a high-level overview on training and prediction stages of the analytics model building process:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float: center;\" src=\"_notebook-related-files/TrainPredictWF.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Helper Functions:  Linear Regression***\n",
    "\n",
    "  The next section can be copy/pasted into a user’s script or adapted to a specific use case. The Helper function block provided below can be used directly to automate the training and prediction stages of DAAL’s Linear Regression algorithm. The helper function is followed be a full usage code example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "'''\n",
    "training() function\n",
    "-----------------\n",
    "Arguments:\n",
    "        train data of type numeric table, train dependent values of type numeric table\n",
    "Returns:\n",
    "        training results object \n",
    "'''\n",
    "def training(trainData,trainDependentVariables):\n",
    "    from daal.algorithms.linear_regression import training\n",
    "    algorithm = training.Batch ()\n",
    "    # Pass a training data set and dependent values to the algorithm\n",
    "    algorithm.input.set (training.data, trainData)\n",
    "    algorithm.input.set (training.dependentVariables, trainDependentVariables)\n",
    "    trainingResult = algorithm.compute ()\n",
    "    return trainingResult\n",
    "\n",
    "'''\n",
    "prediction() function\n",
    "-----------------\n",
    "Arguments:\n",
    "        training result object, test data of type numeric table\n",
    "Returns:\n",
    "        predicted responses of type numeric table\n",
    "'''\n",
    "def prediction(trainingResult,testData):\n",
    "    from daal.algorithms.linear_regression import  prediction, training\n",
    "    algorithm = prediction.Batch()\n",
    "    # Pass a testing data set and the trained model to the algorithm\n",
    "    algorithm.input.setTable(prediction.data, testData)\n",
    "    algorithm.input.setModel(prediction.model, trainingResult.get(training.model))\n",
    "    predictionResult = algorithm.compute ()\n",
    "    predictedResponses = predictionResult.get(prediction.prediction)\t\n",
    "    return predictedResponses\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*To use: copy the complete block of helper function and call training()\n",
    " and prediction() methods.*\n",
    " \n",
    "***<a id='TrainPredictLR-use'>Usage Example: Linear Regression</a>***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#import required modules\n",
    "from daal.data_management import HomogenNumericTable\n",
    "import numpy as np\n",
    "from utils import printNumericTable\n",
    "seeded = np.random.RandomState (42)\n",
    "\n",
    "#set up train and test numeric tables\n",
    "trainData =HomogenNumericTable(seeded.rand(200,10))\n",
    "trainDependentVariables = HomogenNumericTable(seeded.rand (200, 2))\n",
    "testData =HomogenNumericTable(seeded.rand(50,10))\n",
    "testGroundTruth = HomogenNumericTable(seeded.rand (50, 2))\n",
    "\n",
    "#--------------\n",
    "#Training Stage\n",
    "#--------------\n",
    "trainingResult = training(trainData,trainDependentVariables)\n",
    "#--------------\n",
    "#Prediction Stage\n",
    "#--------------\n",
    "predictionResult = prediction(trainingResult, testData)\n",
    "\n",
    "#Print and compare results\n",
    "printNumericTable (predictionResult, \"Linear Regression prediction results: (first 10 rows):\", 10)\n",
    "printNumericTable (testGroundTruth, \"Ground truth (first 10 rows):\", 10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notes:**<br>\n",
    "[Helper function classes](https://github.com/preethivenkatesh/pydaal-getting-started/tree/master/2-pre-built-helper-classes) have been created using Intel DAAL’s low level API for popular algorithms to perform various stages of model building and deployment process. These classes contain training and prediction stages as methods, and are available in daaltces’s GitHub repository. These functions require only input arguments to be passed in each stage as shown in the usage example.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  ###  <a id='QualityMet'> 1.3\tTrained Model Evaluation and Quality Metrics:</a>\n",
    "  \n",
    "  <img style=\"float: left;\" src=\"_notebook-related-files/QualityMetRibbon.png\"> <br>Intel DAAL offers quality metrics classes for binary classifiers, multi-class classifiers and regression algorithms to measure quality of the trained model. Various standard metrics are computed by Intel DAAL quality metrics library for different types of analytics modeling. <br><br><br><br><br><br>\n",
    "  \n",
    "  \n",
    "  \n",
    "**Binary Classification:**<br><br>\n",
    "Accuracy, Precision, Recall, F1-score, Specificity, AUC<br><br>\n",
    "[Click here](https://software.intel.com/en-us/daal-programming-guide-quality-metrics-for-binary-classification-algorithms) for more details on notations and definitions.<br><br>\n",
    "**Multi-class Classification:**<br><br>\n",
    "Average accuracy, Error rate, Micro precision (Precision μ ), Micro recall (Recall μ ), Micro F-score (F-score μ ), Macro precision (Precision M), Macro recall (Recall M), Macro F-score (F-score M)<br><br>\n",
    "[Click here](https://software.intel.com/en-us/daal-programming-guide-quality-metrics-for-multi-class-classification-algorithms) for more details on notations and definitions.\n",
    "\n",
    "\n",
    "**Regression:**<br><br>\n",
    "For regression models, Intel DAAL computes metrics using 2 libraries:\n",
    "1.\tSingle Beta: Computes and produces metrics results based on Individual beta coefficients of trained model.<br>\n",
    "RMSE, Vector of variances, variance-covariance matrices, Z-score statistics<br><br>\n",
    "2.\tGroup Beta: Computes and produces metrics results based on group of beta coefficients of trained model.<br>\n",
    "Mean and Variance of expected responses, Regression Sum of Squares, Sum of Squares of Residuals, Total Sum of Squares, Coefficient of Determination, F-Statistic<br>\n",
    "\n",
    "[Click here](https://software.intel.com/en-us/daal-programming-guide-quality-metrics-for-linear-regression) for more details on notations and definitions.\n",
    "\n",
    "**Notes:**<br>\n",
    "[Helper function classes](https://github.com/preethivenkatesh/pydaal-getting-started/tree/master/2-pre-built-helper-classes) have been created using Intel DAAL’s low level API for popular algorithms to perform various stages of model building and deployment process. These classes contain quality metrics methods, and are available in daaltces’s GitHub repository. These functions require only input arguments to be passed in each stage as shown in the usage example.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <a id='ModelPort'>1.4\tTrained Model Storage and Portability:</a>\n",
    "  <img style=\"float: left;\" src=\"_notebook-related-files/ModelPortRibbon.png\"> <br>Trained models can be serialized into byte-type numpy arrays and deserialized using Intel DAAL’s data archive classes to: <br>\n",
    "  \n",
    "•\tSupport data transmission between devices.<br>\n",
    "•\tSave and restore from disk at a later date to predict response for an incoming observation or re-train the model with a set of new observations.<br>\n",
    "\n",
    "Optionally, to reduce network traffic and memory footprint, serialized models can further be compressed and later decompressed using the deserialization method.\n",
    "\n",
    "##### Steps to attain model portability in Intel DAAL:\n",
    "\n",
    "*  Serialization:\n",
    "\n",
    "    a.\tSerialize training stage results(trainingResults) into Intel DAAL’s  Input Data Archive object.<br>\n",
    "    b.\tCreate an empty byte type numpy array object(bufferArray) of size Input Data Archive object. <br>\n",
    "    c.\tPopulate bufferArray with Input Data Archive contents.<br>\n",
    "    d.\tCompress bufferArray to numpy array object (optional).<br>\n",
    "    e.\tSave bufferArray as .npy file to disk (optional).<br>\n",
    "\n",
    "\n",
    "*  Deserialization:\n",
    "\n",
    "    a.\tLoad .npy file from disk to numpy array object(if Serialization step 1e was performed).<br>\n",
    "    b.\tDecompress numpy array object to bufferArray (if Serialization step 1d was performed).<br>\n",
    "    c.\tCreate Intel DAAL’s Output Data Archive object with bufferArray contents.<br>\n",
    "    d.\tCreate an empty original training stage results object (trainingResults).<br>\n",
    "    e.\tDeserialize Output Data Archive contents into trainingResults.<br>\n",
    "\n",
    "**Note:** As mentioned in deserialization step 2d, an empty original training results object is required for Intel DAAL’s data archive methods to deserialize the serialized training results object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***<a id='SerDesLR-use'>Helper Functions:  Linear Regression</a>***\n",
    "  \n",
    "  The next section can be copy/pasted into a user’s script or adapted to a specific use case. The Helper function block provided below can be used directly to automate the training and prediction stages of DAAL’s Linear Regression algorithm. The helper function is followed be a full usage code example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import warnings\n",
    "from daal.data_management import  (HomogenNumericTable, InputDataArchive, OutputDataArchive, \\\n",
    "                                   Compressor_Zlib, Decompressor_Zlib, level9, DecompressionStream, CompressionStream)\n",
    "'''\n",
    "Arguments: serialized numpy array\n",
    "Returns Compressed numpy array\n",
    "'''\n",
    "\n",
    "def compress(arrayData):\n",
    "    compressor = Compressor_Zlib ()\n",
    "    compressor.parameter.gzHeader = True\n",
    "    compressor.parameter.level = level9\n",
    "    comprStream = CompressionStream (compressor)\n",
    "    comprStream.push_back (arrayData)\n",
    "    compressedData = np.empty (comprStream.getCompressedDataSize (), dtype=np.uint8)\n",
    "    comprStream.copyCompressedArray (compressedData)\n",
    "    return compressedData\n",
    "\n",
    "'''\n",
    "Arguments: deserialized numpy array\n",
    "Returns decompressed numpy array\n",
    "'''\n",
    "def decompress(arrayData):\n",
    "    decompressor = Decompressor_Zlib ()\n",
    "    decompressor.parameter.gzHeader = True\n",
    "    # Create a stream for decompression\n",
    "    deComprStream = DecompressionStream (decompressor)\n",
    "    # Write the compressed data to the decompression stream and decompress it\n",
    "    deComprStream.push_back (arrayData)\n",
    "    # Allocate memory to store the decompressed data\n",
    "    bufferArray = np.empty (deComprStream.getDecompressedDataSize (), dtype=np.uint8)\n",
    "    # Store the decompressed data\n",
    "    deComprStream.copyDecompressedArray (bufferArray)\n",
    "    return bufferArray\n",
    "\n",
    "#-------------------\n",
    "#***Serialization***\n",
    "#-------------------\n",
    "'''\n",
    "Method 1:\n",
    "    Arguments: data(type nT/model)\n",
    "    Returns  dictionary with serailized array (type object) and object Information (type string)\n",
    "Method 2:\n",
    "    Arguments: data(type nT/model), fileName(.npy file to save serialized array to disk)\n",
    "    Saves serialized numpy array as \"fileName\" argument\n",
    "    Saves object information as \"filename.txt\"\n",
    " Method 3:\n",
    "    Arguments: data(type nT/model), useCompression = True\n",
    "    Returns  dictionary with compressed array (type object) and object information (type string)\n",
    "Method 4:\n",
    "    Arguments: data(type nT/model), fileName(.npy file to save serialized array to disk), useCompression = True\n",
    "    Saves compresseed numpy array as \"fileName\" argument\n",
    "    Saves object information as \"filename.txt\"\n",
    "'''\n",
    "\n",
    "def serialize(data, fileName=None, useCompression= False):\n",
    "    buffArrObjName = (str(type(data)).split()[1].split('>')[0]+\"()\").replace(\"'\",'')\n",
    "    dataArch = InputDataArchive()\n",
    "    data.serialize (dataArch)\n",
    "    length = dataArch.getSizeOfArchive()\n",
    "    bufferArray = np.zeros(length, dtype=np.ubyte)\n",
    "    dataArch.copyArchiveToArray(bufferArray)\n",
    "    if useCompression == True:\n",
    "        if fileName != None:\n",
    "            if len (fileName.rsplit (\".\", 1)) == 2:\n",
    "                fileName = fileName.rsplit (\".\", 1)[0]\n",
    "            compressedData = compress(bufferArray)\n",
    "            np.save (fileName, compressedData)\n",
    "        else:\n",
    "            comBufferArray = compress (bufferArray)\n",
    "            serialObjectDict = {\"Array Object\":comBufferArray,\n",
    "                                \"Object Information\": buffArrObjName}\n",
    "            return serialObjectDict\n",
    "    else:\n",
    "        if fileName != None:\n",
    "            if len (fileName.rsplit (\".\", 1)) == 2:\n",
    "                fileName = fileName.rsplit (\".\", 1)[0]\n",
    "            np.save(fileName, bufferArray)\n",
    "        else:\n",
    "            serialObjectDict = {\"Array Object\": bufferArray,\n",
    "                                \"Object Information\": buffArrObjName}\n",
    "            return serialObjectDict\n",
    "    infoFile = open (fileName + \".txt\", \"w\")\n",
    "    infoFile.write (buffArrObjName)\n",
    "    infoFile.close ()\n",
    "\n",
    "\n",
    "#---------------------\n",
    "#***Deserialization***\n",
    "#---------------------\n",
    "'''\n",
    "Returns deserialized/ decompressed numeric table/model\n",
    "Input can be serialized/ compressed numpy array or serialized/ compressed .npy file saved to disk\n",
    "'''\n",
    "def deserialize(serialObjectDict = None, fileName=None,useCompression = False):\n",
    "    import daal\n",
    "    if fileName!=None and serialObjectDict == None:\n",
    "        bufferArray = np.load(fileName)\n",
    "        buffArrObjName = open(fileName.rsplit (\".\", 1)[0]+\".txt\",\"r\").read()\n",
    "    elif  fileName == None and any(serialObjectDict):\n",
    "        bufferArray = serialObjectDict[\"Array Object\"]\n",
    "        buffArrObjName = serialObjectDict[\"Object Information\"]\n",
    "    else:\n",
    "         warnings.warn ('Expecting \"bufferArray\" or \"fileName\" argument, NOT both')\n",
    "         raise SystemExit\n",
    "    if useCompression == True:\n",
    "        bufferArray = decompress(bufferArray)\n",
    "    dataArch = OutputDataArchive (bufferArray)\n",
    "    try:\n",
    "        deSerialObj = eval(buffArrObjName)\n",
    "    except AttributeError :\n",
    "        deSerialObj = HomogenNumericTable()\n",
    "    deSerialObj.deserialize(dataArch)\n",
    "    return deSerialObj\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*To use: copy the complete block of helper function and call serialize()\n",
    " and deserialize() methods.*<br>\n",
    " \n",
    " ***Usage Example: Linear Regression***\n",
    " \n",
    " The example below Implements serialize() and deserialize() functions on Linear Regression trainingResult. (Refer [Linear Regression](#TrainPredictLR-use) usage example from section Build and Predict with PyDAAL Analytics Models to compute trainingResult )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Serialize\n",
    "serialTrainingResultArray = serialize(trainingResult) # Run Usage Example: Linear Regression from section 1.2 \n",
    "#Deserialize\n",
    "deserialTrainingResult = deserialize(serialTrainingResultArray)\n",
    "\n",
    "#predict\n",
    "predictionResult = prediction(deserialTrainingResult, testData)\n",
    "\n",
    "#Print and compare results\n",
    "printNumericTable (predictionResult, \"Linear Regression deserialized prediction results: (first 10 rows):\", 10)\n",
    "printNumericTable (testGroundTruth, \"Ground truth (first 10 rows):\", 10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examples below implement other combinations of serialize()  and deserialize() methods with different input arguments "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "#---compress and serialize\n",
    "serialTrainingResultArray = serialize(trainingResult, useCompression=True)\n",
    "#---decompress and deserialize \n",
    "deserialTrainingResult = deserialize(serialTrainingResultArray, useCompression=True)\n",
    "\n",
    "\n",
    "\n",
    "#---serialize and save to disk as numpy array\n",
    "serialize(trainingResult,fileName=\"trainingResult\")\n",
    "#---deserialize file from disk\n",
    "deserialTrainingResult = deserialize(fileName=\"trainingResult.npy\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:**<br>\n",
    "[Helper function classes](https://github.com/preethivenkatesh/pydaal-getting-started/tree/master/2-pre-built-helper-classes) have been created using Intel DAAL’s low level API for popular algorithms to perform various stages of model building and deployment process. These classes contain model storage and portability stages as methods, and are available in daaltces’s GitHub repository. These functions require only input arguments to be passed in each stage as shown in the usage example.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.  Distributed Learning with PyDAAL and MPI:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyDAAL and mpi4py can be used to easily distribute model training for many of DAAL’s algorithm implementations using the Single Program Multiple Data (SPMD) technique. Other Python machine learning libraries allow for the trivial implementation of a parameter-tuning grid search, mainly because it’s an “embarrassingly parallel” process. What sets Intel DAAL apart is the included IA-optimized distributed versions of many of its model training classes. This means acceleration of single model training is enabled with similar syntax to batch learning. For these implementations, the DAAL engineering team has provided a slave method to compute partial training results on row-grouped chunks of data, and then a master method for reduction of the partial results into a final model result.\n",
    "\n",
    "###   2.1 Serialization and Message Passings:\n",
    "\n",
    "Messages passed with MPI4Py are passed as serialized objects. MPI4Py uses the popular Python object serialization library Pickle under the hood during this process. PyDAAL uses SWIG (Simplified Wrapper and Interface Generator) as its wrapper interface. Unfortunately SWIG is not compatible with Pickle. Fortunately, DAAL has built-in serialized and deserialization functionality. See Trained Model Portability section for details.  The table below demonstrates the master and slave methods for the distributed version of PyDAAL’s covariance model method. \n",
    "\n",
    "*NOTE: The serialize() and deserialize() helper functions are provided in the [Trained Model Portability](#SerDesLR-use) section of this volume.*\n",
    "\n",
    "<img style=\"float: center;\" src=\"_notebook-related-files/TrainPredictWF.png\">\n",
    "\n",
    "*** Helper Functions: Covariance Matrix***\n",
    "\n",
    "*Note: The upcoming helper function will require [customUtlis](https://github.com/preethivenkatesh/pydaal-getting-started/tree/master/3-custom-modules/customUtils)  module to be imported from daaltces GitHub Repository*\n",
    "\n",
    "The next section can be copy/pasted into a user’s script or adapted to a specific use case. The helper function block provided below can be used carry out distributed computation of the covariance matrix, but can be adapted for fitting other types of models. See [Computation Modes](https://software.intel.com/en-us/daal-programming-guide-computation-modes) section in developer’s docs for more details on distributed model fitting.  The helper function is followed be a full usage code example.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "''' \n",
    "Defined Slave and Master Routines as Python Functions \n",
    "Returns serialized partial model result. Input is serialized partial numeric table \n",
    "'''\n",
    "from CustomUtils import getBlockOfNumericTable, serialize, deserialize\n",
    "from daal.data_management import HomogenNumericTable\n",
    "from daal.algorithms.covariance import (\n",
    "    Distributed_Step1LocalFloat64DefaultDense, data, partialResults,\n",
    "    Distributed_Step2MasterFloat64DefaultDense\n",
    ")\n",
    "\n",
    "   \n",
    "def computestep1Local(serialnT):\n",
    "   # Deseralize using Helper Function\n",
    "   partialnT = deserialize(serialnT)\n",
    "   # Create partial model object\n",
    "   model = Distributed_Step1LocalFloat64DefaultDense()\n",
    "   # Set input data for the model\n",
    "   model.input.set(data, partialnT)\n",
    "   # Get the computed partial estimate result\n",
    "   partialResult = model.compute()\n",
    "   # Seralize using Helper Function\n",
    "   serialpartialResult = serialize(partialResult)\n",
    "    \n",
    "   return serialpartialResult\n",
    "\n",
    "# Define master compute routine\n",
    "''' \n",
    "Imports global variable finalResult. Computes master version of model and sets full model result into finalResult. Inputs are array of serialized partial results and MPI world size \n",
    "'''\n",
    "def computeOnMasterNode(serialPartialResult, size):\n",
    "    global finalResult\n",
    "    # Create master model object\n",
    "    model = Distributed_Step2MasterFloat64DefaultDense()\n",
    "    # Add partial results to the distributed master model\n",
    "    for i in range(size):        \n",
    "        # Deseralize using Helper Function\n",
    "        partialResult = deserialize(serialPartialResult[i])  \n",
    "        # Set input objects for the model\n",
    "        model.input.add(partialResults, partialResult)\n",
    "    # Recompute a partial estimate after combining partial results\n",
    "    model.compute()\n",
    "    # Finalize the result in the distributed processing mode\n",
    "    finalResult = model.finalizeCompute()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*** Usage Example:  Covariance Matrix***\n",
    "\n",
    "The below example uses the complete block of helper functions given above and implements computestep1Local(), computeOnMasterNode() functions with mpi4py to construct a Covariance Matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from mpi4py import MPI\n",
    "from CustomUtils import getBlockOfNumericTable, serialize, deserialize\n",
    "from daal.data_management import HomogenNumericTable\n",
    "\n",
    "''' \n",
    "Begin MPI Initialization and Run Options \n",
    "'''\n",
    "# Get MPI vars\n",
    "size = MPI.COMM_WORLD.Get_size()\n",
    "rank = MPI.COMM_WORLD.Get_rank()\n",
    "name = MPI.Get_processor_name()    \n",
    "   \n",
    "# Initialize result vars to fill\n",
    "serialPartialResults = [0] * size\n",
    "finalResult = None\n",
    "\n",
    "''' \n",
    "Begin Data Set Creation \n",
    "\n",
    "The below example variable values can be used:\n",
    "numRows, numCols = 1000, 100\n",
    "\n",
    "'''\n",
    "# Create random array for demonstration\n",
    "# numRows, numCols defined by user\n",
    "seeded = np.random.RandomState(42)\n",
    "fullArray = seeded.rand(numRows, numCols) \n",
    "\n",
    "# Build seeded random data matrix, and slice into chunks\n",
    "# rowStart and rowEnd determined by size of the chunks\n",
    "sizeOfChunk = int(numRows/size)\n",
    "rowStart = rank*sizeOfChunk\n",
    "rowEnd = ((rank+1)*sizeOfChunk)-1\n",
    "array = fullArray[rowStart:rowEnd, :]\n",
    "partialnT = HomogenNumericTable(array)\n",
    "serialnT = serialize(partialnT)\n",
    "\n",
    "\n",
    "'''\n",
    "Begin Distributed Execution \n",
    "'''\n",
    "\n",
    "if rank == 0:\n",
    "\n",
    "   serialPartialResults[rank] = computestep1Local(serialnT)\n",
    "   \n",
    "   if size > 1:\n",
    "      # Begin receive slave partial results on master\n",
    "      for i in range(1, size):\n",
    "         rank, size, name, serialPartialResults[rank] = MPI.COMM_WORLD.recv(source=MPI.ANY_SOURCE, tag=1)\n",
    "\n",
    "   computeOnMasterNode(serialPartialResults,size)\n",
    "\n",
    "else:\n",
    "   serialPartialResult =  computestep1Local(serialnT)\n",
    "   MPI.COMM_WORLD.send((rank, size, name, serialPartialResult), dest=0, tag=1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion:\n",
    "\n",
    "Previous volumes (Volume 1 and Volume 2) demonstrated Intel® Data Analytics Acceleration Library’s (Intel® DAAL) Numeric Table data structure and basic operations on Numeric Tables. Volume 3 discussed Intel® DAAL’s algorithm component and performing analytical modelling through different stages with both batch and distributed processing. Also, Volume 3 demonstrated how to achieve model probability (Serialization) and perform model evaluation (Quality Metrics) process. Furthermore, this volume utilized Intel® DAAL classes to provide helper functions that deliver a standalone solution in model fitting and deployment process. \n",
    "\n",
    "## Other Related Links:\n",
    "\n",
    "•\t[Gentle Introduction to PyDAAL: Vol 1 of 3 Data Structures](volume-1-data-structures.ipynb)<br>\n",
    "•\t[Gentle Introduction to PyDAAL: Vol 2 of 3 Basic Operations on Numeric Tables](volume-2-basic-operations-on-numeric-tables.ipynb)<br>\n",
    "•\t[Developer Guide for Intel® DAAL](https://software.intel.com/en-us/daal-programming-guide)<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}