welcome.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome to the Iguazio MLOps Platform\n",
    "\n",
    "An initial introduction to the Iguazio MLOps Platform and the platform tutorials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Platform Overview](#platform-overview)\n",
    "- [Data Science Workflow](#data-science-workflow)\n",
    "- [The Tutorial Notebooks](#the-tutorial-notebooks)\n",
    "- [Getting-Started Tutorial](#getting-started-tutorial)\n",
    "- [End-to-End Use-Case Application and How-To Demos](#demos)\n",
    "- [Installing and Updating the MLRun Python Package](#mlrun-python-pkg-install-n-update)\n",
    "- [Additional Platform Resources](#platform-resources)\n",
    "- [Miscellaneous](#misc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"platform-overview\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Platform Overview\n",
    "\n",
    "The Iguazio MLOps Platform (**\"the platform\"**) is a fully integrated and secure data science platform as a service (PaaS), which simplifies development, accelerates performance, facilitates collaboration, and addresses operational challenges.\n",
    "The platform incorporates the following components:\n",
    "\n",
    "- A data science workbench that includes Jupyter Notebook, integrated analytics engines, and Python packages\n",
    "- The [MLRun](https://mlrun.readthedocs.io) open-source MLOps orchestration framework for ML model management with experiments tracking and pipeline automation\n",
    "- Managed data and machine-learning (ML) services over a scalable Kubernetes cluster\n",
    "- A real-time serverless functions framework for model serving ([Nuclio](https://nuclio.io/))\n",
    "- An extremely fast and secure data layer that supports SQL, NoSQL, time-series databases, files (simple objects), and streaming\n",
    "- Integration with third-party data sources such as Amazon S3, HDFS, SQL databases, and streaming or messaging protocols\n",
    "- Real-time dashboards based on Grafana\n",
    "\n",
    "<br><img src=\"./assets/images/igz-self-service-platform.png\" alt=\"Self-service data science platform\" width=\"650\"/><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"data-science-workflow\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Science Workflow\n",
    "\n",
    "The platform provides a complete data science workflow in a single ready-to-use platform that includes all the required building blocks for creating data science applications from research to production:\n",
    "\n",
    "- Collect, explore, and label data from various real-time or offline sources\n",
    "- Run ML training and validation at scale over multiple CPUs and GPUs\n",
    "- Deploy models and applications into production with serverless functions\n",
    "- Log, monitor, and visualize all your data and services\n",
    "\n",
    "![Data Science Workflow](./assets/images/igz-data-science-workflow.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"the-tutorial-notebooks\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Tutorial Notebooks\n",
    "\n",
    "The home directory of the platform's running-user directory (**/User/&lt;running user&gt;**) contains pre-deployed tutorial Jupyter notebooks with code samples and documentation to assist you in your development &mdash; including a [**demos**](demos/README.ipynb) directory with end-to-end use-case applications (see the next section).\n",
    "\n",
    "> **Note:**\n",
    "> - To view and run the tutorials from the platform, use your IDE. See [Setting up your Environment](https://www.iguazio.com/docs/latest-release/intro/setup-env/). (Alternatively, you can create a Jupyter Notebook service).\n",
    "> - The **welcome.ipynb** notebook and main **README.md** file provide the same introduction in different formats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-tutorial\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting-Started Tutorial\n",
    "\n",
    "Start out by running the Quick start tutorial to familiarize yourself with the platform and experience firsthand some of its main capabilities.\n",
    "\n",
    "<a href=\"https://docs.mlrun.org/en/stable/tutorials/01-mlrun-basics.html\"><img src=\"./assets/images/view-tutorial-button.png\" alt=\"View tutorial\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"demos\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End-to-End Use-Case Application and How-To Demos\n",
    "\n",
    "Iguazio provides full end-to-end use-case application and how-to demos that demonstrate how to use the platform, its MLRun service, and related tools to address data science requirements for different industries and implementations.\n",
    "These demos are available in the [MLRun demos repository](https://github.com/mlrun/demos).\n",
    "Use the provided [**update-demos.sh**](./update-demos.sh) script to get updated demos from this repository.\n",
    "By default, the script retrieves the files from the latest release that matches the version of the installed `mlrun` package (see [Installing and Updating the MLRun Python Package](#mlrun-python-pkg-install-n-update)).\n",
    "The files are copied to the **/v3io/users/&lt;username&gt;/demos** directory, where `<username>` is the name of the running user (`$V3IO_USERNAME`) unless you set the `-u|--user` flag to another username.\n",
    "> **Note:** Before running the script, close any open files in the **demos** directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get additional demos\n",
    "!/User/update-demos.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For full usage instructions, run the script with the `-h` or `--help` flag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!/User/update-demos.sh --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"end-to-end-use-case-applications\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### End-to-End Use-Case Application Demos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "    <tr align=\"left\" style=\"border-bottom: 1pt solid black;\">\n",
    "    <th>Demo</th>\n",
    "    <th/>\n",
    "    <th/>\n",
    "    <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td><b>Mask detection</b></td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a href=\"demos/mask-detection/README.md\"><img src=\"./assets/images/Jupyter-Logo-32px.png\"/><br>Open locally</a>\n",
    "        </td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a target=\"_blank\" href=\"https://github.com/mlrun/demos/tree/release/v1.6.x-latest/mask-detection/\">\n",
    "                <img src=\"./assets/images/GitHub-Mark-32px.png\"/><br>View on GitHub</a>\n",
    "        </td>\n",
    "        <td>This demo contains 3 notebooks where we:\n",
    "      1. Train and evaluate a model for detecting if a person is wearing a mask in an image by using Tensorflow.Keras or PyTorch.<br>\n",
    "      2. Serve the model as a serverless function in a http endpoint.<br>\n",
    "      3. Write an automatic pipeline where we download a dataset of images, train and evaluate the model, then optimize the model (using ONNX) and serve it.\n",
    "        </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td><b>Fraud Prevention - Iguazio Feature Store</b></td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a href=\"demos/fraud-prevention-feature-store/README.md\"><img src=\"./assets/images/Jupyter-Logo-32px.png\"/><br>Open locally</a>\n",
    "        </td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a target=\"_blank\" href=\"https://github.com/mlrun/demos/tree/release/v1.6.x-latest/fraud-prevention-feature-store/\">\n",
    "                <img src=\"./assets/images/GitHub-Mark-32px.png\"/><br>View on GitHub</a>\n",
    "        </td>\n",
    "        <td>Demonstrates the feature store usage for fraud prevention: Data ingestion & preparation; Model training & testing; Model serving; Building An Automated ML Pipeline.\n",
    "        </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td><b>News Article Summarization and Keyword Extraction via NLP</b></td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a href=\"demos/news-article-nlp/README.md\"><img src=\"./assets/images/Jupyter-Logo-32px.png\"/><br>Open locally</a>\n",
    "        </td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a target=\"_blank\" href=\"https://github.com/mlrun/demos/tree/release/v1.6.x-latest/news-article-nlp/\"><img src=\"./assets/images/GitHub-Mark-32px.png\"/><br>View on GitHub</a>\n",
    "        </td>\n",
    "        <td>This demo creates an NLP pipeline that summarizes and extract keywords from a news article URL. We will be using state-of-the-art transformer models. such as BERT. to perform these NLP tasks.\n",
    "Additionally, we will use MLRun's real-time inference graphs to create the pipeline. This allows for easy containerization and deployment of the pipeline on top of a production-ready Kubernetes cluster.\n",
    "        </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td><b>NetOps Demo: Predictive Network Operations/Telemetry</b></td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a href=\"demos/network-operations/README.md\"><img src=\"./assets/images/Jupyter-Logo-32px.png\"/><br>Open locally</a>\n",
    "        </td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a target=\"_blank\" href=\"https://github.com/mlrun/demos/tree/release/v1.6.x-latest/network-operations/\"><img src=\"./assets/images/GitHub-Mark-32px.png\"/><br>View on GitHub</a>\n",
    "        </td>\n",
    "        <td>This demo demonstrates how to build an automated machine-learning (ML) pipeline for predicting network outages based on network-device telemetry, also known as Network Operations (NetOps).\n",
    "The demo implements feature engineering, model training, testing, inference, and model monitoring (with concept-drift detection).\n",
    "The demo uses a offline/real-time metrics simulator to generate semi-random network telemetry data that is used across the pipeline.\n",
    "        </td>\n",
    "    </tr>\n",
    "\t <tr>\n",
    "        <td><b>Stocks Prediction</b></td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a href=\"demosstocke-prediction/README.md\"><img src=\"./assets/images/Jupyter-Logo-32px.png\"/><br>Open locally</a>\n",
    "        </td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a target=\"_blank\" href=\"https://github.com/mlrun/demos/tree/release/v1.6.x-latest/stocks-prediction/\"><img src=\"./assets/images/GitHub-Mark-32px.png\"/><br>View on GitHub</a>\n",
    "        </td>\n",
    "        <td>This demo illustrates using Iguazio's latest technologies and methods for model serving, the platform feature store, and the MLRun frameworks (sub-modules for the most commonly \n",
    "\t\tused machine and deep learning frameworks, providing features such as automatic logging, model management, and distributed training). The demo predicts stock prices, \n",
    "\t\tand it creates a Grafana dashbord for model analysis.\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"howto-demos\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How-To Demos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "    <tr align=\"left\" style=\"border-bottom: 1pt solid black;\">\n",
    "    <th>Demo</th>\n",
    "    <th/>\n",
    "    <th/>\n",
    "    <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td><b>How-To: Converting existing ML code to an MLRun project</b></td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a href=\"demos/howto/converting-to-mlrun/mlrun-code.ipynb\"><img src=\"./assets/images/Jupyter-Logo-32px.png\"/><br>Open locally</a>\n",
    "        </td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a target=\"_blank\" href=\"https://github.com/mlrun/demos/tree/release/v1.6.x-latest/howto/converting-to-mlrun\"><img src=\"./assets/images/GitHub-Mark-32px.png\"/><br>View on GitHub</a>\n",
    "        </td>\n",
    "        <td>Demonstrates how to convert existing ML code to an MLRun project.\n",
    "            The demo implements an MLRun project for taxi ride-fare prediction based on a <a href=\"https://www.kaggle.com/jsylas/python-version-of-top-ten-rank-r-22-m-2-88\">Kaggle notebook</a> with an ML Python script that uses data from the <a href=\"https://www.kaggle.com/c/new-york-city-taxi-fare-prediction\">New York City Taxi Fare Prediction competition</a>.\n",
    "        </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td><b>How-To: Running a Spark job for reading a CSV file</b></td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a href=\"demos/howto/spark/spark-mlrun-read-csv.ipynb\"><img src=\"./assets/images/Jupyter-Logo-32px.png\"/><br>Open locally</a>\n",
    "        </td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a target=\"_blank\" href=\"https://github.com/mlrun/demos/tree/release/v1.6.x-latest/howto/spark/spark-mlrun-read-csv.ipynb\"><img src=\"./assets/images/GitHub-Mark-32px.png\"/><br>View on GitHub</a>\n",
    "        </td>\n",
    "        <td>Demonstrates how to run a Spark job that reads a CSV file and logs the data set to an MLRun database.\n",
    "        </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td><b>How-To: Running a Spark job for analyzing data</b></td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a href=\"demos/howto/spark/spark-mlrun-describe.ipynb\"><img src=\"./assets/images/Jupyter-Logo-32px.png\"/><br>Open locally</a>\n",
    "        </td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a target=\"_blank\" href=\"https://github.com/mlrun/demos/tree/release/v1.6.x-latest/howto/spark/spark-mlrun-describe.ipynb\"><img src=\"./assets/images/GitHub-Mark-32px.png\"/><br>View on GitHub</a>\n",
    "        </td>\n",
    "        <td>Demonstrates how to create and run a Spark job that generates a profile report from an Apache Spark DataFrame based on pandas profiling.\n",
    "        </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td><b>How-To: Running a Spark Job with Spark Operator</b></td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a href=\"demos/howto/spark/spark-operator.ipynb\"><img src=\"./assets/images/Jupyter-Logo-32px.png\"/><br>Open locally</a>\n",
    "        </td>\n",
    "        <td align=\"center\", style=\"min-width:45px; padding: 10px;\">\n",
    "            <a target=\"_blank\" href=\"https://github.com/mlrun/demos/tree/release/v1.6.x-latest/howto/spark/spark-operator.ipynb\"><img src=\"./assets/images/GitHub-Mark-32px.png\"/><br>View on GitHub</a>\n",
    "        </td>\n",
    "        <td>Demonstrates how to use <a target=\"_blank\" href=\"https://github.com/GoogleCloudPlatform/spark-on-k8s-operator\">Spark Operator</a> to run a Spark job over Kubernetes with MLRun.\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"mlrun-python-pkg-install-n-update\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing and Updating the MLRun Python Package\n",
    "\n",
    "The demo applications and many of the platform tutorials use [MLRun](https://docs.mlrun.org) &mdash; Iguazio's end-to-end open-source MLOps solution for managing and automating your entire analytics and machine-learning life cycle, from data ingestion through model development to full pipeline deployment in production.\n",
    "MLRun is available in the platform via a default (pre-deployed) shared platform service (`mlrun`).\n",
    "However, to use MLRun from Python code (such as in the demo and tutorial notebooks), you also need to install the [MLRun Python package](https://docs.mlrun.org) (`mlrun`).\n",
    "The version of the installed package must match the version of the platform's MLRun service and must be updated whenever the service's version is updated.\n",
    "\n",
    "The platform provides an [**align_mlrun.sh**](./align_mlrun.sh) script for simplifying the MLRun package installation and version synchronization with the MLRun service.\n",
    "The script is available in the running-user directory (your Jupyter home directory), which is accessible via the `/User` data mount.\n",
    "Use the following command to run this script for the initial package installation (after creating a new Jupyter Notebook service) and whenever the MLRun service is updated; (the command should be run for each Jupyter Notebook service):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!/User/align_mlrun.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"platform-resources\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Platform Resources\n",
    "\n",
    "You can find more information and resources in the MLRun documentation:\n",
    "<br>\n",
    "**&#x25B6; [View the MLRun documentation](https://docs.mlrun.org)**\n",
    "\n",
    "You might also find the following resources useful:\n",
    "\n",
    "- [Introduction video](https://www.youtube.com/watch?v=GS_4XUIudsY)\n",
    "- [Platform Services](https://www.iguazio.com/docs/latest-release/services/)\n",
    "- [Python SDK for management APIs](https://www.iguazio.com/docs/latest-release/cluster-mgmt/mgmt-sdk/): A Python SDK for controlling and performing operations on the the Iguazio system via REST-API.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"misc\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Miscellaneous"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"creating-virtual-environments-in-jupyter-notebook\"></a>\n",
    "### Creating Virtual Environments in Jupyter Notebook\n",
    "\n",
    "A virtual environment is a named, isolated, working copy of Python that maintains its own files, directories, and paths so that you can work with specific versions of libraries or Python itself without affecting other Python projects.\n",
    "Virtual environments make it easy to cleanly separate projects and avoid problems with different dependencies and version requirements across components.\n",
    "See the [virtual-env](virtual-env.ipynb) tutorial notebook for step-by-step instructions for using conda to create your own Python virtual environments, which will appear as custom kernels in Jupyter Notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"update-notebooks\"></a>\n",
    "### Updating the Tutorial Notebooks\n",
    "\n",
    "You can use the provided **igz-tutorials-get.sh** script to get updated platform tutorials from the [tutorials GitHub repository](https://github.com/v3io/tutorials/).\n",
    "By default, the script retrieves the files from the latest release that matches the current platform version.\n",
    "For details, see the [**update-tutorials.ipynb**](update-tutorials.ipynb) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"v3io-dir\"></a>\n",
    "### The v3io Directory\n",
    "\n",
    "The **v3io** directory that you see in the file browser of the Jupyter UI displays the contents of the `v3io` data mount for browsing the platform data containers. For information about the platform's data containers and how to reference data in these containers, see [Data Containers](https://www.iguazio.com/docs/latest-release/services/data-layer/containers/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"support\"></a>\n",
    "### Support\n",
    "\n",
    "The Iguazio [support team](mailto:support@iguazio.com) will be happy to assist with any questions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}