how_does_dask_work.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How does Dask work? Why should I use Dask?\n",
    "\n",
    "Dask allows you to scale up your most favorite functions such as Pandas, NumPy, and Scikit-learn to use on Big Data. \n",
    "\n",
    "Arrays, Dataframes, and semi-structured/unstructured data take up a great deal of RAM when processed directly with Pandas, Numpy, and Scikit-learn. These functions are not great at memory management so they front-load their work which puts stress on your RAM and causes it to slow down when fed in too much data.\n",
    "\n",
    "<img src=\"./Example-Structured-Unstructured-Semi-Structured-Data-e1638423518970.jpg\"/>\n",
    "\n",
    "Dask resolves this by efficiently breaking down your data and feeding it to the RAM in bite sized pieces, ultimately combining it back into readable and usable data once complete.\n",
    "\n",
    "Dask has three main \"collections\" called Dask Array, Dask Dataframe, and Dask-ML (Dask Bag) which are alternatives to Pandas DataFrame, Numpy Array and Scikit-learn. \n",
    "\n",
    "Below, we will be descibing how Dask manages each of these collections to make them more efficient.\n",
    "\n",
    "<br>\n",
    "\n",
    "### **Dask Array**\n",
    "A Dask array is made up of smaller Numpy arrays that are fed into an algorithm to allow computation on arrays that are larger than your available RAM memory. During an Array operation, Dask translates the array operation into a task graph which breaks up large Numpy arrays into multiple smaller chunks, and executes the work on each chunk at the same time. This is called parallel computing. Results from each chunk are combined to produce the final output. \n",
    "\n",
    "### **Dask Dataframe**\n",
    "A Dask DataFrame maintains the familiar Pandas code structure and language, making it easy for Pandas users to scale up DataFrame workloads, just replace ```pd``` with ```dd```. During a DataFrame operation, Dask creates a task graph and breaks down the dataframe into smaller parts that reduces memory footprint and increases RAM efficiency by sharing and deleting intermediate results while computing.\n",
    "\n",
    "### **Dask-ML (Dask Bag)**\n",
    "Dask Bag is used on semi-structured/unstructured data such as JSON records, text data, log files, or user-defined Python objects using operations such as filter, fold, map and groupby. This allows you to do things such as hefty text analysis and data scraping that would be difficult without Dask's proper resource management. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Who uses Dask? What can it be used for?\n",
    "\n",
    "\n",
    "\n",
    "### *Life sciences*\n",
    "\n",
    "**Harvard Medical School, Howard Hughes Medical Institute, Chan Zuckerberg Initiative, and the UC Berkeley Advanced Bioimaging Center** all use Dask for high resolution, 4-dimensional, cellular imagery. This generates large amounts of data that is difficult to analyze with traditional methods. \n",
    "- Dask helps them scale their data analysis workflows with its familiar API that resembles NumPy, Pandas, and Scikit-learn code. \n",
    "\n",
    "Dask is also used at the Novartis Institute for Biomedical Research to scale machine learning prototypes.\n",
    "\n",
    "<br>\n",
    "\n",
    "### *Geophysical sciences*\n",
    "\n",
    "Dask is used in Climate Science, Energy, Hydrology, Meteorology, and Satellite Imaging by companies such as NASA, LANL, PANGEO, and the UK Meteorology Office.\n",
    "\n",
    "- With Dask, oceanographers can produce massive simulated datasets of the earth’s oceans and researchers can look at large seismology datasets from sensors around the world, collect a large number of observations from satellites and weather stations, and run big simulations.\n",
    "\n",
    "<br>\n",
    "\n",
    "### *Corporations*\n",
    "**Walmart** uses Dask for forecasting the demand for 500,000,000 store-item combinations. To provide in-demand items in sufficient quantities at all their outlets, they need to run huge computations. Using RAPIDS and XGBoost, supported by Dask, they reached 100x acceleration.\n",
    "\n",
    "**Blue Yonder** uses Dask to process terabytes of data on a daily basis. They can write Pandas-like code in Dask, which can then be pushed directly to production. This helps keep their feedback cycles short and waste low.\n",
    "\n",
    "**Capital One** uses Dask to accelerate their machine learning pipelines.\n",
    "\n",
    "**Barclays** usesd Dask for financial system modeling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sources:\n",
    "\n",
    "### https://en.wikipedia.org/wiki/Dask_(software)#High-Level_collections\n",
    "\n",
    "### https://stories.dask.org/en/latest/\n",
    "\n",
    "### https://earth-env-data-science.github.io/lectures/dask/dask_arrays.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.5 ('mikeys_env')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.5"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "579423154b4eaee7bfe5a430418673f411175c3f3abce33f4969df0aa8270f69"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}