lesson8_part2_lasso_and_ridge.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Занятие 8. Задача восстановления регрессии. Обучение без учителя"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Часть 2. Lasso и Ridge регрессия"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "from __future__ import division, print_function\n",
    "# отключим всякие предупреждения Anaconda\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "%pylab inline\n",
    "# import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.datasets import load_boston\n",
    "from sklearn.cross_validation import cross_val_score, KFold\n",
    "from sklearn.linear_model import LassoCV, RidgeCV, Lasso, Ridge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**Будем работать с набором данных по ценам на дома в Бостоне (репозиторий UCI).**\n",
    "**Загружаем данные.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "boston = load_boston()\n",
    "X, y = boston['data'], boston['target']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**Почитаем описание набора данных:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Boston House Prices dataset\n",
      "===========================\n",
      "\n",
      "Notes\n",
      "------\n",
      "Data Set Characteristics:  \n",
      "\n",
      "    :Number of Instances: 506 \n",
      "\n",
      "    :Number of Attributes: 13 numeric/categorical predictive\n",
      "    \n",
      "    :Median Value (attribute 14) is usually the target\n",
      "\n",
      "    :Attribute Information (in order):\n",
      "        - CRIM     per capita crime rate by town\n",
      "        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.\n",
      "        - INDUS    proportion of non-retail business acres per town\n",
      "        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
      "        - NOX      nitric oxides concentration (parts per 10 million)\n",
      "        - RM       average number of rooms per dwelling\n",
      "        - AGE      proportion of owner-occupied units built prior to 1940\n",
      "        - DIS      weighted distances to five Boston employment centres\n",
      "        - RAD      index of accessibility to radial highways\n",
      "        - TAX      full-value property-tax rate per $10,000\n",
      "        - PTRATIO  pupil-teacher ratio by town\n",
      "        - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\n",
      "        - LSTAT    % lower status of the population\n",
      "        - MEDV     Median value of owner-occupied homes in $1000's\n",
      "\n",
      "    :Missing Attribute Values: None\n",
      "\n",
      "    :Creator: Harrison, D. and Rubinfeld, D.L.\n",
      "\n",
      "This is a copy of UCI ML housing dataset.\n",
      "http://archive.ics.uci.edu/ml/datasets/Housing\n",
      "\n",
      "\n",
      "This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.\n",
      "\n",
      "The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic\n",
      "prices and the demand for clean air', J. Environ. Economics & Management,\n",
      "vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics\n",
      "...', Wiley, 1980.   N.B. Various transformations are used in the table on\n",
      "pages 244-261 of the latter.\n",
      "\n",
      "The Boston house-price data has been used in many machine learning papers that address regression\n",
      "problems.   \n",
      "     \n",
      "**References**\n",
      "\n",
      "   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.\n",
      "   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.\n",
      "   - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(boston.DESCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**Признаки:**\n",
    "- CRIM - количество преступлений на душу населения\n",
    "- ZN - процент жилых участков площадью больше 25 тыс. кв. футов (~ 23 сотки)\n",
    "- INDUS - процент площадей под оптовую торговлю \n",
    "- CHAS - протекает ли река\n",
    "- NOX - концентрация оксидов азота\n",
    "- RM - среднее число комнат в здании\n",
    "- AGE - доля зданий, построенных до 1940 года \n",
    "- DIS - взвешенное расстояние до 5 деловых центров Бостона\n",
    "- RAD - индекс доступности скоростных магистралей\n",
    "- TAX - уровень налогов\n",
    "- PTRATIO - среднее число учащихся на одного преподавателя \n",
    "- B - процент афроамериканцев\n",
    "- LSTAT - процент граждан с низким уровнем жизни\n",
    "- MEDV (целевой) - медианная стоимости домов в районе"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD',\n",
       "       'TAX', 'PTRATIO', 'B', 'LSTAT'],\n",
       "      dtype='<U7')"
      ]
     },
     "execution_count": 4,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "boston.feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**Посмотрим на первые 2 записи.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  6.32000000e-03,   1.80000000e+01,   2.31000000e+00,\n",
       "          0.00000000e+00,   5.38000000e-01,   6.57500000e+00,\n",
       "          6.52000000e+01,   4.09000000e+00,   1.00000000e+00,\n",
       "          2.96000000e+02,   1.53000000e+01,   3.96900000e+02,\n",
       "          4.98000000e+00],\n",
       "       [  2.73100000e-02,   0.00000000e+00,   7.07000000e+00,\n",
       "          0.00000000e+00,   4.69000000e-01,   6.42100000e+00,\n",
       "          7.89000000e+01,   4.96710000e+00,   2.00000000e+00,\n",
       "          2.42000000e+02,   1.78000000e+01,   3.96900000e+02,\n",
       "          9.14000000e+00]])"
      ]
     },
     "execution_count": 5,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Lasso-регрессия"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Lasso-регрессия решает задачу минимизации среднеквардатичной ошибки с L1-регуляризацией:\n",
    "$$\\Large error(X, y, \\beta) = \\frac{1}{2n} \\sum_{i=1}^n {(y_i - \\beta^Tx_i)}^2 + \\alpha \\sum_{i=1}^n |\\beta_i|$$\n",
    "\n",
    "где $y = \\beta^Tx$ - уравнение гиперплоскости, зависящее от параметров модели $\\beta$, $n$-число объектов в выборке X, y - значения целевого признака, $\\alpha$ - коэффициент регуляризации."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**Обучим Lasso-регрессию с небольшим коэффициентом $\\alpha$ (слабая регуляризация). Обнуляется только коэффициент при признаке NOX (концентрация оксидов азота). Значит, он наименее важен для предсказания целевого признака - медианной стоимости домов в районе.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ -9.05881706e-02,   4.97201161e-02,  -1.76515804e-02,\n",
       "         0.00000000e+00,  -0.00000000e+00,   3.09182210e+00,\n",
       "        -9.33420115e-04,  -1.04019584e+00,   2.80048399e-01,\n",
       "        -1.52748917e-02,  -7.72353814e-01,   9.98417006e-03,\n",
       "        -6.17479065e-01])"
      ]
     },
     "execution_count": 37,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso = Lasso(alpha=0.3)\n",
    "lasso.fit(X,y)\n",
    "lasso.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**Обучим Lasso-регрессию с коэффициентом $\\alpha=10$. Теперь ненулевые коэффициенты остались только при признаках ZN (процент жилых участков площадью больше 25 тыс. кв. футов), TAX (уровень налогов), B (процент афроамериканцев) и LSTAT (процент граждан с низким уровнем жизни).**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.52336312307213451"
      ]
     },
     "execution_count": 35,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso = Lasso(alpha=10)\n",
    "lasso.fit(X,y)\n",
    "lasso.score(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**Таким образом, Lasso-регрессия служит методом *отбора* признаков.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "c5b0fa3c28ff9b08be773382c6e684bc79e487d7"
     }
    }
   ],
   "source": [
    "n_alphas = 200\n",
    "alphas = np.linspace(0.1, 10, n_alphas)\n",
    "model = Lasso()\n",
    "\n",
    "coefs = []\n",
    "for a in alphas:\n",
    "    model.set_params(alpha=a)\n",
    "    model.fit(X, y)\n",
    "    coefs.append(model.coef_)\n",
    "    \n",
    "ax = plt.gca()\n",
    "ax.set_color_cycle(['b', 'r', 'g', 'c', 'k', 'y', 'm'])\n",
    "\n",
    "ax.plot(alphas, coefs)\n",
    "ax.set_xscale('log')\n",
    "ax.set_xlim(ax.get_xlim()[::-1])  # reverse axis\n",
    "plt.xlabel('alpha')\n",
    "plt.ylabel('weights')\n",
    "plt.title('Lasso coefficients as a function of the regularization')\n",
    "plt.axis('tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**Теперь определим лучшее значение $\\alpha$ в процессе кросс-валидации.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LassoCV(alphas=array([  0.1    ,   0.14975, ...,   9.95025,  10.     ]),\n",
       "    copy_X=True, cv=3, eps=0.001, fit_intercept=True, max_iter=1000,\n",
       "    n_alphas=100, n_jobs=1, normalize=False, positive=False,\n",
       "    precompute='auto', random_state=42, selection='cyclic', tol=0.0001,\n",
       "    verbose=False)"
      ]
     },
     "execution_count": 9,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso_cv = LassoCV(alphas=alphas, \n",
    "                   cv=3, random_state=42)\n",
    "lasso_cv.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.09633634,  0.04943349, -0.02730583,  0.16245012, -0.        ,\n",
       "        3.55546459, -0.00723939, -1.13353947,  0.28039514, -0.0150312 ,\n",
       "       -0.78020119,  0.01032709, -0.58332373])"
      ]
     },
     "execution_count": 10,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso_cv.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.14974874371859298"
      ]
     },
     "execution_count": 11,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso_cv.alpha_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**В Scikit-learn версии 0.17 все еще не исправлен [баг](https://github.com/scikit-learn/scikit-learn/issues/2439): cross_val_score может возвращать отрицательные значения MSE. В любом случае, можно смотреть на модуль ошибки - чем меньше, тем лучше.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_squared_error was renamed to neg_mean_squared_error in version 0.18 and will be removed in 0.20.\n",
      "  sample_weight=sample_weight)\n",
      "/usr/local/lib/python3.5/dist-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_squared_error was renamed to neg_mean_squared_error in version 0.18 and will be removed in 0.20.\n",
      "  sample_weight=sample_weight)\n",
      "/usr/local/lib/python3.5/dist-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_squared_error was renamed to neg_mean_squared_error in version 0.18 and will be removed in 0.20.\n",
      "  sample_weight=sample_weight)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([-16.90196564, -37.60029071, -72.42092752])"
      ]
     },
     "execution_count": 12,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(Lasso(lasso_cv.alpha_), X, y, cv=3, scoring='mean_squared_error')#'neg_mean_squared_error'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42.30772795343924"
      ]
     },
     "execution_count": 13,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "abs(np.mean(cross_val_score(Lasso(lasso_cv.alpha_), X, y, cv=3, \n",
    "                        scoring='neg_mean_squared_error')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "53.722668480656019"
      ]
     },
     "execution_count": 14,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "abs(np.mean(cross_val_score(Lasso(9.95), X, y, cv=3 , scoring='neg_mean_squared_error')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**Еще один неоднозначный момент: LassoCV сортирует значения параметров по убыванию - так проще оптимизировать. Из-за этого может показаться, что оптимизация параметра $\\alpha$ работает неправильно**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.1       ,  0.14974874,  0.19949749,  0.24924623,  0.29899497,\n",
       "        0.34874372,  0.39849246,  0.44824121,  0.49798995,  0.54773869])"
      ]
     },
     "execution_count": 15,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso_cv.alphas[:10] # значения параметров на входе"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 10.        ,   9.95025126,   9.90050251,   9.85075377,\n",
       "         9.80100503,   9.75125628,   9.70150754,   9.65175879,\n",
       "         9.60201005,   9.55226131])"
      ]
     },
     "execution_count": 16,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso_cv.alphas_[:10] # преобразованные значения параметров"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "0dc97864b6d8315beef76d780c7b4f8efab7354d"
     }
    }
   ],
   "source": [
    "plot(lasso_cv.alphas, lasso_cv.mse_path_.mean(1)) # неверно\n",
    "axvline(lasso_cv.alpha_, c='g');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "4c29c5dd9b7bdfd9e79a8560ff32bda77f3aac9c"
     }
    }
   ],
   "source": [
    "plot(lasso_cv.alphas_, lasso_cv.mse_path_.mean(1)) # верно\n",
    "axvline(lasso_cv.alpha_, c='g');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Ridge-регрессия"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Ridge-регрессия (или гребневая регрессия) решает задачу минимизации среднеквардатичной ошибки с L2-регуляризацией:\n",
    "$$\\Large error(X, y, \\beta) = \\frac{1}{2n} \\sum_{i=1}^n {(y_i - \\beta^Tx_i)}^2 + \\alpha \\sum_{i=1}^n \\beta_i^2$$\n",
    "\n",
    "где $y = \\beta^Tx$ - уравнение гиперплоскости, зависящее от параметров модели $\\beta$, $n$-число объектов в выборке X, y - значения целевого признака, $\\alpha$ - коэффициент регуляризации."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "В sklearn реализован специальный класс [RidgeCV](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.html#sklearn.linear_model.RidgeCV) для кросс-валидации с Ridge-регрессией."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "n_alphas = 200\n",
    "ridge_alphas = np.logspace(-2, 6, n_alphas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RidgeCV(alphas=array([  1.00000e-02,   1.09699e-02, ...,   9.11589e+05,   1.00000e+06]),\n",
       "    cv=3, fit_intercept=True, gcv_mode=None, normalize=False,\n",
       "    scoring='mean_squared_error', store_cv_values=False)"
      ]
     },
     "execution_count": 39,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_cv = RidgeCV(alphas=ridge_alphas, \n",
    "                   scoring='mean_squared_error',\n",
    "                   cv=3)\n",
    "ridge_cv.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "264.30814869741084"
      ]
     },
     "execution_count": 40,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_cv.alpha_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**В случае Ridge-регрессии никакие праметры не зануляются - они могут быть очень малыми, но не нулевыми.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.0992728 ,  0.05560678, -0.04988281,  0.30212213, -0.09167075,\n",
       "        1.42147115,  0.01290084, -0.99854933,  0.33047941, -0.01661163,\n",
       "       -0.82006053,  0.00883866, -0.72358474])"
      ]
     },
     "execution_count": 22,
     "metadata": {
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_cv.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "06aec18cc8b7cf896da20c39ab231e597567d254"
     }
    }
   ],
   "source": [
    "n_alphas = 200\n",
    "ridge_alphas = np.logspace(-2, 6, n_alphas)\n",
    "model = Ridge()\n",
    "\n",
    "coefs = []\n",
    "for a in alphas:\n",
    "    model.set_params(alpha=a)\n",
    "    model.fit(X, y)\n",
    "    coefs.append(model.coef_)\n",
    "    \n",
    "ax = plt.gca()\n",
    "ax.set_color_cycle(['b', 'r', 'g', 'c', 'k', 'y', 'm'])\n",
    "\n",
    "ax.plot(alphas, coefs)\n",
    "ax.set_xscale('log')\n",
    "ax.set_xlim(ax.get_xlim()[::-1])  # reverse axis\n",
    "plt.xlabel('alpha')\n",
    "plt.ylabel('weights')\n",
    "plt.title('Ridge coefficients as a function of the regularization')\n",
    "plt.axis('tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Ссылки\n",
    "- [Обощеннные линейные модели](http://scikit-learn.org/stable/modules/linear_model.html) (Generalized Linear Models, GLM) в Scikit-learn\n",
    "- [LinearRegression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression), [Lasso](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html#sklearn.linear_model.Lasso), [LassoCV](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html#sklearn.linear_model.LassoCV), [Ridge](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html) и [RidgeCV](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.html#sklearn.linear_model.RidgeCV) в Scikit-learn\n",
    "- [Статья](https://habrahabr.ru/post/264915/) \"Методы отбора фич\" на Хабрахабре с упоминанием Lasso-регрессии "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (Ubuntu Linux)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "name": "lesson8_part1_kmeans.ipynb"
 },
 "nbformat": 4,
 "nbformat_minor": 0
}