20180920 문제인식 및 보고.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2018/09/20 문제 인식 및 실험 개시"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Library 호출"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np #수치계산 라이브러리\n",
    "import pandas as pd #구조데이터 처리 라이브러리\n",
    "import matplotlib.pyplot as plt #그래프 라이브러리\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 파일 호출"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "#피클 파일 호출\n",
    "with open('ten_raw_data.pickle','rb') as handle:\n",
    "    raw_data = pickle.load(handle)\n",
    "#삼성전자, SK하이닉스, 현대차, PSOCO, LG화학, NAVER, 삼성물산, SK텔레콤, 현대모비스, KB금융\n",
    "#\"005930\",\"000660\",\"005380\",\"047050\",\"051910\",\"035420\",\"028260\",\"017670\",\"012330\",\"105560\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "stock_name=['삼성전자', 'SK하이닉스', '현대차', 'PSOCO', 'LG화학', 'NAVER','삼성물산','SK텔레콤', '현대모비스', 'KB금융']\n",
    "stock_num=[\"005930\",\"000660\",\"005380\",\"047050\",\"051910\",\"035420\",\"028260\",\"017670\",\"012330\",\"105560\"]\n",
    "#dictionary key를 code에서 주식 이름으로 교체\n",
    "for i in range(10):\n",
    "    raw_data[stock_name[i]] = raw_data.pop(stock_num[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "del raw_data['삼성물산']\n",
    "del raw_data['SK텔레콤']\n",
    "del raw_data['SK하이닉스']\n",
    "del raw_data['현대모비스']\n",
    "del raw_data['KB금융']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Adj Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2012-01-02</th>\n",
       "      <td>21400.0</td>\n",
       "      <td>21600.0</td>\n",
       "      <td>21300.0</td>\n",
       "      <td>21600.0</td>\n",
       "      <td>21.233131</td>\n",
       "      <td>13151350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-01-03</th>\n",
       "      <td>21860.0</td>\n",
       "      <td>22100.0</td>\n",
       "      <td>21840.0</td>\n",
       "      <td>22100.0</td>\n",
       "      <td>21.724644</td>\n",
       "      <td>16927750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-01-04</th>\n",
       "      <td>22100.0</td>\n",
       "      <td>22200.0</td>\n",
       "      <td>21500.0</td>\n",
       "      <td>21600.0</td>\n",
       "      <td>21.233131</td>\n",
       "      <td>17103700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-01-05</th>\n",
       "      <td>21460.0</td>\n",
       "      <td>21580.0</td>\n",
       "      <td>21100.0</td>\n",
       "      <td>21100.0</td>\n",
       "      <td>20.741627</td>\n",
       "      <td>17298400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-01-06</th>\n",
       "      <td>21120.0</td>\n",
       "      <td>21320.0</td>\n",
       "      <td>20600.0</td>\n",
       "      <td>20800.0</td>\n",
       "      <td>20.446722</td>\n",
       "      <td>18816250</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Open     High      Low    Close  Adj Close    Volume\n",
       "Date                                                               \n",
       "2012-01-02  21400.0  21600.0  21300.0  21600.0  21.233131  13151350\n",
       "2012-01-03  21860.0  22100.0  21840.0  22100.0  21.724644  16927750\n",
       "2012-01-04  22100.0  22200.0  21500.0  21600.0  21.233131  17103700\n",
       "2012-01-05  21460.0  21580.0  21100.0  21100.0  20.741627  17298400\n",
       "2012-01-06  21120.0  21320.0  20600.0  20800.0  20.446722  18816250"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_data['삼성전자'].loc['2012-01-01':'2018-07-31'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "          \n",
    "                   \n",
    "                    \n",
    "                    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "           "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Adj Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2018-07-25</th>\n",
       "      <td>46250.0</td>\n",
       "      <td>46550.0</td>\n",
       "      <td>45900.0</td>\n",
       "      <td>46150.0</td>\n",
       "      <td>46150.0</td>\n",
       "      <td>7222471</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-07-26</th>\n",
       "      <td>46100.0</td>\n",
       "      <td>47000.0</td>\n",
       "      <td>46000.0</td>\n",
       "      <td>46900.0</td>\n",
       "      <td>46900.0</td>\n",
       "      <td>7374946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-07-27</th>\n",
       "      <td>46450.0</td>\n",
       "      <td>47000.0</td>\n",
       "      <td>46450.0</td>\n",
       "      <td>46900.0</td>\n",
       "      <td>46900.0</td>\n",
       "      <td>4762460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-07-30</th>\n",
       "      <td>46550.0</td>\n",
       "      <td>46800.0</td>\n",
       "      <td>46350.0</td>\n",
       "      <td>46500.0</td>\n",
       "      <td>46500.0</td>\n",
       "      <td>5723035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-07-31</th>\n",
       "      <td>46200.0</td>\n",
       "      <td>46450.0</td>\n",
       "      <td>46000.0</td>\n",
       "      <td>46250.0</td>\n",
       "      <td>46250.0</td>\n",
       "      <td>8033607</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Open     High      Low    Close  Adj Close   Volume\n",
       "Date                                                              \n",
       "2018-07-25  46250.0  46550.0  45900.0  46150.0    46150.0  7222471\n",
       "2018-07-26  46100.0  47000.0  46000.0  46900.0    46900.0  7374946\n",
       "2018-07-27  46450.0  47000.0  46450.0  46900.0    46900.0  4762460\n",
       "2018-07-30  46550.0  46800.0  46350.0  46500.0    46500.0  5723035\n",
       "2018-07-31  46200.0  46450.0  46000.0  46250.0    46250.0  8033607"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_data['삼성전자'].loc['2012-01-01':'2018-07-31'].tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature 추가"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shift 조정 코드 간략 설명"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2010-01-04         NaN\n",
       "2010-01-05    0.016069\n",
       "2010-01-06    0.023114\n",
       "2010-01-07   -0.033294\n",
       "2010-01-08    0.009840\n",
       "2010-01-11   -0.029233\n",
       "2010-01-12    0.015056\n",
       "2010-01-13   -0.014833\n",
       "2010-01-14    0.037641\n",
       "Name: Close, dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#수익률 계산\n",
    "raw_data['삼성전자'].Close.pct_change().head(9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2010-01-04         NaN\n",
       "2010-01-05         NaN\n",
       "2010-01-06         NaN\n",
       "2010-01-07         NaN\n",
       "2010-01-08         NaN\n",
       "2010-01-11         NaN\n",
       "2010-01-12    0.016069\n",
       "2010-01-13    0.023114\n",
       "2010-01-14   -0.033294\n",
       "Name: Close, dtype: float64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Shift조정\n",
    "#5일 후의 데이터를 예측하기 위해 5일 전의 데이터를 저장\n",
    "raw_data['삼성전자'].Close.pct_change().shift(5).head(9)\n",
    "\n",
    "#2010년 1월 12일의 0.016069는 1월 5일의 수익률이다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature 추가"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in raw_data.keys():\n",
    "    \"\"\"\n",
    "    X\n",
    "    \"\"\"\n",
    "    # 1일, 5일을 shift 조정한 수익률 계산\n",
    "    raw_data[i] = raw_data[i].assign(sh1_pct_change = raw_data[i].Close.pct_change().shift(1))\n",
    "    raw_data[i] = raw_data[i].assign(sh5_pct_change = raw_data[i].Close.pct_change().shift(5))\n",
    "    # 실제 당일의 수익률 계산\n",
    "    raw_data[i] = raw_data[i].assign(pct_change = raw_data[i].Close.pct_change().fillna(0))\n",
    "    \n",
    "    # 1일, 5일을 shift 조정한 3일 주가 이동평균 계산\n",
    "    raw_data[i] = raw_data[i].assign(sh1_close_ma_3 = raw_data[i].Close.rolling(window =3).mean().shift(1))\n",
    "    raw_data[i] = raw_data[i].assign(sh5_close_ma_3 = raw_data[i].Close.rolling(window =3).mean().shift(5))\n",
    "    \n",
    "    # 1일, 5일을 shift 조정한 7일, 21일 역사적 변동성 계산\n",
    "    raw_data[i] = raw_data[i].assign(sh1_his_vol_7 = raw_data[i][\"pct_change\"].rolling(window = 7).std().shift(1)*(252**0.5))\n",
    "    raw_data[i] = raw_data[i].assign(sh5_his_vol_7 = raw_data[i][\"pct_change\"].rolling(window = 7).std().shift(5)*(252**0.5))\n",
    "    \n",
    "    raw_data[i] = raw_data[i].assign(sh1_his_vol_21 = raw_data[i][\"pct_change\"].rolling(window = 21).std().shift(1)*(252**0.5))\n",
    "    raw_data[i] = raw_data[i].assign(sh5_his_vol_21 = raw_data[i][\"pct_change\"].rolling(window = 21).std().shift(5)*(252**0.5))\n",
    "   \n",
    "    # 1일, 5일을 shift 조정한 거래량 계싼\n",
    "    raw_data[i] = raw_data[i].assign(sh1_volume = raw_data[i].Volume.shift(1))\n",
    "    raw_data[i] = raw_data[i].assign(sh5_volume = raw_data[i].Volume.shift(5))\n",
    "    \n",
    "    \"\"\"\n",
    "    Y\n",
    "    \"\"\"\n",
    "    # 주가를 활용한 값vs평균 Y값 계산\n",
    "    raw_data[i] = raw_data[i].assign(Y1 = raw_data[i].Close - raw_data[i].Close.rolling(window =7).mean().shift(1))\n",
    "    raw_data[i] = raw_data[i].assign(Y1 = np.where(raw_data[i].Y1>=0, 1, 0))\n",
    "    # 주가를 활용한 평균vs평균 Y값 계산\n",
    "    raw_data[i] = raw_data[i].assign(Y3 = raw_data[i].Close.rolling(window=7).mean() - raw_data[i].Close.rolling(window =7).mean().shift(1))\n",
    "    raw_data[i] = raw_data[i].assign(Y3 = np.where(raw_data[i].Y3>=0, 1, 0))\n",
    "    \n",
    "    \"\"\"\n",
    "    Drop\n",
    "    \"\"\"\n",
    "    # 하나라도 NaN Value가 있는 행or열 제거\n",
    "    raw_data[i] = raw_data[i].dropna(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Volume</th>\n",
       "      <th>sh1_volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2010-02-08</th>\n",
       "      <td>21891500</td>\n",
       "      <td>25705750.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-09</th>\n",
       "      <td>16834200</td>\n",
       "      <td>21891500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-10</th>\n",
       "      <td>13174200</td>\n",
       "      <td>16834200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-11</th>\n",
       "      <td>13208300</td>\n",
       "      <td>13174200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-12</th>\n",
       "      <td>22879300</td>\n",
       "      <td>13208300.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Volume  sh1_volume\n",
       "Date                            \n",
       "2010-02-08  21891500  25705750.0\n",
       "2010-02-09  16834200  21891500.0\n",
       "2010-02-10  13174200  16834200.0\n",
       "2010-02-11  13208300  13174200.0\n",
       "2010-02-12  22879300  13208300.0"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 정확히 하루씩 밀린 모습\n",
    "raw_data['삼성전자'][['Volume', 'sh1_volume']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Volume</th>\n",
       "      <th>sh5_volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2010-02-08</th>\n",
       "      <td>21891500</td>\n",
       "      <td>25037200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-09</th>\n",
       "      <td>16834200</td>\n",
       "      <td>19677150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-10</th>\n",
       "      <td>13174200</td>\n",
       "      <td>16467700.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-11</th>\n",
       "      <td>13208300</td>\n",
       "      <td>11971700.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-12</th>\n",
       "      <td>22879300</td>\n",
       "      <td>25705750.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-16</th>\n",
       "      <td>7723550</td>\n",
       "      <td>21891500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-17</th>\n",
       "      <td>13997150</td>\n",
       "      <td>16834200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-18</th>\n",
       "      <td>7251900</td>\n",
       "      <td>13174200.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Volume  sh5_volume\n",
       "Date                            \n",
       "2010-02-08  21891500  25037200.0\n",
       "2010-02-09  16834200  19677150.0\n",
       "2010-02-10  13174200  16467700.0\n",
       "2010-02-11  13208300  11971700.0\n",
       "2010-02-12  22879300  25705750.0\n",
       "2010-02-16   7723550  21891500.0\n",
       "2010-02-17  13997150  16834200.0\n",
       "2010-02-18   7251900  13174200.0"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 정확히 5일씩 밀린 모습\n",
    "raw_data['삼성전자'][['Volume', 'sh5_volume']].head(8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2015년 8월 1일부터 2018년 7월 31일까지의 데이터만 사용\n",
    "data = {}\n",
    "for i in raw_data.keys():\n",
    "    data[i] = raw_data[i].loc['2015-08-01':'2018-07-31']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Adj Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>sh1_pct_change</th>\n",
       "      <th>sh5_pct_change</th>\n",
       "      <th>pct_change</th>\n",
       "      <th>sh1_close_ma_3</th>\n",
       "      <th>sh5_close_ma_3</th>\n",
       "      <th>sh1_his_vol_7</th>\n",
       "      <th>sh5_his_vol_7</th>\n",
       "      <th>sh1_his_vol_21</th>\n",
       "      <th>sh5_his_vol_21</th>\n",
       "      <th>sh1_volume</th>\n",
       "      <th>sh5_volume</th>\n",
       "      <th>Y1</th>\n",
       "      <th>Y3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2015-08-03</th>\n",
       "      <td>23680.0</td>\n",
       "      <td>23680.0</td>\n",
       "      <td>23320.0</td>\n",
       "      <td>23500.0</td>\n",
       "      <td>241.185562</td>\n",
       "      <td>9844550</td>\n",
       "      <td>-0.024691</td>\n",
       "      <td>0.000814</td>\n",
       "      <td>-0.008439</td>\n",
       "      <td>24420.000000</td>\n",
       "      <td>24620.000000</td>\n",
       "      <td>0.330462</td>\n",
       "      <td>0.206070</td>\n",
       "      <td>0.322405</td>\n",
       "      <td>0.282936</td>\n",
       "      <td>19579450.0</td>\n",
       "      <td>12152000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-08-04</th>\n",
       "      <td>23500.0</td>\n",
       "      <td>23740.0</td>\n",
       "      <td>23400.0</td>\n",
       "      <td>23640.0</td>\n",
       "      <td>242.622421</td>\n",
       "      <td>12844350</td>\n",
       "      <td>-0.008439</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.005957</td>\n",
       "      <td>23833.333333</td>\n",
       "      <td>24593.333333</td>\n",
       "      <td>0.326568</td>\n",
       "      <td>0.135040</td>\n",
       "      <td>0.314871</td>\n",
       "      <td>0.282633</td>\n",
       "      <td>9844550.0</td>\n",
       "      <td>13381350.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-08-05</th>\n",
       "      <td>23640.0</td>\n",
       "      <td>23640.0</td>\n",
       "      <td>23160.0</td>\n",
       "      <td>23180.0</td>\n",
       "      <td>237.901352</td>\n",
       "      <td>12802500</td>\n",
       "      <td>0.005957</td>\n",
       "      <td>0.026829</td>\n",
       "      <td>-0.019459</td>\n",
       "      <td>23613.333333</td>\n",
       "      <td>24820.000000</td>\n",
       "      <td>0.335496</td>\n",
       "      <td>0.215603</td>\n",
       "      <td>0.301013</td>\n",
       "      <td>0.297478</td>\n",
       "      <td>12844350.0</td>\n",
       "      <td>13734450.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-08-06</th>\n",
       "      <td>23100.0</td>\n",
       "      <td>23120.0</td>\n",
       "      <td>22300.0</td>\n",
       "      <td>22300.0</td>\n",
       "      <td>228.869736</td>\n",
       "      <td>20199050</td>\n",
       "      <td>-0.019459</td>\n",
       "      <td>-0.038005</td>\n",
       "      <td>-0.037964</td>\n",
       "      <td>23440.000000</td>\n",
       "      <td>24720.000000</td>\n",
       "      <td>0.341812</td>\n",
       "      <td>0.308978</td>\n",
       "      <td>0.304783</td>\n",
       "      <td>0.314385</td>\n",
       "      <td>12802500.0</td>\n",
       "      <td>15419600.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-08-07</th>\n",
       "      <td>22400.0</td>\n",
       "      <td>22740.0</td>\n",
       "      <td>22300.0</td>\n",
       "      <td>22720.0</td>\n",
       "      <td>233.180313</td>\n",
       "      <td>13787100</td>\n",
       "      <td>-0.037964</td>\n",
       "      <td>-0.024691</td>\n",
       "      <td>0.018834</td>\n",
       "      <td>23040.000000</td>\n",
       "      <td>24420.000000</td>\n",
       "      <td>0.377340</td>\n",
       "      <td>0.330462</td>\n",
       "      <td>0.327678</td>\n",
       "      <td>0.322405</td>\n",
       "      <td>20199050.0</td>\n",
       "      <td>19579450.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Open     High      Low    Close   Adj Close    Volume  \\\n",
       "Date                                                                   \n",
       "2015-08-03  23680.0  23680.0  23320.0  23500.0  241.185562   9844550   \n",
       "2015-08-04  23500.0  23740.0  23400.0  23640.0  242.622421  12844350   \n",
       "2015-08-05  23640.0  23640.0  23160.0  23180.0  237.901352  12802500   \n",
       "2015-08-06  23100.0  23120.0  22300.0  22300.0  228.869736  20199050   \n",
       "2015-08-07  22400.0  22740.0  22300.0  22720.0  233.180313  13787100   \n",
       "\n",
       "            sh1_pct_change  sh5_pct_change  pct_change  sh1_close_ma_3  \\\n",
       "Date                                                                     \n",
       "2015-08-03       -0.024691        0.000814   -0.008439    24420.000000   \n",
       "2015-08-04       -0.008439        0.000000    0.005957    23833.333333   \n",
       "2015-08-05        0.005957        0.026829   -0.019459    23613.333333   \n",
       "2015-08-06       -0.019459       -0.038005   -0.037964    23440.000000   \n",
       "2015-08-07       -0.037964       -0.024691    0.018834    23040.000000   \n",
       "\n",
       "            sh5_close_ma_3  sh1_his_vol_7  sh5_his_vol_7  sh1_his_vol_21  \\\n",
       "Date                                                                       \n",
       "2015-08-03    24620.000000       0.330462       0.206070        0.322405   \n",
       "2015-08-04    24593.333333       0.326568       0.135040        0.314871   \n",
       "2015-08-05    24820.000000       0.335496       0.215603        0.301013   \n",
       "2015-08-06    24720.000000       0.341812       0.308978        0.304783   \n",
       "2015-08-07    24420.000000       0.377340       0.330462        0.327678   \n",
       "\n",
       "            sh5_his_vol_21  sh1_volume  sh5_volume  Y1  Y3  \n",
       "Date                                                        \n",
       "2015-08-03        0.282936  19579450.0  12152000.0   0   0  \n",
       "2015-08-04        0.282633   9844550.0  13381350.0   0   0  \n",
       "2015-08-05        0.297478  12844350.0  13734450.0   0   0  \n",
       "2015-08-06        0.314385  12802500.0  15419600.0   0   0  \n",
       "2015-08-07        0.322405  20199050.0  19579450.0   0   0  "
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['삼성전자'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1일 조정한 5가지의 feature\n",
    "li1 = [\"sh1_pct_change\",\"sh1_close_ma_3\",\"sh1_volume\",\"sh1_his_vol_7\",\"sh1_his_vol_21\"]\n",
    "# 5일 조정한 5가지의 feature\n",
    "li5 = [\"sh5_pct_change\",\"sh5_close_ma_3\",\"sh5_volume\",\"sh5_his_vol_7\",\"sh5_his_vol_21\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train & Test 셋 분리"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1일 데이터를 조정한 훈련, 테스트 데이터셋 만들기\n",
    "train1 = {}\n",
    "train1_idx = {}\n",
    "test1 = {}\n",
    "test1_idx = {}\n",
    "\n",
    "# 5일 데이터를 조정한 훈련, 테스트 데이터셋 만들기\n",
    "train5 = {}\n",
    "train5_idx = {}\n",
    "test5 = {}\n",
    "test5_idx = {}\n",
    "\n",
    "# 수익률, 3일 주가 이동평균, 거래량, 7일 변동성, 21일 변동성\n",
    "# 위 5개 feature와 평균vs평균의 Y값으로 예측할 train, test set 생성\n",
    "for code, df in data.items():\n",
    "    train1[code] = df.iloc[:int(len(df)*0.8)][li1]\n",
    "    train1_idx[code] = df.iloc[:int(len(df)*0.8)][\"Y3\"]    \n",
    "    test1[code] = df.iloc[int(len(df)*0.8):][li1]\n",
    "    test1_idx[code] = df.iloc[int(len(df)*0.8):][\"Y3\"]\n",
    "    \n",
    "    train5[code] = df.iloc[:int(len(df)*0.8)][li5]\n",
    "    train5_idx[code] = df.iloc[:int(len(df)*0.8)][\"Y3\"]    \n",
    "    test5[code] = df.iloc[int(len(df)*0.8):][li5]\n",
    "    test5_idx[code] = df.iloc[int(len(df)*0.8):][\"Y3\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 위와 동일한 코드, Scale 조정을 하기 위한 데이터셋 추가\n",
    "train1_scale = {}\n",
    "train1_idx_scale = {}\n",
    "test1_scale = {}\n",
    "test1_idx_scale = {}\n",
    "\n",
    "train5_scale = {}\n",
    "train5_idx_scale = {}\n",
    "test5_scale = {}\n",
    "test5_idx_scale = {}\n",
    "\n",
    "for code, df in data.items():\n",
    "    train1_scale[code] = df.iloc[:int(len(df)*0.8)][li1]\n",
    "    train1_idx_scale[code] = df.iloc[:int(len(df)*0.8)][\"Y3\"]    \n",
    "    test1_scale[code] = df.iloc[int(len(df)*0.8):][li1]\n",
    "    test1_idx_scale[code] = df.iloc[int(len(df)*0.8):][\"Y3\"]\n",
    "    \n",
    "    train5_scale[code] = df.iloc[:int(len(df)*0.8)][li5]\n",
    "    train5_idx_scale[code] = df.iloc[:int(len(df)*0.8)][\"Y3\"]    \n",
    "    test5_scale[code] = df.iloc[int(len(df)*0.8):][li5]\n",
    "    test5_idx_scale[code] = df.iloc[int(len(df)*0.8):][\"Y3\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sh1_pct_change</th>\n",
       "      <th>sh1_close_ma_3</th>\n",
       "      <th>sh1_volume</th>\n",
       "      <th>sh1_his_vol_7</th>\n",
       "      <th>sh1_his_vol_21</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2015-08-03</th>\n",
       "      <td>-0.024691</td>\n",
       "      <td>24420.000000</td>\n",
       "      <td>19579450.0</td>\n",
       "      <td>0.330462</td>\n",
       "      <td>0.322405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-08-04</th>\n",
       "      <td>-0.008439</td>\n",
       "      <td>23833.333333</td>\n",
       "      <td>9844550.0</td>\n",
       "      <td>0.326568</td>\n",
       "      <td>0.314871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-08-05</th>\n",
       "      <td>0.005957</td>\n",
       "      <td>23613.333333</td>\n",
       "      <td>12844350.0</td>\n",
       "      <td>0.335496</td>\n",
       "      <td>0.301013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-08-06</th>\n",
       "      <td>-0.019459</td>\n",
       "      <td>23440.000000</td>\n",
       "      <td>12802500.0</td>\n",
       "      <td>0.341812</td>\n",
       "      <td>0.304783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-08-07</th>\n",
       "      <td>-0.037964</td>\n",
       "      <td>23040.000000</td>\n",
       "      <td>20199050.0</td>\n",
       "      <td>0.377340</td>\n",
       "      <td>0.327678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            sh1_pct_change  sh1_close_ma_3  sh1_volume  sh1_his_vol_7  \\\n",
       "Date                                                                    \n",
       "2015-08-03       -0.024691    24420.000000  19579450.0       0.330462   \n",
       "2015-08-04       -0.008439    23833.333333   9844550.0       0.326568   \n",
       "2015-08-05        0.005957    23613.333333  12844350.0       0.335496   \n",
       "2015-08-06       -0.019459    23440.000000  12802500.0       0.341812   \n",
       "2015-08-07       -0.037964    23040.000000  20199050.0       0.377340   \n",
       "\n",
       "            sh1_his_vol_21  \n",
       "Date                        \n",
       "2015-08-03        0.322405  \n",
       "2015-08-04        0.314871  \n",
       "2015-08-05        0.301013  \n",
       "2015-08-06        0.304783  \n",
       "2015-08-07        0.327678  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train1['삼성전자'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4개의 실험군 중 2개의 실험군 Standard Normalization 실시"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scikit-learn의 preprocessing 모듈 안에서 StandardScaler 객체 호출\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# 표준 정규화 Scale 실시\n",
    "for code in data.keys():\n",
    "    # Create StandardScaler object\n",
    "    std_scaler1 = StandardScaler()\n",
    "    std_scaler5 = StandardScaler()\n",
    "    # fit\n",
    "    std_scaler1.fit(train1_scale[code][li1].values)\n",
    "    std_scaler5.fit(train5_scale[code][li5].values)\n",
    "    # transform\n",
    "    train1_scale[code][li1] = std_scaler1.transform(train1_scale[code][li1].values)\n",
    "    test1_scale[code][li1] = std_scaler1.transform(test1_scale[code][li1].values)\n",
    "    train5_scale[code][li5] = std_scaler5.transform(train5_scale[code][li5].values)\n",
    "    test5_scale[code][li5] = std_scaler5.transform(test5_scale[code][li5].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sh1_pct_change</th>\n",
       "      <th>sh1_close_ma_3</th>\n",
       "      <th>sh1_volume</th>\n",
       "      <th>sh1_his_vol_7</th>\n",
       "      <th>sh1_his_vol_21</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2017-12-22</th>\n",
       "      <td>-2.543971</td>\n",
       "      <td>0.780330</td>\n",
       "      <td>0.218415</td>\n",
       "      <td>0.416899</td>\n",
       "      <td>0.660755</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-26</th>\n",
       "      <td>0.654199</td>\n",
       "      <td>0.533468</td>\n",
       "      <td>-0.323032</td>\n",
       "      <td>0.563198</td>\n",
       "      <td>0.644052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-27</th>\n",
       "      <td>-1.625364</td>\n",
       "      <td>-0.019503</td>\n",
       "      <td>0.269266</td>\n",
       "      <td>0.946825</td>\n",
       "      <td>0.936332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-28</th>\n",
       "      <td>1.348883</td>\n",
       "      <td>0.016704</td>\n",
       "      <td>-0.378840</td>\n",
       "      <td>1.655515</td>\n",
       "      <td>1.349283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-01-02</th>\n",
       "      <td>1.806610</td>\n",
       "      <td>0.224068</td>\n",
       "      <td>-0.593986</td>\n",
       "      <td>2.390017</td>\n",
       "      <td>0.850949</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            sh1_pct_change  sh1_close_ma_3  sh1_volume  sh1_his_vol_7  \\\n",
       "Date                                                                    \n",
       "2017-12-22       -2.543971        0.780330    0.218415       0.416899   \n",
       "2017-12-26        0.654199        0.533468   -0.323032       0.563198   \n",
       "2017-12-27       -1.625364       -0.019503    0.269266       0.946825   \n",
       "2017-12-28        1.348883        0.016704   -0.378840       1.655515   \n",
       "2018-01-02        1.806610        0.224068   -0.593986       2.390017   \n",
       "\n",
       "            sh1_his_vol_21  \n",
       "Date                        \n",
       "2017-12-22        0.660755  \n",
       "2017-12-26        0.644052  \n",
       "2017-12-27        0.936332  \n",
       "2017-12-28        1.349283  \n",
       "2018-01-02        0.850949  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit_transform을 적용하여 test set의 mean과 std로 정규화를 적용했을 때\n",
    "std_scaler = StandardScaler()\n",
    "test1['삼성전자'][li1] = std_scaler.fit_transform(test1['삼성전자'][li1].values)\n",
    "test1['삼성전자'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sh1_pct_change</th>\n",
       "      <th>sh1_close_ma_3</th>\n",
       "      <th>sh1_volume</th>\n",
       "      <th>sh1_his_vol_7</th>\n",
       "      <th>sh1_his_vol_21</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2017-12-22</th>\n",
       "      <td>-2.993225</td>\n",
       "      <td>1.507612</td>\n",
       "      <td>0.678656</td>\n",
       "      <td>0.748069</td>\n",
       "      <td>1.080292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-26</th>\n",
       "      <td>0.618040</td>\n",
       "      <td>1.459368</td>\n",
       "      <td>-0.139840</td>\n",
       "      <td>0.855585</td>\n",
       "      <td>1.070141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-27</th>\n",
       "      <td>-1.955966</td>\n",
       "      <td>1.351300</td>\n",
       "      <td>0.755527</td>\n",
       "      <td>1.137513</td>\n",
       "      <td>1.247774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-28</th>\n",
       "      <td>1.402453</td>\n",
       "      <td>1.358376</td>\n",
       "      <td>-0.224203</td>\n",
       "      <td>1.658330</td>\n",
       "      <td>1.498744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-01-02</th>\n",
       "      <td>1.919304</td>\n",
       "      <td>1.398902</td>\n",
       "      <td>-0.549435</td>\n",
       "      <td>2.198117</td>\n",
       "      <td>1.195882</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            sh1_pct_change  sh1_close_ma_3  sh1_volume  sh1_his_vol_7  \\\n",
       "Date                                                                    \n",
       "2017-12-22       -2.993225        1.507612    0.678656       0.748069   \n",
       "2017-12-26        0.618040        1.459368   -0.139840       0.855585   \n",
       "2017-12-27       -1.955966        1.351300    0.755527       1.137513   \n",
       "2017-12-28        1.402453        1.358376   -0.224203       1.658330   \n",
       "2018-01-02        1.919304        1.398902   -0.549435       2.198117   \n",
       "\n",
       "            sh1_his_vol_21  \n",
       "Date                        \n",
       "2017-12-22        1.080292  \n",
       "2017-12-26        1.070141  \n",
       "2017-12-27        1.247774  \n",
       "2017-12-28        1.498744  \n",
       "2018-01-02        1.195882  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit 실시 후 transform을 적용하여 train set의 mean과 std로 정규화를 적용했을 때\n",
    "test1_scale['삼성전자'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 모델 호출"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from xgboost import XGBClassifier\n",
    "\n",
    "from sklearn import metrics\n",
    "\n",
    "# Sklearn과 XGBOOST에서 모델 호출\n",
    "\n",
    "xgb_clf = XGBClassifier(random_state=42)\n",
    "log_clf = LogisticRegression()\n",
    "svm_clf = SVC()\n",
    "rf_clf = RandomForestClassifier(random_state=42)\n",
    "gnb_clf = GaussianNB()\n",
    "knn_clf = KNeighborsClassifier()\n",
    "dt_clf = DecisionTreeClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "models=[xgb_clf,rf_clf,log_clf,svm_clf,dt_clf,gnb_clf,knn_clf]\n",
    "model_name=['xgb','rf','log','svm','dt','gnb','knn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils.validation import NotFittedError\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 예측 함수 제작"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python 오류 문구 제거 코드\n",
    "import warnings\n",
    "warnings.simplefilter('ignore', DeprecationWarning)\n",
    "\n",
    "\n",
    "def modelfit_with_score(train, train_idx, test, test_idx):\n",
    "    result = {} #Test 결과를 저장할 Dictionary 자료구조 생성\n",
    "    conf_mx = {} #Confusion Matrix를 저장한 Dictionary 자료구조 생성\n",
    "    for i in data.keys():\n",
    "        # Train, Test Set 이름 일반화\n",
    "        X_train = train[i].values\n",
    "        y_train = train_idx[i].values\n",
    "        X_test = test[i].values\n",
    "        y_test = test_idx[i].values\n",
    "        score = {} #주식별로 모델별 Test Score를 저장할 dict 생성\n",
    "        confusion = {} #주식별로 모델별 Confusion Matrix를 저장할 dict 생성\n",
    "        for clf, name in zip(models, model_name):\n",
    "            model_score = [] #각 모델별 score가 저장될 list\n",
    "            confusion_mat = [] #각 모델별 confusion matrix가 저장될 list\n",
    "            clf.fit(X_train, y_train) #fitting\n",
    "            y_pred = clf.predict(X_test) #predict\n",
    "            try:\n",
    "                y_prob = clf.predict_proba(X_test) #클래스 확률 계산\n",
    "            # SVM의 경우, predict_proba 메서드를 실시하기 위해\n",
    "            # 모델의 Probability를 True로 바꿔줘야한다.\n",
    "            # 이를 위한 코드\n",
    "            except NotFittedError:\n",
    "                clf.probability = True\n",
    "                y_prob = clf.predict_proba(X_test)\n",
    "            # 모델 별 score 및 confusion matrix를 list에 저장\n",
    "            model_score.append(metrics.accuracy_score(y_test, y_pred))\n",
    "            model_score.append(metrics.precision_score(y_test, y_pred))\n",
    "            model_score.append(metrics.recall_score(y_test, y_pred))\n",
    "            model_score.append(metrics.f1_score(y_test, y_pred))\n",
    "            model_score.append(metrics.roc_auc_score(y_test, y_prob[:,1]))\n",
    "            confusion_mat.append(metrics.confusion_matrix(y_test, y_pred))\n",
    "            # 모델별 score및 confusion matrix를 모델별로 저장\n",
    "            score[name] = model_score\n",
    "            confusion[name] = confusion_mat\n",
    "        # 주식별 score와 confusion matrix를 저장\n",
    "        result[i] = score\n",
    "        conf_mx[i] = confusion\n",
    "        \n",
    "    # 결과값을 return으로 받음\n",
    "    return result, conf_mx\n",
    "\n",
    "def print_result(result):\n",
    "    for i in data.keys():\n",
    "        print(i)\n",
    "        print(pd.DataFrame(result[i],index=['accuracy', 'precision', 'recall', 'f1', 'auc']).apply(lambda x : round(x,4)*100),\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 실험 개시\n",
    "- predictor(feature) : 수익률, 거래량, 3일 주가 이동평균, 7일 변동성, 21일 변동성\n",
    "- target : 주가의 7일 평균 vs 전일부터 7일의 주가 평균"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 첫번째 실험군 : 1일후 예측, 표준 정규화 X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "삼성전자\n",
      "              dt    gnb    knn    log     rf     svm    xgb\n",
      "accuracy   59.18  49.66  49.66  59.18  49.66   50.34  54.42\n",
      "precision  57.95   0.00   0.00  58.97  50.00   50.34  60.00\n",
      "recall     68.92   0.00   0.00  62.16  74.32  100.00  28.38\n",
      "f1         62.96   0.00   0.00  60.53  59.78   66.97  38.53\n",
      "auc        59.12  62.46  50.00  62.37  51.77   50.00  53.47 \n",
      "\n",
      "현대차\n",
      "              dt    gnb    knn    log     rf     svm    xgb\n",
      "accuracy   57.14  53.74  54.42  39.46  52.38   46.94  48.98\n",
      "precision  53.75  50.54  51.28  36.49  49.28   46.94  46.34\n",
      "recall     62.32  68.12  57.97  39.13  49.28  100.00  55.07\n",
      "f1         57.72  58.02  54.42  37.76  49.28   63.89  50.33\n",
      "auc        57.44  54.25  47.02  36.05  54.66   50.00  53.60 \n",
      "\n",
      "PSOCO\n",
      "              dt    gnb    knn    log     rf    svm    xgb\n",
      "accuracy   62.59  58.50  63.27  57.14  61.22  49.66  65.99\n",
      "precision  60.92  59.70  62.20  56.32  61.64   0.00  65.00\n",
      "recall     71.62  54.05  68.92  66.22  60.81   0.00  70.27\n",
      "f1         65.84  56.74  65.38  60.87  61.22   0.00  67.53\n",
      "auc        62.52  65.83  66.31  63.46  65.30  50.00  70.00 \n",
      "\n",
      "LG화학\n",
      "              dt    gnb    knn    log     rf     svm    xgb\n",
      "accuracy   63.27  51.02  53.06  48.30  56.46   48.98  58.50\n",
      "precision  59.78  50.00  51.43  48.63  56.25   48.98  56.32\n",
      "recall     76.39  94.44  75.00  98.61  50.00  100.00  68.06\n",
      "f1         67.07  65.38  61.02  65.14  52.94   65.75  61.64\n",
      "auc        63.53  42.74  54.65  56.37  57.94   50.00  65.81 \n",
      "\n",
      "NAVER\n",
      "              dt    gnb    knn     log     rf     svm    xgb\n",
      "accuracy   50.34  44.90  48.98   43.54  53.74   42.86  55.10\n",
      "precision  44.44  43.28  43.75   43.15  46.48   42.86  48.51\n",
      "recall     63.49  92.06  66.67  100.00  52.38  100.00  77.78\n",
      "f1         52.29  58.88  52.83   60.29  49.25   60.00  59.76\n",
      "auc        51.98  45.69  49.31   56.63  55.57   50.00  64.22 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 1일 후를 예측, 표준 정규화 X\n",
    "result1, conf_mx1 = modelfit_with_score(train1, train1_idx, test1, test1_idx)\n",
    "print_result(result1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 두번째 실험군 : 5일 후를 예측, 표준 정규화 X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "삼성전자\n",
      "              dt    gnb    knn     log     rf     svm    xgb\n",
      "accuracy   64.63  48.98  46.94   52.38  63.95   50.34  62.59\n",
      "precision  62.50  49.18  47.96   51.39  64.00   50.34  71.11\n",
      "recall     74.32  40.54  63.51  100.00  64.86  100.00  43.24\n",
      "f1         67.90  44.44  54.65   67.89  64.43   66.97  53.78\n",
      "auc        64.56  59.20  44.08   50.98  67.75   50.00  65.35 \n",
      "\n",
      "현대차\n",
      "              dt    gnb    knn    log     rf     svm    xgb\n",
      "accuracy   55.78  57.82  54.42  46.94  57.82   46.94  64.63\n",
      "precision  52.27  57.78  51.25  41.82  53.93   46.94  58.25\n",
      "recall     66.67  37.68  59.42  33.33  69.57  100.00  86.96\n",
      "f1         58.60  45.61  55.03  37.10  60.76   63.89  69.77\n",
      "auc        56.41  58.58  54.94  47.27  62.86   50.00  67.22 \n",
      "\n",
      "PSOCO\n",
      "              dt    gnb    knn    log     rf    svm    xgb\n",
      "accuracy   48.98  53.74  53.74  53.74  48.30  49.66  53.06\n",
      "precision  49.44  54.69  54.17  53.85  48.44   0.00  53.33\n",
      "recall     59.46  47.30  52.70  56.76  41.89   0.00  54.05\n",
      "f1         53.99  50.72  53.42  55.26  44.93   0.00  53.69\n",
      "auc        48.91  57.63  57.36  56.68  47.85  50.00  52.73 \n",
      "\n",
      "LG화학\n",
      "              dt    gnb    knn     log     rf     svm    xgb\n",
      "accuracy   48.30  55.78  51.02   48.98  48.30   48.98  50.34\n",
      "precision  46.43  55.93  50.00   48.98  46.00   48.98  48.89\n",
      "recall     36.11  45.83  61.11  100.00  31.94  100.00  30.56\n",
      "f1         40.63  50.38  55.00   65.75  37.70   65.75  37.61\n",
      "auc        48.06  59.43  49.58   38.59  51.00   50.00  58.48 \n",
      "\n",
      "NAVER\n",
      "              dt    gnb    knn     log     rf     svm    xgb\n",
      "accuracy   51.02  44.22  46.94   42.86  59.18   42.86  54.42\n",
      "precision  44.44  42.86  40.96   42.86  52.05   42.86  47.96\n",
      "recall     57.14  90.48  53.97  100.00  60.32  100.00  74.60\n",
      "f1         50.00  58.16  46.58   60.00  55.88   60.00  58.39\n",
      "auc        51.79  52.87  46.99   52.74  62.57   50.00  63.00 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 5일 후를 예측, 표준 정규화 X\n",
    "result5, conf_mx5 = modelfit_with_score(train5, train5_idx, test5, test5_idx)\n",
    "print_result(result5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 세번째 실험군 : 1일 후를 예측, 표준 정규화 O"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "삼성전자\n",
      "              dt    gnb    knn    log     rf    svm    xgb\n",
      "accuracy   51.02  51.70  65.99  51.02  53.74  63.27  54.42\n",
      "precision  51.28  51.81  63.04  51.32  55.77  59.26  54.55\n",
      "recall     54.05  58.11  78.38  52.70  39.19  86.49  56.76\n",
      "f1         52.63  54.78  69.88  52.00  46.03  70.33  55.63\n",
      "auc        51.00  54.87  68.51  58.05  58.17  62.99  61.64 \n",
      "\n",
      "현대차\n",
      "              dt    gnb    knn    log     rf    svm    xgb\n",
      "accuracy   53.74  58.50  60.54  63.95  52.38  58.50  48.98\n",
      "precision  50.62  55.88  57.33  61.11  49.28  55.41  46.34\n",
      "recall     59.42  55.07  62.32  63.77  49.28  59.42  55.07\n",
      "f1         54.67  55.47  59.72  62.41  49.28  57.34  50.33\n",
      "auc        54.07  58.23  62.40  66.65  54.66  63.34  53.60 \n",
      "\n",
      "PSOCO\n",
      "              dt    gnb    knn    log     rf    svm    xgb\n",
      "accuracy   57.82  60.54  61.22  60.54  61.22  67.35  65.99\n",
      "precision  57.14  60.53  60.00  59.30  61.64  66.25  65.00\n",
      "recall     64.86  62.16  68.92  68.92  60.81  71.62  70.27\n",
      "f1         60.76  61.33  64.15  63.75  61.22  68.83  67.53\n",
      "auc        57.77  64.90  64.87  67.77  65.30  73.11  70.00 \n",
      "\n",
      "LG화학\n",
      "              dt    gnb    knn    log     rf    svm    xgb\n",
      "accuracy   61.90  57.82  51.02  59.18  56.46  62.59  58.50\n",
      "precision  60.00  54.63  50.00  55.08  56.25  60.49  56.32\n",
      "recall     66.67  81.94  66.67  90.28  50.00  68.06  68.06\n",
      "f1         63.16  65.56  57.14  68.42  52.94  64.05  61.64\n",
      "auc        62.00  63.81  57.52  69.54  57.94  63.87  65.81 \n",
      "\n",
      "NAVER\n",
      "              dt    gnb    knn    log     rf    svm    xgb\n",
      "accuracy   47.62  53.74  59.18  57.14  53.74  55.10  55.10\n",
      "precision  42.71  48.03  51.85  50.00  46.48  48.65  48.51\n",
      "recall     65.08  96.83  66.67  82.54  52.38  85.71  77.78\n",
      "f1         51.57  64.21  58.33  62.28  49.25  62.07  59.76\n",
      "auc        49.80  67.27  63.91  65.78  55.57  68.82  64.22 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 1일 후를 예측, 표준 정규화 O\n",
    "result1s, conf_mx1s = modelfit_with_score(train1_scale, train1_idx_scale, test1_scale, test1_idx_scale)\n",
    "print_result(result1s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 네번째 실험군 : 5일 후를 예측, 표준 정규화 O"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "삼성전자\n",
      "              dt    gnb    knn    log     rf    svm    xgb\n",
      "accuracy   64.63  60.54  57.82  61.22  63.95  61.22  62.59\n",
      "precision  62.79  62.90  60.00  63.93  64.00  66.67  71.11\n",
      "recall     72.97  52.70  48.65  52.70  64.86  45.95  43.24\n",
      "f1         67.50  57.35  53.73  57.78  64.43  54.40  53.78\n",
      "auc        64.57  65.57  62.02  66.14  67.75  64.72  65.35 \n",
      "\n",
      "현대차\n",
      "              dt    gnb    knn    log     rf    svm    xgb\n",
      "accuracy   52.38  64.63  58.50  64.63  57.82  61.22  65.31\n",
      "precision  49.44  66.67  54.44  62.69  53.93  56.52  58.82\n",
      "recall     63.77  49.28  71.01  60.87  69.57  75.36  86.96\n",
      "f1         55.70  56.67  61.64  61.76  60.76  64.60  70.18\n",
      "auc        53.04  69.86  62.96  68.75  62.77  68.52  67.28 \n",
      "\n",
      "PSOCO\n",
      "              dt    gnb    knn    log     rf    svm    xgb\n",
      "accuracy   48.30  50.34  53.06  59.86  48.30  57.82  53.06\n",
      "precision  48.86  50.70  53.33  61.19  48.44  60.00  53.33\n",
      "recall     58.11  48.65  54.05  55.41  41.89  48.65  54.05\n",
      "f1         53.09  49.66  53.69  58.16  44.93  53.73  53.69\n",
      "auc        48.23  49.78  53.65  63.44  48.21  59.02  52.73 \n",
      "\n",
      "LG화학\n",
      "              dt    gnb    knn    log     rf    svm    xgb\n",
      "accuracy   48.98  56.46  50.34  59.86  48.30  59.86  50.34\n",
      "precision  47.17  55.71  49.12  58.23  46.00  62.75  48.89\n",
      "recall     34.72  54.17  38.89  63.89  31.94  44.44  30.56\n",
      "f1         40.00  54.93  43.41  60.93  37.70  52.03  37.61\n",
      "auc        48.69  62.48  54.30  66.02  50.92  62.71  58.43 \n",
      "\n",
      "NAVER\n",
      "              dt    gnb    knn    log     rf    svm    xgb\n",
      "accuracy   53.74  51.70  60.54  57.14  59.18  59.86  54.42\n",
      "precision  46.84  46.88  52.94  50.00  52.05  51.92  47.96\n",
      "recall     58.73  95.24  71.43  80.95  60.32  85.71  74.60\n",
      "f1         52.11  62.83  60.81  61.82  55.88  64.67  58.39\n",
      "auc        54.37  68.05  59.83  65.97  62.57  63.34  63.00 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 5일 후를 예측, 표준 정규화 O\n",
    "result5s, conf_mx5s = modelfit_with_score(train5_scale, train5_idx_scale, test5_scale, test5_idx_scale)\n",
    "print_result(result5s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "- 데이터를 재희형과 Cross Check하여 정확한 구성 실시\n",
    "- Standard Scale하는 부분도 정확하게 Train의 평균과 분산으로 실시\n",
    "- 4개의 실험군을 나눠 실험 개시\n",
    "- modelfit_with_score 함수를 만들어 예측 실시\n",
    "  - 이 함수는 fit-predict의 역할을 수행\n",
    "  - 함수가 길어지게 된 이유는 보조적인 score를 저장하는 과정에서 길어지게 됨\n",
    "  - 이에 대한 설명을 위에 함수에 요약 설명"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "- 코드를 체크했음에도 불구하고 오히려 더 낮은 예측 정확도를 보임\n",
    "- SVM의 경우 일전에 본 Precision이 0되는 경우는 Scale 조정을 하지 않아서 예측을 못했던 것으로 판명\n",
    "  - 실제로 SVM이 Best Estimator가 되는 경우도 발견됨\n",
    "  - 이를 통해 SVM으로도 Performance를 향상시킬 수 있을 것으로 조명됨"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 향후..?\n",
    "- 우리가 사용하는 Y가 실제로 돈을 벌 수 있는, 유용한 Information인지 확실치 않음\n",
    "- Data에 문제가 존재하지 않았고 이는 feature가 제대로 갖춰지지 않음을 의미\n",
    "- Grid Search로 돌리지는 않았으나 70~80%로 끌어올리는 것에는 무리가 있을 것으로 판단\n",
    "- 새로운 Methodology(Feature 및 target 선정)를 도입하는 것도 방법이라 사려됨"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}