20190107 Reviewing Everything.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 공통"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "\n",
    "from matplotlib import font_manager, rc\n",
    "font_name = font_manager.FontProperties(fname=\"c:/Windows/Fonts/malgun.ttf\").get_name()\n",
    "rc('font', family=font_name)\n",
    "matplotlib.rcParams['axes.unicode_minus']=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-08-13\n",
    "- Baseline 구축\n",
    "- X : MACD, MACD>0 & MACDDiff>0 (Long&Short), returns, ewm\n",
    "  - For loop돌 때 ewm 반복된 할당\n",
    "- Y : Class of returns\n",
    "  - shift 오류\n",
    "- Train, Test Split\n",
    "- Standard Scaler로 Scale 조정\n",
    "- GNB, KNN, RF\n",
    "  - Predict 평균 50\n",
    "- Grid Search\n",
    "  - best estimator로 재예측"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-08-16\n",
    "- X : 5~30 Moving Average, minmaxVolume, minmaxClose\n",
    "- Y : 수익률 - 수익률.rolling(5).mean()\n",
    "  - shift 오류\n",
    "- GNB, KNN, RF\n",
    "  - Predict 평균 50~60\n",
    "- Grid Search\n",
    "  - 시간 상 KeyboardInterrupt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-08-23\n",
    "- fix_yahoo_finance에서 주식 데이터 read\n",
    "- X : Close Moving Average, Volume Moving Average, Historical Volatility\n",
    "- Y : Class of\n",
    " - Y1 = Close - Close_MA_7\n",
    " - Y2 = Returns - Returns_MA_7\n",
    " - Y3 = Close.rolling(7).sum().pct_change()\n",
    "   - shift 오류\n",
    "- 2012-01-01 : ~\n",
    "- XGBoost 첫 사용\n",
    "- modelfit 함수, useTrainCV, dMatrix 사용\n",
    "- Feature Importance 계산\n",
    "- Train에서 높은 Fitting을 보여주지만 Test에서 별다른 성과를 얻지 못함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def modelfit(alg, dtrain, predictors, target, useTrainCV = True, cv_folds = 5, early_stopping_rounds = 50):\n",
    "    \n",
    "    start = time.time()\n",
    "    \n",
    "    if useTrainCV:\n",
    "        xgb_param = alg.get_xgb_params()\n",
    "        xgtrain = xgb.DMatrix(dtrain[predictors].values, label=dtrain[target].values)\n",
    "        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], nfold=cv_folds,\n",
    "            metrics='auc', early_stopping_rounds=early_stopping_rounds, show_stdv=False)\n",
    "        alg.set_params(n_estimators=cvresult.shape[0])\n",
    "    \n",
    "    #Fit the algorithm on the data\n",
    "    alg.fit(dtrain[predictors], dtrain[target], eval_metric='auc')\n",
    "        \n",
    "    #Predict training set:\n",
    "    dtrain_predictions = alg.predict(dtrain[predictors])\n",
    "    dtrain_predprob = alg.predict_proba(dtrain[predictors])[:,1]\n",
    "        \n",
    "    #Print model report:\n",
    "    print(\"Model Report\")\n",
    "    print(\"Accuracy : %.4g\" % metrics.accuracy_score(dtrain[target].values, dtrain_predictions))\n",
    "    print(\"AUC Score (Train): %f\" % metrics.roc_auc_score(dtrain[target], dtrain_predprob))\n",
    "                    \n",
    "    imp_fear_name = pd.Series(predictors)[alg.feature_importances_ > 0.01]\n",
    "    imp_fear_name.index = range(len(imp_fear_name))\n",
    "    imp_fear = pd.concat((imp_fear_name, \n",
    "               pd.Series(alg.feature_importances_[alg.feature_importances_ > 0.01])),axis=1).sort_values(by=1,ascending=True)\n",
    "    imp_fear.index = range(len(imp_fear))\n",
    "    imp_fear.columns = [\"feature_name\", \"feature_importance\"]\n",
    "#     {i:j for i,j in zip(imp_fear.index, imp_fear[\"feature_name\"])}\n",
    "    imp_fear.plot(y='feature_importance', x='feature_name', kind='barh', legend=False, figsize=(8,5))\n",
    "    \n",
    "#     feat_imp = pd.Series(alg.feature_importances_).sort_values(ascending=False)\n",
    "#     feat_imp.plot(kind='bar', title='Feature Importances', figsize=(200,5))\n",
    "#     plt.ylabel('Feature Importance Score')\n",
    "    \n",
    "    elapse = time.time() - start\n",
    "    print(\"elapse time : {}\".format(round(elapse,2)))\n",
    "    \n",
    "    return alg, imp_fear"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-08-26\n",
    "- X : Close Moving Average, Volume Moving Average, Historical Volatility\n",
    "- Y : Class of\n",
    " - Y1 = Close - Close_MA_7\n",
    " - Y2 = Returns - Returns_MA_7\n",
    " - Y3 = Close.rolling(7).sum().pct_change()\n",
    "   - shift 오류\n",
    "- 예측 결과로 Simulation실시\n",
    "- Y2와 XGBClassifier로 예측 실시\n",
    "  - Predict 80% up\n",
    "  - 허나 이는 잘못된 결과임을 곧 알게된다.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-09-10\n",
    "- 고대교수님 세미나 준비\n",
    "- X, Y 변화 없음\n",
    "- GNB, KNN, RF, DT, LOG, SVM, XGB 사용\n",
    "- time 계산 후 Feature Selection으로 시간이 얼마나 단축되는지 계산"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-09-13\n",
    "- modelfit함수 수정\n",
    "- 기타 잘못된 방법으로 모델 학습 진행"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def modelfit2(alg, dtrain, test, predictors, target, useTrainCV = True, \n",
    "             cv_folds = 5, early_stopping_rounds = 50, top = 10, \n",
    "             figure = True, model_report = True):\n",
    "    \n",
    "    start = time.time()\n",
    "    \n",
    "    if useTrainCV:\n",
    "        xgb_param = alg.get_xgb_params()\n",
    "        xgtrain = xgb.DMatrix(dtrain[predictors].values, label=dtrain[target].values)\n",
    "        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], nfold=cv_folds,\n",
    "            metrics='auc', early_stopping_rounds=early_stopping_rounds, show_stdv=False)\n",
    "        alg.set_params(n_estimators=cvresult.shape[0])\n",
    "    \n",
    "    #Fit the algorithm on the data\n",
    "    alg.fit(dtrain[predictors], dtrain[target], eval_metric='auc')\n",
    "        \n",
    "    #Predict training set:\n",
    "    dtrain_predictions = alg.predict(dtrain[predictors])\n",
    "    dtrain_predprob = alg.predict_proba(dtrain[predictors])[:,1]\n",
    "    test_predictions = alg.predict(test[predictors])\n",
    "    test_predprob = alg.predict_proba(test[predictors])[:,1]\n",
    "        \n",
    "    if model_report:\n",
    "        #Print model report:\n",
    "        print(\"Model Report\")\n",
    "        print(\"Accuracy (Train): %.4g\" % metrics.accuracy_score(dtrain[target].values, dtrain_predictions))\n",
    "        print(\"Accuracy (Test): %.4g\" % metrics.accuracy_score(test[target].values, test_predictions))\n",
    "        print(\"AUC Score (Train): %f\" % metrics.roc_auc_score(dtrain[target], dtrain_predprob))\n",
    "        print(\"AUC Score (Test): %f\" % metrics.roc_auc_score(test[target], test_predprob))\n",
    "                    \n",
    "    imp_fear_name = pd.Series(predictors)[alg.feature_importances_ > 0.01]\n",
    "    imp_fear_name.index = range(len(imp_fear_name))\n",
    "    imp_fear = pd.concat((imp_fear_name, \n",
    "               pd.Series(alg.feature_importances_[alg.feature_importances_ > 0.01])),axis=1).sort_values(by=1,ascending=True)\n",
    "    imp_fear.index = range(len(imp_fear))\n",
    "    imp_fear.columns = [\"feature_name\", \"feature_importance\"]\n",
    "#     {i:j for i,j in zip(imp_fear.index, imp_fear[\"feature_name\"])}\n",
    "    if figure:\n",
    "        if top:\n",
    "            imp_fear.iloc[-top:].plot(y='feature_importance', x='feature_name', kind='barh', legend=False, figsize=(8,5))\n",
    "        else:\n",
    "            imp_fear.plot(y='feature_importance', x='feature_name', kind='barh', legend=False, figsize=(8,5))\n",
    "    \n",
    "#     feat_imp = pd.Series(alg.feature_importances_).sort_values(ascending=False)\n",
    "#     feat_imp.plot(kind='bar', title='Feature Importances', figsize=(200,5))\n",
    "#     plt.ylabel('Feature Importance Score')\n",
    "    \n",
    "    elapse = time.time() - start\n",
    "    print(\"elapse time : {} sec\".format(round(elapse,2)))\n",
    "    \n",
    "    return alg, imp_fear, elapse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-09-17\n",
    "- 잘못된 것을 깨닫고 학습 진행\n",
    "- Cross Val Predict 실습 진행\n",
    "- Y shift 수정\n",
    "- MACD, Stochastic, RSI로 위기를 모면하고자 함수화 진행\n",
    "- 예측률 떡락 (66%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def modelfit3(alg, dtrain, test, predictors, target, useTrainCV = True, \n",
    "             cv_folds = 5, early_stopping_rounds = 50):\n",
    "\n",
    "    if useTrainCV:\n",
    "        xgb_param = alg.get_xgb_params()\n",
    "        xgtrain = xgb.DMatrix(dtrain[predictors].values, label=dtrain[target].values)\n",
    "        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], nfold=cv_folds,\n",
    "            metrics='auc', early_stopping_rounds=early_stopping_rounds, show_stdv=False)\n",
    "        alg.set_params(n_estimators=cvresult.shape[0])\n",
    "    \n",
    "    #Fit the algorithm on the data\n",
    "    alg.fit(dtrain[predictors], dtrain[target], eval_metric='auc')\n",
    "        \n",
    "    #Predict training set:\n",
    "    test_predictions = alg.predict(test[predictors])\n",
    "    test_predprob = alg.predict_proba(test[predictors])[:,1]\n",
    "        \n",
    "    fi_dict = {feature : importance for feature, importance in zip(predictors, alg.feature_importances_)}\n",
    "\n",
    "    return alg, fi_dict, test_predictions, test_predprob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-09-18\n",
    "- TestAccuracy 함수 제작\n",
    "- 예측률 77%\n",
    "- 주식 소거 기준 제공\n",
    "  - del_li = [key for key, df in raw_data.items() if len(df) < 252*2]\n",
    "  - del_li2 = ['BGF', 'HDC', 'SK디스커버리', '대우조선해양', '대한항공', '동아쏘시오홀딩스', '동양', '롯데지주', '오리온홀딩스', '우리은행', '쿠쿠홀딩스', '팬오션', '한국타이어월드와이드', '한라홀딩스', '현대중공업', '효성'] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def TestAccuracy(train, test, predictors, target, alg):\n",
    "    score_set = pd.DataFrame(columns=['accuracy', 'precision', 'recall', 'auc', 'f1'])\n",
    "    for i in raw_data.keys():\n",
    "        X_train = train[i][predictors].values\n",
    "        y_train = train[i][target].values\n",
    "        X_test = test[i][predictors].values\n",
    "        y_test = test[i][target].values\n",
    "        a=[]\n",
    "        alg.fit(X_train, y_train)\n",
    "        y_pred = alg.predict(X_test)\n",
    "        a.append(metrics.accuracy_score(y_test, y_pred))\n",
    "        value_1=cross_val_predict(alg, X_train, y_train, cv=5,method=\"predict_proba\")\n",
    "        a.append(metrics.precision_score(y_test, y_pred))\n",
    "        a.append(metrics.recall_score(y_test, y_pred))\n",
    "        fpr, tpr, thresholds = metrics.roc_curve(y_train, value_1[:,1])\n",
    "        a.append(metrics.auc(fpr, tpr))\n",
    "        a.append(metrics.f1_score(y_test,y_pred).mean())\n",
    "        score_set.loc[i] = a\n",
    "    return score_set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def temp_calc_feature_importance(alg, train, test, predictors, target):\n",
    "    test_result = {}\n",
    "    fi_dict = {}\n",
    "    for i in train.keys():\n",
    "        model, fear_importance = modelfit(alg, train[i], predictors, target, \n",
    "                                                  cv_folds=5, early_stopping_rounds=20,\n",
    "                                                  figure=False, model_report=False)\n",
    "        pred = model.predict(test[i][predictors])\n",
    "        # Calculating Score\n",
    "        test_result[i] = np.array([\n",
    "            metrics.accuracy_score(pred, test[i][target]),\n",
    "            metrics.f1_score(pred, test[i][target]),\n",
    "            metrics.precision_score(pred, test[i][target]),\n",
    "            metrics.recall_score(pred, test[i][target]),\n",
    "            metrics.roc_auc_score(pred, test[i][target])\n",
    "        ])\n",
    "        fi_dict[i] = {feature:value for feature,value in fear_importance.iloc[-10:].values}\n",
    "    return test_result, fi_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-10-08\n",
    "- Kospi의 수익률을 Feature로 추가\n",
    "- Kaggle의 Sigma competition을 참고, 각종 수익률에 대한 feature를 고려\n",
    "- Y의 Predict 기간을 조정\n",
    "- 학습 준비 파일을 제공 (20181013version)\n",
    "- 2010부터의 데이터를 제공 (20181008version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-10-13\n",
    "- moon_code로 실험"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-10-22\n",
    "- 뜬금없이 Panel로 노선 변경"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_multi_index(col_name):\n",
    "    return pd.MultiIndex.from_arrays([[col_name]*len(panel.loc['Close']),\n",
    "                                      panel.index.get_level_values(1).unique().tolist()])\n",
    "\n",
    "def get_multi_index(col_name):\n",
    "    return pd.MultiIndex.from_product([[col_name],list(raw_data.keys())])\n",
    "\n",
    "def add_feature(q, col_name, expr):\n",
    "    ind = get_multi_index(col_name)\n",
    "    try:\n",
    "        df = pd.DataFrame(expr.values, columns=ind, index=q.index)\n",
    "    except AttributeError:\n",
    "        df = pd.DataFrame(expr, columns=ind, index=q.index)\n",
    "    q = pd.concat((q, df), axis=1)\n",
    "    return q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-10-24\n",
    "- xgb_modelfit 사용\n",
    "- portfolio 제작"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_col_df(df_name, fillna=None):\n",
    "    df = pd.DataFrame()\n",
    "    for stock in model_set.keys():\n",
    "        df = pd.concat((df,validation_[stock][df_name]), axis=1)\n",
    "    df.columns = list(model_set.keys())\n",
    "    if fillna == 'mean':\n",
    "        df = df.fillna(df.mean())\n",
    "    else:\n",
    "        df = df.fillna(0)\n",
    "    col = pd.MultiIndex.from_product([['{}'.format(df_name)],df.columns])\n",
    "    df.columns = col\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import copy\n",
    "\n",
    "def modelfit(alg, train, test, X, y):\n",
    "    alg = copy.deepcopy(alg)\n",
    "    xgb_params = alg.get_xgb_params()\n",
    "    xgtrain = xgb.DMatrix(train[X].values, label=train[y].values)\n",
    "    cvresult=xgb.cv(xgb_params, xgtrain,\n",
    "                    num_boost_round=alg.get_params()['n_estimators'],\n",
    "                    nfold=5, early_stopping_rounds=50,\n",
    "                    show_stdv=False)\n",
    "    alg.set_params(n_estimators=cvresult.shape[0])\n",
    "    alg.fit(train[X], train[y], eval_metric='auc')\n",
    "    dtrain_predictions = alg.predict(train[X])\n",
    "    dtrain_predprob = alg.predict_proba(train[X])[:,1]\n",
    "    dtest_predictions = alg.predict(test[X])\n",
    "    dtest_predprob = alg.predict_proba(test[X])[:,1]\n",
    "    print(\"Train Accuracy : {:.4f}\".format(metrics.accuracy_score(\n",
    "                            train[y].values, dtrain_predictions)))\n",
    "    print(\"Train AUC Score : {:.4f}\".format(metrics.roc_auc_score(\n",
    "                            train[y].values, dtrain_predprob)))\n",
    "    print(\"Test  Accuracy : {:.4f}\".format(metrics.accuracy_score(\n",
    "                            test[y].values, dtest_predictions)))\n",
    "    print(\"Test  AUC Score : {:.4f}\".format(metrics.roc_auc_score(\n",
    "                            test[y].values, dtest_predprob)))\n",
    "    return alg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-11-07\n",
    "- 작업 준비\n",
    "- Y3한정 90% 돌파"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import pickle\n",
    "with open('raw_data_20181008.pickle', 'rb') as handle:\n",
    "    raw_data = pickle.load(handle)\n",
    "ks11 = pd.read_csv('ks11.csv')\n",
    "ks11.index = ks11.Date\n",
    "del ks11['Date']\n",
    "\n",
    "copy_keys = list(raw_data.keys())\n",
    "for stock in copy_keys:\n",
    "    if len(raw_data[stock].loc[:'2013-01-01']) < 1:\n",
    "        del raw_data[stock]\n",
    "        \n",
    "ks11 = ks11.drop(ks11.index[np.where(ks11.Volume==0)[0]])\n",
    "        \n",
    "for key, df in raw_data.items():\n",
    "    raw_data[key] = df.drop(df.index[np.where(df.Volume==0)[0]], axis=0)\n",
    "    \n",
    "# data = ks11.copy()\n",
    "# data = raw_data['삼성전자'].copy()\n",
    "# data = raw_data['NAVER'].copy()\n",
    "# data = raw_data['LG화학'].copy()\n",
    "data = raw_data['미래에셋대우'].copy()\n",
    "\n",
    "data[\"pct_change\"] = data.Close.pct_change()\n",
    "for i in range(3, 31, 2):\n",
    "    data[\"close_ma_{}\".format(i)] = data.Close.rolling(window = i).mean()\n",
    "for i in [7, 21, 30, 60, 90, 91, 100, 120]:\n",
    "    data[\"his_vol_{}\".format(i)] = data[\"pct_change\"].rolling(window = i).std()*(252**0.5)\n",
    "for i in [1, 3, 5, 7, 10, 12, 14]:\n",
    "    data['returnsCC{}'.format(i) ]= np.log(data.Open / data.Close.shift(i))\n",
    "    data['returnsOO{}'.format(i) ] = np.log(data.Open / data.Open.shift(i))\n",
    "    data['returnsCC{}'.format(i) ]= np.log(data.Close / data.Close.shift(i))\n",
    "    data['returnsOC{}'.format(i) ] = np.log(data.Close / data.Open.shift(i))\n",
    "for i in [1, 3, 5, 7, 10, 15, 20, 25, 30, 35, 40, 45, 50,\n",
    "          55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120]:\n",
    "    for shift in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]:\n",
    "        data['Y1_{}_s{}'.format(i, shift)] = (data.Close - data.Close.shift(shift).rolling(i).mean())\n",
    "        data['Y3_{}_s{}'.format(i, shift)] = (data.Close.rolling(i).mean() - data.Close.shift(shift).rolling(i).mean())\n",
    "        data['sh{}_Y1_{}'.format(shift, i)] = np.where((data['Y1_{}_s{}'.format(i, shift)]).shift(-1*shift)>=0, 1, 0)\n",
    "        data['sh{}_Y3_{}'.format(shift, i)] = np.where((data['Y3_{}_s{}'.format(i, shift)]).shift(-1*shift)>=0, 1, 0)\n",
    "        \n",
    "data_ = data.loc['2018-07-01':'2018-10-01']\n",
    "data = data.loc['2013-07-01':'2018-07-01']\n",
    "\n",
    "X_li = [i for i in data.columns if (i.find('sh') == -1) if (i.find('Volume') == -1)]\n",
    "\n",
    "X_li2 = [i for i in data.columns \n",
    "         if (i.find('sh') == -1) \n",
    "         if (i.find('Volume') == -1)\n",
    "         if (i.find('Y') == -1)]\n",
    "\n",
    "%run moon_code\n",
    "%run xgb_modelfit\n",
    "\n",
    "import copy\n",
    "\n",
    "train, test = train_test_split(data, 0.8, dtrain=True)\n",
    "\n",
    "y_li = [i for i in data.columns if i.find('sh') != -1]\n",
    "\n",
    "def modelfit(alg, train, test, X, y):\n",
    "    alg = copy.deepcopy(alg)\n",
    "    xgb_params = alg.get_xgb_params()\n",
    "    xgtrain = xgb.DMatrix(train[X].values, label=train[y].values)\n",
    "    cvresult=xgb.cv(xgb_params, xgtrain,\n",
    "                    num_boost_round=alg.get_params()['n_estimators'],\n",
    "                    nfold=5, early_stopping_rounds=50,\n",
    "                    show_stdv=False)\n",
    "    alg.set_params(n_estimators=cvresult.shape[0])\n",
    "    alg.fit(train[X], train[y], eval_metric='auc')\n",
    "    dtrain_predictions = alg.predict(train[X])\n",
    "    dtrain_predprob = alg.predict_proba(train[X])[:,1]\n",
    "    dtest_predictions = alg.predict(test[X])\n",
    "    dtest_predprob = alg.predict_proba(test[X])[:,1]\n",
    "#     print(\"Train Accuracy : {:.4f}\".format(metrics.accuracy_score(\n",
    "#                             train[y].values, dtrain_predictions)))\n",
    "#     print(\"Train AUC Score : {:.4f}\".format(metrics.roc_auc_score(\n",
    "#                             train[y].values, dtrain_predprob)))\n",
    "#     print(\"Test  Accuracy : {:.4f}\".format(metrics.accuracy_score(\n",
    "#                             test[y].values, dtest_predictions)))\n",
    "#     print(\"Test  AUC Score : {:.4f}\".format(metrics.roc_auc_score(\n",
    "#                             test[y].values, dtest_predprob)))\n",
    "    return alg\n",
    "\n",
    "for idx, stock in enumerate(raw_data.keys()):\n",
    "        \n",
    "    if idx < a : \n",
    "        continue\n",
    "    if idx > b :\n",
    "        break\n",
    "        \n",
    "    print(stock)\n",
    "    data = raw_data[stock].copy()\n",
    "\n",
    "    data[\"pct_change\"] = data.Close.pct_change()\n",
    "    for i in range(3, 31, 2):\n",
    "        data[\"close_ma_{}\".format(i)] = data.Close.rolling(window = i).mean()\n",
    "    for i in [7, 21, 30, 60, 90, 91, 100, 120]:\n",
    "        data[\"his_vol_{}\".format(i)] = data[\"pct_change\"].rolling(window = i).std()*(252**0.5)\n",
    "    for i in [1, 3, 5, 7, 10, 12, 14]:\n",
    "        data['returnsCC{}'.format(i) ]= np.log(data.Open / data.Close.shift(i))\n",
    "        data['returnsOO{}'.format(i) ] = np.log(data.Open / data.Open.shift(i))\n",
    "        data['returnsCC{}'.format(i) ]= np.log(data.Close / data.Close.shift(i))\n",
    "        data['returnsOC{}'.format(i) ] = np.log(data.Close / data.Open.shift(i))\n",
    "    for i in [1, 3, 5, 7, 10, 15, 20, 25, 30, 35, 40, 45, 50,\n",
    "              55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120]:\n",
    "        for shift in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]:\n",
    "            data['Y1_{}_s{}'.format(i, shift)] = (data.Close - data.Close.shift(shift).rolling(i).mean())\n",
    "            data['Y3_{}_s{}'.format(i, shift)] = (data.Close.rolling(i).mean() - data.Close.shift(shift).rolling(i).mean())\n",
    "            data['sh{}_Y1_{}'.format(shift, i)] = np.where((data['Y1_{}_s{}'.format(i, shift)]).shift(-1*shift)>=0, 1, 0)\n",
    "            data['sh{}_Y3_{}'.format(shift, i)] = np.where((data['Y3_{}_s{}'.format(i, shift)]).shift(-1*shift)>=0, 1, 0)\n",
    "\n",
    "    data_ = data.loc['2018-07-01':'2018-10-01']\n",
    "    data = data.loc['2013-07-01':'2018-07-01']\n",
    "\n",
    "    X_li = [i for i in data.columns if (i.find('sh') == -1) if (i.find('Volume') == -1)]\n",
    "\n",
    "    train, test = train_test_split(data, 0.8, dtrain=True)\n",
    "\n",
    "    y_li = [i for i in data.columns if i.find('sh') != -1]\n",
    "\n",
    "    model_set = {}\n",
    "    for y in y_li:\n",
    "        model = modelfit(alg, train, test, X_li, y)\n",
    "        model_set[y] = model\n",
    "        \n",
    "    try:    \n",
    "        y1 = pd.DataFrame()\n",
    "        y3 = pd.DataFrame()\n",
    "        for idx, y in enumerate(y_li):\n",
    "            acc = metrics.accuracy_score(test[y], model_set[y].predict(test[X_li]))\n",
    "            auc = metrics.roc_auc_score(test[y], model_set[y].predict(test[X_li]))\n",
    "            res = pd.DataFrame([acc,auc], index=['accuracy', 'AUC'], columns=[y])\n",
    "            if idx%2==0:\n",
    "                y1 = pd.concat((y1, res), axis=1)\n",
    "            else:\n",
    "                y3 = pd.concat((y3, res), axis=1)\n",
    "\n",
    "        Y1 = np.array([])\n",
    "        for i in range(int(270/10)):\n",
    "            i *= 10\n",
    "            if i == 0:\n",
    "                Y1 = y1.values[:,i:i+10]\n",
    "            else:\n",
    "                Y1 = np.vstack((Y1, y1.values[:,i:i+10]))\n",
    "    except ValueError:\n",
    "        continue\n",
    "\n",
    "    Y3 = np.array([])\n",
    "    for i in range(int(270/10)):\n",
    "        i *= 10\n",
    "        if i == 0:\n",
    "            Y3 = y3.values[:,i:i+10]\n",
    "        else:\n",
    "            Y3 = np.vstack((Y3, y3.values[:,i:i+10]))\n",
    "\n",
    "    Y = np.hstack((Y3, Y1))\n",
    "\n",
    "    y1.to_excel('./20181105_result/y1_{}.xlsx'.format(stock))\n",
    "    y3.to_excel('./20181105_result/y3_{}.xlsx'.format(stock))\n",
    "    pd.DataFrame(Y).to_excel('./20181105_result/Y_{}.xlsx'.format(stock))\n",
    "\n",
    "    with open('./20181105_result/model_set_{}_20181031.pickle'.format(stock), 'wb') as handle:\n",
    "        pickle.dump(model_set, handle, protocol=pickle.HIGHEST_PROTOCOL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2018-11-21\n",
    "- Regression >> Classification으로 넘어가기 위한 준비 단계"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}