Devops support (#167)

* add python3 framework for resource download

* add tag as extra field for api/v2/app_info

* add bk_devops_nav logo

* add is_display to app, for desktop

* update py3 framework to 1.0.2

* add api for bk-sops light app

* refactor : api for bk-sops light app

* refactor 1: api for bk-sops light app

* refactor 2: api for bk-sops light app

* delete syncdb command for django version >= 1.9

* set requests default timeout 5 second and add some logs when exception

* set HTTP_TIMEOUT variable in settings

* add newline in settings_default.py

* support py3 for esb sdk, update translation for some apis, support DELETE/PUT method for request some apis, support get bk apidoc from md files

* up cc some apidocs

* update VERSION to 3.2.3

* upgrade gunicorn to 19.9.0

* add guide for python3 framework

* add app env var admin

* update py3 framework to 1.0.3

* fix js pagination error

* fix js pagination error

* up cc confapis

* Devops support fixbug20190306 (#14)

* add error message where user info (phone, chname) check

* fixbug: light app logo image name ramdom

* refactor

* up cc/sops apis

* up cc/sops confapis

* add try-except for bk_backend.authenticate (#15)

* Bugfix 0311 (#16)

* add try-except for bk_backend.authenticate

* update bk_devops_nav logo

* app.yml support app env init while setup (#17)

* app.yml support app env init while setup

* fix app env key/value validation for app.yml

* env key/value type cast to str before validate

* unify the app_env_var value length to 1000

* support old login account manage page url (#18)

* remove on_migrate

* add doc to release.md: update python framework to support https

* update release.md

* third server password should not be chinese

* fix link too long text-overflow

* fix some issues

* remove header for some files

* bugfix: bktoken decrypt wrong

* batch issues fix

* change version to 3.2.5

* modify default user logo (#51)

* fix some icons display error (#52)

* fix fta imap (#56)

* Update release.md

* fix the too complex of account _is_bk_token_valid (#59)

* Explain login/paas settings different between development/production (#79) (#80)

* login settings SITE_URL diff in develoment/production

* update doc: LOGIN_DOMAIN in paas development

* 修复本地开发登录URL 404问题 (#82)


* fix develop login url 404 (#81)

* 更新 sops 组件，更新cmsi 发送语音通知组件 (#84)

* up sops confapis, cmsi send_voice_msg

* add component conf for cmsi send_voice_msg

* 更新release说明 (#85)

* up sops confapis, cmsi send_voice_msg

* add component conf for cmsi send_voice_msg

* up release doc

* fix login decrypt token code bug (#88)

* paas_agent support python3 (#89)

* fix sync blocking bug when build script times out (#64)

* Upgrade python framework to 2.0.0 (#90)

* upgrade python framework to 2.0.0

* change new framework icon

* fix fta imap (#103)

* Update weixin doc and fix ligth app logo (#105)

* fix light app lose bug

* update weixin notify doc

* update weixin notify doc

* fixbug: import user do not response bug

* fix cc delete_object_attribute apidoc (#106)

* update guide doc of python framework (#108)

* bugfix: 滚动按钮提示内容错误。issue #97 (#113)

* fixbug:Development Language Change #96 (#116)

* add app-env title (#124)

* add resource migration for python doc url and delete united dev-env (#129)

* Add docker (#131)

* add docker docs for quick experience

* update docker support docs

* update docker support docs

* upgrade framework to 2.1.4.11 (#134)

* update link or replace ce with open source version (#135)

* 93:修改BUG：当版本历史为空时, 前端出现两条横线 (#123)

* add static version for css/js (#136)

* Fix user avatar (#142)

* fix user avatar

* add import_common_template for sops

* Bugfix 0717 (#143)

* fix #128; remove django specific version in document

* fix #140; resource link to official website

* use HTTP_SCHEMA for job/cc link int (#145)

* update cc apis, add cmsi send_msg/get_msg_type (#146)

* saas info show settings style (#150)

* fix #147; add/delete user refresh list without query term (#149)

* disable bk_language while the i18n is not support in ce (#155)

* fix resource download icon (#156)

* fix resource download icon

* fix pep8 fail

* fix login api ApiErrorCodeEnumV2 missing USER_NOT_EXISTS2 (#158)

* support logo file in pkg root dir (#164)

* Update docs (#166)

* update readme.md/readme_en.md

Co-authored-by: nannan00 <17491932+nannan00@users.noreply.github.com>
Co-authored-by: jamesge <gejun.coolfriend@gmail.com>
Co-authored-by: alex-smile <443677891@qq.com>
Co-authored-by: wenjaron <45757842+wenjaron@users.noreply.github.com>
Co-authored-by: kindmaketheworldpeace <38469337+kindmaketheworldpeace@users.noreply.github.com>
Co-authored-by: canway-obama <50621369+canway-obama@users.noreply.github.com>
Co-authored-by: molly0755 <16878048@qq.com>

VERSION

@@ -1,2 +1,2 @@
-PaaS      3.2.2
+PaaS      3.2.5
 PaaSAgent 3.1.0

docs/install/ce_paas_agent_install.md

@@ -2,7 +2,7 @@
 
 ## 系统要求
 
-- Python版本: python2.7
+- Python版本: python2.7/python3.6.7
 - Go
 - Nginx
 - Linux环境
@@ -12,21 +12,33 @@ PaaSAgent需要部署在满足系统要求的app服务器上。建议最少准
 
 #### 1. 基础环境初始化
 
-python 2.7环境初始化
+python 环境初始化
 
 ```
 # 确认基础python环境是python2.7
 $ python --version
 Python 2.7.15
 
+# 执行以下命令确定python的安装路径
+$ which python
+
 # 安装 virtualenv 及 预装包
 $ pip install virtualenv virtualenvwrapper
+$ pip install supervisor==3.3.3
 $ pip install -r paas-ce/paasagent/etc/build/packages/requirements.txt
 
-# 查看python路径
-$ which python
+# 如果使用python3，建议将python3安装到/opt/py36目录下
+# 从源码包编译安装python3.6.7时，可能需要手动建立python和pip的软链
+$ cd /opt/py36/bin
+$ ln -s python3.6 python
+$ ln -s pip3 pip
+
+# 确认python3的版本
+$ /opt/py36/bin/python --version
+Python 3.6.7
 ```
 
+
 构建目录及新建用户
 
 ```
@@ -62,7 +74,7 @@ $ sed -i "s#TPL_AGENT_ROOT#${AGENT_ROOT}#g" paas_agent_config.yaml nginx/paasage
 $ chmod +x build/virtualenv/build  build/virtualenv/saas/buildsaas
 ```
 
-注意: `build` 和 `buildsaas` 中的 `VIRTUALENVWRAPPER_PYTHON` 变量需要设置正确的 python 路径，默认`/usr/bin/python`, 如果上一步`which python`的返回结果不是`/usr/bin/python`, 需要修改为对应正确的路径
+注意: `build` 和 `buildsaas` 中的 `VIRTUALENVWRAPPER_PYTHON` 变量需要设置正确的 python 路径，默认`/usr/bin/python`, 如果上一步`which python`的返回结果不是`/usr/bin/python`, 需要修改为对应正确的路径；如果用到了python3，`PYTHON3_PATH`也需要设置成正确的路径，默认`/opt/py36`
 
 #### 3. 蓝鲸智云开发者中心注册服务器
 

docs/wiki/container-support.md

@@ -0,0 +1,79 @@
+# 极速体验
+
+为了能让大家最快的体验蓝鲸智云PaaS平台，社区推出了蓝鲸智云PaaS平台的容器部署方式，秒级体验PaaS平台服务。
+
+注意:
+- 镜像仅用于快速体验, 不能用于生产; 
+- 目前可以体验login(统一登录)/paas(开发者中心)/esb(API网关), 暂时还不能体验paas_agent(应用引擎: 应用部署流程); 目前处理中, 后续会加入
+
+## Docker环境
+
+蓝鲸智云PaaS平台容器化部署同时支持让你在Windows, macOS, Linux环境下进行部署。
+
+如果你已经有docker的环境，可以直接进入部署环节。如果你没有docker环境，也想体验，请参照下面的官方链接进行部署。
+
+* windows
+
+    docker官方社区目前只支持Windows10(64位)版本，并启用了Hyper-V服务。Docker for Windows的安装
+    包在[这里](https://download.docker.com/win/stable/19507/Docker%20for%20Windows%20Installer.exe)
+    下载并安装。安装细节可以参考[这里](https://docs.docker.com/docker-for-windows/install/#what-to-know-before-you-install)。
+
+* macOS
+
+    安装命令如下。安装完成后，直接启动docker即可。
+    ```shell
+    brew cask install docker
+    ```
+
+ * linux
+    由于linux的发行版本比较多，这里就不逐一列出，大家可以参考docker官方的文档进行安装。
+    - [CentOS](https://docs.docker.com/install/linux/docker-ce/centos/)
+    - [Debian](https://docs.docker.com/install/linux/docker-ce/debian/)
+    - [Fedora](https://docs.docker.com/install/linux/docker-ce/fedora/)
+    - [Ubuntu](https://docs.docker.com/install/linux/docker-ce/ubuntu/)
+
+## 部署
+
+#### 1. 拉取镜像
+
+当你有了docker的环境后，就可以在**终端**拉取bk-PaaS的体验镜像了。
+
+bk-PaaS用于体验的镜像存储于腾讯云的镜像公共镜像仓库。 仓库地址在[这里](https://console.cloud.tencent.com/tke/registry/qcloud/default/detail/tag?rid=1&reponame=bk.io%252Fpaas-standalone)
+
+目前有一个tag为**latest**的镜像, 方便大家体验最新版本; 未来会提供与release版本一致的镜像，
+
+
+镜像名称：ccr.ccs.tencentyun.com/bk.io/paas-standalone
+
+镜像tag: latest
+
+
+拉取命令：
+
+```shell
+docker pull ccr.ccs.tencentyun.com/bk.io/paas-standalone:latest
+```
+
+#### 2. 启动容器
+
+执行如下命令，bk-PaaS容器服务即可启动。
+
+```shell
+docker run -d --name="bk-paas" -p 8000-8003:8000-8003 ccr.ccs.tencentyun.com/bk.io/paas-standalone:latest
+```
+
+这里端口映射: `appengine 8000/ paas 8001 / esb 8002 / login 8003`
+
+#### 3. 配置host及访问
+
+
+配置host
+
+```
+127.0.0.1 www.bking.com
+```
+
+此时你可以打开浏览器，访问`http://www.bking.com:8001`, 使用默认用户名密码(admin/admin)登录, 即可体验最新版的蓝鲸智云PaaS平台服务。
+
+**恭喜你，你正在体验最新的蓝鲸智云PaaS平台服务。**
+

paas-ce/paas/VERSION

@@ -1 +1 @@
-3.2.2
+3.2.5

paas-ce/paas/appengine/requirements.txt

@@ -1,6 +1,6 @@
 django==1.8.11
 pymysql==0.6.7
 requests==2.21.0
-gunicorn==19.6.0
+gunicorn==19.9.0
 uWSGI==2.0.13.1
 gevent==1.1.2

paas-ce/paas/esb/apps/sdk_management/sdk_tmpl/blueking/component/base.py

@@ -40,7 +40,7 @@ def __call__(self, *args, **kwargs):
         self.url = self.get_url_with_api_ver()
         try:
             return self._call(*args, **kwargs)
-        except ComponentAPIException, e:
+        except ComponentAPIException as e:
             # Combine log message
             log_message = [e.error_message, ]
             log_message.append('url=%(url)s' % {'url': e.api_obj.url})
@@ -75,7 +75,7 @@ def _call(self, *args, **kwargs):
         # Request remote server
         try:
             resp = self.client.request(self.method, self.url, params=params, data=data)
-        except Exception, e:
+        except Exception as e:
             logger.exception('Error occurred when requesting method=%s url=%s',
                              self.method, self.url)
             raise ComponentAPIException(self, u'Request component error, Exception: %s' % str(e))

paas-ce/paas/esb/apps/sdk_management/sdk_tmpl/blueking/component/client.py

@@ -13,8 +13,8 @@
 import time
 import random
 import logging
-import urlparse
 
+from .compat import urlparse
 from . import conf
 from . import collections
 from .utils import get_signature
@@ -133,7 +133,7 @@ def request(self, method, url, params=None, data=None, **kwargs):
         if method == 'POST':
             params = {}
 
-        url_path = urlparse.urlparse(url).path
+        url_path = urlparse(url).path
         # signature always in GET params
         params.update({
             'bk_timestamp': int(time.time()),

paas-ce/paas/esb/apps/sdk_management/sdk_tmpl/blueking/component/compat.py

@@ -0,0 +1,26 @@
+# -*- coding: utf-8 -*-
+"""
+Tencent is pleased to support the open source community by making 蓝鲸智云PaaS平台社区版 (BlueKing PaaS Community Edition) available.
+Copyright (C) 2017-2018 THL A29 Limited, a Tencent company. All rights reserved.
+Licensed under the MIT License (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+http://opensource.org/licenses/MIT
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+""" # noqa
+import sys
+
+_ver = sys.version_info
+
+is_py2 = (_ver[0] == 2)
+
+is_py3 = (_ver[0] == 3)
+
+
+if is_py2:
+    from urlparse import urlparse
+
+    str = unicode
+
+elif is_py3:
+    from urllib.parse import urlparse  # noqa
+
+    str = str

paas-ce/paas/esb/apps/sdk_management/sdk_tmpl/blueking/component/utils.py

@@ -11,6 +11,8 @@
 import hmac
 import hashlib
 
+from .compat import str
+
 
 def get_signature(method, path, app_secret, params=None, data=None):
     """generate signature
@@ -23,8 +25,10 @@ def get_signature(method, path, app_secret, params=None, data=None):
         kwargs['data'] = data
     kwargs = '&'.join([
         '%s=%s' % (k, v)
-        for k, v in sorted(kwargs.iteritems(), key=lambda x: x[0])
+        for k, v in sorted(kwargs.items(), key=lambda x: x[0])
     ])
     orignal = '%s%s?%s' % (method, path, kwargs)
-    signature = base64.b64encode(hmac.new(str(app_secret), orignal, hashlib.sha1).digest())
-    return signature
+    app_secret = app_secret.encode('utf-8') if isinstance(app_secret, str) else app_secret
+    orignal = orignal.encode('utf-8') if isinstance(orignal, str) else orignal
+    signature = base64.b64encode(hmac.new(app_secret, orignal, hashlib.sha1).digest())
+    return signature if isinstance(signature, str) else signature.decode('utf-8')

paas-ce/paas/esb/common/base_loggers.py

@@ -7,6 +7,7 @@
 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 """ # noqa
 import json
+import copy
 
 from common.log import logger_api, logger
 from common.base_utils import datetime_format
@@ -23,6 +24,11 @@ def write(self, request, response):
             kwargs = request.g.kwargs_copy
         else:
             kwargs = request.g.kwargs
+
+        if request.g.system_name == 'CMSI' and request.g.component_alias_name == 'send_mail':
+            kwargs = copy.copy(kwargs)
+            kwargs.pop('attachments', None)
+
         msecs_cost = (request.g.ts_request_end - request.g.ts_request_start) * 1000
         if isinstance(response, dict):
             message = response and response.get('message', '')

paas-ce/paas/esb/common/base_utils.py

@@ -215,9 +215,9 @@ def get_client_real_ip(request):
 
 
 def get_request_params(request):
-    if request.method not in ('GET', 'POST'):
-        raise error_codes.COMMON_ERROR.format_prompt(
-            'Request method error, please apply GET or POST request.', replace=True)
+    # if request.method not in ('GET', 'POST'):
+    #     raise error_codes.COMMON_ERROR.format_prompt(
+    #         'Request method error, please apply GET or POST request.', replace=True)
     # "GET"方法
     if request.method == 'GET':
         request_params = dict(request.GET.items())
@@ -262,3 +262,26 @@ def load_yaml(path):
 def read_file(path):
     with open(path, 'r') as fp:
         return fp.read()
+
+
+def html_escape(html, is_json=False):
+    """
+    Replace special characters "&", "<" and ">" to HTML-safe sequences.
+    If the optional flag quote is true, the quotation mark character (")
+    is also translated.
+    rewrite the cgi method
+    @param html: html代码
+    @param is_json: 是否为json串（True/False） ，默认为False
+    """
+    # &转换
+    if not is_json:
+        html = html.replace('&', '&amp;')  # Must be done first!
+    # <>转换
+    html = html.replace("<", "&lt;")
+    html = html.replace(">", "&gt;")
+    # 单双引号转换
+    if not is_json:
+        # html = html.replace(' ', "&nbsp;")
+        html = html.replace('"', "&quot;")
+        html = html.replace("'", "&#39;")
+    return html

paas-ce/paas/esb/common/constants.py

@@ -26,12 +26,17 @@
 API_TYPE_OP = 'operate'
 API_TYPE_Q = 'query'
 
+HTTP_METHOD = FancyDict((
+    ('GET', 'GET'),
+    ('POST', 'POST'),
+))
+
 
 BK_SYSTEMS = {
     'BK_LOGIN': {
         'name': 'BK_LOGIN',
-        'label': _(u'蓝鲸登录平台'),
-        'remark': u'蓝鲸登录平台，管理用户登录验证，及用户信息',
+        'label': _(u'蓝鲸统一登录'),
+        'remark': u'蓝鲸统一登录，管理用户登录验证，及用户信息',
     },
     'BK_PAAS': {
         'name': 'BK_PAAS',

paas-ce/paas/esb/common/errors.py

@@ -144,7 +144,7 @@ class ErrorCodes(object):
     """
 
     error_codes = (
-        # 13063xx, user error
+        # 13064xx, user error
         ErrorCode('OPERATOR_REQUIRED', 1306401, 'You must specify the current operator'),
         ErrorCode('USER_PERMISSION_DENIED', 1306402, 'User permission is insufficient'),
         ErrorCode('APP_PERMISSION_DENIED', 1306403, 'APP permission is insufficient'),

paas-ce/paas/esb/components/bk/apis/bk_login/get_user.py

@@ -71,6 +71,14 @@ class GetUser(Component):
     class Form(BaseComponentForm):
         bk_token = forms.CharField(label='login token', required=False)
         username = forms.CharField(label='username', required=False)
+        bk_username = forms.CharField(label='bk_username', required=False)
+
+        def clean(self):
+            data = self.cleaned_data
+            return {
+                'bk_token': data['bk_token'],
+                'username': data['username'] or data['bk_username'],
+            }
 
     def handle(self):
         self.response.payload = self.outgoing.http_client.get(

paas-ce/paas/esb/components/bk/apis/cc/toolkit/tools.py

@@ -45,8 +45,8 @@ def post_request(self, host, path, data=None, headers_json=True, **kwargs):
         except Exception:
             logger.exception('response: %s', response)
             raise CommonAPIError(
-                'An exception occurred while requesting CC interface, '
-                'please contact the component developer to handle it.')
+                'An error occurred while requesting CC interface, '
+                'the response content does not contain code field.')
 
         if code != '0':
             return {

paas-ce/paas/esb/components/bk/apis/fta/toolkit/imap.py

@@ -174,20 +174,22 @@ def search(self, charset=None, unseen=None, before=None, since=None, size_limit=
     def filter(self, mails, sent_from=None, sent_to=None, subject=None, before=None, since=None,
                fetch_header_already=False):
         result_list = []
-        mails = mails if fetch_header_already else self.iter_fetch_chunks(mails=mails, criteria=self.CRITERIA_HEADER,)
+        mails = mails
+        if not fetch_header_already:
+            for index in range(0, len(mails)):
+                self.fetch_header(mails[index * 100:(index + 1) * 100])
         sent_from = sent_from.split(",") if sent_from else ()
         sent_to = sent_to.split(",") if sent_to else ()
         subject = [subject] if subject else []
         for mail in mails:
-            if ((before and before <= mail.time) or (since and since > mail.time)):
+            if mail.time is None or (before and before <= mail.time) or (since and since > mail.time):
                 continue
             if sent_from and not contain_one(mail.sender, sent_from):
                 continue
             if sent_to and not contain_one(mail.receiver, sent_to):
                 continue
             if subject and not contain_one(mail.subject, subject):
                 continue
-
             result_list.append(mail)
         return result_list
 
@@ -197,7 +199,7 @@ def fetch_header(self, mails):
 
     def fetch(self, mails, criteria=None):
         mail_mappings = {str(mail.uid): mail for mail in mails}
-        status, result = self.imap_client.fetch(",".join(mail_mappings.keys()), criteria or self.CRITERIA,)
+        status, result = self.imap_client.fetch(",".join(mail_mappings.keys()), criteria or self.CRITERIA)
         if status != "OK":
             raise Exception("fetch mail[%s] failed", mail_mappings.keys())
 

paas-ce/paas/esb/components/bk/apisv2/cc/bind_role_privilege.py

@@ -33,7 +33,7 @@ class BindRolePrivilege(Component):
 
     | {{ _("字段") }}      |  {{ _("类型") }}      | {{ _("必选") }}   |  {{ _("描述") }}      |
     |-----------|------------|--------|------------|
-    | bk_supplier_account |  string    | {{ _("是") }}     | {{ _("开发商帐号") }} |
+    | bk_supplier_account |  string    | {{ _("是") }}     | {{ _("开发商账号") }} |
     | bk_obj_id           |  string    | {{ _("是") }}     | {{ _("模型ID") }} |
     | bk_property_id      |  string    | {{ _("是") }}     | {{ _("模型对应用户角色属性ID") }}   |
     | data                |  list      | {{ _("否") }}     | {{ _("角色权限数据，输入为空数组则不绑定任何权限") }}   |