Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gitlab Integration #1040

Open
wants to merge 80 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
d47b2f7
Initial Setup
shariff6 Sep 13, 2024
39ff4b3
Adds blueprint for gitlabIssues, Projects, Merge Requests and Groups
shariff6 Sep 13, 2024
31af127
Adds gitlabProject, gitlabMergeRequests, gitlabIssues and gitlabGroup…
shariff6 Sep 13, 2024
31c7b6b
Update lock file
shariff6 Sep 13, 2024
b341e5c
Adds config.yaml file
shariff6 Sep 13, 2024
cb1f6e1
Adds kinds gitlabGroup, gitlProject , gitlabIssue and gitlabMergeRequest
shariff6 Sep 13, 2024
2e107e0
corrects project identifier, reames to gitlabProject
shariff6 Sep 13, 2024
7019701
Initialise client
shariff6 Sep 13, 2024
6ae8c50
Client for gitlab api, get projects, issues, mr's and groups
shariff6 Sep 14, 2024
1468851
fethc, projects, groups, issues and mrs async with ocean
shariff6 Sep 14, 2024
86ce514
adds gitlab token to env
shariff6 Sep 14, 2024
b78b8f8
Error Message Fix
shariff6 Sep 14, 2024
81e29e7
Uses AsyncGenerator class for fetching and processing
shariff6 Sep 14, 2024
51e07b9
removes pagination for now. Reduces page size to 10
shariff6 Sep 14, 2024
d3ad53a
Updates main.py to work with Async Generator type,
shariff6 Sep 14, 2024
c7d5cae
Adds a fetch_items to fetch resources asynchronously.
shariff6 Sep 14, 2024
868a767
Implements rate_limiting using asyncio, Semaphore.
shariff6 Sep 14, 2024
e48676a
Removes Record Limit, Stops fetching when completed
shariff6 Sep 14, 2024
e6c22a6
Refactor resync handling: Split single on_resync function into indivi…
shariff6 Sep 14, 2024
f49a27e
Adds a gitlab_rate_limiter class, that implements rate limiting based…
shariff6 Sep 14, 2024
37fa4b1
Fixes bug with issues and mrs, fetches the authroised users mrs, proj…
shariff6 Sep 15, 2024
77e180d
Simplifies gitlab handler, moves resource specific mapping to mapper …
shariff6 Sep 15, 2024
d7fdeb4
Moves rate limiting to helper class
shariff6 Sep 15, 2024
c693736
Updates main to work with new mapper helper
shariff6 Sep 15, 2024
34bf883
Adds endpoint to communicate with Gitlab Webhooks API
shariff6 Sep 15, 2024
c1f7bc5
Adds Webhook Secret and Wehbook Url to port configs
shariff6 Sep 15, 2024
231ac46
Modifies params in resource fetching
shariff6 Sep 15, 2024
e6def0b
Removes lopp in fetch_paginated_resources.
shariff6 Sep 17, 2024
4d9952a
Uses initials gitlab handler, yield response on resync.
shariff6 Sep 17, 2024
c9bfb06
Adds a gitlab_config for configuring specific params, and loading add…
shariff6 Sep 17, 2024
08848f0
Removes redundant error checks
shariff6 Sep 17, 2024
d7b8f80
Removes webhook logic from gitlab handler class, moves
shariff6 Sep 17, 2024
ab1f204
Adds an init client to initalise gitlab handler,
shariff6 Sep 17, 2024
2d472fc
Fixes bug ewhen additional data is not defined
shariff6 Sep 18, 2024
7499ffd
Cleans up file for bettter readability,
shariff6 Sep 18, 2024
084f894
Add semaphore to regulate concurrent requests
shariff6 Sep 18, 2024
e961c44
Updates propertires to match blueprint
shariff6 Sep 18, 2024
9ce86c9
Adds logic for group webhooks
shariff6 Sep 18, 2024
e353543
Removes inline comments
shariff6 Sep 18, 2024
316fa2c
Removes inline comments
shariff6 Sep 18, 2024
bed744b
Specifies token as gitlab token
shariff6 Sep 18, 2024
5b353f7
Maps identifiers to string type
shariff6 Sep 18, 2024
a8ac9a7
Removes unused mapper factory
shariff6 Sep 18, 2024
10ee430
Updates gitlab params with those allowed by permission
shariff6 Sep 18, 2024
eaf25c1
Merge branch 'main' of https://github.com/shariff6/ocean into gitlab_v1
shariff6 Sep 18, 2024
8b3a4dc
Merge pull request #3 from shariff6/gitlab_v1
shariff6 Sep 18, 2024
6aadbc9
Adds support for multiple tokens
shariff6 Sep 18, 2024
3cb9e22
Mutliple token support
shariff6 Sep 18, 2024
39fc89d
Merge pull request #4 from shariff6/gitlab_v1
shariff6 Sep 18, 2024
3c6109b
Merge branch 'main' into main
shariff6 Sep 18, 2024
7f3978e
Adds description and requires gitlabToken
shariff6 Sep 18, 2024
8e26bad
Removes check on webhook secret gor incoming post from gitlab
shariff6 Sep 18, 2024
383900d
Generates secret fif webhook secret is not provided
shariff6 Sep 18, 2024
0c04db1
Removes unused project webhook code
shariff6 Sep 18, 2024
1892e9b
moves gitlab integration class to its own file cleaning up main file
shariff6 Sep 18, 2024
dd8b3de
Adds events to be attached to webhook, configures events through yaml…
shariff6 Sep 18, 2024
a119fec
removes gitlab prefix from kinds
shariff6 Sep 18, 2024
2069374
Update CHANGELOG.md
shariff6 Sep 18, 2024
d8c87d1
Removes configuration from launch.json
shariff6 Sep 18, 2024
381ee2b
Merge branch 'gitlab_v1' of https://github.com/shariff6/ocean into gi…
shariff6 Sep 18, 2024
98d8afb
Update CONTRIBUTING.md
shariff6 Sep 18, 2024
298c6b8
Merge branch 'gitlab_v1' of https://github.com/shariff6/ocean into gi…
shariff6 Sep 18, 2024
4d26340
Uncomments aws credentials check in pre-commit
shariff6 Sep 18, 2024
5e5273d
Merge pull request #5 from shariff6/gitlab_v1
shariff6 Sep 18, 2024
d0ecd05
Merge branch 'port-labs:main' into main
shariff6 Sep 19, 2024
b54bec1
Adds support for all events to the setup group webhooks
shariff6 Sep 19, 2024
2bbb3bd
Merge pull request #6 from shariff6/gitlab_v1
shariff6 Sep 19, 2024
c80f27a
Adds generic resync
shariff6 Sep 20, 2024
d7de9e9
Merge pull request #7 from shariff6/gitlab_v1
shariff6 Sep 20, 2024
9603ee7
Merge branch 'port-labs:main' into main
shariff6 Sep 20, 2024
395c837
--no-verify
shariff6 Sep 23, 2024
ea590a0
uses token comma separated string instead of array
shariff6 Sep 23, 2024
8372bb2
Comments out unused additonal data for this task
shariff6 Sep 23, 2024
b76b7c1
Cleans up code, combines webhook events for better handling
shariff6 Sep 23, 2024
a34df09
Merge pull request #8 from shariff6/gitlab_v1
shariff6 Sep 23, 2024
4998d7b
Merge branch 'port-labs:main' into main
shariff6 Sep 23, 2024
34259f6
Fixes pre-commit config
shariff6 Sep 23, 2024
c9ff417
Updates env.example
shariff6 Sep 23, 2024
a1876a0
Updates ocean to latest version
shariff6 Sep 23, 2024
e189035
Merge pull request #9 from shariff6/gitlab_v1
shariff6 Sep 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions integrations/gitlab_v1/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
OCEAN__PORT__CLIENT_ID="<port-client-id>"
OCEAN__PORT__CLIENT_SECRET="<port-client-secret>"
OCEAN__INTEGRATION__CONFIG__GITLAB_TOKEN="<gitlab-token>,<gitlab-token>,<gitlab-token>"
OCEAN__INTEGRATION__CONFIG__APP_HOST=
OCEAN__INTEGRATION__CONFIG__WEBHOOK_SECRET=
153 changes: 153 additions & 0 deletions integrations/gitlab_v1/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/
1 change: 1 addition & 0 deletions integrations/gitlab_v1/.port/resources/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
!.gitignore
228 changes: 228 additions & 0 deletions integrations/gitlab_v1/.port/resources/blueprints.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,228 @@
[
{
"identifier": "gitlabGroup",
"title": "Group",
"icon": "GitLab",
"schema": {
"properties": {
"visibility": {
"icon": "Lock",
"title": "Visibility",
"type": "string",
"enum": [
"public",
"internal",
"private"
],
"enumColors": {
"public": "red",
"internal": "yellow",
"private": "green"
}
},
"url": {
"title": "URL",
"format": "url",
"type": "string",
"icon": "Link"
},
"description": {
"title": "Description",
"type": "string",
"icon": "BlankPage"
}
},
"required": []
},
"mirrorProperties": {},
"calculationProperties": {},
"aggregationProperties": {},
"relations": {}
},
{
"identifier": "gitlabProject",
"title": "Project",
"icon": "Microservice",
"schema": {
"properties": {
"url": {
"title": "URL",
"type": "string",
"format": "url"
},
"readme": {
"title": "README",
"type": "string",
"format": "markdown"
},
"description": {
"title": "Description",
"type": "string"
},
"language": {
"title": "Language",
"type": "string"
},
"namespace": {
"title": "Namespace",
"type": "string"
},
"fullPath": {
"title": "Full Path",
"type": "string"
},
"defaultBranch": {
"title": "Default Branch",
"type": "string"
}
},
"required": []
},
"mirrorProperties": {},
"calculationProperties": {},
"relations": {
"group": {
"title": "Group",
"target": "gitlabGroup",
"required": true,
"many": false
}
}
},
{
"identifier": "gitlabMergeRequest",
"title": "Merge Request",
"icon": "GitVersion",
"schema": {
"properties": {
"creator": {
"title": "Creator",
"type": "string"
},
"status": {
"title": "Status",
"type": "string",
"enum": [
"opened",
"closed",
"merged",
"locked"
],
"enumColors": {
"opened": "yellow",
"closed": "red",
"merged": "green",
"locked": "blue"
}
},
"createdAt": {
"title": "Created At",
"type": "string",
"format": "date-time"
},
"updatedAt": {
"title": "Updated At",
"type": "string",
"format": "date-time"
},
"mergedAt": {
"title": "Merged At",
"type": "string",
"format": "date-time"
},
"link": {
"title": "Link",
"format": "url",
"type": "string"
},
"reviewers": {
"title": "Reviewers",
"type": "array",
"items": {
"type": "string"
}
}
},
"required": []
},
"mirrorProperties": {},
"calculationProperties": {
"lead_time_days": {
"title": "Lead time (Days)",
"calculation": "if .properties.status == \"merged\" then ((.properties.mergedAt | sub(\"\\\\.[0-9]+\\\\+00:00$\"; \"Z\") | strptime(\"%Y-%m-%dT%H:%M:%SZ\") | mktime) - (.properties.createdAt | sub(\"\\\\.[0-9]+\\\\+00:00$\"; \"Z\") | strptime(\"%Y-%m-%dT%H:%M:%SZ\") | mktime)) / 86400 | tonumber else null end",
"type": "number"
}
},
"aggregationProperties": {},
"relations": {
"service": {
"title": "Project",
"target": "project",
"required": false,
"many": false
}
}
},
{
"identifier": "gitlabIssue",
"title": "Issue",
"icon": "GitLab",
"schema": {
"properties": {
"link": {
"title": "Link",
"type": "string",
"format": "url"
},
"description": {
"title": "Description",
"type": "string",
"format": "markdown"
},
"createdAt": {
"title": "Created At",
"type": "string",
"format": "date-time"
},
"closedAt": {
"title": "Closed At",
"type": "string",
"format": "date-time"
},
"updatedAt": {
"title": "Updated At",
"type": "string",
"format": "date-time"
},
"creator": {
"title": "Creator",
"type": "string"
},
"status": {
"title": "Status",
"type": "string",
"enum": ["opened", "closed"],
"enumColors": {
"opened": "green",
"closed": "purple"
}
},
"labels": {
"title": "Labels",
"type": "array",
"items": {
"type": "string"
}
}
}
},
"relations": {
"service": {
"target": "project",
"title": "Project",
"required": true,
"many": false
}
}
}
]
Loading