-
Notifications
You must be signed in to change notification settings - Fork 61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Integration][async-gitlab] - GitLab integration with Async support #1121
Closed
Closed
Changes from all commits
Commits
Show all changes
26 commits
Select commit
Hold shift + click to select a range
8d0f592
feat(async-gitlab): adds blue print and spec config
victor-devv 6aa7665
feat(async-gitlab): adds github client and utils file
victor-devv 6a3c1d5
feat(async-gitlab): adds groups, projects, MRs and issue re-syncs
victor-devv f58e65a
feat(async-gitlab): completes resync functions for all kinds
victor-devv 1851442
feat(async-gitlab): refactors code and adds webhook integration
victor-devv cf36947
chore(async-gitlab): cleanup
victor-devv ffeb379
ref(async-gitlab): adds spec defaults, adds test cases, shrinks clien…
victor-devv 00d253a
chore(async-gitlab): cleanup
victor-devv 7c960a4
ref(async-gitlab): uses toString flag to convert numeric resource ids
victor-devv 7cb7217
Merge branch 'main' into feat/async-gitlab
victor-devv e1807a8
ref(async-gitlab): uses single resync func for resources
victor-devv 7fbf950
ref(async-gitlab): shrinks webhook handler
victor-devv 6dd6ecc
feat(async-gitlab): sets up system hooks on startup
victor-devv d30b7f4
chore(async-gitlab): cleanup
victor-devv 392e4fc
chore(async-gitlab): cleanup
victor-devv a45102b
ref(async-gitlab): disposes api ver method
victor-devv 29282d6
chore(async-gitlab): cleanup
victor-devv fee1291
chore(async-gitlab): cleanup
victor-devv 5f58503
feat(async-gitlab): adds support for both system and group webhooks
victor-devv 3eee70a
chore(async-gitlab): cleanup
victor-devv cf622e6
chore(async-gitlab): cleanup
victor-devv 6375f16
ref(sync-gitlab): cleanup
victor-devv 5669f87
ref(async-gitlab): adds checks for app host before webhook setup
victor-devv 0c1ec05
feat(async-gitlab): uses gitlab rate limiter headers to set async lim…
victor-devv 86f683c
chore(async-gitlab): cleanup
victor-devv f04eb39
chore: lints code
victor-devv File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
OCEAN__PORT__CLIENT_ID="<port-client-id>" | ||
OCEAN__PORT__CLIENT_SECRET="<port-client-secret>" | ||
OCEAN__INTEGRATION__CONFIG__GITLAB_HOST="<gitlab_host_url>" | ||
OCEAN__INTEGRATION__CONFIG__ACCESS_TOKEN="<gitlab_access_token>" | ||
OCEAN__EVENT_LISTENER={"type": "POLLING"} | ||
OCEAN__INTEGRATION="async-gitlab" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,153 @@ | ||
# Byte-compiled / optimized / DLL files | ||
__pycache__/ | ||
*.py[cod] | ||
*$py.class | ||
|
||
# C extensions | ||
*.so | ||
|
||
# Distribution / packaging | ||
.Python | ||
build/ | ||
develop-eggs/ | ||
dist/ | ||
downloads/ | ||
eggs/ | ||
.eggs/ | ||
lib/ | ||
lib64/ | ||
parts/ | ||
sdist/ | ||
var/ | ||
wheels/ | ||
share/python-wheels/ | ||
*.egg-info/ | ||
.installed.cfg | ||
*.egg | ||
MANIFEST | ||
|
||
# PyInstaller | ||
# Usually these files are written by a python script from a template | ||
# before PyInstaller builds the exe, so as to inject date/other infos into it. | ||
*.manifest | ||
*.spec | ||
|
||
# Installer logs | ||
pip-log.txt | ||
pip-delete-this-directory.txt | ||
|
||
# Unit test / coverage reports | ||
htmlcov/ | ||
.tox/ | ||
.nox/ | ||
.coverage | ||
.coverage.* | ||
.cache | ||
nosetests.xml | ||
coverage.xml | ||
*.cover | ||
*.py,cover | ||
.hypothesis/ | ||
.pytest_cache/ | ||
cover/ | ||
|
||
# Translations | ||
*.mo | ||
*.pot | ||
|
||
# Django stuff: | ||
*.log | ||
local_settings.py | ||
db.sqlite3 | ||
db.sqlite3-journal | ||
|
||
# Flask stuff: | ||
instance/ | ||
.webassets-cache | ||
|
||
# Scrapy stuff: | ||
.scrapy | ||
|
||
# Sphinx documentation | ||
docs/_build/ | ||
|
||
# PyBuilder | ||
.pybuilder/ | ||
target/ | ||
|
||
# Jupyter Notebook | ||
.ipynb_checkpoints | ||
|
||
# IPython | ||
profile_default/ | ||
ipython_config.py | ||
|
||
# pyenv | ||
# For a library or package, you might want to ignore these files since the code is | ||
# intended to run in multiple environments; otherwise, check them in: | ||
# .python-version | ||
|
||
# pipenv | ||
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. | ||
# However, in case of collaboration, if having platform-specific dependencies or dependencies | ||
# having no cross-platform support, pipenv may install dependencies that don't work, or not | ||
# install all needed dependencies. | ||
#Pipfile.lock | ||
|
||
# poetry | ||
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. | ||
# This is especially recommended for binary packages to ensure reproducibility, and is more | ||
# commonly ignored for libraries. | ||
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control | ||
#poetry.lock | ||
|
||
# pdm | ||
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. | ||
#pdm.lock | ||
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it | ||
# in version control. | ||
# https://pdm.fming.dev/#use-with-ide | ||
.pdm.toml | ||
|
||
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm | ||
__pypackages__/ | ||
|
||
# Celery stuff | ||
celerybeat-schedule | ||
celerybeat.pid | ||
|
||
# SageMath parsed files | ||
*.sage.py | ||
|
||
# Environments | ||
.env | ||
.venv | ||
env/ | ||
venv/ | ||
ENV/ | ||
env.bak/ | ||
venv.bak/ | ||
|
||
# Spyder project settings | ||
.spyderproject | ||
.spyproject | ||
|
||
# Rope project settings | ||
.ropeproject | ||
|
||
# mkdocs documentation | ||
/site | ||
|
||
# mypy | ||
.mypy_cache/ | ||
.dmypy.json | ||
dmypy.json | ||
|
||
# Pyre type checker | ||
.pyre/ | ||
|
||
# pytype static type analyzer | ||
.pytype/ | ||
|
||
# Cython debug symbols | ||
cython_debug/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
!.gitignore |
228 changes: 228 additions & 0 deletions
228
integrations/async-gitlab/.port/resources/blueprints.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,228 @@ | ||
[ | ||
{ | ||
"identifier": "gitlabGroup", | ||
"title": "Group", | ||
"icon": "GitLab", | ||
"schema": { | ||
"properties": { | ||
"visibility": { | ||
"icon": "Lock", | ||
"title": "Visibility", | ||
"type": "string", | ||
"enum": [ | ||
"public", | ||
"internal", | ||
"private" | ||
], | ||
"enumColors": { | ||
"public": "red", | ||
"internal": "yellow", | ||
"private": "green" | ||
} | ||
}, | ||
"url": { | ||
"title": "URL", | ||
"format": "url", | ||
"type": "string", | ||
"icon": "Link" | ||
}, | ||
"description": { | ||
"title": "Description", | ||
"type": "string", | ||
"icon": "BlankPage" | ||
} | ||
}, | ||
"required": [] | ||
}, | ||
"mirrorProperties": {}, | ||
"calculationProperties": {}, | ||
"aggregationProperties": {}, | ||
"relations": {} | ||
}, | ||
{ | ||
"identifier": "project", | ||
"title": "Project", | ||
"icon": "Microservice", | ||
"schema": { | ||
"properties": { | ||
"url": { | ||
"title": "URL", | ||
"type": "string", | ||
"format": "url" | ||
}, | ||
"readme": { | ||
"title": "README", | ||
"type": "string", | ||
"format": "markdown" | ||
}, | ||
"description": { | ||
"title": "Description", | ||
"type": "string" | ||
}, | ||
"language": { | ||
"title": "Language", | ||
"type": "string" | ||
}, | ||
"namespace": { | ||
"title": "Namespace", | ||
"type": "string" | ||
}, | ||
"fullPath": { | ||
"title": "Full Path", | ||
"type": "string" | ||
}, | ||
"defaultBranch": { | ||
"title": "Default Branch", | ||
"type": "string" | ||
} | ||
}, | ||
"required": [] | ||
}, | ||
"mirrorProperties": {}, | ||
"calculationProperties": {}, | ||
"relations": { | ||
"group": { | ||
"title": "Group", | ||
"target": "gitlabGroup", | ||
"required": true, | ||
"many": false | ||
} | ||
} | ||
}, | ||
{ | ||
"identifier": "gitlabMergeRequest", | ||
"title": "Merge Request", | ||
"icon": "GitVersion", | ||
"schema": { | ||
"properties": { | ||
"creator": { | ||
"title": "Creator", | ||
"type": "string" | ||
}, | ||
"status": { | ||
"title": "Status", | ||
"type": "string", | ||
"enum": [ | ||
"opened", | ||
"closed", | ||
"merged", | ||
"locked" | ||
], | ||
"enumColors": { | ||
"opened": "yellow", | ||
"closed": "red", | ||
"merged": "green", | ||
"locked": "blue" | ||
} | ||
}, | ||
"createdAt": { | ||
"title": "Created At", | ||
"type": "string", | ||
"format": "date-time" | ||
}, | ||
"updatedAt": { | ||
"title": "Updated At", | ||
"type": "string", | ||
"format": "date-time" | ||
}, | ||
"mergedAt": { | ||
"title": "Merged At", | ||
"type": "string", | ||
"format": "date-time" | ||
}, | ||
"link": { | ||
"title": "Link", | ||
"format": "url", | ||
"type": "string" | ||
}, | ||
"reviewers": { | ||
"title": "Reviewers", | ||
"type": "array", | ||
"items": { | ||
"type": "string" | ||
} | ||
} | ||
}, | ||
"required": [] | ||
}, | ||
"mirrorProperties": {}, | ||
"calculationProperties": { | ||
"lead_time_days": { | ||
"title": "Lead time (Days)", | ||
"calculation": "if .properties.status == \"merged\" then ((.properties.mergedAt | sub(\"\\\\.[0-9]+\\\\+00:00$\"; \"Z\") | strptime(\"%Y-%m-%dT%H:%M:%SZ\") | mktime) - (.properties.createdAt | sub(\"\\\\.[0-9]+\\\\+00:00$\"; \"Z\") | strptime(\"%Y-%m-%dT%H:%M:%SZ\") | mktime)) / 86400 | tonumber else null end", | ||
"type": "number" | ||
} | ||
}, | ||
"aggregationProperties": {}, | ||
"relations": { | ||
"project": { | ||
"title": "Project", | ||
"target": "project", | ||
"required": false, | ||
"many": false | ||
} | ||
} | ||
}, | ||
{ | ||
"identifier": "gitlabIssue", | ||
"title": "Issue", | ||
"icon": "GitLab", | ||
"schema": { | ||
"properties": { | ||
"link": { | ||
"title": "Link", | ||
"type": "string", | ||
"format": "url" | ||
}, | ||
"description": { | ||
"title": "Description", | ||
"type": "string", | ||
"format": "markdown" | ||
}, | ||
"createdAt": { | ||
"title": "Created At", | ||
"type": "string", | ||
"format": "date-time" | ||
}, | ||
"closedAt": { | ||
"title": "Closed At", | ||
"type": "string", | ||
"format": "date-time" | ||
}, | ||
"updatedAt": { | ||
"title": "Updated At", | ||
"type": "string", | ||
"format": "date-time" | ||
}, | ||
"creator": { | ||
"title": "Creator", | ||
"type": "string" | ||
}, | ||
"status": { | ||
"title": "Status", | ||
"type": "string", | ||
"enum": ["opened", "closed"], | ||
"enumColors": { | ||
"opened": "green", | ||
"closed": "purple" | ||
} | ||
}, | ||
"labels": { | ||
"title": "Labels", | ||
"type": "array", | ||
"items": { | ||
"type": "string" | ||
} | ||
} | ||
} | ||
}, | ||
"relations": { | ||
"project": { | ||
"target": "project", | ||
"title": "Project", | ||
"required": true, | ||
"many": false | ||
} | ||
} | ||
} | ||
] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why is lead time a calculation property here, The calculation should be done in the mapping configuration and default to null if the pr hasn't been merged. we have access to both mergedAt and createdAt at the time of ingesting. this would help the reuse of leadTime. calculation properties are done on the fly. there are instances that we need to use leadTime.