Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Integration][async-gitlab] - GitLab integration with Async support #1121

Closed
wants to merge 26 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
8d0f592
feat(async-gitlab): adds blue print and spec config
victor-devv Nov 4, 2024
6aa7665
feat(async-gitlab): adds github client and utils file
victor-devv Nov 4, 2024
6a3c1d5
feat(async-gitlab): adds groups, projects, MRs and issue re-syncs
victor-devv Nov 4, 2024
f58e65a
feat(async-gitlab): completes resync functions for all kinds
victor-devv Nov 5, 2024
1851442
feat(async-gitlab): refactors code and adds webhook integration
victor-devv Nov 5, 2024
cf36947
chore(async-gitlab): cleanup
victor-devv Nov 5, 2024
ffeb379
ref(async-gitlab): adds spec defaults, adds test cases, shrinks clien…
victor-devv Nov 7, 2024
00d253a
chore(async-gitlab): cleanup
victor-devv Nov 7, 2024
7c960a4
ref(async-gitlab): uses toString flag to convert numeric resource ids
victor-devv Nov 8, 2024
7cb7217
Merge branch 'main' into feat/async-gitlab
victor-devv Nov 8, 2024
e1807a8
ref(async-gitlab): uses single resync func for resources
victor-devv Nov 11, 2024
7fbf950
ref(async-gitlab): shrinks webhook handler
victor-devv Nov 11, 2024
6dd6ecc
feat(async-gitlab): sets up system hooks on startup
victor-devv Nov 11, 2024
d30b7f4
chore(async-gitlab): cleanup
victor-devv Nov 11, 2024
392e4fc
chore(async-gitlab): cleanup
victor-devv Nov 15, 2024
a45102b
ref(async-gitlab): disposes api ver method
victor-devv Nov 15, 2024
29282d6
chore(async-gitlab): cleanup
victor-devv Nov 15, 2024
fee1291
chore(async-gitlab): cleanup
victor-devv Nov 18, 2024
5f58503
feat(async-gitlab): adds support for both system and group webhooks
victor-devv Nov 18, 2024
3eee70a
chore(async-gitlab): cleanup
victor-devv Nov 18, 2024
cf622e6
chore(async-gitlab): cleanup
victor-devv Nov 18, 2024
6375f16
ref(sync-gitlab): cleanup
victor-devv Nov 21, 2024
5669f87
ref(async-gitlab): adds checks for app host before webhook setup
victor-devv Nov 21, 2024
0c1ec05
feat(async-gitlab): uses gitlab rate limiter headers to set async lim…
victor-devv Nov 21, 2024
86f683c
chore(async-gitlab): cleanup
victor-devv Nov 22, 2024
f04eb39
chore: lints code
victor-devv Nov 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file added integrations/__init__.py
Empty file.
6 changes: 6 additions & 0 deletions integrations/async-gitlab/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
OCEAN__PORT__CLIENT_ID="<port-client-id>"
OCEAN__PORT__CLIENT_SECRET="<port-client-secret>"
OCEAN__INTEGRATION__CONFIG__GITLAB_HOST="<gitlab_host_url>"
OCEAN__INTEGRATION__CONFIG__ACCESS_TOKEN="<gitlab_access_token>"
OCEAN__EVENT_LISTENER={"type": "POLLING"}
OCEAN__INTEGRATION="async-gitlab"
153 changes: 153 additions & 0 deletions integrations/async-gitlab/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/
1 change: 1 addition & 0 deletions integrations/async-gitlab/.port/resources/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
!.gitignore
228 changes: 228 additions & 0 deletions integrations/async-gitlab/.port/resources/blueprints.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,228 @@
[
{
"identifier": "gitlabGroup",
"title": "Group",
"icon": "GitLab",
"schema": {
"properties": {
"visibility": {
"icon": "Lock",
"title": "Visibility",
"type": "string",
"enum": [
"public",
"internal",
"private"
],
"enumColors": {
"public": "red",
"internal": "yellow",
"private": "green"
}
},
"url": {
"title": "URL",
"format": "url",
"type": "string",
"icon": "Link"
},
"description": {
"title": "Description",
"type": "string",
"icon": "BlankPage"
}
},
"required": []
},
"mirrorProperties": {},
"calculationProperties": {},
"aggregationProperties": {},
"relations": {}
},
{
"identifier": "project",
"title": "Project",
"icon": "Microservice",
"schema": {
"properties": {
"url": {
"title": "URL",
"type": "string",
"format": "url"
},
"readme": {
"title": "README",
"type": "string",
"format": "markdown"
},
"description": {
"title": "Description",
"type": "string"
},
"language": {
"title": "Language",
"type": "string"
},
"namespace": {
"title": "Namespace",
"type": "string"
},
"fullPath": {
"title": "Full Path",
"type": "string"
},
"defaultBranch": {
"title": "Default Branch",
"type": "string"
}
},
"required": []
},
"mirrorProperties": {},
"calculationProperties": {},
"relations": {
"group": {
"title": "Group",
"target": "gitlabGroup",
"required": true,
"many": false
}
}
},
{
"identifier": "gitlabMergeRequest",
"title": "Merge Request",
"icon": "GitVersion",
"schema": {
"properties": {
"creator": {
"title": "Creator",
"type": "string"
},
"status": {
"title": "Status",
"type": "string",
"enum": [
"opened",
"closed",
"merged",
"locked"
],
"enumColors": {
"opened": "yellow",
"closed": "red",
"merged": "green",
"locked": "blue"
}
},
"createdAt": {
"title": "Created At",
"type": "string",
"format": "date-time"
},
"updatedAt": {
"title": "Updated At",
"type": "string",
"format": "date-time"
},
"mergedAt": {
"title": "Merged At",
"type": "string",
"format": "date-time"
},
"link": {
"title": "Link",
"format": "url",
"type": "string"
},
"reviewers": {
"title": "Reviewers",
"type": "array",
"items": {
"type": "string"
}
}
},
"required": []
},
"mirrorProperties": {},
"calculationProperties": {
"lead_time_days": {
"title": "Lead time (Days)",
"calculation": "if .properties.status == \"merged\" then ((.properties.mergedAt | sub(\"\\\\.[0-9]+\\\\+00:00$\"; \"Z\") | strptime(\"%Y-%m-%dT%H:%M:%SZ\") | mktime) - (.properties.createdAt | sub(\"\\\\.[0-9]+\\\\+00:00$\"; \"Z\") | strptime(\"%Y-%m-%dT%H:%M:%SZ\") | mktime)) / 86400 | tonumber else null end",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is lead time a calculation property here, The calculation should be done in the mapping configuration and default to null if the pr hasn't been merged. we have access to both mergedAt and createdAt at the time of ingesting. this would help the reuse of leadTime. calculation properties are done on the fly. there are instances that we need to use leadTime.

"type": "number"
}
},
"aggregationProperties": {},
"relations": {
"project": {
"title": "Project",
"target": "project",
"required": false,
"many": false
}
}
},
{
"identifier": "gitlabIssue",
"title": "Issue",
"icon": "GitLab",
"schema": {
"properties": {
"link": {
"title": "Link",
"type": "string",
"format": "url"
},
"description": {
"title": "Description",
"type": "string",
"format": "markdown"
},
"createdAt": {
"title": "Created At",
"type": "string",
"format": "date-time"
},
"closedAt": {
"title": "Closed At",
"type": "string",
"format": "date-time"
},
"updatedAt": {
"title": "Updated At",
"type": "string",
"format": "date-time"
},
"creator": {
"title": "Creator",
"type": "string"
},
"status": {
"title": "Status",
"type": "string",
"enum": ["opened", "closed"],
"enumColors": {
"opened": "green",
"closed": "purple"
}
},
"labels": {
"title": "Labels",
"type": "array",
"items": {
"type": "string"
}
}
}
},
"relations": {
"project": {
"target": "project",
"title": "Project",
"required": true,
"many": false
}
}
}
]
Loading