Add GitLab functionalities #1098

alessandrozago · 2024-08-07T16:54:01Z

add code to handle Gitlab repository, add axios to project packages, add env variables for Gitlab

alessandrozago · 2024-08-08T08:29:07Z

Here the list of file changed or added in order to add the ability to use GitLab instead of GitHub as store repo. For each file, a simple description is given to describe its purpose within the application and the changes made for GitLab:

package.json (and package-lock.json)
added axios library and related dependencies for gitlab API calls.
.env.example
It's the file that contains the tokens for the repositories.
In addition to the original OTA_ENGINE_GITHUB_TOKEN variable, 2 new variables were added, OTA_ENGINE_GITLAB_TOKEN (used for the declarations-related repository) and OTA_ENGINE_GITLAB_RELEASES_TOKEN (used for the versions-related repository on which to save datasets in ZIP file format).
config/default.json
The file was modified by adding some new elements within the JSON structure of the file. In the "reporter" element, the same structure as "githubIssues" is expected for gitlab, but with the name "gitlabIssues". In the "declaration" field the name of the repository should be given in the form "<owner_name>/<project_name>"
engine/src/index.js
The file is the module related to the declarations tracking procedure (performed with the "npx ota track" command).
It has been modified by adding checks for the presence of GitHub and GitLab tokens. In either case, the relevant Reporter (for GitHub) or ReporterGitLab (for GitLab) classes are instantiated.
engine/src/reporterGitlab/gitlab.js
This file contains the definition of the Gitlab class and all its related methods for operating on a repository: getRepositoryLabels, createLabel, createIssues, setIssueLabels, openIssue, closeIssue, getIssue, addCommentToIssue, closeIssueWithCommentIfExists, and createOrUpdateIssue. It contains some settings such as the url where to reach the API of the GitLab repo you want to use (for example: https://gitlab.com/api/v4 )
engine/src/reporterGitlab/index.js
This file contains the definition of the Reporter class related to the use of GitLab. Within it, the related methods of the GitLab class described above are invoked.
engine/src/reporterGitlab/labels.json
This file contains, via a JSON structure, the list of labels that must be present on the repository in order to properly create issues. In case no labels are present in the repository thanks to this file they are automatically created during the first run.
engine/src/reporterGitlab/labels.test.js
It's a Javascript file used to test the file "labels.json" to verify that all the info inserted complains with the GitLab constraints.
engine/scripts/dataset/index.js
The file is the module related to the creation of releases and datasets (performed with the "npx ota dataset" command). It has been modified by adding checks for the presence of GitHub and GitLab tokens.
In either case, the relevant publishRelease (for GitHub) or publishReleaseGitLab (for GitLab) classes are instantiated.
engine/scripts/dataset/assets/README.templateGitLab.js
This file contains the template with all the texts and links that are used into every releases created during the "dataset" process.
engine/scripts/dataset/publishGitLab/index.js
The file contains the function to publish a release on a GitLab repository. This function creates a release and the zip file that is attached as asset link to the release.

MattiSG · 2024-08-08T09:05:34Z

Hello,
This pull request seems to be a duplicate of #1095, without handling the requested changes. In particular, #1095 (comment) and #1098 (comment) are copy-pasted.
As explained in #1095 (review), squashing commits is not welcome. Please have a look at the CONTRIBUTING guidelines.

@fabianospinelli @alessandrozago could you please clarify what is the relationship between #1095 and #1098, and close the potentially obsolete PR? The European Commission that employs you certainly provides enough resources to minimise the burden on maintainers 🙂

In general, introducing yourselves and your intentions is a welcome first step in open source collaboration before requesting maintainers to allocate time to review your work. You can find a good introduction to open source collaboration on the Open Source Guide, in particular for the case at hand in section 5, “How to submit a contribution”:

Before you open an issue or pull request, check the project’s contributing docs (usually a file called CONTRIBUTING, or in the README), to see whether you need to include anything specific. For example, they may ask that you follow a template, or require that you use tests. If you want to make a substantial contribution, open an issue to ask before working on it.

I am happy to have a video call with your team if useful.

MattiSG

Since merging code into the codebase entails long-term maintenance commitment, we will also need automated tests to pass before considering review.
If you'd like to speed up the review process for this GitLab integration to be available within your organisation, please consider sponsoring!

MattiSG · 2024-08-08T09:07:51Z

config/default.json

@@ -59,6 +59,7 @@
    "dataset": {
      "title": "sandbox",
      "versionsRepositoryURL": "https://github.com/OpenTermsArchive/sandbox",
+      "versionsRepositoryURLGitLab": "https://gitlab.com/p2b/contrib-versions",


The account p2b on GitLab is reserved. Do you own it? If not, it does not seem appropriate to send notifications to that account by default.

The url was only used as an example, but it was our mistake since we did not realize it was pointing to an existing account. This was removed, and replaced with an obvious fake url as an example. It will be available when the pull request will be updated.

MattiSG · 2024-08-08T09:09:04Z

package.json

@@ -100,7 +100,8 @@
    "swagger-jsdoc": "^6.2.8",
    "swagger-ui-express": "^5.0.0",
    "winston": "^3.3.3",
-    "winston-mail": "^2.0.0"
+    "winston-mail": "^2.0.0",
+    "axios": "^1.7.2"


Adding a dependency is not a lightweight change as it introduces security risks and maintenance burden. It seems this dependency is used only for 3 HTTP calls. Please use the native http module or one of the existing dependencies.
Please also use npm install --save to maintain alphabetical order and avoid future churn.

This is solved and will be available when the pull request will be updated. Axios was removed and was replaced with node-fetch (already used in the project).

MattiSG · 2024-08-08T09:09:47Z

scripts/dataset/publishGitLab/index.js

+
+import config from 'config';
+import dotenv from 'dotenv';
+//import { Octokit } from 'octokit';


Do not keep commented-out lines, please erase them.

This is solved (along with other similar cases) and will be available when the pull request will be updated

alessandrozago · 2024-08-12T15:34:04Z

Hi @MattiSG , thank you for the feedback and the resources provided.
As discussed, the obsolete pull request #1095 was closed, I opened an issue: #1099 and I am working on fixing the issues mentioned in the comments.
We expect to complete the issues for the first weeks of September. I will update the pull request when ready, or if there is any other update.

MattiSG · 2024-08-13T09:11:10Z

src/reporterGitlab/labels.test.js

+        expect(descriptionLength).to.be.lessThan(GITLAB_LABEL_DESCRIPTION_MAX_LENGTH);
+      });
+
+      it('complies with the GitHub constraints for color', () => {


This still mentions GitHub when it's related to GitLab 😉

This is solved and will be available when the pull request will be updated.

MattiSG · 2024-08-13T09:12:12Z

src/reporterGitlab/labels.json

Do you really need to duplicate this file? It seems to be a complete duplicate of the GitHub reporter labels. Why not simply import the same file? 🙂

The color format is different; although it is just the "#" character at the beginning of the color code.
We could find some way to use the same test file and simply add the "#" character via code for GitLab, but I do not think this is a good solution.
Considering GitHub and GitLab use different formats, it's also possible they might change in the future. To us, it makes sense to have them separate.
Please let us know if you prefer to unify the tests anyway. In this case, it might also make sense to move the "labels.json" in a more generic directory/path.

MattiSG · 2024-08-13T09:21:46Z

scripts/dataset/assets/README.templateGitLab.js

This seems to be a complete duplicate from the GitHub template. Duplicating this file means that they are more likely to not be updated properly. Is there a reason for not simply importing the existing template? 🙂

The only change in that file is, in fact, just a variable name (compared to the GitHub counterpart).
To use a single template, we could:

pass that variable as an input for the template and modify the code in the Release generation to handle both the GitHub and GitLab parts

use only one variable in the configuration file for "versionsRepositoryURL"; this will make the publishing part not consistent with the reporting part, where GitHub and GitLab use different configurations (reporter.githubIssues and reporter.gitlabIssues).

Please let us know if there is a preferred solution, otherwise we could proceed with n°1.

add test file for GitLab, fix formatting in files, refactor GitLab code to remove axios library

remove the GitLab releases repository with a new one used as an example

remove duplicate readme for gitlab and handle the new input parameter

Fix the release zip workflow using GitLab packages to upload the file

src/reporterGitlab/gitlab.js

aligned format and minor wording fixes

Changes were handled

Ndpnt · 2024-11-19T08:42:33Z

Integrated by #1116

Add Gtilab functionalities

fd97234

add code to handle Gitlab repository, add axios to project packages, add env variables for Gitlab

MattiSG requested changes Aug 8, 2024

View reviewed changes

alessandrozago mentioned this pull request Aug 12, 2024

Add support for GitLab repositories #1099

Closed

MattiSG reviewed Aug 13, 2024

View reviewed changes

MattiSG changed the title ~~Add Gtilab functionalities~~ Add GitLab functionalities Aug 13, 2024

alessandrozago and others added 9 commits October 21, 2024 10:15

Fix new gitlab files using eslint

7a94ea5

Add tests for GitLab class, remove axios and format fixes

e540a1b

add test file for GitLab, fix formatting in files, refactor GitLab code to remove axios library

remove existing GitLab repository in the config

ec13bb0

remove the GitLab releases repository with a new one used as an example

add EUPL-1.2 copyright

12005fb

use only a single template for README

5fcdc37

remove duplicate readme for gitlab and handle the new input parameter

Fix release zip upload not working

aa9ea5b

Fix the release zip workflow using GitLab packages to upload the file

Update changelog

1d8a3bd

Merge branch 'main' into main

c10aa7c

Fix typo on changelog

c7956af

MattiSG linked an issue Oct 28, 2024 that may be closed by this pull request

Add support for GitLab repositories #1099

Closed

MattiSG previously requested changes Oct 29, 2024

View reviewed changes

src/reporterGitlab/gitlab.js Outdated Show resolved Hide resolved

alessandrozago and others added 3 commits October 29, 2024 11:12

Remove explicit EUPL-1.2 copyright in files

dcd4a03

Align logging messages format with latest releases

5045281

aligned format and minor wording fixes

Merge branch 'main' into main

c99ce82

Merge branch 'main' into main

b20423c

Ndpnt mentioned this pull request Nov 6, 2024

Refactor GitLab integration #1116

Merged

Ndpnt closed this Nov 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GitLab functionalities #1098

Add GitLab functionalities #1098

alessandrozago commented Aug 7, 2024

alessandrozago commented Aug 8, 2024

MattiSG commented Aug 8, 2024

MattiSG left a comment

MattiSG Aug 8, 2024

alessandrozago Sep 12, 2024

MattiSG Aug 8, 2024

alessandrozago Sep 12, 2024

MattiSG Aug 8, 2024

alessandrozago Sep 12, 2024

alessandrozago commented Aug 12, 2024

MattiSG Aug 13, 2024

alessandrozago Sep 12, 2024

MattiSG Aug 13, 2024

alessandrozago Sep 12, 2024

MattiSG Aug 13, 2024

alessandrozago Sep 12, 2024

Ndpnt commented Nov 19, 2024

Add GitLab functionalities #1098

Add GitLab functionalities #1098

Conversation

alessandrozago commented Aug 7, 2024

alessandrozago commented Aug 8, 2024

MattiSG commented Aug 8, 2024

MattiSG left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alessandrozago commented Aug 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ndpnt commented Nov 19, 2024