RSD v2 release

[dmijatovic]

RSD 2 release

This PR collects all features to be included in v2 release.
include BREAKING CHANGES in the last commit. - DONE in #927

Closes #655
Closes #789
Closes #792
Closes #803
Closes #806
Closes #823
Closes #824
Closes #838
Closes #888
Closes #896
Closes #916
Closes #913
Closes #924
Closes #922
Closes #932
Closes #933
Closes #934
Closes #935
Closes #936
Closes #937
Closes #938
Closes #955
Closes #976
Closes #979
Closes #980
Closes #982

Changes proposed in this pull request:

• See separate issues 

Conclusion

The load tests show that RSD v2 server capacity dropped significantly compared to v1. The additional features introduced in v2 require more resources. However, the current capacity of RSD hardware is sufficient to offer additional v2 features under the current traffic. After implementing few optimizations the page load of v2 is comparable to v1 when analysed using Lighthouse.

RSD capacity difference RSD-v1.26.0 vs RSD-v2.0.0

In the K6 load test below, performed on the same hardware, we see that achieving same througoutput (100 users) with v2 is not possible. However, our current hardware CPU capacity on RSD VM seem sufficient. We utilize <10% of the hardware CPU capacity on average day. Based on this test results I expect hardware's CPU utilization to increase to ~30%. For the partners is it important to check their current hardware utilization and take into a count required increase in capacity when upgrading to v2. The data traffic is estimated to be ~150% of v1. Partners are adviced to test v2 performance before commiting to upgrade.

Performance improvements

Based on performed load tests on pages and the api we identified few improvement points. These are implemented in #973. Although the performance of specific api calls is improved significantly the total server capacity remains significantly lower compared to v1 as shown in the image below.

Lighthouse audit improvements

In addition to api improvements we implemented lighthouse audit suggestions. This lead to significant improvement in the lighthouse scores especially on software overview page. The project overview page, for example, have slightly better score in v2 than in v1 (see image).

Software overview

Project overview

How to test:

• make start to build app
• test all functionalities specified in the separate issues

PR Checklist:

• Increase version numbers in docker-compose.yml
• Link to a GitHub issue
• Update documentation
• Tests

[sonarcloud]

SonarCloud Quality Gate failed.   

4 Bugs
0 Vulnerabilities
9 Security Hotspots
59 Code Smells

No Coverage information
3.7% Duplication

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[sonarcloud]

[authentication] Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[sonarcloud]

[scrapers] Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[dmijatovic]

@cmeessen @fembau @jmaassen @ewan-escience @ctwhome I rebased this PR onto the last release of v1.27.0. Can you test this version of PR as the first release candidate of v2.0.0, please?

I would like to merge this PR into main as soon as possible and set it on our dev server for further testing. Before official release we want to invite some "power users" to have a look at this version before official release. I expect official release of v2 to be ~1-2 weeks from now if no major errors are discovered.

[sonarcloud]

[rsd-database] Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
30.2% Duplication

[sonarcloud]

[rsd-frontend] SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
20 Code Smells

54.0% Coverage
3.5% Duplication

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[dmijatovic]

@cmeessen Can you perform final test of this PR on your side. NOTE! We made one minor DB update on rpc in 100-xxx script.

-- programming language filter for software
-- used by software_overview func
CREATE FUNCTION prog_lang_filter_for_software() RETURNS TABLE (
	software UUID,
	prog_lang TEXT[]
) LANGUAGE plpgsql STABLE AS
$$
BEGIN
	RETURN QUERY
	SELECT
		repository_url.software,
		(SELECT
			ARRAY_AGG(p_lang ORDER BY repository_url.languages -> p_lang DESC)
		FROM
			JSONB_OBJECT_KEYS(repository_url.languages) p_lang
		) AS "prog_lang"
	FROM
		repository_url
	;
END
$$;

If you have any questions let me know :-).

[cmeessen]

Great job, I am looking forward to seeing this live!