[Integration][GCP] Added Rate Limiting for ProjectsV3GetRequestsPerMinutePerProject Quota #1304

oiadebayo · 2025-01-08T10:23:58Z

Description

What

Implemented rate limiting for the ProjectsV3GetRequestsPerMinutePerProject quota.
Adjusted real-time event handling to respect the newly implemented rate limits.

Why

To address issues reported where real-time event handling exceeded GCP quota limits, resulting in 429 Quota Exceeded errors.
Improve system reliability and data consistency by adhering to quota limits.

How

Introduced a rate limiting mechanism for the ProjectsV3GetRequestsPerMinutePerProject quota using the existing rate limiter
Added support in the resolve_request_controllers method to fetch and apply this specific quota.
Integrated the rate limiter into real-time event processing methods, particularly for project-related operations.

Type of change

Please leave one option from the following and delete the rest:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
New Integration (non-breaking change which adds a new integration)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Non-breaking change (fix of existing functionality that will not change current behavior)
Documentation (added/updated documentation)

All tests should be run against the port production environment(using a testing org).

Core testing checklist

Integration able to create all default resources from scratch
Resync finishes successfully
Resync able to create entities
Resync able to update entities
Resync able to detect and delete entities
Scheduled resync able to abort existing resync and start a new one
Tested with at least 2 integrations from scratch
Tested with Kafka and Polling event listeners
Tested deletion of entities that don't pass the selector

Integration testing checklist

Integration able to create all default resources from scratch
Resync able to create entities
Resync able to update entities
Resync able to detect and delete entities
Resync finishes successfully
If new resource kind is added or updated in the integration, add example raw data, mapping and expected result to the examples folder in the integration directory.
If resource kind is updated, run the integration with the example data and check if the expected result is achieved
If new resource kind is added or updated, validate that live-events for that resource are working as expected
Docs PR link here

Preflight checklist

Handled rate limiting
Handled pagination
Implemented the code in async
Support Multi account

Screenshots

Include screenshots from your environment showing how the resources of the integration will look.

API Documentation

Provide links to the API documentation used for this integration.

…cover-projects-v-3-get-requests-per-minute-per-project

mk-armah

great work... 👏🏽 left some comments

mk-armah · 2025-01-13T09:19:59Z

integrations/gcp/gcp_core/search/resource_searches.py

@@ -311,22 +327,27 @@ async def feed_event_to_resource(
    config: Optional[ProtoConfig] = None,
 ) -> RAW_ITEM:
    resource = None
+    live_event_projects_rate_limiter, _ = await resolve_request_controllers(
+        AssetTypesWithSpecialHandling.PROJECT, method="get"


I'm not sure this is scalable enough, there are other quota's under the same tree that use "get" and "search" prefix, see if you can pass the quota id itself in there in place of method, and rename "method" to "quota_id"

mk-armah · 2025-01-13T09:22:04Z

integrations/gcp/gcp_core/utils.py

@@ -181,10 +187,36 @@ def get_service_account_project_id() -> str:


 async def get_quotas_for_project(
-    project_id: str, kind: str
+    project_id: str, kind: str, **kwargs: Any


Suggested change

project_id: str, kind: str, **kwargs: Any

project_id: str, kind: str, quota_id: Optional[str] = None

Lets be more precise here

mk-armah · 2025-01-13T09:22:51Z

integrations/gcp/gcp_core/utils.py

 ) -> Tuple["AsyncLimiter", "BoundedSemaphore"]:
    try:
        match kind:
+            case AssetTypesWithSpecialHandling.PROJECT:
+                method = kwargs.get("method")


Suggested change

method = kwargs.get("method")

mk-armah · 2025-01-13T09:24:00Z

integrations/gcp/gcp_core/utils.py

 ) -> Tuple["AsyncLimiter", "BoundedSemaphore"]:
    try:
        match kind:
+            case AssetTypesWithSpecialHandling.PROJECT:
+                method = kwargs.get("method")
+                if method == "search":


Suggested change

if method == "search":

if quota_id == "apiSearchAllResourcesQpmPerProject":

mk-armah · 2025-01-13T10:07:13Z

integrations/gcp/main.py

@@ -106,7 +109,10 @@ async def resync_organizations(kind: str) -> ASYNC_GENERATOR_RESYNC_TYPE:

 @ocean.on_resync(kind=AssetTypesWithSpecialHandling.PROJECT)
 async def resync_projects(kind: str) -> ASYNC_GENERATOR_RESYNC_TYPE:
-    async for batch in search_all_projects():
+    resync_projects_rate_limiter, _ = await resolve_request_controllers(
+        kind, method="search"


mk-armah · 2025-01-13T10:08:01Z

integrations/gcp/main.py

@@ -62,7 +62,10 @@ async def _resolve_resync_method_for_resource(
        case AssetTypesWithSpecialHandling.ORGANIZATION:
            return search_all_organizations()
        case AssetTypesWithSpecialHandling.PROJECT:
-            return search_all_projects()
+            project_rate_limiter, _ = await resolve_request_controllers(
+                kind, method="search"


…cover-projects-v-3-get-requests-per-minute-per-project

oiadebayo added 2 commits January 8, 2025 11:09

Added ProjectV3GetRequestsPerMinutePerProject quota limit for project

291a4c3

Update resource_searches.py

07a7777

oiadebayo requested a review from a team as a code owner January 8, 2025 10:23

github-actions bot added the size/M label Jan 8, 2025

oiadebayo marked this pull request as draft January 8, 2025 10:28

oiadebayo and others added 4 commits January 9, 2025 10:42

Added quota handling support for search project

ba22e2e

Fix: Lint issues

cba6966

Merge branch 'main' into PORT-12220-bug-extend-gcp-quota-handling-to-…

a8cf8b9

…cover-projects-v-3-get-requests-per-minute-per-project

updated version

f395076

oiadebayo marked this pull request as ready for review January 9, 2025 10:13

oiadebayo and others added 4 commits January 9, 2025 12:08

Updated test case

07400e8

Merge branch 'main' into PORT-12220-bug-extend-gcp-quota-handling-to-…

14fbb0d

…cover-projects-v-3-get-requests-per-minute-per-project

fix: lint

0fa8730

Merge branch 'main' into PORT-12220-bug-extend-gcp-quota-handling-to-…

933b3e6

…cover-projects-v-3-get-requests-per-minute-per-project

mk-armah requested changes Jan 13, 2025

View reviewed changes

oiadebayo and others added 3 commits January 14, 2025 03:34

Merge branch 'main' into PORT-12220-bug-extend-gcp-quota-handling-to-…

d1aa1b8

…cover-projects-v-3-get-requests-per-minute-per-project

Attended to reviewer comments

3bd0f79

Merge branch 'main' into PORT-12220-bug-extend-gcp-quota-handling-to-…

7cecf83

…cover-projects-v-3-get-requests-per-minute-per-project

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Integration][GCP] Added Rate Limiting for ProjectsV3GetRequestsPerMinutePerProject Quota #1304

[Integration][GCP] Added Rate Limiting for ProjectsV3GetRequestsPerMinutePerProject Quota #1304

oiadebayo commented Jan 8, 2025

mk-armah left a comment

mk-armah Jan 13, 2025

mk-armah Jan 13, 2025

mk-armah Jan 13, 2025

mk-armah Jan 13, 2025

mk-armah Jan 13, 2025

mk-armah Jan 13, 2025

mk-armah Jan 13, 2025

	project_id: str, kind: str, **kwargs: Any
	project_id: str, kind: str, quota_id: Optional[str] = None

	if method == "search":
	if quota_id == "apiSearchAllResourcesQpmPerProject":

[Integration][GCP] Added Rate Limiting for ProjectsV3GetRequestsPerMinutePerProject Quota #1304

Are you sure you want to change the base?

[Integration][GCP] Added Rate Limiting for ProjectsV3GetRequestsPerMinutePerProject Quota #1304

Conversation

oiadebayo commented Jan 8, 2025

Description

Type of change

All tests should be run against the port production environment(using a testing org).

Core testing checklist

Integration testing checklist

Preflight checklist

Screenshots

API Documentation

mk-armah left a comment

Choose a reason for hiding this comment

mk-armah Jan 13, 2025

Choose a reason for hiding this comment

mk-armah Jan 13, 2025

Choose a reason for hiding this comment

mk-armah Jan 13, 2025

Choose a reason for hiding this comment

mk-armah Jan 13, 2025

Choose a reason for hiding this comment

mk-armah Jan 13, 2025

Choose a reason for hiding this comment

mk-armah Jan 13, 2025

Choose a reason for hiding this comment

mk-armah Jan 13, 2025

Choose a reason for hiding this comment