chore: raw workspace configs

[achettyiitr]

Description

• Current Implementation: if the poller is set, it makes an API call to get the workspace config, stores it in the cache, and informs the client through the notification. The client then gets the configs from the cache.
• The initial idea was to implement WithoutCache and WithoutParsing.
• WithoutCache:
  • If the poller is set, then it makes an API call to get the workspace config, discards the response because of the no-cache policy, and informs the client through the notification. The client then makes the API call to fetch the response since it's not present in the cache. Since the response fetched by the poller is not being used, no need to have the poller. It can also send the data through the Notification, but in that case, it would not work for incremental updates because the client needs to store the relevant stuff.
  • If the poller is not set. then there is nothing in the cache and the client will need to fetch the response.
  • In general, it would not work for incremental updates because the client needs to store the relevant stuff and merge the configs
• WithoutParsing:
  • It was getting complicated to support both the raw JSON and modelv2.WorkspaceConfigs within the cache. So introduced a new API at an SDK level GetRawWorkspaceConfigs to make the API call and get the config.
  • In general, it would not work for incremental updates because the client needs to store the relevant stuff and merge the configs

Linear Ticket

• Resolves PIPE-751

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[fracasula]

I understand the need for GetRawWorkspaceConfigs and I'm cool with that.
I think it makes sense for a client to have all three methods.

The problem though is that we're still using CPU cycles to parse the response into a set of structs that we don't really need and we're also keeping them in memory (see poller).

I think we need a way to either disable the poller altogether, or have the poller just poll and not cache (which also makes sense).

[achettyiitr]

I understand the need for GetRawWorkspaceConfigs and I'm cool with that. I think it makes sense for a client to have all three methods.

The problem though is that we're still using CPU cycles to parse the response into a set of structs that we don't really need and we're also keeping them in memory (see poller).

I think we need a way to either disable the poller altogether, or have the poller just poll and not cache (which also makes sense).

If we set the interval to 0, the poller is disabled.

rudder-cp-sdk/sdk.go

Line 114 in 9d8fc91

if cp.pollingInterval == 0 {

[fracasula]

If we set the interval to 0, the poller is disabled.

rudder-cp-sdk/sdk.go

Line 114 in 9d8fc91

if cp.pollingInterval == 0 {

Cool. Why would the poller cache though? They are two separate concepts. Now we would have to poll ourselves.
Maybe we can figure out a way to poll and not cache.

Let's discuss tomorrow in the call, maybe we'll find a nice way 👍