Why is it the OPAL client and not the server which fetches data

[blacksails]

I was looking at the OPAL architecture, and I was wondering why it's the responsibility of the OPAL client to fetch data when you have a server component which in a way also fetches data (if you think of the policy and data.json, in git as data). My concern is that the OPAL clients will each have to pull the same data from the data sources.

In my mind it would make more sense architecturally to have the server component pull the data once, and bundle it with the policy from git. Could you enlighten me on this design decision?

[orweis]

Hi @blacksails :)
I appreciate you coming in and asking :). It is, I think, one of the more subtle and interesting parts of the architecture.

The answer is simple: privacy/security and realtime performance/scale

1. This allows the server and client to run in seperate clusters, VPCs, and even clouds/networks - and the client can still have direct access to data-sources right beside it (such as the database of the app its serving authorization for). In Permit.io for example this allows the service to orchestrate everything from the cloud without being exposed at all to the customer's data.

2. OPAL is designed to be realtime in its delivery of updates (i.e. we want clients to know about changes as soon as possible, even before they are aware of the content of the change itself (this is a first step in a bigger plan for full consistency and solving the new-enemy problem). You can already see the benefit of this with the transaction and policy health-check features ; where OPAL allows OPA (and other parts of the application) to be aware of an incomplete data picture as part of their decision making.
   To add to that in general websocket channels are less suited for passing large amounts of data.

3. This allows for more flexible work load balancing and QoS, consider having multiple sources of data with different levels of importance, frequency of change , and/or volume. Instead of always having all the clients get their data from one point fighting for resource/bandwidth allocation (OPAL-server ) potentially have some data volumes starving out others... you can send different clients to different sources and apply smart prioritization.

4. You can still have it your way, by simply having the OPAL server point at itself (another route that you add to its FastAPI, or another service next to it) - you can still poll the data you want in the server (even by reusing the DataFetchers module, or even running a client next to the server), and simply point the clients to fetch it from there (as I wrote in 2. - Anyway you wouldn't want to pass large volumes of data on the pub/sub websocket).
   Let me know if you'd like some help in setting this up.

Hope this sheds some more light and hope it also makes some sense 😅 😇

[blacksails]

Thank you for the thorough reply. I had a look at the Styra DAS as well, but will probably test OPAL this week.

[orweis]

Cool.
Happy to help with anything else (also on our Slack).

BTW - While OPAL has some small overlap with DAS, they are mostly complimentary.
You can see some discussions on our Slack.

[blacksails]

Awesome, will have a look at that as well.