Reuse badger disk storage for other purposes

[adamwasila]

My use case is that I have a component that uses OPA in library mode (as embedded component). Coincidentally my project storage is also backed up by BadgerDB. Therefore it seems quite natural and simplifying to use just one instance of BadgerDB on the disk.

So there are two updates in my approach: it is possible to provice BadgerDB DB instance that was created externally in other parts of the code and it is possible to add common, global prefix for all keys inserted by OPA storage which gives effectively a namespace so conflicts with other parts of the code can be avoided.

Do you think such addition would be valuable for general use? Currently I'm hacking over copy of disk storage implementation (copied directly from the project into my codebase) but I could easily provide Pull Request with necessary changes based on my work.

[srenatus]

Honestly, this is not something we've thought about much.

How are you using OPA? Through the SDK? Or via the rego packages, or something even lower-level?

I think we'd hesitate introducing any sort of "bring your own badgerdb" feature any time soon. It's still early days for the disk storage in OPA, and I'd be cautious with this -- we don't know how to deal with migrations properly, for example, and having to consider other users of the same database instance would definitely complicate things. It's easier to deal with a database you own completely.

But all that said, you could write your own storage implementation that would use your shared badgerdb instance, with the prefixing dealt with as you see fit.

Btw, do you know other badger-embedding projects that offer this kind of functionality? 🤔

[adamwasila]

Thanks for your quick answer. I'm using rego packages and embed OPA inside my application which is kind of specialized database that uses badgerdb as storage layer. OPA is expected to provide user-configurable authorization layer. And no, I don't I know any project example that would use unmanaged database handle in a way I proposed here.

Writing own storage implementation seems to be the best choice but interface for it (Storage) makes it quite big, scary and tiresome task.
I used Casbin before and integrated it into few projects and while I recognize OPA to be more advanced and mature I still miss how easy was to adapt data source and how many adapters community has provided. For your reference: https://casbin.org/docs/en/adapters

Therefore I would like to move this problem into different direction now: is there a space for discussion to refactor/simplify disk storage into something casbin have (trivial 2-3 methods interface but relatively low-level)?

[srenatus]

Writing own storage implementation seems to be the best choice but interface for it (Storage) makes it quite big, scary and tiresome task.

Oh I didn't think you should start from scratch here. I thought you could take the existing code, copy it, change the initialization of the badger db to do as you see fit. Your changes would reflect what would be necessary to share the database. Also, frankly, the maintenance burden would be on you: if the "official" (in-tree) disk storage backend would start needing to do some migration logic, etc, you'd have to deal with doing the same in your setup.

Therefore I would like to move this problem into different direction now: is there a space for discussion to refactor/simplify disk storage into something casbin have (trivial 2-3 methods interface but relatively low-level)?

I'm afraid casbin doesn't have anything like OPA's data tree structure, does it? I'm no expert there. Would you have a suggestion on how OPA's storage layer could be done instead? 🤔

[adamwasila]

So basically you confirm what I was afraid of to write and maintain custom implementation. I made a quick prototype already and it seems to work fine, code is also packaged nicely and self-contained, no problems here whatsoever.

As for direct comparison - yes, casbin adapters are quite trivial. Policies are just list of fixed length arrays (closest model would be a CSV file or sql table) that is read/saved all at once (as if whole file would be read). Therefore it just needs two methods to handle persistence.
OPA has hierarchical data model that needs to be updated/read in user defined chunks - this is main source of complexity, right?

Maybe I'll hold with any suggestions until I study implementation more. Right now I have few ideas but can't say for sure which one have a chance to work with current codebase.

By the way, you've mentioned migrations twice already. Is there any material/discussions that I could read about this subject? It sounds interesting and possibly conflicting with my targets a bit so it's worth to learn about, I guess.

[srenatus]

Frankly, I'm a little conflicted: For one thing, the persistent storage option in OPA is called disk storage (and not Badger) for a reason -- whether it's badger or sqlite should be considered an implementation detail that most users wouldn't care about. When I was talking about "migrations", what I really meant is that right now, whatever happens underneath the disk storage feature is self-contained: for the sake of the argument, an upgrade to the schema used could drop all tables and recreate them. If the underlying badger store was shared, dropping all keys perhaps wouldn't be an option. I've got similar (and similarly vague) concerns about things like backup and restore -- much stuff becomes more complicated when you cannot assume that you're the only user of a data store.

On the other hand, if you look at the code for the badger storage, it's got all the wrapping to project OPA's data storage (hierarchical, static json data, dynamic rego policies) needs down to key-value pairs. So it would probably be possible to refactor the code such that new backends could more easily be plugged in, if only they provide Set and Delete for key-val pairs.

I'm afraid the second idea isn't on our roadmap right now.

[adamwasila]

Understood, though I can't agree that users will not care about storage laoyout. For any serious use case, to have control over data persistence will be crucial thing. It's plain boring ops stuff - to have it distributed (partitioned for performance/scalability and replicated for availability), to have proper backup for disaster recovery, to completely understood data you store (not to be locked in by current solution and to troubleshoot inevitable bugs and problems).

But ok, enough ranting. I have enough data now and can proceed with my project now and if it will lead me into something usefull and interesting I'll share my results. Thanks for your comments, it is valuable insight and helps me to move forward.