Proposal: Ability to use Postgres (+TimescaleDB) instead of kafka, clickhouse and broker

[ilijaNL]

Currently if you want to self host you are required to spin up a lot of infrastructure. I understand this is necessary to have high scale and high availability, however in many cases a self hosted instance does not have such high load and traffic.

My proposal is to replace kafka and clickhouse with postgres and timescaledb extension for usage metrics. This will result in reducing infra by a significant amount

What do you think guys?

I am willing to contribute on this matter.

[kamilkisiela]

Hi, thanks for proposing it <3

Pluggable storage means we (maintainers) won't be able to test it very well, but we will have to fix bugs and support it :)

It will also bring additional maintenance cost:

• separate migration scripts
• extra code complexity (abstractions and two stores that needs to be maintained)
• dedicated CI flow (to test it a bit)
• docker-related files to deal with
• and probably like 5 more things

I'm not against having a pluggable storage, in fact we wanted to do it at the very beginning (but decided not to for the reasons I mentioned).
It's just that I can't ship a lightweight version of Hive that I'm not using anywhere, cannot test it in production environment, but yet respond to people's bug reports and feature requests.

I hope you understand my points ❤️

If you have a good plan on how all of the above points could be solved (and future proof), then I'm happy to help you and I will accept a Pull Request.

[ilijaNL]

Thanks for your reply. You are correct that it will increase the maintenance cost. Especially enforcing same data structure support in the stores can be hard. The question is, is it worth it. Personally I think so since it will simplify the infrastructure and have a lower entry point for new users of the platform. Hosting your own kafka broker and clickhouse is not that simple and can be time consuming and resource heavy. My proposal is the following:

So instead of "pluggable" storage you get "pluggable" architecture. With the example above you don't need to host clickhouse, kafka broker, usage-ingestor and perhaps more?

The only tricky part I see is that you need to have a adapter for retrieving data from timescaledb instead of clickhouse and I am not certain where all that kind of logic lives.

[kamilkisiela]

The pluggable storage concept is still there, as we need to read data from TimescaleDB.
Every time we want to make a change, we now will have to do it twice and there's a big chance we will break the timescaledb version users.
I can accept a Pull Request, with tests and will see how far we can go with it, but I can't promise we will keep it forever.
Once the maintenance burden is too big and none of contributors are willing to help, we might drop supporting it.

[kamilkisiela]

I can explain all the bits and pieces, to help you with the PR

[dotansimha]

Hi @ilijaNL!

Just wanted to share my point of view here:

• We have an interface to define the storage layer, so this makes it easier for users to add new implementations for different kinds of DBs.
• With that being said, I think it will be an overhead for our team to maintain a new implementation for a database we are not familiar with (in terms of capabilities, performance, and language).
• We are very open to new ideas here. However, I believe that the path to having alternative databases is by forking, implementing, and testing it on your use case. Hive is fully open source and you can fork and leverage the existing test suites to get there.
• At the point where a new implementation for the database is implemented and fully tested (to make sure it matches the Hive requirements), we can consider getting this as part of the main codebase.

If you are willing to work on that, please let us know, and we can create a faster communication channel (Slack?) and help you with that process.

Thanks!