Goal: Billing #166

zolotokrylin · 2024-04-07T00:36:41Z

Specs

Blocker

Problems

To decide

~~Problem: truflation/tsn-synchronizer repo is not created~~ (cancelled)

Billing/Data provider tasks

TSN Tasks

Supabase tasks

truflation/website#426
truflation/website#480
truflation/website#481
truflation/website#436

Website tasks

https://github.com/truflation/website/issues/497
truflation/website#553
https://github.com/truflation/website/issues/568

zolotokrylin · 2024-04-09T16:52:57Z

@KwilLuke @brennanjl do you guys already have it in place?

brennanjl · 2024-04-10T03:45:52Z

This is currently implementable with oracles and extensions.

Once Truflation is ready to add this as a feature, we're happy to jump in to help prototype, show how it can be done right now, and then make additions to the Kwil product as needed.

Ideally, it follows a process similar to how we have handled streams, where:

we help prototype an initial version using the current Kwil feature set
we analyze what could be better for Truflation
our team makes core additions / changes to Kwil based on learnings from the initial version

zolotokrylin · 2024-04-10T17:13:12Z

@brennanjl please point me to the documentation, if any. Thank you!

KwilLuke · 2024-04-10T19:28:13Z

@zolotokrylin you can check out the docs for oracles and extensions here: https://docs.kwil.com/docs/extensions/oracles-overview

There is also an example for an eth oracle here: https://github.com/kwilteam/kwil-db/tree/main/extensions/listeners/eth_deposits

For Truflation's billing, the implementation would require some creativity to get data payment rails into an extension. For example:

Should the billing be done in TRUF? USDC?
Are you doing crypto payments only? Or Fiat? Could there be payment rails?
What is the billing for? To read data? Write data? Create new indexes? All of the above?

Lot's to think through here, but have a look at the docs and let us know if this looks interesting for what you are thinking.

brennanjl · 2024-04-11T03:10:28Z

Yeah, we do not yet have a place describing how to do pricing per-se; it would likely be a framework we develop via extensions.

If you have an outline on requirements, I can try to propose more specifics on a solution.

zolotokrylin · 2024-04-11T15:00:14Z

Thank you, guys!

I pinged @itscameronlee to provide us the initial input directly in the Spec doc (see description of the issue).
Once we have this input, we can iterate over it.

zolotokrylin · 2024-05-22T02:01:46Z

@itscameronlee a kind reminder to provide initial inputs here.

itscameronlee · 2024-05-22T09:29:34Z

Will do. Assigning to myself so I don't lose track of this.

rsoury · 2024-06-22T01:57:37Z

Notes from our meeting on 18th of June,

Billing begins centralised and simple.

Once a user subscribes via the existing billing mechanism,

Customer can register Wallet Public Key(s) to automatically whitelist for access to the TSN
This way paying subscribers can query the TSN directly.

Side note on Trust:
Centralised billing is OK as the core value proposition of TSN is trust-minimised (decentralised) data management. No single party can tamper with the dataset.
However, facilitating money management is not the core value prop - unlike Ethereum, or other chains competing with VISA.

@zolotokrylin - did you wish to add to this?

zolotokrylin · 2024-06-22T05:12:12Z

@rsoury thank you for your message.
Also, we have agreed that we need to work on the system to allow us to measure the usage per Data Consumer.

However, facilitating money management is not the core value prop - unlike Ethereum, or other chains competing with VISA.

Not in the first version, but eventually that's the end goal.

zolotokrylin · 2024-06-25T07:26:23Z

@rsoury could you please propose a high-level system in the Spec doc (see description of this issue)?
Let's iterate over the specs, so we can all agree on the implementation idea.

rsoury · 2024-06-27T03:42:26Z

We have a start on a technical spec internally, will be incrementally added to the doc today/tomorrow.

zolotokrylin · 2024-06-28T03:01:07Z

@rsoury I have replied in the Specs doc.

rsoury · 2024-06-29T05:06:01Z

@zolotokrylin - We've made a good start to this spec.
@outerlook - To continue finalising the technical architecture.
@brennanjl - KGW will likely need an update to serve traceable access (and other) related metrics over socket connection, local file, etc. to be supported by Vector as a sidecar.

zolotokrylin · 2024-06-29T07:53:05Z

@truflation/team-tsn-core I have left comments in the doc.

rsoury · 2024-07-01T02:39:37Z

@zolotokrylin - Replied.

Will leave my https://calendly.com/ryansoury/chat here too.

zolotokrylin · 2024-07-01T07:52:51Z

@rsoury booked the meeting for tomorrow.
@markholdex I have also invited you.

zolotokrylin · 2024-07-02T05:34:30Z

We had a call with @rsoury . We are aligned on how to proceed further. He will share the recording of our meeting here.

Next sequential steps:

finalise the Specs
start development

rsoury · 2024-07-02T05:42:56Z

@truflation/team-tsn @truflation/team-tsn-core - Please review the Spec and share any thoughts or questions.

Once reviewed, please confirm awareness and alignment on the Spec.

To review the meeting:
https://drive.google.com/file/d/1QZrMnCD_8zo-Uh4EWio0pXwuQnOZf9Qy/view?usp=sharing

zolotokrylin · 2024-07-03T02:04:41Z

I just left an additional comment in the document, @rsoury.

rsoury · 2024-07-03T21:57:19Z

Replied @zolotokrylin @Jarryd-pretorius @outerlook

rsoury · 2024-07-05T02:39:41Z

@zolotokrylin - Discussed with @Jarryd-pretorius , and we will use Supabase as the core database. No intermediary cache is necessary.

Furthermore, no SQS or event management is necessary, as I realised it to be easier to just consolidate all metadata updates across Truflation streams within the TSN.
ie. A Metadata Admin Procedure that manages cross-contract calls all within a single transaction.

Vadim, and @outerlook -

Please review the latest architectural update.
Once happy, let's proceed.
It's as simple as we can get it now.

markholdex · 2024-07-15T08:12:51Z

@MicBun @outerlook do you need other members participating to this goal? If not please unassig them and leave only those who are directly involved. Thank you!

markholdex · 2024-07-19T07:59:05Z

Good job guys on the progress of this goal @MicBun @outerlook. 👍 What do you think is the ETA for completion? Do you have any blockers that need to be escalated?

MicBun · 2024-07-19T08:41:00Z

Do you have any blockers that need to be escalated?

Hi @markholdex, there are two major blockers for now:

which block tasks related to it as:

What do you think is the ETA for completion?

Includes the wait for the blocker to be cleared, I think 3 weeks. It can be as fast as 1 week if it cleared, as most of the functions and methods are set up and then we can sync with the established Supabase database.

outerlook · 2024-07-25T11:42:06Z

@zolotokrylin @markholdex

See that #383 is important just to see how much data our subscribers are consuming, as we're not using any kind of usage cap it won't modify the permission logic right now.

Said that, if you want to split that analytics step into another goal, the billing logic wouldn't be affected. This is just a suggestion in case other goals can take priority here.

zolotokrylin · 2024-07-25T14:56:31Z

@outerlook @markholdex

Said that, if you want to split that analytics step into another goal, the billing logic wouldn't be affected. This is just a suggestion in case other goals can be prioritized here.

Yes, please. Let's split so we can already have a basic billing released and start playing with it.

markholdex · 2024-07-25T17:04:05Z

@zolotokrylin @markholdex

See that #383 is important just to see how much data our subscribers are consuming, as we're not using any kind of usage cap it won't modify the permission logic right now.

Said that, if you want to split that analytics step into another goal, the billing logic wouldn't be affected. This is just a suggestion in case other goals can take priority here.

@outerlook how about @MicBun comment #383 (comment) to match our specs document and track the gas costs for the usage?

Of course, we can split the specs into multiple versions and extend the billing to support later a pay as you go plan, and for now, just keep a fixed billing cost. This would mean we don't necessarily track consumer usage but just track the fact that the services are paid.

outerlook · 2024-07-25T18:00:36Z

The comment is valid, but is inserted in more context here:

The TSN roadmap includes autonomous billing mechanisms that are cryptographically enforced upon data consumption. The goal is for data consumers to pay "gas" that covers the cost of the data consumed. Based on the dependency tree of the data stream selected for data consumption, intermediary analysts—stream deployers and typically data providers—will be rewarded a portion of the gas. Additionally, this gas may serve to compensate Node Operators.

While this concept is ideal, several questions and considerations need to be addressed to achieve this paradigm. These include determining what is considered high-value data, how long data should be stored, and whether data is permanent by default or if a "rent" mechanism should be introduced (similar to Solana Rent).

The TSN is a new form of decentralised network with the unique purpose of delivering "true" indexes. "True" means as high quality as possible, with cryptographic security against censorship and tampering. To achieve this vision, data is required to drive correct iteration. TSN can iterate on features that extend its data, as the data is the true value proposition. Using a credibly neutral database minimises trust in the data and its governance, with additional features enhancing user experience.

Therefore, the first version of the billing mechanism aims to ensure that paying Truflation customers can directly and securely access the credibly neutral database. Currently, Truflation is the only authorised data consumer.

as for the intention of already tracking some metrics is about this part:

Kwil Gateway (KGW) Metrics:

Modify the KGW to produce access-related OpenTelemetry metrics. Kwil should enable this.

Metrics Collection:

Metrics should be pushed to a socket (TCP or Unix) or HTTP endpoint.

[Vector](https://vector.dev/) can support as a KGW sidecar to collect metrics and push them to SQS.

These metrics can also surface in [the public metrics interface](https://docs.google.com/document/d/10BWNbBT2VINq0-YNRHxdcD_ZvXBzgaVZlSfJUW5nD9Y/edit#heading=h.t332lgsrqz4x), especially regarding TSN query/consumption usage.

Metrics and Whitelist Management:

A function for collecting SQS metrics and pushing them to a database supporting internal Grafana, for analysis and management.

Eventually, evolve this function will be evolved to adjust the whitelist database to temporarily deactivate access for customers who exceed the usage limit. A monthly scheduled function will reset all temporarily deactivated accesses.

So, the need for the metrics to be there is knowledge about consumption, for us to start checking how we will eventually set gas-based billing. I.e., if we release it and someone with a subscription spams requests, we won't know easily this is happening from a single user or even how they truly use it.

But my suggestion is to be ok with that for now, just to focus on other goals

markholdex · 2024-07-25T19:47:08Z

@outerlook alright, I'll convert into a Goal

outerlook · 2024-07-29T12:12:10Z

I executed a test to see if it works:

use fetch from a random wallet and see not permitted

~$ kwil-cli database call -a=get_index date_from:2023-06-17 date_to:2023-06-17 -n=stf389ad7681059ca7750dda907735b2 -o=4710a8d8f0d845da110086812a32de6d90d7ff5c --private-key 0000000000000000000000000000000000000000000000000000000000000018 --provider https://staging.tsn.test.truflation.com
error calling action: call action: jsonrpc.Error: code = -300, message = ERROR: wallet not allowed to read (SQLSTATE P0001), data = null
err code = -300, msg = ERROR: wallet not allowed to read (SQLSTATE P0001)

create a tsn-access row manually whitelisting this wallet

executed the lambda manually:

START RequestId: a241cb00-2cba-48e0-8a44-53d3b893773f Version: $LATEST
2024/07/29 12:05:18 Starting billing synchronizer
2024/07/29 12:05:19 Retrieving category directories
2024/07/29 12:05:19 Reading ID mapping file
2024/07/29 12:05:19 Reading taxonomy file
2024/07/29 12:05:19 Reading ID mapping file
2024/07/29 12:05:19 Reading taxonomy file
2024/07/29 12:05:19 Reading ID mapping file
2024/07/29 12:05:19 Reading taxonomy file
2024/07/29 12:05:19 Reading ID mapping file
2024/07/29 12:05:19 Reading taxonomy file
2024/07/29 12:05:19 Reading ID mapping file
2024/07/29 12:05:19 Reading taxonomy file
2024/07/29 12:05:19 Found 81 root streams
2024/07/29 12:05:19 Fetching active subscriptions
2024/07/29 12:05:19 Found a total of 5 active subscriptions
2024/07/29 12:05:20 Found a total of 450 streams
2024/07/29 12:05:22 Found a total of 450 streams with whitelist information
2024/07/29 12:05:22 Starting to apply permission differences...
2024/07/29 12:05:22 Executing 450 whitelist instructions
2024/07/29 12:05:22 Total of 450 whitelist instructions executed
2024/07/29 12:05:22 No wallets to revoke
2024/07/29 12:05:22 Successfully applied permission differences.
2024/07/29 12:05:22 Waiting for transactions to be mined
2024/07/29 12:05:22 Waiting for transaction 859F12B1E766E574505C50725C87B2EA9C8EF5E42181E11821E89C08F4D642D0
2024/07/29 12:05:27 Synchronization complete
END RequestId: a241cb00-2cba-48e0-8a44-53d3b893773f
REPORT RequestId: a241cb00-2cba-48e0-8a44-53d3b893773f	Duration: 8724.81 ms	Billed Duration: 8885 ms	Memory Size: 128 MB	Max Memory Used: 45 MB	Init Duration: 159.44 ms

try fetch again and see if it works

~$ kwil-cli database call -a=get_index date_from:2023-06-17 date_to:2023-06-17 -n=stf389ad7681059ca7750dda907735b2 -o=4710a8d8f0d845da110086812a32de6d90d7ff5c --private-key 0000000000000000000000000000000000000000000000000000000000000018 --provider https://staging.tsn.test.truflation.com
| date_value |  value  |
+------------+---------+
| 2023-06-16 | 154.248 |

remove from the tns-access, execute lambda, and see access is revoked

START RequestId: 14f7b0e5-4099-46f0-9e27-2a74b9ea09b5 Version: $LATEST
2024/07/29 12:09:30 Starting billing synchronizer
2024/07/29 12:09:30 Retrieving category directories
2024/07/29 12:09:30 Reading ID mapping file
2024/07/29 12:09:30 Reading taxonomy file
2024/07/29 12:09:30 Reading ID mapping file
2024/07/29 12:09:30 Reading taxonomy file
2024/07/29 12:09:30 Reading ID mapping file
2024/07/29 12:09:30 Reading taxonomy file
2024/07/29 12:09:30 Reading ID mapping file
2024/07/29 12:09:30 Reading taxonomy file
2024/07/29 12:09:30 Reading ID mapping file
2024/07/29 12:09:30 Reading taxonomy file
2024/07/29 12:09:30 Found 81 root streams
2024/07/29 12:09:30 Fetching active subscriptions
2024/07/29 12:09:30 Found a total of 4 active subscriptions
2024/07/29 12:09:31 Found a total of 450 streams
2024/07/29 12:09:32 Found a total of 450 streams with whitelist information
2024/07/29 12:09:32 Starting to apply permission differences...
2024/07/29 12:09:32 No wallets to whitelist
2024/07/29 12:09:32 Executing 450 revoke instructions
2024/07/29 12:09:32 Total of 450 revoke instructions executed
2024/07/29 12:09:32 Successfully applied permission differences.
2024/07/29 12:09:32 Waiting for transactions to be mined
2024/07/29 12:09:32 Waiting for transaction 77585E775220FC0DE683F0BA6471B853FA259A25840717D7C65CA88ABCBDB8BB
2024/07/29 12:10:26 Synchronization complete
END RequestId: 14f7b0e5-4099-46f0-9e27-2a74b9ea09b5
REPORT RequestId: 14f7b0e5-4099-46f0-9e27-2a74b9ea09b5	Duration: 56584.41 ms	Billed Duration: 56585 ms	Memory Size: 128 MB	Max Memory Used: 48 MB

~$ kwil-cli database call -a=get_index date_from:2023-06-17 date_to:2023-06-17 -n=stf389ad7681059ca7750dda907735b2 -o=4710a8d8f0d845da110086812a32de6d90d7ff5c --private-key 0000000000000000000000000000000000000000000000000000000000000018 --provider https://staging.tsn.test.truflation.com
error calling action: call action: jsonrpc.Error: code = -300, message = ERROR: wallet not allowed to read (SQLSTATE P0001), data = null
err code = -300, msg = ERROR: wallet not allowed to read (SQLSTATE P0001)

outerlook · 2024-07-29T12:17:15Z

also, no error on sync since it was deployed, executing on 5-minute rate

markholdex · 2024-08-07T12:29:30Z

TSN pages are loading now. I will proceed and change the wallet address and test if the billing access is working.

https://truflation.com/dashboard?feed=us-inflation-rate&tsn=true

markholdex · 2024-08-07T12:52:28Z

All work with the new wallet address. Good job @outerlook and @MicBun

rsoury changed the title ~~Gol: Billing~~ Goal: Billing Apr 8, 2024

itscameronlee self-assigned this May 22, 2024

zolotokrylin added the type: goal label May 23, 2024

rsoury mentioned this issue Jun 22, 2024

Goal: TSN Connect - Analyse Verified Transparency of Data Providers' Data Lineage #318

Open

zolotokrylin mentioned this issue Jun 24, 2024

Goal: real-time Indexes and Primitives #199

Open

zolotokrylin assigned rsoury Jun 26, 2024

zolotokrylin unassigned itscameronlee Jun 27, 2024

This comment was marked as resolved.

Sign in to view

rsoury mentioned this issue Jul 12, 2024

Problem: the backlog is not clear for some developers #386

Closed

zolotokrylin mentioned this issue Jul 12, 2024

Goal: Improve Documentation over System Contracts Architecture #341

Closed

9 tasks

This was referenced Jul 14, 2024

Problem: nodes and indexers are prone to crashes #403

Closed

Problem: local environment is not similar to deployed server #401

Closed

Problem: stable kwil-db doesn't include @foreign_caller #399

Closed

outerlook unassigned zolotokrylin, rsoury and markholdex Jul 15, 2024

This was referenced Jul 22, 2024

Goal: 99.9% TSN Reliability #415

Closed

Goal: onboard Kwil as Node Operator #393

Closed

Problem: Enable OTel Compatible Metrics from KGW #383

Open

This comment has been minimized.

Sign in to view

zolotokrylin mentioned this issue Aug 6, 2024

Goal: officially launch TSN with 12 node operators #438

Closed

12 tasks

zolotokrylin assigned morenonaitana Aug 6, 2024

MicBun mentioned this issue Aug 6, 2024

Problem: need to replace generate_dbid as it is built-in #429

Closed

markholdex closed this as completed Aug 7, 2024

outerlook mentioned this issue Sep 16, 2024

Goal: Get Usage Metrics From TSN #427

Open

7 tasks

MicBun unassigned morenonaitana Nov 23, 2024

Goal: Billing #166

Goal: Billing #166

Comments

zolotokrylin commented Apr 7, 2024 • edited by MicBun Loading

Specs

Blocker

Related

Problems

To decide

Billing/Data provider tasks

TSN Tasks

Supabase tasks

Website tasks

zolotokrylin commented Apr 9, 2024

brennanjl commented Apr 10, 2024

zolotokrylin commented Apr 10, 2024

KwilLuke commented Apr 10, 2024

brennanjl commented Apr 11, 2024

zolotokrylin commented Apr 11, 2024

zolotokrylin commented May 22, 2024

itscameronlee commented May 22, 2024

rsoury commented Jun 22, 2024

zolotokrylin commented Jun 22, 2024

zolotokrylin commented Jun 25, 2024 • edited Loading

rsoury commented Jun 27, 2024

This comment was marked as resolved.

zolotokrylin commented Jun 28, 2024

rsoury commented Jun 29, 2024

zolotokrylin commented Jun 29, 2024

rsoury commented Jul 1, 2024

zolotokrylin commented Jul 1, 2024

zolotokrylin commented Jul 2, 2024

rsoury commented Jul 2, 2024

zolotokrylin commented Jul 3, 2024

rsoury commented Jul 3, 2024

rsoury commented Jul 5, 2024

markholdex commented Jul 15, 2024

markholdex commented Jul 19, 2024 • edited Loading

MicBun commented Jul 19, 2024 • edited by outerlook Loading

outerlook commented Jul 25, 2024

zolotokrylin commented Jul 25, 2024

markholdex commented Jul 25, 2024 • edited Loading

outerlook commented Jul 25, 2024

markholdex commented Jul 25, 2024

outerlook commented Jul 29, 2024

outerlook commented Jul 29, 2024

This comment has been minimized.

markholdex commented Aug 7, 2024 • edited Loading

markholdex commented Aug 7, 2024

zolotokrylin commented Apr 7, 2024 •

edited by MicBun

Loading

zolotokrylin commented Jun 25, 2024 •

edited

Loading

markholdex commented Jul 19, 2024 •

edited

Loading

MicBun commented Jul 19, 2024 •

edited by outerlook

Loading

markholdex commented Jul 25, 2024 •

edited

Loading

markholdex commented Aug 7, 2024 •

edited

Loading