Goal: every-second stream updates

[markholdex]

Specs

Problems

Node

• Problem: new contracts are not benchmarked #770
• Problem: missing exact storage requirements for data #774
• Problem: infrastructure might cannot handle rapid updates
• Problem: Unix benchmark report lacks information #775

Truf-Data-Provider

• https://github.com/trufnetwork/truf-data-provider/issues/383
• https://github.com/trufnetwork/truf-data-provider/issues/386
• https://github.com/trufnetwork/truf-data-provider/issues/392
• https://github.com/trufnetwork/truf-data-provider/issues/396
• https://github.com/trufnetwork/truf-data-provider/issues/397
• https://github.com/trufnetwork/truf-data-provider/issues/399
• https://github.com/trufnetwork/truf-data-provider/issues/400
• https://github.com/trufnetwork/truf-data-provider/issues/403
• https://github.com/trufnetwork/truf-data-provider/issues/405
• https://github.com/trufnetwork/truf-data-provider/issues/408
• trufnetwork/truf-data-provider#411
• https://github.com/trufnetwork/truf-data-provider/issues/415
• trufnetwork/truf-data-provider#416

ADAPTERS

• Problem: can't batch insert on frequent ingestion adapters#55

SDK-PY

• Problem: unable to push data to TN based on UNIX timestamp sdk-py#5

SDK-GO

• Problem(go): current contracts do not support timestamp sdk-go#62
• Problem: unable to push data to TN based on UNIX timestamp sdk-go#64

SDK-JS

• Problem: unable to push data to TN based on UNIX timestamp sdk-js#30

ETA: 2025-01-17 2025-01-18 2025-01-21

[markholdex]

@zolotokrylin @outerlook @MicBun please check the specs document and leave your comments/suggestions there. Let's have a call after that.

[rsoury]

@markholdex
Is this a clone of #199?

Is it just the granularity that we need, or are we shooting for real-time subscription to data streams?

[zolotokrylin]

@rsoury granularity.

[MicBun]

Log: updated the spec document according to the latest call.

[MicBun]

Log: I've broken down the spec into smaller problems, feel free to adjust or break it down into smaller parts if needed.

[markholdex]

@MicBun what is the ETA of this Goal?

[MicBun]

I've updated the issue description with the ETA @markholdex

[outerlook]

just to aggregate information about our capacity, based on

• Problem: Unix benchmark report lacks information #775 (comment)
• Problem: missing exact storage requirements for data #774 (comment)

with a frequency of 1 record per 5 minutes, based on experiments, we have:

• to achieve under 5 seconds to fetch[1] (arbitrary QoS), we can handle a composition of ~50 streams [2]
• 500 streams with this frequency take ~5GB of data with data for a year (theoretical)

notes:
[1] fetching 1 month of data. If we fetch 1 day of data, we can handle 30x more streams. It's nearly proportional on the new contract.
[2] only if we use the 0.9 optimizations (easy to apply). Using the current version of contracts we can compose ~15 streams

*5 min is based on coingecko's availability

[markholdex]

Hey @MicBun are we on track with this goal?

[MicBun]

Hi @markholdex, unfortunately, I discovered some issues and need to hold back the deployment

• https://github.com/trufnetwork/truf-data-provider/issues/408
• https://github.com/trufnetwork/truf-data-provider/issues/405

I'll still try to finish it on 2025-01-18

[MicBun]

Apologies for the further delay. @markholdex
The push service is deployed but not stable yet. It ran into crashes. I will investigate it and might need 1-2 days to resolve.

• https://github.com/trufnetwork/truf-data-provider/issues/415

Bitcoin data: