Goal: More performant architecture with Kwil v0.10

[outerlook]

Specs

Blocks

• Bug: can't retrieve all streams sdk-js#34
• Goal: Argentina SEPA historical data adapters#52
• Goal: Argentina SEPA QC adapters#54

Blocked By

• Problem: kwil-db 0.10 documentation is missing #798
• Problem: unable to execute SQL queries on node v0.10.0-beta.3 #802

Issues

Contract And Node Upgrade

• Problem(2.0): node version is not updated to v0.10.0 #795
• Problem: Previous stream behaviors are not listed #800
• Problem: Stream "Authorization" behaviors are untested #808
• Problem: Stream "Querying" behaviors are untested #809
• Problem: Stream "Insertion" behaviors are untested #810
• Problem: Stream "Composition" behaviors are untested #811
• Problem: Stream "Other" behaviors are untested #812
• Problem: missing stream delete implementation #822
• Problem: missing data insertion implementation #821
• Problem: missing primitive retrieval implementation #823
• Problem: missing composed weight retrieval implementation #824
• Problem: missing read stream authorization #825
• Problem: missing compose stream authorization #826
• Problem: missing write stream authorization #827
• Problem: missing owner transfer implementation #828
• Problem: Batch mechanisms incompatible with new architecture
• Problem: Genesis file new chain missing
• Problem: CDK not configured for new nodes
• Problem: System contract not deployed from TrufNetwork wallet

SDKs Upgrade

• TBD

Data Provider & Adapter

• Problem: Data Provider doesn't support new SDKs
• Problem: Adapters not migrated to new architecture
• Problem: Nuon contract missing for new structure
• Problem: Data ingestion flows not set up for new system

Deprecate Old Chain

• Problem: Migration guide for TN users missing
• Problem: Migration guide for node operators missing
• Problem: Nuon migration missing
• Problem: Website migration missing
• Problem: Index.fun migration missing
• Problem: Deprecation timeline not scheduled or announced

[outerlook]

Hey @rsoury @markholdex, I'm just creating this goal to log that @brennanjl is investigating and suggesting alternative approaches where our contracts could perform 25x better related to speed, and use less memory too.

It's related to the next Kwil version, because it would require some additional features, SQL-wise, from Kwil.

It does have compromises, like adopting an architecture similar to the one discussed on #137, so we also know it would work (with the compromise of not having flexible logics, which isn't widely used at the moment)

the discussion could go through this goal and specs

[markholdex]

@outerlook can you kickstart the specs with the info you know?

[MicBun]

Should this goal be closed? #763
Looking at the versioning the v0.10 should be newer than 0.9, right?

@outerlook

[outerlook]

I don't think so, as each is a bit different:

• The v0.9 goal is about performance enhancements we're ready to support (e.g. using indexed access)
• v0.10 is next version, by the time I created, there were some ongoing discussions about being able to improve time speeds up to 25x, and maybe changing our streams architecture to achieve that massive gain. However, I'm in debt of giving more information in a specs -- because I don't have more concrete information yet.

In short if some of the goals is off, maybe it's this one. But is it worth keeping just to have on our map that this intention exists? And, as soon as kwil team has more concrete information about the modifications of the new version, we begin the specs?

[markholdex]

@MicBun @outerlook let's move the discussion to a spec document, please!

[markholdex]

Hey @brennanjl @KwilLuke do you guys have any updates on the new architecture? We are keen to begin the work around the Billing mechanism we discussed.

• Goal: TRUF Network billing #740

[brennanjl]

Hi @markholdex

Apologies for the delayed response; the Kwil team takes off the 7 days between Christmas and New Years, so we are just getting back to it.

Our team is close to having the new version ready. We ran into some hiccups, which combined with the holidays, delayed timelines a little bit. For the purposes of your team's planning, we should break down Kwil's milestones into 2 sections:

• v0.10 beta
• billing

The v0.10 beta will likely be ready ~1 week before billing (since e2e testing for billing requires a somewhat stable v0.10). We are having a team-wide meeting tomorrow to discuss timelines for the beta, so I will have more accurate timelines for you then, but my current estimate is around the 14th. I will let you know if our estimates after our meeting tomorrow are different from this.

Billing will follow this as closely as possible. As mentioned above, we should have something usable ~1 week after v0.10 beta, but please keep in mind that real tokens should not yet be used with it. The UX/DX should be pretty set (so that your team can begin development and experimentation with EVM testnets), but there is still auditing and testing that needs to be performed before it is safe to use real funds.

[brennanjl]

Hi team,

We are still running a few days behind. We have hit some last minute bugs, and are ironing them out before we get your team's hands on it. We will keep you closely updated with our progress as we fix these.

[markholdex]

Hey @brennanjl,

Thanks for keeping us updated. Looking forward!

[brennanjl]

Hi team,

We should have things ready to be used on Tuesday. Thank you for your patience.

[markholdex]

@brennanjl hope you don't mind. I moved your message into a Google Doc in the description of this goal. It's easier for us to follow-up with specific questions there. I'll hide your message from this feed for now.

[zolotokrylin]

@markholdex if you will be talking to Stefan, please raise the issue that we need more resources to get this Goal achieved ASAP. Otherwise, we will never start working on it because we will always have more urgent things.

[rsoury]

Added to the Spec. Outlines a plan to upgrade in parallel to current work.

Looping in @Jarryd-pretorius re #673 (comment)

TL;DR - This is becoming increasingly more important.
Delays are surfacing in #673 (comment)

[zolotokrylin]

Yes, we need to upgrade ASAP.

[outerlook]

if I get some authorization, I can immediately work on creating a fork where our two nodes work in isolation from other operators, to have a quicker response time to our business requirements until we deploy the next architecture. The alternative is to continue as is waiting for operators to fix
@rsoury @markholdex

[markholdex]

@outerlook check my comments in the specs. I might have answered you.

@outerlook @MicBun let's cleanup the specs into a clear plan and begin the work.

[outerlook]

added some issue statements to the description to tackle the goal. Also edited the specs document for better organization