accounts + native transfers #1210

kyscott18 · 2024-11-01T19:44:22Z

Closes #1157 #1198 #1205

Tasks

`sync`

`sync-historical`

`sync-realtime`

fetchBlockEventData
handleBlock
factory support

API

AccountSource
ponder.config.ts
buildConfigAndIndexingFunctions

packages/core/src/sync-historical/index.ts

kyscott18 · 2024-11-04T02:21:16Z

packages/core/src/sync-historical/index.ts

+      if (block.transactions.find((t) => t.hash === hash) === undefined) {
        throw new Error(
-          `Detected inconsistent RPC responses. 'trace.blockHash' ${callTrace.blockHash} does not match 'block.hash' ${block.hash}`,
+          `Detected inconsistent RPC responses. 'trace.transactionHash' ${hash} not found in 'block.transactions' ${block.hash}`,
        );
      }


kyscott18 · 2024-11-04T02:22:35Z

packages/core/src/sync-historical/index.ts

+    let traces = await _debug_traceBlockByNumber(args.requestQueue, {
+      blockNumber: toHex(blockNumber),
+    });


When I was evaluating the debug rpc methods, I notice that traceBlockByHash was often way faster than traceBlockByNumber. It would require a different algorithm because block hash isn't immediately available, but something we should try before shipping

kyscott18 · 2024-11-04T02:47:21Z

packages/core/src/sync-historical/index.ts

+    const syncTraces = await _debug_traceBlockByNumber(args.requestQueue, {
+      blockNumber: toHex(blockNumber),
+    });


We should start to think about how we can avoid requesting the same block twice if there are multiple transfer and transaction filters

kyscott18 · 2024-11-05T03:28:24Z

packages/core/src/sync-realtime/index.ts

+    // });
+
+    // Remove traces that dont match transaction or transfer filter
+    traces = traces.filter((trace) => {


I think we need to do a depth first search over all the traces, either in this block or above where traces is declared.

It performs three functions:

processes child traces (each call can have several sub-calls)

orders traces for checkpoint.eventIndex

determine failed traces

See sync-historical

ponder/packages/core/src/sync-historical/index.ts

Lines 324 to 368 in 2117cec

const dfs = (

syncTrace: SyncTrace["result"][],

transactionHash: Hex,

parentTrace: SyncTrace["result"] | undefined,

) => {

for (const trace of syncTrace) {

if (filter.type === "transfer") {

if (

isTransferFilterMatched({

filter,

block: { number: toHex(blockNumber) },

trace,

})

) {

// TODO(kyle) handle factory

traces.push({ trace, transactionHash, position });

}

} else {

if (

isTransactionFilterMatched({

filter,

block: { number: toHex(blockNumber) },

trace,

})

) {

// TODO(kyle) handle factory

traces.push({ trace, transactionHash, position });

}

}

if (trace.error !== undefined) {

failedTraces.set(trace, trace.error);

} else if (parentTrace && failedTraces.has(parentTrace)) {

failedTraces.set(trace, failedTraces.get(parentTrace)!);

}

position++;

if (trace.calls) {

dfs(trace.calls, transactionHash, trace);

}

}

};

kyscott18 · 2024-11-05T03:29:53Z

packages/core/src/sync-realtime/index.ts


    return {
      block,
      logs,
      factoryLogs,
-      callTraces,
+      traces,


Should this type be the same as ??

ponder/packages/core/src/sync-historical/index.ts

Lines 313 to 317 in 2117cec

const traces: {

trace: SyncTrace["result"];

transactionHash: Hex;

position: number;

}[] = [];

kyscott18 · 2024-11-05T03:33:48Z

packages/core/src/sync-realtime/index.ts

    if (shouldRequestTraces) {
-      const traces = await _trace_block(args.requestQueue, {
+      traces = await _debug_traceBlockByNumber(args.requestQueue, {


Should use _debug_traceBlockByHash. It's more robust when reorgs occur because it unambiguously refers to the exact block

…or traces

…eBlock

…ounts

kyscott18 · 2024-11-20T19:38:41Z

packages/core/src/sync-historical/index.ts

+          isTransactionFilterMatched({
+            filter: {
+              ...filter,
+              fromAddress: fromAddress,
+              toAddress: toAddress,
+            },
+            block,
+            transaction,
+          })


I like this, do you think we could get around overriding filter.fromAddress by adding an optional parameter fromChildAddresses: Set<Address>? Then we could reuse this in sync-realtime as well

kyscott18 · 2024-11-20T20:56:14Z

packages/core/src/sync-realtime/index.ts

@@ -173,6 +208,36 @@ export const createRealtimeSync = (
      }
    }

+    const isFactoryAddressMatched = ({


Could get rid of this if we update is[type]FilterMatched to handle factories

typedarray · 2024-11-25T19:49:18Z

packages/core/src/build/configAndIndexingFunctions.ts

@@ -27,6 +32,36 @@ export type IndexingFunctions = {
  [eventName: string]: (...args: any) => any;
 };

+const flattenSource = <


flattenSources ?

typedarray · 2024-11-25T19:50:44Z

packages/core/src/build/configAndIndexingFunctions.ts

+
+      if (!sourceEventName) {
+        throw new Error(
+          `Validation failed: Invalid event '${eventName}', expected format '{sourceName}:{eventName}' or '{sourceName}.{eventName}'.`,


or '{sourceName}.{functionName}' ?

typedarray · 2024-11-25T20:01:17Z

packages/core/src/build/configAndIndexingFunctions.ts

      ...(config.blocks ?? {}),
    }).find((_sourceName) => _sourceName === sourceName);

    if (!matchedSourceName) {
      // Multi-network has N sources, but the hint here should not have duplicates.
      const uniqueSourceNames = dedupe(
-        Object.keys({ ...(config.contracts ?? {}), ...(config.blocks ?? {}) }),
+        Object.keys({


I think you already have a sourceNames set in scope here from above, could use that

typedarray · 2024-11-25T20:13:35Z

packages/core/src/indexing/service.ts

-        const result = await executeLog(indexingService, { event });
-        if (result.status !== "success") {
-          return result;
+    const toErrorMeta = () => {


I feel like this should be a static function that doesn't rely on the event object being in scope (accepts it as an arg). I think as-is this function will get reinitialized inside the hot loop.

typedarray · 2024-11-25T20:15:33Z

packages/core/src/indexing/service.ts

@@ -455,9 +428,9 @@ const executeSetup = async (
  return { status: "success" };
 };

-const executeLog = async (
+const executeEvent = async (


Nice simplifications to this file

typedarray · 2024-11-25T21:14:31Z

packages/core/src/sync-store/migrations.ts

+        .addColumn("to", "varchar(42)")
+        .addColumn("gas", "numeric(78, 0)", (col) => col.notNull())
+        .addColumn("gasUsed", "numeric(78, 0)", (col) => col.notNull())
+        .addColumn("input", "text", (col) => col.notNull())


Are you sure this is not null? I've definitely seen traces with an "empty" input, but maybe it's an empty string.

https://github.com/alloy-rs/alloy/blob/5fcbb2d87100923f4274a13e3ddb54cde3bbd681/crates/rpc-types-trace/src/geth/call.rs#L24

typedarray · 2024-11-25T21:20:42Z

packages/core/src/sync-store/migrations.ts

+        .addColumn("subcalls", "integer", (col) => col.notNull())
+        .execute();
+
+      // `getEvents` benefits from an index on


Hmm, this definitely feels over-indexed, but I'm happy to leave it as-is until we tackle the sync tables overhaul.

it definitely felt like overkill but not sure which ones should be left out

typedarray · 2024-11-25T21:24:38Z

packages/core/src/sync/events.ts

+      case "account": {
+        switch (source.filter.type) {
+          case "transaction": {
+            // TODO(kyle) what if toAddress and fromAddress are both undefined?


Not sure what you mean here

How do we know where do display {sourceName}:transaction:from or ${sourceName}:transaction:to if the user creates an account with { address: undefined }. I think we should actually validate against this

typedarray · 2024-11-25T21:27:54Z

packages/core/src/types/eth.ts

+  /** Number of subcalls. */
+  subcalls: number;


Won't this still count non-CALL traces? eg

/** Number of child traces produced during the execution of this trace. */ subtraces: number;

yes, the naming is not super clear. Trace has a propery calls, but the Traces inside calls don't have to have type === "CALL"

typedarray · 2024-11-25T21:41:59Z

packages/core/src/utils/rpc.ts

+      let index = 0;
+      // all traces that weren't included because the trace has an error
+      // or the trace's parent has an error, mapped to the error string
+      const failedTraces = new Map<


Seems fine but I think this Map will contain frames across multiple transactions within the block

kevinfaveri · 2024-11-28T14:49:23Z

Hello guys; I have a pressing timeline on getting something for native transfers and would like to explore using this here... Do you think it will be ready within next 1/2 weeks? Any timeline in mind?

We're hoping to get a prerelease out in the next few days, but don't want to make any promises on when we will release v0.8.

@kyscott18 Hey, curious on the technical implementation here. I see you use debug RPC APIs (trace by block number). Why a simple block by number wouldn't suffice (like, by filtering msg.value > 0 as a transfer of native and using tx.origin as the "from")? I suppose it is because you want the internal calls, but what information are you missing on a native transfer that you would need the debug/trace RPC vs the normal get block by number?

I can think of one transfer that Alice > Bob > Charlie would be interpreted as Alice > Charlie; but too practical effects, on that transaction, the money went from Alice to Charlie at the end of the block confirmation, right? What is the imporstance of knowing bob in the middle of the internal call?

Is this for example some custom contract that would send part of a deposit from user to 2 users, which could create the Alice > Bob > Charlie situation? And then if you only consider in / out of block you wouldnt see that Bob got a cut too?

khaidarkairbek self-assigned this Nov 3, 2024

kyscott18 commented Nov 4, 2024

View reviewed changes

This was linked to issues Nov 5, 2024

[Feature] Native transfers #1157

Open

feat: filter types #1198

Open

feat: use debug namespace for inner transactions #1201

Open

kyscott18 commented Nov 5, 2024

View reviewed changes

Base automatically changed from v0.7 to main November 8, 2024 17:52

kyscott18 and others added 21 commits November 10, 2024 11:40

start new filter impl

2daf668

debug rpc types

35d5d00

withLog

3cfd157

add debug helper functions

f987047

syncTrace in sync-historical added with filter matching

cc5a3f6

syncTrace filtering modified, blockwise processing added

1a61820

sync-realtime: added tx/transfer factories

d15808d

sync: fragments and encoding for transaction/transfer filters

600304d

sync-realtime: transaction/trasfer filter added

b318ead

sync-realtime: handleBlock is modified for new filters

0ab76da

sync-realtime: fetchBlockEventData changed to debug and new filters f…

57bfc82

…or traces

update historical sync

3f3f82d

build checkpoint for traces

06b9060

record failed traces

3d944f0

implement insert traces db method

479be95

.

6681242

cleanup realtime

41412e6

corrections: minor changes

edbbea0

sync-historical: traceCache added

51e87df

sync-realtime: trace traversal added in fetchBlockEventData and handl…

a7052f3

…eBlock

sync: buildEvents and decodeEvents modfied

e7dcfbb

kyscott18 force-pushed the accounts branch from 8237db7 to e7dcfbb Compare November 10, 2024 16:41

fix debug type

5640a6a

khaidarkairbek and others added 7 commits November 19, 2024 16:59

sync: handle factory address

b12eacc

cleanup

f0fb96f

Merge branch 'accounts' of https://github.com/0xOlias/ponder into acc…

c9f4ed9

…ounts

fix transport tests

c9a40aa

realtime sync tests

3b4cd1f

fix typecheck

04c934e

account specific test

0ca8af0

kyscott18 marked this pull request as ready for review November 20, 2024 01:37

kyscott18 added 2 commits November 19, 2024 21:13

accounts example

0c97911

fix virtual types

7c2cfa4

kyscott18 commented Nov 20, 2024

View reviewed changes

kyscott18 and others added 10 commits November 20, 2024 16:00

cleanup sync

587e3de

note

ecb3482

transaction includeReverted

c9f5e1a

cleanup

aae6b18

fix: config

5bef6b0

sync: factory filter

ec42276

sync: factory filter cont.

017466c

set

597a65f

feature-accounts

74c0246

cleanup

36631a7

kyscott18 requested a review from typedarray November 23, 2024 17:26

trace type

8d6b189

typedarray reviewed Nov 25, 2024

View reviewed changes

kyscott18 added 3 commits November 25, 2024 18:40

nits

731d2bb

fix

dc78175

cleanup

1e249e2

kyscott18 merged commit 7a83bb2 into v0.8 Nov 26, 2024
8 checks passed

kyscott18 deleted the accounts branch November 26, 2024 21:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

accounts + native transfers #1210

accounts + native transfers #1210

kyscott18 commented Nov 1, 2024 •

edited

Loading

kyscott18 Nov 4, 2024

kyscott18 Nov 4, 2024

kyscott18 Nov 4, 2024

kyscott18 Nov 5, 2024

kyscott18 Nov 5, 2024

kyscott18 Nov 5, 2024

kyscott18 Nov 20, 2024 •

edited

Loading

kyscott18 Nov 20, 2024

typedarray Nov 25, 2024

typedarray Nov 25, 2024

typedarray Nov 25, 2024

typedarray Nov 25, 2024

typedarray Nov 25, 2024

typedarray Nov 25, 2024

kyscott18 Nov 25, 2024

typedarray Nov 25, 2024

kyscott18 Nov 25, 2024

typedarray Nov 25, 2024

kyscott18 Nov 26, 2024

typedarray Nov 25, 2024

kyscott18 Nov 25, 2024

typedarray Nov 25, 2024

kevinfaveri commented Nov 28, 2024

	const dfs = (
	syncTrace: SyncTrace["result"][],
	transactionHash: Hex,
	parentTrace: SyncTrace["result"] \| undefined,
	) => {
	for (const trace of syncTrace) {
	if (filter.type === "transfer") {
	if (
	isTransferFilterMatched({
	filter,
	block: { number: toHex(blockNumber) },
	trace,
	})
	) {
	// TODO(kyle) handle factory

	traces.push({ trace, transactionHash, position });
	}
	} else {
	if (
	isTransactionFilterMatched({
	filter,
	block: { number: toHex(blockNumber) },
	trace,
	})
	) {
	// TODO(kyle) handle factory

	traces.push({ trace, transactionHash, position });
	}
	}

	if (trace.error !== undefined) {
	failedTraces.set(trace, trace.error);
	} else if (parentTrace && failedTraces.has(parentTrace)) {
	failedTraces.set(trace, failedTraces.get(parentTrace)!);
	}

	position++;

	if (trace.calls) {
	dfs(trace.calls, transactionHash, trace);
	}
	}
	};

	const traces: {
	trace: SyncTrace["result"];
	transactionHash: Hex;
	position: number;
	}[] = [];

accounts + native transfers #1210

accounts + native transfers #1210

Conversation

kyscott18 commented Nov 1, 2024 • edited Loading

Tasks

sync

sync-historical

sync-realtime

API

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kyscott18 Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinfaveri commented Nov 28, 2024

kyscott18 commented Nov 1, 2024 •

edited

Loading

`sync`

`sync-historical`

`sync-realtime`

kyscott18 Nov 20, 2024 •

edited

Loading