Add token transfer processor #195

aditya1702 · 2025-06-13T21:18:34Z

What

Add a token transfer processor that uses the corresponding ttp processor to extract debit/credit events. Once the events are extracted, the processor generates state changes with the appropriate information (account ID, operation ID, tx hash, claimable balance ID etc....).

Note: only DEBIT, CREDIT, MINT and BURN state changes are generated.

The following operations are processed:

CreateAccount
AccountMerge
Payment
CreateClaimableBalance
ClaimClaimableBalance
Clawback, ClawbackClaimableBalance
LiquidityPoolDeposit, LiquidityPoolWithdraw
PathPaymentStrictSend, PathPaymentStrictReceive
ManageBuyOffer, ManageSellOffer
InvokeHostFunction (SEP-41)

The unit tests build a transaction for each case and asserts the proper creation of transfer events.

Why

Part of the state change indexer work

Known limitations

N/A

Issue that this PR addresses

Closes #194

Checklist

PR Structure

It is not possible to break this PR down into smaller PRs.
This PR does not mix refactoring changes with feature changes.
This PR's title starts with name of package that is most changed in the PR, or all if the changes are broad or impact many packages.

Thoroughness

This PR adds tests for the new functionality or fixes.
All updated queries have been tested (refer to this check if the data set returned by the updated query is expected to be same as the original one).

Release

This is not a breaking change.
This is ready to be tested in development.
The new functionality is gated with a feature flag if this is not ready for production.

marcelosalloum

Looking great so far 👏

internal/db/migrations/2025-06-10.4-create_indexer_table_state_changes.sql

internal/processors/token_transfer.go

marcelosalloum

I haven't finished reviewing the PR yet, but will leave my initial comments here.

marcelosalloum · 2025-06-24T14:38:24Z

internal/processors/state_change_builder.go

+			b.base.Token = sql.NullString{String: "native"}
+		} else if issuedAsset := asset.GetIssuedAsset(); issuedAsset != nil {
+			b.base.Token = sql.NullString{String: fmt.Sprintf("%s:%s", issuedAsset.GetAssetCode(), issuedAsset.GetIssuer())}
+		}
+	} else {
+		b.base.ContractID = sql.NullString{String: contractAddress}


👋 The usage of sql.NullString seems wrong here because you need to also set Valid: true. You can use the utils.SQLNullString helper here instead.

wallet-backend/internal/utils/sql.go

Lines 8 to 13 in 8be1927

func SQLNullString(s string) sql.NullString {

return sql.NullString{

String: s,

Valid: s != "",

}

}

marcelosalloum · 2025-06-24T14:40:51Z

internal/processors/state_change_builder.go

+// WithClaimableBalance sets the claimable balance ID
+func (b *StateChangeBuilder) WithClaimableBalance(balanceID string) *StateChangeBuilder {
+	b.base.ClaimableBalanceID = sql.NullString{String: balanceID, Valid: true}
+	return b
+}
+
+// WithLiquidityPool sets the liquidity pool ID
+func (b *StateChangeBuilder) WithLiquidityPool(poolID string) *StateChangeBuilder {
+	b.base.LiquidityPoolID = sql.NullString{String: poolID, Valid: true}
+	return b
+}


I can see you correctly set Valid: true on these ones ✔️. Still, the utils function can be applied here for simplification as well:

wallet-backend/internal/utils/sql.go

Lines 8 to 13 in 8be1927

func SQLNullString(s string) sql.NullString {

return sql.NullString{

String: s,

Valid: s != "",

}

}

marcelosalloum · 2025-06-24T14:42:32Z

internal/processors/token_transfer.go

+var ErrOperationNotFound = errors.New("operation not found")
+
+// TokenTransferProcessor processes Stellar transactions and extracts token transfer events.
+// It converts Stellar operations (payments, account merges, etc.) into standardized state changes.


💡 the word standardized seems to mean it's a kind of object that is standard across the Stellar ecosystem, which is not the case. I suggest updating the comment a bit:

Suggested change

// It converts Stellar operations (payments, account merges, etc.) into standardized state changes.

// It converts Stellar operations (payments, account merges, etc.) into our internal (synthetic) state changes representation.

JakeUrban · 2025-06-24T19:48:22Z

internal/processors/state_change_builder.go

+func (b *StateChangeBuilder) WithAsset(asset *asset.Asset, contractAddress string) *StateChangeBuilder {
+	if asset != nil {
+		if asset.GetNative() {
+			b.base.Token = sql.NullString{String: "native"}
+		} else if issuedAsset := asset.GetIssuedAsset(); issuedAsset != nil {
+			b.base.Token = sql.NullString{String: fmt.Sprintf("%s:%s", issuedAsset.GetAssetCode(), issuedAsset.GetIssuer())}
+		}
+	} else {
+		b.base.ContractID = sql.NullString{String: contractAddress}


I would advocate for always using the contract id for tokens when possible, even for native or classic assets. For that same reason, maybe we change the name of the function to withToken?

JakeUrban · 2025-06-24T19:49:17Z

internal/processors/state_change_builder.go

+type StateChangeBuilder struct {
+	base types.StateChange
+}


Other than the withAsset function, this builder class seems unnecessary and we can just use StateChange, wdyt?

I'm assuming there will be more fields to set when we support more state change types, in which case this would help make things more readable in the processors.

JakeUrban · 2025-06-24T19:53:46Z

internal/processors/token_transfer.go

+	// Process both fee events (transaction costs) and operation events (transfers, mints, etc.)
+	allEvents := append(txEvents.FeeEvents, txEvents.OperationEvents...)


From an ordering perspective, are we sure that we should have fee events proceed operation events? We may want to confirm with @karthikiyer56 or even a core engineer working on P23 how to think about this.

This is precisely why we break out free events into their own segment when reading events from the events from transaction level.
The coalescing happens at the event from ledger level, where I break down the use cases based on pre-protocol 23 and post protocol 23 behavior.

As to whether you need to order events in the chronological order, depends entirely on what you might do with them.
If you simply store them in a DB, for retrieval purposes in the future , then you are probably okay.
You might still want to take a look at the latest code in the protocol 23 branch of TTP, and if needed, emulate that behavior in your own logic here

JakeUrban · 2025-06-24T20:27:55Z

internal/processors/token_transfer.go

+			opID, opType, opSourceAccount, err = p.parseOperationDetails(tx, ledgerNumber, txIdx, opIdx)
+			if err != nil {
+				if errors.Is(err, ErrOperationNotFound) {
+					// Skip events for operations that couldn't be found
+					continue
+				}
+				return nil, fmt.Errorf("parsing operation details for transaction hash: %s, operation index: %d, err: %w", txHash, opIdx, err)
+			}


When would an operation not be found? Can't we assume that e.GetMeta().GetOperationIndex() will always return a valid index for the transaction from which e was produced? If so, then if this error is returned, it indicates a bug in our code, in which case we shouldn't fail silently and continue.

JakeUrban · 2025-06-24T20:32:11Z

internal/processors/token_transfer.go

+}
+
+// handleDefaultTransfer handles normal transfers and liquidity pool interactions.
+// For LP interactions, creates single state change with LP ID. For regular transfers, creates debit/credit pair.


I think we should clarify why we're handling transfers from/to LPs here, because its not obvious. Explicit deposit and withdraw operations are handled in handleTransfer(), but path payments could involve using an LP to swap assets.

JakeUrban · 2025-06-24T20:35:36Z

internal/processors/token_transfer.go

+// handleFee processes transaction fee events.
+// Positive amounts are debits (fee charged), negative amounts are credits (fee refunds).
+func (p *TokenTransferProcessor) handleFee(fee *ttp.Fee, contractAddress string, builder *StateChangeBuilder) ([]types.StateChange, error) {
+	amount := fee.GetAmount()
+	var category types.StateChangeCategory
+	var finalAmount string
+
+	// Negative fee amounts represent refunds (common in Soroban transactions)
+	if strings.HasPrefix(amount, "-") {
+		category = types.StateChangeCategoryCredit
+		// Store the refund as positive credit amount
+		finalAmount = strings.TrimPrefix(amount, "-")
+	} else {
+		// Positive fee amounts are normal transaction costs
+		category = types.StateChangeCategoryDebit
+		finalAmount = amount
+	}
+
+	change := builder.WithCategory(category).
+		WithAccount(fee.GetFrom()).
+		WithAmount(finalAmount).
+		WithAsset(fee.GetAsset(), contractAddress).
+		Build()
+
+	return []types.StateChange{change}, nil
+}


Have we decided whether we should consolidate fee events into a net fee? I know core breaks this down more granularly but I'm not familiar with the reasons why, and I think wallets are only interested in the net fee per transaction.

I wonder what can cause a negative fee (refund), it's the first time I see it 🤔

It's probably worth to follow @karthikiyer56's suggestion and consolidate it into a FeeChanges parameter that can be used to compute the resulting fee at the end.

marcelosalloum

This is looking pretty good!

My main suggestion is around reducing this scope a bit so we can get this PR merged sooner and then move forward to cases that require effects processing, like offers, path payments, and liquidity pools (already started in this PR).

marcelosalloum · 2025-06-24T22:16:44Z

internal/processors/token_transfer.go

+	if fromIsLP || toIsLP {
+		if fromIsLP {
+			// LP is sending tokens to account (e.g., path payment buying from LP)
+			change := p.createLiquidityPoolChange(types.StateChangeCategoryCredit, to, from, amount, asset, contractAddress, builder)
+			return []types.StateChange{change}, nil
+		}
+		// LP is receiving tokens from account (e.g., path payment selling to LP)
+		change := p.createLiquidityPoolChange(types.StateChangeCategoryDebit, from, to, amount, asset, contractAddress, builder)
+		return []types.StateChange{change}, nil
+	}


Considering it's not possible for both fromIsLP and toIsLP to be true, can we simplify things a bit?

Suggested change

if fromIsLP || toIsLP {

if fromIsLP {

// LP is sending tokens to account (e.g., path payment buying from LP)

change := p.createLiquidityPoolChange(types.StateChangeCategoryCredit, to, from, amount, asset, contractAddress, builder)

return []types.StateChange{change}, nil

}

// LP is receiving tokens from account (e.g., path payment selling to LP)

change := p.createLiquidityPoolChange(types.StateChangeCategoryDebit, from, to, amount, asset, contractAddress, builder)

return []types.StateChange{change}, nil

}

if fromIsLP {

// LP is sending tokens to account (e.g., path payment buying from LP)

change := p.createLiquidityPoolChange(types.StateChangeCategoryCredit, to, from, amount, asset, contractAddress, builder)

return []types.StateChange{change}, nil

} else if toIsLP {

// LP is receiving tokens from account (e.g., path payment selling to LP)

change := p.createLiquidityPoolChange(types.StateChangeCategoryDebit, from, to, amount, asset, contractAddress, builder)

return []types.StateChange{change}, nil

}

marcelosalloum · 2025-06-24T22:26:45Z

internal/processors/token_transfer.go

+// handleFee processes transaction fee events.
+// Positive amounts are debits (fee charged), negative amounts are credits (fee refunds).
+func (p *TokenTransferProcessor) handleFee(fee *ttp.Fee, contractAddress string, builder *StateChangeBuilder) ([]types.StateChange, error) {
+	amount := fee.GetAmount()
+	var category types.StateChangeCategory
+	var finalAmount string
+
+	// Negative fee amounts represent refunds (common in Soroban transactions)
+	if strings.HasPrefix(amount, "-") {
+		category = types.StateChangeCategoryCredit
+		// Store the refund as positive credit amount
+		finalAmount = strings.TrimPrefix(amount, "-")
+	} else {
+		// Positive fee amounts are normal transaction costs
+		category = types.StateChangeCategoryDebit
+		finalAmount = amount
+	}
+
+	change := builder.WithCategory(category).
+		WithAccount(fee.GetFrom()).
+		WithAmount(finalAmount).
+		WithAsset(fee.GetAsset(), contractAddress).
+		Build()
+
+	return []types.StateChange{change}, nil
+}


I wonder what can cause a negative fee (refund), it's the first time I see it 🤔

It's probably worth to follow @karthikiyer56's suggestion and consolidate it into a FeeChanges parameter that can be used to compute the resulting fee at the end.

marcelosalloum · 2025-06-24T22:31:49Z

internal/processors/token_transfer.go

+	case xdr.OperationTypeSetTrustLineFlags, xdr.OperationTypeAllowTrust:
+		// Skip events generated by these operations since they involve only an LP and Claimable Balance ID
+		return nil, nil
+


Operations that pertain to liquidity pools, path payments and offers can all generate additional effects: modify funds on offers and liquidity pools. That's not entirely covered in this PR though and I don't think it should.

I think we could disable those operation types for now (as you did for OperationTypeAllowTrust and OperationTypeSetTrustLineFlags), and open a dedicated PR to handle them after we wrap this one. That way you'll simplify this PR and move this complexity to the next one, where you can focus solely on these specific cases.

aditya1702 added 16 commits June 10, 2025 16:19

Add redis store

1242f34

Add contract store

be9dc3d

Add tests for contract and redis cache

b1ff2ff

Add contract cache mock

a5f3571

go mod tidy

022d771

fix "make check" issues

c57d61f

Remove HGetAll method

972dc69

update comment

c0b7bca

Add token transfer processor - 1

8f43670

use in-memory cache

d52a50d

remove health.go

7265cac

fix test

3b3f382

increase default expiration to 7 days

00cddc8

Merge remote-tracking branch 'upstream/main' into indexer-processors

622921a

Add liquidity pool ID to state change table

2333029

process token transfer events

19a245a

aditya1702 changed the base branch from main to indexer June 13, 2025 21:18

aditya1702 added 2 commits June 13, 2025 17:19

Merge remote-tracking branch 'upstream/main' into indexer

0df6658

Merge remote-tracking branch 'upstream/indexer' into indexer-processors

a7a604d

aditya1702 changed the title ~~Add Token Transfer Processor~~ [WIP] Add Token Transfer Processor Jun 13, 2025

marcelosalloum reviewed Jun 13, 2025

View reviewed changes

internal/db/migrations/2025-06-10.4-create_indexer_table_state_changes.sql Outdated Show resolved Hide resolved

internal/processors/token_transfer.go Outdated Show resolved Hide resolved

internal/processors/token_transfer.go Outdated Show resolved Hide resolved

aditya1702 added 9 commits June 14, 2025 19:09

add migration for contracts table

d313932

Add tests for contract

abf6e20

remove cascade option

5bf45bd

Use RunInTransaction to update contracts table in db

ead8dc4

Update cache + db functionality

db0ccc9

fix make check errors

d7d0b23

Add trigger for update_at and send error on cache initialization

9ef9569

Run GetByID in same txn as update/insert

837f908

Fix the concurrency test

2e17c0f

aditya1702 added 10 commits June 20, 2025 13:52

Create a new copy of objects

c32e67f

Add more tests

5705686

Create 2 state changes per Burn event

5e99514

deduplicate code

b2adcd0

make check

e8acc31

Update token_transfer.go

cb41f71

Move code to separate files

1320546

Add comments

e0708c1

Add comments and improve code readability

871f93b

Add tests for SEP-41 contract transfers

e6aff12

aditya1702 marked this pull request as ready for review June 23, 2025 18:36

aditya1702 changed the title ~~[WIP] Add Token Transfer Processor~~ Add Token Transfer Processor Jun 23, 2025

aditya1702 added 2 commits June 23, 2025 15:11

Generate 2 state changes per mint event

b058a43

Update token_transfer.go

da8053b

aditya1702 requested review from marcelosalloum and JakeUrban June 23, 2025 19:19

aditya1702 changed the title ~~Add Token Transfer Processor~~ Add token transfer processor Jun 23, 2025

convert "contract" to "tokenContract"

a52afec

marcelosalloum assigned aditya1702 Jun 24, 2025

marcelosalloum reviewed Jun 24, 2025

View reviewed changes

aditya1702 added 4 commits June 24, 2025 10:50

Add tests for data/contracts

64c9018

fix make check issues

0b2fd34

update metrics calculation

782418d

Fix tests

cb7105f

Base automatically changed from indexer to main June 24, 2025 15:43

aditya1702 added 3 commits June 24, 2025 15:34

Merge branch 'indexer' into indexer-processors

4c11f66

Merge remote-tracking branch 'upstream/main' into indexer-processors

8b6de7c

add op and txn details to error

af8bdc4

JakeUrban reviewed Jun 24, 2025

View reviewed changes

marcelosalloum reviewed Jun 24, 2025

View reviewed changes

	func SQLNullString(s string) sql.NullString {
	return sql.NullString{
	String: s,
	Valid: s != "",
	}
	}

	// It converts Stellar operations (payments, account merges, etc.) into standardized state changes.
	// It converts Stellar operations (payments, account merges, etc.) into our internal (synthetic) state changes representation.

		// Process both fee events (transaction costs) and operation events (transfers, mints, etc.)
		allEvents := append(txEvents.FeeEvents, txEvents.OperationEvents...)

Add token transfer processor #195

Are you sure you want to change the base?

Add token transfer processor #195

Uh oh!

Conversation

aditya1702 commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Known limitations

Issue that this PR addresses

Checklist

PR Structure

Thoroughness

Release

Uh oh!

marcelosalloum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marcelosalloum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcelosalloum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aditya1702 commented Jun 13, 2025 •

edited

Loading