services: Implement call to `getLedgers` and start parsing the `xdr.LedgerCloseMeta` #197

marcelosalloum · 2025-06-18T23:40:07Z

What

Update the indexer code to stream from {RPC}.getLedgers and break down the results from ledgerRange -> ledgers -> transactions.

Also, (pseudo) process the ledgers in parallel.

Why

As part of the new indexer implementation.

Future Work

evaluate transactions, filter by participant account & store in the DB
evaluate operations, filter by participant account & store in the DB
generate state changes
unlock channel accounts

I haven't added many tests to the ingest service yet, since this code will still evolve quite a bit.

Issue that this PR addresses

Partially address #128

Checklist

PR Structure

It is not possible to break this PR down into smaller PRs.
This PR does not mix refactoring changes with feature changes.
This PR's title starts with name of package that is most changed in the PR, or all if the changes are broad or impact many packages.

Thoroughness

This PR adds tests for the new functionality or fixes.
All updated queries have been tested (refer to this check if the data set returned by the updated query is expected to be same as the original one).

Release

This is not a breaking change.
This is ready to be tested in development.
The new functionality is gated with a feature flag if this is not ready for production.

… a new IngestStoreModel, since it relates to a different database table.

… used by the stellar-rpc package dependency

…xecuted by a dbTx.

…e fetched from the RPC, based on the RPC available ledgers and the local "curstor"

…d limit, since this is how the API is meant to be used.

… transactions.

Copilot

Pull Request Overview

This PR implements changes for processing ledgers in the new indexer service. Key updates include the addition of a sqlExec parameter to several functions for improved transactional control, modifications to the rpcService error messages, and a new ingestion flow with an advisory lock and parallel ledger processing.

Reviewed Changes

Copilot reviewed 20 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
internal/signing/store/types.go	Added sqlExec parameter to UnassignTxAndUnlockChannelAccounts for transaction consistency.
internal/signing/store/mocks.go	Updated mock signature for the new sqlExec parameter.
internal/services/rpc_service*.go	Modified getLedgers interface and error messages for clarity.
internal/services/ingest.go	Introduced a new ingestion Run function with advisory locking and a worker pool for parallel ledger processing; adjusted metrics updates.
internal/ingest/ingest.go	Deprecated the old Run function and updated logging messages.
internal/indexer/types/types.go	Changed LedgerNumber type from int64 to uint32.
internal/data/models.go & ingest_store*	Refactored ledger synced cursor logic by moving it from PaymentModel to IngestStoreModel.
go.mod & .golangci.yml	Added new dependencies and refined linter exclusion patterns.

Comments suppressed due to low confidence (1)

internal/signing/store/types.go:26

Introducing the sqlExec parameter improves transactional control; ensure that all callers are updated accordingly to pass the correct SQL executer.

	UnassignTxAndUnlockChannelAccounts(ctx context.Context, sqlExec db.SQLExecuter, txHashes ...string) (int64, error)

internal/services/ingest.go

marcelosalloum · 2025-06-19T00:17:48Z

internal/services/ingest.go

+	mu := sync.Mutex{}
+	var allTxHashes []string
+	var errs []error
+
+	// Submit tasks to the pool
+	for _, ledger := range getLedgersResponse.Ledgers {
+		ledger := ledger // Create a new variable to avoid closure issues
+		pool.Submit(func() {
+			txHashes, err := m.processLedger(ctx, ledger)
+			mu.Lock()
+			if err != nil {


I plan to use toid to order the transactions in the database. It's calculated like this.

…can be executed by a dbTx." This reverts commit c3d0372.

aditya1702 · 2025-06-24T16:15:03Z

internal/ingest/ingest.go

@@ -38,7 +38,7 @@ func Ingest(cfg Configs) error {
 	}

 	if err = ingestService.Run(ctx, uint32(cfg.StartLedger), uint32(cfg.EndLedger)); err != nil {
-		log.Ctx(ctx).Fatalf("Running ingest from %d to %d: %v", cfg.StartLedger, cfg.EndLedger, err)
+		log.Ctx(ctx).Fatalf("Error running 'ingest' from %d to %d: %v", cfg.StartLedger, cfg.EndLedger, err)


small nit

Suggested change

log.Ctx(ctx).Fatalf("Error running 'ingest' from %d to %d: %v", cfg.StartLedger, cfg.EndLedger, err)

log.Ctx(ctx).Fatalf("running 'ingest' from %d to %d: %v", cfg.StartLedger, cfg.EndLedger, err)

internal/services/ingest.go

aditya1702 · 2025-06-24T18:59:25Z

internal/services/ingest.go

+func (m *ingestService) fetchNextLedgersBatch(ctx context.Context, startLedger uint32) (GetLedgersResponse, error) {
+	rpcHealth, err := m.rpcService.GetHealth()
+	if err != nil {
+		return GetLedgersResponse{}, fmt.Errorf("getting rpc health: %w", err)


In cases where RPC becomes unhealthy for a few mins and then healthy again, this would stop the ingestion and terminate right? How about we terminate ingestion process only if RPC stays unhealthy for a certain threshold. Otherwise we keep letting it try and fetch the ledgers batch until then.

Hm, good point! I'll work on that 👍

aditya1702 · 2025-06-24T19:09:42Z

internal/services/ingest.go

+	if latestLedgerSynced >= rpcNewestLedger {
+		return LedgerSeqRange{}, true
+	}
+	ledgerRange.StartLedger = max(latestLedgerSynced+1, rpcOldestLedger)


If we do a max() here would that mean if wallet-backend is at 5 and rpc's oldest ledger is at 8, it would skip ingesting 6, 7? If we are ingesting using GetLedgers by getting ledgers in bulk, do we want to allow WB catch up to RPC instead of skipping ledgers?

If we do a max() here would that mean if wallet-backend is at 5 and rpc's oldest ledger is at 8, it would skip ingesting 6, 7?

I think we don't have another option here, right? If we have [oldestLedger: 8, newestLedger: 20], we can only ingest within that window, as the RPC doesn't have anything older than ledger 8 available for ingestion.

aditya1702 · 2025-06-24T19:11:59Z

internal/services/ingest.go

+
+func (m *ingestService) processLedgerResponse(ctx context.Context, getLedgersResponse GetLedgersResponse) error {
+	// Create a worker pool with
+	const poolSize = 16


nit: lets move this with other constants at the top

aditya1702 · 2025-06-24T19:18:03Z

internal/services/ingest.go

+	log.Ctx(ctx).Debugf("🚧 Got %d transactions for ledger %d", len(transactions), xdrLedgerCloseMeta.LedgerSequence())
+	txHashes := make([]string, 0, len(transactions))
+	for _, tx := range transactions {
+		txHashes = append(txHashes, tx.Hash.HexString())


Since this is still in TODO, I am assuming this will return a list of types.Transaction structs instead of tx hash strings? Because what would be the use of the tx hashes?

Yes, you're right! I'm returning txHashes because I'd like to log something short. But the output will definitely be actual objects!

marcelosalloum added 13 commits June 18, 2025 12:04

Install stellar-rpc.

eee53fa

Migrate the latest ledger synched tracker from the PaymentsModel into…

002f77a

… a new IngestStoreModel, since it relates to a different database table.

Change the LedgerNumber from int64 to uint32, which is the same being…

4b53943

… used by the stellar-rpc package dependency

Implement the rpcService.getLedgers.

c8b7c7d

Implement the AcquireAdvisoryLock and ReleaseAdvisoryLock functions.

b59f92a

Update the UnassignTxAndUnlockChannelAccounts method so it can be e…

c3d0372

…xecuted by a dbTx.

Implement the getLedgerSeqRange to calculate the Ledgers range to b…

3ce48f3

…e fetched from the RPC, based on the RPC available ledgers and the local "curstor"

Instead of calculating startLedger and lastLedger, use startLedger an…

e92b73f

…d limit, since this is how the API is meant to be used.

Temporarily rename Run to DeprecatedRun

f3e4097

Start streaming the Ledgers and logging the start/end ledgers range

78ca141

storing network passphrase in the ingest service.

fd49b04

Fix ticker logic

82eb1f1

Start breaking down ledger ranges into individual ledgers and then to…

35d3c4b

… transactions.

marcelosalloum self-assigned this Jun 18, 2025

marcelosalloum requested a review from Copilot June 18, 2025 23:46

Copilot AI reviewed Jun 18, 2025

View reviewed changes

internal/services/ingest.go Show resolved Hide resolved

internal/services/ingest.go Outdated Show resolved Hide resolved

Address Copilot code review suggestions.

ca3bb92

marcelosalloum commented Jun 19, 2025

View reviewed changes

marcelosalloum marked this pull request as ready for review June 19, 2025 00:18

marcelosalloum changed the title ~~[WIP] services: Implement call to getLedgers and start parsing the xdr.LedgerCloseMeta~~ services: Implement call to getLedgers and start parsing the xdr.LedgerCloseMeta Jun 19, 2025

marcelosalloum requested a review from aditya1702 June 19, 2025 00:18

marcelosalloum added 2 commits June 20, 2025 14:54

Add unit tests for getLedgerTransactions.

e88ec56

Revert "Update the UnassignTxAndUnlockChannelAccounts method so it …

11f4205

…can be executed by a dbTx." This reverts commit c3d0372.

marcelosalloum changed the title ~~services: Implement call to getLedgers and start parsing the xdr.LedgerCloseMeta~~ services: Implement call to getLedgers and start parsing the xdr.LedgerCloseMeta Jun 23, 2025

marcelosalloum mentioned this pull request Jun 23, 2025

State Change Indexer: Create Ingestion Pipeline #128

Open

marcelosalloum added this to the Wallet Backend Indexer milestone Jun 23, 2025

aditya1702 removed this from the Wallet Backend Indexer milestone Jun 23, 2025

aditya1702 reviewed Jun 24, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

services: Implement call to `getLedgers` and start parsing the `xdr.LedgerCloseMeta` #197

services: Implement call to `getLedgers` and start parsing the `xdr.LedgerCloseMeta` #197

marcelosalloum commented Jun 18, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

marcelosalloum Jun 19, 2025

Uh oh!

aditya1702 Jun 24, 2025

Uh oh!

Uh oh!

aditya1702 Jun 24, 2025

Uh oh!

marcelosalloum Jun 24, 2025

Uh oh!

aditya1702 Jun 24, 2025

Uh oh!

marcelosalloum Jun 24, 2025

Uh oh!

aditya1702 Jun 24, 2025

Uh oh!

aditya1702 Jun 24, 2025

Uh oh!

marcelosalloum Jun 24, 2025

Uh oh!

Uh oh!

	log.Ctx(ctx).Fatalf("Error running 'ingest' from %d to %d: %v", cfg.StartLedger, cfg.EndLedger, err)
	log.Ctx(ctx).Fatalf("running 'ingest' from %d to %d: %v", cfg.StartLedger, cfg.EndLedger, err)

services: Implement call to getLedgers and start parsing the xdr.LedgerCloseMeta #197

Are you sure you want to change the base?

services: Implement call to getLedgers and start parsing the xdr.LedgerCloseMeta #197

Conversation

marcelosalloum commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Future Work

Issue that this PR addresses

Checklist

PR Structure

Thoroughness

Release

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

services: Implement call to `getLedgers` and start parsing the `xdr.LedgerCloseMeta` #197

services: Implement call to `getLedgers` and start parsing the `xdr.LedgerCloseMeta` #197

marcelosalloum commented Jun 18, 2025 •

edited

Loading