tap-agent taking too long to start up

[gusinacio]

Problem statement

Indexers with high performance create tons of receipts and have multiple allocations. We need to speed up start-up by parallelizing the SenderAllocation creation and also look at indexes in our database that may be affecting the initialization.

This can be done by spawning tasks for the initialization for SenderAccounts and SenderAllocations.

The code is currently sync:

• indexer-rs/tap-agent/src/agent/sender_accounts_manager.rs

  Lines 149 to 152 in 6f24558

  	state.sender_ids.insert(sender_id);
  	state
  	.create_or_deny_sender(myself.get_cell(), sender_id, allocation_ids)
  	.await;

• indexer-rs/tap-agent/src/agent/sender_account.rs

  Lines 487 to 490 in 6f24558

  	// Create a sender allocation for each allocation
  	state
  	.create_sender_allocation(myself.clone(), *allocation_id)
  	.await?;

This is just a suggestion. If you find any other performance improvement opportunity, feel free to add to the PR as well.

There are indexers that have around 600 allocations open. It would be good to have a benchmark for this.

Also we need to create some indexes for ravs

[YashBit]

@gusinacio Please assign this to me.

[YashBit]

@gusinacio I have a few questions:

1. Which library would you recommend for parallelisation?

There are indexers that have around 600 allocations open. It would be good to have a benchmark for this.

Could you please elaborate on this?

This can be done by spawning tasks for the initialization for SenderAccounts and SenderAllocations.

Parallel tasks?

Also we need to create some indexes for ravs

Please can you elaborate on this?

Sorry, I am new to indexer-rs

[gusinacio]

@gusinacio I have a few questions:

1. Which library would you recommend for parallelization?

I think just using tokio::spawn is enough to speed up. Right now one can only be created after the previous one was created. Maybe you could spawn all the tasks and loop over and join them using JoinSet.

There are indexers that have around 600 allocations open. It would be good to have a benchmark for this.

Could you please elaborate on this?

Maybe this is a different PR related to #8. Don't worry about this for now.

This can be done by spawning tasks for the initialization for SenderAccounts and SenderAllocations.

Parallel tasks?

Yeah, using tokio::spawn and joining all of them before proceeding.

Also we need to create some indexes for ravs

Please can you elaborate on this?

Actually, you don't need to worry about this, we need to create indexes inside the database but this is in graphprotocol/indexer and not here.

[gusinacio]

Hey, are you still planning to do this?

[YashBit]

@gusinacio Yes. I have a FT job, I mostly start issues on the weekends.

Is that okay?

[gusinacio]

Hey, I have to take this one because it became a higher priority.