[PROD] Attestation is not included in any AggregateAndProof

[twoeths]

Describe the bug

Validator 270124 submitted timely attestation with correct data:

May-22 21:36:27.524[]                ^[[34mdebug^[[39m: Signed attestation slot=3869280, index=11, head=0xe9469bca16c51752f43a15e59ef19102b528e36283da7da764a474fa0c5698e9, validatorIndex=270114
May-22 21:36:27.525[]                ^[[34mdebug^[[39m: Signed attestation slot=3869280, index=4, head=0xe9469bca16c51752f43a15e59ef19102b528e36283da7da764a474fa0c5698e9, validatorIndex=270124

the attestation were sent to 15 peers

May-22 21:36:28.062[VMON]            ^[[34mdebug^[[39m: Local validator published unaggregated attestation validatorIndex=270114, slot=3869280, committeeIndex=11, subnet=11, sentPeers=12, delaySec=0.8840000629425049
May-22 21:36:28.062[VMON]            ^[[34mdebug^[[39m: Local validator published unaggregated attestation validatorIndex=270124, slot=3869280, committeeIndex=4, subnet=4, sentPeers=15, delaySec=0.8840000629425049

however it's not included in any AggregateAndProof and became missed

May-22 21:49:07.209[VMON]            ^[[34mdebug^[[39m: Failed attestation in previous epoch validatorIndex=270124, prevEpoch=120915, isPrevSourceAttester=false, isPrevHeadAttester=false, isPrevTargetAttester=false, inclusionDistance=null

note that validator 270114 submitted attestation at same slot and same time but it was not missed

Expected behavior

attestation is not missed

[twoeths]

it's noticed that the delaySec is 0.88s, and attestation of validator 270114 was not included in other AggregateAndProof as well, it's not missed because it's an aggregator in that committee

May-22 21:36:31.227[VMON]            ^[[34mdebug^[[39m: Local validator published aggregated attestation validatorIndex=270114, slot=3869280, committeeIndex=11, sentPeers=50, delaySec=0.04999995231628418

the reason of almost 1s delay is probably due to the I/O lag in #4002, it takes almost 0.5s to get a response from our beacon node

May-22 21:36:27.008[]                ^[[34mdebug^[[39m: HttpClient request routeId=produceAttestationData
May-22 21:36:27.277[]              ^[[36mverbose^[[39m: Found new chain head slot=3869280, head=0xe9469bca16c51752f43a15e59ef19102b528e36283da7da764a474fa0c5698e9, previouDuty=0x040b4a15cd4fbeeda4ff7695a7ede488770eeb97a0b87e7353f970fb3e404cdc, currentDuty=0xa1a1dcbb9951c94bb63a6a0b4b595e001b93b2527b2c28013a5635935008ca80
May-22 21:36:27.517[]                ^[[34mdebug^[[39m: HttpClient response routeId=produceAttestationData
May-22 21:36:27.524[]                ^[[34mdebug^[[39m: Signed attestation slot=3869280, index=11, head=0xe9469bca16c51752f43a15e59ef19102b528e36283da7da764a474fa0c5698e9, validatorIndex=270114
May-22 21:36:27.525[]                ^[[34mdebug^[[39m: Signed attestation slot=3869280, index=4, head=0xe9469bca16c51752f43a15e59ef19102b528e36283da7da764a474fa0c5698e9, validatorIndex=270124
May-22 21:36:27.529[]                ^[[34mdebug^[[39m: HttpClient request routeId=submitPoolAttestations
May-22 21:36:28.064[]                ^[[34mdebug^[[39m: HttpClient response routeId=submitPoolAttestations

it takes 0.5s for produceAttestationData request to reach beacon node, and 0.38s for submitPoolAttestations request to reach beacon node

May-22 21:36:27.516[API]             ^[[34mdebug^[[39m: Req req-129x 127.0.0.1 produceAttestationData
May-22 21:36:27.517[API]             ^[[34mdebug^[[39m: Res req-129x produceAttestationData - 200

May-22 21:36:27.884[API]             ^[[34mdebug^[[39m: Req req-129y 127.0.0.1 submitPoolAttestations

normally it takes almost no time to get an attestation data

May-22 21:37:00.442[]                ^[[34mdebug^[[39m: HttpClient request routeId=produceAttestationData
May-22 21:37:00.449[]                ^[[34mdebug^[[39m: HttpClient response routeId=produceAttestationData

also note that 3869280 is an epoch transition slot

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.