Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace single course grained filter with batch request of fine grained topic filters #12361

Closed
wants to merge 13 commits into from
Closed

Replace single course grained filter with batch request of fine grained topic filters #12361

wants to merge 13 commits into from
+867 −262

Commits on Apr 25, 2024

  1. Replace single course grained filter with batch request of fine grain… 

    …ed topic filters

This replaces Filter() with EthGetLogsReqs(), adding
fetchLogsByBlockHash, sendBatchRequests, and a handful of other helper
methods to support this

It also replaces these fields in logPoller struct:

lp.filter
lp.filterDirty
lp.cachedEventSigs
lp.cachedAddresses

With new fields, used for implementing caching and indexing of a collection of batch requests, insetad of a single filter:

lp.newFilters                  map[string]struct{}                    // Set of filter names which have been added since cached reqs indices were last rebuilt
lp.removedFilters              []Filter                               // Slice of filters which have been removed or replaced since cached reqs indices were last rebuilt
lp.cachedReqsByAddress         map[common.Address][]*GetLogsBatchElem // Index of cached GetLogs requests, by contract address
lp.cachedReqsByEventsTopicsKey map[string]*GetLogsBatchElem           // Index of cached GetLogs requests, by eventTopicsKey
    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    90196c9 View commit details
    Browse the repository at this point in the history

  2. Some refactoring & debugging 

    Trace -> Debug for BatchCallContext()
    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    226ad58 View commit details
    Browse the repository at this point in the history

  3. Pass fromBlock & toBlock as strings instead of *big.Int 

    Production rpc servers will fail if these are passed as *big.Int instead
of a string, so our SimultaedBackedClient was giving a false positive
PASS by handling the conversion. Must be converted in the code before calling BatchCallContext()
to work on live networks; now will also be required for simulated geth
    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    7999d65 View commit details
    Browse the repository at this point in the history

  4. Fix lint

    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    b38c957 View commit details
    Browse the repository at this point in the history

  5. addresses -> address

    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    51d1ad7 View commit details
    Browse the repository at this point in the history

  6. Don't ever keep trailing nils in outer topics slice 

    Instead of always being length 4, it needs to be the same length
(or less) than the number of topics emitted by logs it matches
    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    aea6fd1 View commit details
    Browse the repository at this point in the history

  7. Remove deletion of trailing nils from SimulatedBackendClient 

    This can cause false positives, since it's not done by production rpc servers
    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    75311a8 View commit details
    Browse the repository at this point in the history

  8. Update test

    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    697b8cb View commit details
    Browse the repository at this point in the history

  9. Don't skip nil topics if there are non-nil ones with a higher index

    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    ca70228 View commit details
    Browse the repository at this point in the history

  10. Clean up some things, and deep copy topics just in case

    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    6ffc705 View commit details
    Browse the repository at this point in the history

  11. debugging, slight improvement

    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    784965a View commit details
    Browse the repository at this point in the history

  12. Finally found the bug! 

    There's a case where Keepers registers a filter with one name, then
unregisters it, then re-registers a filter with the same name. There was
an optimization where instead of computing the hash of the events &
topics for a filter in two different for loops, it saves it the first time
and looks it up in a map the second time. But the keys for the map were
the filter names. The first loop deals with removed filters and the
second with new filters. They both had the same filter name, but
different addresses & topics.  This might be a bug in Keepers, but I
can at least make this code more robust since it should be possible to
update a filter's topics & addresses but keep the name the same.
    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    1b6d8c0 View commit details
    Browse the repository at this point in the history

  13. Remove debug statements

    @reductionista
    reductionista committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    8d8ab99 View commit details
    Browse the repository at this point in the history