Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement gateway/node-optimization heuristic lib #138

Open
SgtPooki opened this issue Aug 31, 2023 · 0 comments
Open

feat: implement gateway/node-optimization heuristic lib #138

SgtPooki opened this issue Aug 31, 2023 · 0 comments
Labels
dif/hard Having worked on the specific codebase is important effort/days Estimated to take multiple days, but less than a week Epic kind/enhancement A net-new feature or improvement to an existing feature need/community-input Needs input from the wider community need/maintainers-input Needs input from the current maintainer(s) P1 High: Likely tackled by core team if no one steps up status/ready Ready to be worked

Comments

@SgtPooki
Copy link
Member

[...] create a library that builds gateway reputation over time (sliding window) and changes the order of gateways, prioritizing the one that responds the fastest. [...]

ps. this extra heuristic is not specific to IPLD Explorer, the ipfs-geoip used on Peers screen in webui would benefit from the same thing – currently it tries gateways in serial.
Perhaps worth extracting to standalone project and reusing in both places? (follow-up issue/pr)

I think we should create a library for this, as I've been thinking about this since I starting working with IPFS, but always in the context of a flat "gateway-racer" that queries given gateways in parallel.

It would be much more useful to the entire ecosystem if we built a library that queried various endpoints (kubo node, network via helia, gateways via trustless-fetch, IPNI routing, etc) and kept a weight-sorted array that auto-adjusted based on response timings and success frequencies. Each consumer fetching content have their own latencies from various gateways/endpoints/etc, as well as various spikes to popular services that may be overloaded at a particular time.

A library that automatically corrects to the fastest, most successful content provider would reduce load hot-spots throughout the IPFS network while improving every consumer's experience.

I think this would make sense both in the "helia-fetch" library we've discussed creating a number of times and as a plugin/addon/consumer of that library.

See ipfs/ipld-explorer-components#379 (comment) for more information about the origin of this issue.

cc @lidel @whizzzkid

@SgtPooki SgtPooki added the need/triage Needs initial labeling and prioritization label Aug 31, 2023
@github-project-automation github-project-automation bot moved this to Needs Grooming in IPFS-GUI (PL EngRes) Sep 1, 2023
@SgtPooki SgtPooki moved this from Needs Grooming to Planned / Backlog in IPFS-GUI (PL EngRes) Sep 1, 2023
@SgtPooki SgtPooki added kind/enhancement A net-new feature or improvement to an existing feature dif/hard Having worked on the specific codebase is important P1 High: Likely tackled by core team if no one steps up need/maintainers-input Needs input from the current maintainer(s) need/community-input Needs input from the wider community effort/days Estimated to take multiple days, but less than a week status/ready Ready to be worked Epic and removed need/triage Needs initial labeling and prioritization labels Sep 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dif/hard Having worked on the specific codebase is important effort/days Estimated to take multiple days, but less than a week Epic kind/enhancement A net-new feature or improvement to an existing feature need/community-input Needs input from the wider community need/maintainers-input Needs input from the current maintainer(s) P1 High: Likely tackled by core team if no one steps up status/ready Ready to be worked
Projects
No open projects
Status: Planned / Backlog
Development

No branches or pull requests

1 participant