Design and implement rate limiting policy

[yarikoptic]

• Slow response times from asset list endpoint #1891
  showed the need for means to either balance out or just to limit "heavy users" of the API server. A quick attempt is being worked on in
• Rate limit assets list endpoint for unauthenticated users #1899
  to provide a quick mitigation to that issue, but it is just a stop gap measure.

As many other public (or paid) services have rate limiting, we can and should have one too.

• it needs to be designed with current utilization patterns in mind to not cause unnecessary slow downs etc. May be it could be somehow made to be enforced only/when server experiencing high load etc. The point is that it should not unnecessarily complicate/negatively effect access to our services.
• well documented and described in the Terms & Polices
• dandi-cli should be ensured to continue working, gracefully handling rate limiting, and possibly alerting users about need to authenticate to gain better access, or consider composition of more efficient scripts, etc.

[yarikoptic]

Additional freshier references:

• ran into an article on medium https://medium.com/@mathur.danduprolu/throttle-traffic-to-your-django-app-implementing-a-rate-limiting-algorithm-afc58643ed4a ("Throttling Traffic to Your Django App: Implementing a Rate Limiting Algorithm") stating

Implementing rate limiting in Django is a critical step in ensuring the security, performance, and fair usage of your web application.

• Happen we want some unified reverse proxy ahead of all our components -- https://doc.traefik.io/traefik/ seems to be quite well known and has https://doc.traefik.io/traefik/middlewares/http/ratelimit/
• An "alternative" to rate limiting could be some kind of QoS, filed recently Prevent API service from becoming totally unresponsive/DoS #2085

@waxlamp what are you thoughts on re-approaching implementing rate limiting? It might not provide some ultimate solution to address all our issues, but it could add some guards.