Safeguard for many status requests

[sverhoeven]

In i-VRESSE/haddock3-webapp#110 the queued/running page refreshes. This causes a lot more calls to the /api/job/{id} endpoint. And even more when there a bunch of people waiting for the job to complete.

Getting the state of a job from a scheduler like slurm or dirac can be expensive.

We should benchmark how stress-full it is for those schedulers to have lets say a 100(0) users each with 1 job queued or running being checked every 10 seconds.

This could be resolved in different ways:

rate limit

Moves problem to consuming webapp

cache

It would be nice if the state of a job is cached for a configurable duration like 1 minute. So if job state was fetched 5 seconds ago and you fetch again you would get the 5 second old state. If you wait a minute and fetch again you will get a fresh state from the scheduler. This would need saving a last checked timestamp in the database.