We currently have a constant rate limit on the number of jobs to run per period. We can change this number, but it’s not depending on the backend capacity/load.

Nice to have:
To add automatic dynamic rate limiting for running background jobs, we need to interact more with the backend and monitor its current load and see if more jobs can be spawned per minute. Seems non-trivial, as the expected load per queued job is variable.

Submitted by Elmer van Chastelet on 9 October 2023 at 16:50