Konstantina Skovola fca9078d6c Exponentially backoff when out of background workers
The scheduler detects the following three types of job failures:

1.Jobs that fail to launch (due to shortage of background workers)
2.Jobs that throw a runtime error
3.Jobs that crash due to a process crashing

In cases 2 and 3, additive backoff is applied in calculating the next
start time of a failed job.
In case 1 we previously retried to launch all jobs that failed to launch
simultaneously.

This commit introduces exponential backoff in case 1,
randomly selecting a wait time in [2, 2 + 2^f] seconds at microsecond granularity.
The aim is to reduce the collision probability for jobs that compete
for a background worker. The maximum backoff value is 1 minute.
It does not change the behavior for cases 2 and 3.

Fixes #4562
2022-09-01 15:13:38 +03:00
..
2022-08-17 20:18:09 -03:00
2022-01-24 14:12:56 +01:00
2022-01-24 14:12:56 +01:00
2022-08-25 10:38:03 +03:00
2020-05-18 20:16:03 -04:00
2020-05-18 20:16:03 -04:00
2022-08-02 11:38:32 +02:00