Konstantina Skovola
fca9078d6c
Exponentially backoff when out of background workers
...
The scheduler detects the following three types of job failures:
1.Jobs that fail to launch (due to shortage of background workers)
2.Jobs that throw a runtime error
3.Jobs that crash due to a process crashing
In cases 2 and 3, additive backoff is applied in calculating the next
start time of a failed job.
In case 1 we previously retried to launch all jobs that failed to launch
simultaneously.
This commit introduces exponential backoff in case 1,
randomly selecting a wait time in [2, 2 + 2^f] seconds at microsecond granularity.
The aim is to reduce the collision probability for jobs that compete
for a background worker. The maximum backoff value is 1 minute.
It does not change the behavior for cases 2 and 3.
Fixes #4562
2022-09-01 15:13:38 +03:00
..
2022-08-25 10:38:03 +03:00
2022-09-01 15:13:38 +03:00
2022-08-30 18:32:30 +02:00
2022-08-25 10:38:03 +03:00
2022-08-25 10:38:03 +03:00
2022-08-25 10:38:03 +03:00
2022-08-25 10:38:03 +03:00
2022-08-25 15:29:28 +03:00
2022-08-25 15:29:28 +03:00
2022-05-06 14:30:00 -03:00
2022-08-25 15:29:28 +03:00
2022-08-25 10:38:03 +03:00
2022-08-25 10:38:03 +03:00
2022-08-17 10:23:40 +03:00
2022-08-17 10:23:40 +03:00
2022-08-17 10:23:40 +03:00
2022-08-17 10:23:40 +03:00
2022-08-25 17:49:09 -03:00
2022-08-25 17:49:09 -03:00
2022-08-25 17:49:09 -03:00
2022-08-25 17:49:09 -03:00
2022-08-25 15:29:28 +03:00
2022-08-12 14:30:10 -03:00
2022-01-24 14:12:56 +01:00
2022-08-25 10:38:03 +03:00
2022-08-25 15:29:28 +03:00
2022-08-25 15:29:28 +03:00
2022-08-25 15:29:28 +03:00
2022-08-25 15:29:28 +03:00
2022-01-24 14:12:56 +01:00
2022-05-18 11:38:58 -03:00
2022-05-18 11:38:58 -03:00
2022-05-18 11:38:58 -03:00
2022-08-25 15:29:28 +03:00
2022-01-24 14:12:56 +01:00
2022-08-02 11:38:32 +02:00
2022-08-17 10:23:40 +03:00
2022-08-24 17:14:36 -04:00
2022-08-25 10:38:03 +03:00
2022-04-18 21:46:50 +05:30
2022-08-25 15:29:28 +03:00
2022-07-27 17:19:01 +02:00
2022-09-01 07:31:07 +05:30
2022-04-26 17:24:02 +02:00
2022-04-26 17:24:02 +02:00
2022-04-26 17:24:02 +02:00
2022-06-03 11:41:01 +03:00
2022-08-15 13:34:00 +02:00
2022-08-25 15:29:28 +03:00
2022-08-25 15:29:28 +03:00
2022-06-21 11:53:18 +02:00
2022-08-18 09:32:21 -04:00
2021-10-12 23:28:57 +03:00
2022-08-25 15:29:28 +03:00
2021-10-29 16:15:58 +03:00
2022-08-25 10:38:03 +03:00
2022-08-17 10:30:13 +02:00
2022-08-29 17:44:16 +03:00
2022-08-25 15:29:28 +03:00
2022-08-25 15:29:28 +03:00
2022-08-25 15:29:28 +03:00
2022-08-18 09:32:21 -04:00
2022-08-18 09:32:21 -04:00
2022-08-18 09:32:21 -04:00
2022-08-15 13:34:00 +02:00
2022-05-06 22:13:17 +05:30
2022-08-17 10:30:13 +02:00
2022-07-29 11:40:00 +02:00
2022-08-17 10:30:13 +02:00
2022-07-06 20:09:09 +05:30
2022-08-17 10:30:13 +02:00
2022-05-06 14:30:00 -03:00
2022-02-18 17:35:24 +03:00
2022-08-25 12:59:05 +02:00
2022-02-18 17:35:24 +03:00
2022-05-18 11:38:58 -03:00
2022-05-18 11:38:58 -03:00
2022-05-18 11:38:58 -03:00
2022-06-07 13:10:28 +02:00
2022-05-19 17:37:09 +02:00
2022-06-23 11:14:40 +03:00
2022-03-17 17:46:01 +01:00
2022-03-17 17:46:01 +01:00
2022-03-17 17:46:01 +01:00
2022-08-17 10:30:13 +02:00
2022-01-17 17:36:33 +01:00
2022-02-16 20:50:01 +03:00
2022-08-18 09:32:21 -04:00
2022-08-18 09:32:21 -04:00
2022-08-18 09:32:21 -04:00
2022-04-22 11:06:52 +02:00
2021-10-25 18:20:11 +03:00
2021-12-20 10:02:57 +01:00
2022-08-17 10:30:13 +02:00
2022-08-25 13:17:21 -04:00
2022-08-25 13:17:21 -04:00
2022-08-25 13:17:21 -04:00
2022-03-17 17:46:01 +01:00
2022-08-25 13:17:21 -04:00
2022-08-25 13:17:21 -04:00
2022-08-25 13:17:21 -04:00
2022-08-25 10:38:03 +03:00