Konstantina Skovola
fca9078d6c
Exponentially backoff when out of background workers
...
The scheduler detects the following three types of job failures:
1.Jobs that fail to launch (due to shortage of background workers)
2.Jobs that throw a runtime error
3.Jobs that crash due to a process crashing
In cases 2 and 3, additive backoff is applied in calculating the next
start time of a failed job.
In case 1 we previously retried to launch all jobs that failed to launch
simultaneously.
This commit introduces exponential backoff in case 1,
randomly selecting a wait time in [2, 2 + 2^f] seconds at microsecond granularity.
The aim is to reduce the collision probability for jobs that compete
for a background worker. The maximum backoff value is 1 minute.
It does not change the behavior for cases 2 and 3.
Fixes #4562
2022-09-01 15:13:38 +03:00
..
2022-07-21 23:10:45 +05:30
2022-08-25 17:49:09 -03:00
2022-08-17 20:18:09 -03:00
2022-08-25 10:38:03 +03:00
2022-09-01 15:13:38 +03:00
2022-08-30 18:32:30 +02:00
2022-06-06 16:22:22 +03:00
2022-01-24 14:12:56 +01:00
2022-01-24 14:12:56 +01:00
2022-01-24 14:12:56 +01:00
2022-01-24 14:12:56 +01:00
2022-01-24 14:12:56 +01:00
2022-05-06 14:30:00 -03:00
2022-08-25 15:29:28 +03:00
2022-08-25 10:38:03 +03:00
2022-08-25 10:38:03 +03:00
2022-08-17 10:23:40 +03:00
2022-01-24 14:12:56 +01:00
2022-08-25 17:49:09 -03:00
2022-08-25 17:49:09 -03:00
2022-08-25 17:49:09 -03:00
2022-08-25 17:49:09 -03:00
2022-08-25 15:29:28 +03:00
2022-08-12 14:30:10 -03:00
2022-01-24 14:12:56 +01:00
2022-08-16 00:16:36 +02:00
2022-08-25 15:29:28 +03:00
2022-01-24 14:12:56 +01:00
2022-01-24 14:12:56 +01:00
2022-01-24 14:12:56 +01:00
2022-08-25 15:29:28 +03:00
2022-01-24 14:12:56 +01:00
2022-07-20 12:31:26 +05:30
2022-08-17 10:23:40 +03:00
2022-08-24 17:14:36 -04:00
2022-08-25 17:49:09 -03:00
2022-03-03 12:21:07 +01:00
2020-05-18 20:16:03 -04:00
2022-04-18 21:46:50 +05:30
2020-05-18 20:16:03 -04:00
2021-12-17 10:51:33 -05:00
2022-07-27 17:19:01 +02:00
2022-09-01 07:31:07 +05:30
2021-03-01 06:52:36 +01:00
2022-04-26 17:24:02 +02:00
2021-05-24 18:03:47 -04:00
2021-06-25 17:33:33 +02:00
2019-10-29 19:02:58 -04:00
2022-08-15 13:34:00 +02:00
2022-08-25 15:29:28 +03:00
2022-08-25 15:29:28 +03:00
2021-03-09 09:04:09 +01:00
2022-06-21 11:53:18 +02:00
2022-08-02 11:38:32 +02:00
2021-10-12 23:28:57 +03:00
2022-01-18 14:05:52 +05:30
2020-08-29 23:15:20 +02:00
2021-03-09 09:04:09 +01:00
2021-03-09 09:04:09 +01:00
2021-10-29 16:15:58 +03:00
2022-08-17 10:30:13 +02:00
2022-08-17 10:30:13 +02:00
2022-08-29 17:44:16 +03:00
2022-02-07 10:31:46 +03:00
2022-08-17 10:30:13 +02:00
2022-05-18 22:14:50 +03:00
2021-12-10 14:40:34 +03:00
2022-08-17 10:30:13 +02:00
2022-07-29 11:40:00 +02:00
2022-08-17 10:30:13 +02:00
2022-07-06 20:09:09 +05:30
2022-08-17 10:30:13 +02:00
2022-02-18 17:35:24 +03:00
2022-02-18 17:35:24 +03:00
2022-08-25 12:59:05 +02:00
2022-02-18 17:35:24 +03:00
2020-04-14 23:12:15 +02:00
2020-09-30 15:14:17 +03:00
2022-04-06 12:41:14 +02:00
2022-05-19 17:37:09 +02:00
2022-06-23 11:14:40 +03:00
2021-10-25 12:02:03 +02:00
2022-08-17 10:30:13 +02:00
2022-01-17 17:36:33 +01:00
2022-02-16 20:50:01 +03:00
2022-05-03 19:26:40 +05:30
2020-05-27 17:31:09 +02:00
2020-05-27 17:31:09 +02:00
2021-12-09 20:44:07 +05:30
2021-07-02 16:43:36 +03:00
2021-12-20 10:02:57 +01:00
2021-04-10 00:10:09 +02:00
2022-08-17 10:30:13 +02:00
2021-10-09 00:15:23 +02:00
2021-05-19 19:02:28 +02:00
2021-10-09 00:15:23 +02:00
2022-03-03 12:21:07 +01:00
2019-04-26 13:08:00 -04:00