250 Commits

Author SHA1 Message Date
Markus Pilman
d4ee7be1d7 Reduce excessive tracing and fail after 1M traces 2022-04-19 21:11:51 -06:00
Ata E Husain Bohra
933e5bbd2e
EncryptKeyProxy server APIs for simulation runs. (#6727)
* EncryptKeyProxy server APIs for simulation runs.

Description

  diff-2: FlowSingleton util class
              Bug fixes
  diff-1: Expected errors returned to the caller

Major changes proposed are:
1. EncryptKeyProxy server APIs:
 1.1. Lookup Cipher details via BaseCipherId
 1.2. Lookup latest Cipher details via encryption domainId.
2. EncyrptKeyProxy implements caches indexed by: baseCipherId &
   encyrptDomainId
3. Periodic task to refresh domainId indexed cache to support
   'limiting cipher lifetime' abilities if supported by
   external KMS solutions.

Testing

EncyrptKeyProxyTest workload to validate the newly added code.
2022-04-11 09:08:42 -07:00
Yi Wu
994b8c92f8
Add option to limit resident memory and remove default memory limit (#6719)
Changing `memory` option to limit resident memory instead of virtual memory, in config file and fdbserver/fdbbackup/fdbcli command-line argument. Since `rlimit` doesn't support limiting virtual memory, the current implementation have both of fdbmonitor and the fdbserver/fdbbackup process checking process RSS periodically and kill and restart the process if the limit is exceeded.

Adding a new `memory_vsize` option to limit virtual memory, if backward-compatible behavior is desired.

closes #6671, closes #6672
2022-04-06 20:06:24 -07:00
Renxuan Wang
465ff712b6
Move Hostname to its own files. (#6759)
* Change DNS cache to use std::map.

Revert commit 90c259d84e95dd35e01149c0a86bd18e82e33930, because if we use unordered_map, toString() can be inconsistent.

* Move ClientKnob::COORDINATOR_HOSTNAME_RESOLVE_DELAY to FlowKnob::HOSTNAME_RESOLVE_DELAY.

* Move Hostname to its own files.

Also, add resolve-related variables and functions in Hostname.
2022-04-04 19:04:51 -07:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Xiaoge Su
99b030c2f6 Allow the TOML file assign knobs during test
In this patch, for a given test, it is possible to override the knob
values, e.g.

[[test]]

    [[test.knobs]]
    watch_timeout = 999

will set the client knob WATCH_TIMEOUT to 999 during the test. The
original value will be recovered after the test is over.
2022-03-17 16:53:10 -07:00
sfc-gh-tclinkenbeard
d01f035848 Increase ITERATIONS_PER_REACTOR_CHECK default 2022-02-17 13:45:17 -08:00
sfc-gh-tclinkenbeard
a32547cbbe Merge remote-tracking branch 'origin/main' into flow-enhancements 2022-02-15 15:56:42 -08:00
Ray Jenkins
e1ae0e03b9 Update flow/Knobs.cpp
Co-authored-by: Lukas Joswiak <lukas.joswiak@snowflake.com>
2022-02-07 15:49:35 -08:00
Ray Jenkins
a5681c52bf Force loopback addr when in simulation mode. 2022-02-07 15:49:35 -08:00
Ray Jenkins
226236600e Preserving original LISTENER_PORT knob and adding new Knob TRACING_UDB_LISTENER_ADDR 2022-02-07 15:49:35 -08:00
Ray Jenkins
1bd084810c Refactor TRACING_UDP_LISTENER_PORT to TRACING_UDP_LISTENER_ADDR. 2022-02-07 15:49:35 -08:00
Steven Li
36c6e23700
Add TraceEvent severity knob (#6326)
* add knob for trace event severity

* add knob for TraceEvent severity

* fix format

* fix switch format

* moved intToSeverity call inside __test initialization

* updated knob name

* fix line length format

* fix format

* git clang-format
2022-02-02 11:36:17 -08:00
sfc-gh-tclinkenbeard
8076806e76 Add ITERATIONS_PER_REACTOR_CHECK 2022-01-01 20:14:57 -08:00
Steve Atherton
035e0d6e52
Merge branch 'master' into bit-flipping-workload 2021-11-16 14:42:22 -08:00
Steve Atherton
508429f30d
Redwood chunked file growth and low priority IO starvation prevention (#5936)
* Redwood files now growth in large page chunks controlled by a knob to reduce truncate() calls for expansion.   PriorityMultiLock has limit on consecutive same-priority lock release.  Increased Redwood max priority level to 3 for more separation at higher BTree levels.

* Simulation fix, don't mark certain IO timeout errors as injected unless the simulated process has been set to have an unreliable disk.

* Pager writes now truncate gradually upward, one chunk at a time, in response to writes, which wait on only the necessary truncate operations.   Increased buggified chunk size because truncate can be very slow in simulation.

* In simulation, ioTimeoutError() and ioDegradedOrTimeoutError() will wait until at least the target timeout interval past the point when simulation is sped up.

* PriorityMultiLock::toString() prints more info and is now public.

* Added queued time to PriorityMultiLock.

* Bug fix to handle when speedUpSimulation changes later than the configured time.

* Refactored mutation application in leaf nodes to do fewer comparisons and do in place value updates if the new value is the same size as the old value.

* Renamed updatingInPlace to updatingDeltaTree for clarity.  Inlined switchToLinearMerge() since it is only used in one place.

* Updated extendToCover to be more clear by passing in the old extension future as a parameter.  Fixed initialization warning.
2021-11-12 13:47:07 -08:00
Lukas Joswiak
c93052121f Fix issue where transaction spans would not be recorded 2021-11-10 13:33:49 -08:00
negoyal
1e7338b6c3 Merge branch 'master' into bit-flipping-workload 2021-10-28 14:24:49 -07:00
A.J. Beamon
abab45760d Add some additional logging if the network thread finishes, fails with an error, gets stopped, or is blocked. 2021-10-19 10:05:29 -07:00
negoyal
f913dfed97 Merge branch 'master' into bit-flipping-workload 2021-10-11 16:34:57 -07:00
Lukas Joswiak
e034f66fc2 Add knob to control trace recording percentage 2021-10-01 16:30:46 -07:00
negoyal
3b34423248 Merge branch 'master' into bit-flipping-workload 2021-08-31 12:14:51 -07:00
Lukas Joswiak
d554205db3 Move common knob initialization macros to header 2021-08-23 11:41:01 -07:00
Lukas Joswiak
e2d897a2c2 Clean up types 2021-08-23 11:41:01 -07:00
Lukas Joswiak
7998ec4232 Add atomicity to knobs 2021-08-23 11:41:00 -07:00
sfc-gh-tclinkenbeard
76bf75e026 Replace parseIntegral with standard parsing functions 2021-08-03 15:30:14 -07:00
negoyal
9e7197faba Bunch of changes based on review comments and discussions. 2021-07-30 01:32:43 -07:00
negoyal
4b87716475 Turn the chaos knob off by default. 2021-07-28 18:19:55 -07:00
negoyal
40b4f3b2f1 Merge branch 'master' into bit-flipping-workload 2021-07-28 18:06:07 -07:00
negoyal
050c218502 New Disk Delay Logic and ChaosMetrics. 2021-07-28 16:03:37 -07:00
Steve Atherton
f596a81073 Rename ::TRUE and ::FALSE in BooleanParams to ::True and ::False so as to not conflict with the TRUE and FALSE macros provided by the Windows and MacOS SDKs. 2021-07-17 00:11:40 -07:00
Josh Slocum
02f243c821
Merge pull request #5110 from sfc-gh-jslocum/tss_streaming_master
Added Streaming Read comparison for TSS
2021-07-12 10:44:15 -07:00
sfc-gh-tclinkenbeard
382533c4e1 Add FDB_ prefix to BOOLEAN_PARAM macros 2021-07-09 05:42:14 -07:00
negoyal
2b5a96f745 Single code path for sim and non-sim modes. 2021-07-07 23:58:14 -07:00
sfc-gh-tclinkenbeard
020371a78f Merge remote-tracking branch 'origin/master' into add-boolean-param 2021-07-07 16:50:51 -07:00
Trevor Clinkenbeard
f5ade03538
Merge pull request #4233 from sfc-gh-tclinkenbeard/encrypt-backup-files
Added AsyncFileEncrypted
2021-07-07 13:28:28 -07:00
sfc-gh-tclinkenbeard
8cc40e3a2b Expand use of BOOLEAN_PARAM 2021-07-02 21:41:50 -07:00
Josh Slocum
9f1afbb709 Added Streaming Read comparison for TSS 2021-07-01 19:00:05 -05:00
negoyal
df39c5a44e Implement Disk Throttling Chaos workload. 2021-06-30 17:05:04 -07:00
Josh Slocum
ab0ab765c8
Merge pull request #4965 from sfc-gh-jslocum/tss_mapping_changes
TSS Mismatch Changes
2021-06-29 08:24:25 -07:00
sfc-gh-tclinkenbeard
81b8292094 Merge remote-tracking branch 'origin' into encrypt-backup-files 2021-06-25 12:21:12 -07:00
Evan Tschannen
fcb8bd6475
Revert "Make the sim2 run loop match the behavior of the net2 run loop." 2021-06-22 14:50:01 -07:00
Evan Tschannen
154332a94b Merge branch 'master' of https://github.com/apple/foundationdb into feature-sim-time-batching
# Conflicts:
#	fdbserver/VersionedBTree.actor.cpp
2021-06-22 09:37:40 -07:00
sfc-gh-tclinkenbeard
9d62887f35 Addressed review comments 2021-06-16 11:34:20 -07:00
Josh Slocum
56dadaa428 TSS Mismatch Changes 2021-06-11 23:13:16 +00:00
sfc-gh-tclinkenbeard
c272304e60 Manage global flow knobs with global knob collection 2021-06-09 22:33:00 -07:00
sfc-gh-tclinkenbeard
83a0e473e8 Refactor IKnobCollection code 2021-06-09 20:50:00 -07:00
Evan Tschannen
08a5f17660 Merge branch 'master' of https://github.com/apple/foundationdb into feature-sim-time-batching
# Conflicts:
#	fdbserver/DataDistribution.actor.cpp
2021-06-08 10:04:06 -07:00
Evan Tschannen
52ef8b94fb added comments 2021-06-08 09:57:37 -07:00
sfc-gh-tclinkenbeard
6fc5bd3480 Remove TestID from stateful configuration database classes, and fix some bugs 2021-06-07 00:29:36 -07:00