204 Commits

Author SHA1 Message Date
Ata E Husain Bohra
333aadb903
Interface to enable clients to send/receive REST requests/responses (#6866)
* Interface to enable clients to send/receive REST requests/responses

Description

Major changes:
1. Add RESTClient interface enabling client to send/receive REST HTTP
   requests. Support REST APIs are: get, head, put, post, delete, trace
2. Add RESTUtil file introducing below interfaces:
 2.1. RESTUrl - Extract URI information: host, service, request-parameters.
 2.2. RESTConnectionPool-
      Connection establishment, life-cycle management, connection-pool (TTL)
 2.3. RESTClientKnobs - supports REST Knob parameter management and updates

Testing

Unit test - fdbrpc/RESTClient, fdbrpc/RESTUtils
2022-04-27 12:17:52 -07:00
Markus Pilman
bbb1392aad Merge remote-tracking branch 'origin/main' into features/validate-trace-events-in-simulation 2022-04-21 08:24:18 -06:00
Renxuan Wang
e40cc8722c
A few hostname improvements. (#6825)
* Add tryResolveHostnames() in connection string.

* Add missing hostname to related interfaces.

* Do not pass RequestStream into *GetReplyFromHostname() functions.

Because we are using new RequestStream for each request anyways. Also, the passed in pointer could be nullptr, which results in seg faults.

* Add dynamic hostname resolve and reconnect intervals.

* Address comments.
2022-04-20 13:42:46 -07:00
Markus Pilman
d4ee7be1d7 Reduce excessive tracing and fail after 1M traces 2022-04-19 21:11:51 -06:00
Ata E Husain Bohra
933e5bbd2e
EncryptKeyProxy server APIs for simulation runs. (#6727)
* EncryptKeyProxy server APIs for simulation runs.

Description

  diff-2: FlowSingleton util class
              Bug fixes
  diff-1: Expected errors returned to the caller

Major changes proposed are:
1. EncryptKeyProxy server APIs:
 1.1. Lookup Cipher details via BaseCipherId
 1.2. Lookup latest Cipher details via encryption domainId.
2. EncyrptKeyProxy implements caches indexed by: baseCipherId &
   encyrptDomainId
3. Periodic task to refresh domainId indexed cache to support
   'limiting cipher lifetime' abilities if supported by
   external KMS solutions.

Testing

EncyrptKeyProxyTest workload to validate the newly added code.
2022-04-11 09:08:42 -07:00
Yi Wu
994b8c92f8
Add option to limit resident memory and remove default memory limit (#6719)
Changing `memory` option to limit resident memory instead of virtual memory, in config file and fdbserver/fdbbackup/fdbcli command-line argument. Since `rlimit` doesn't support limiting virtual memory, the current implementation have both of fdbmonitor and the fdbserver/fdbbackup process checking process RSS periodically and kill and restart the process if the limit is exceeded.

Adding a new `memory_vsize` option to limit virtual memory, if backward-compatible behavior is desired.

closes #6671, closes #6672
2022-04-06 20:06:24 -07:00
Renxuan Wang
465ff712b6
Move Hostname to its own files. (#6759)
* Change DNS cache to use std::map.

Revert commit 90c259d84e95dd35e01149c0a86bd18e82e33930, because if we use unordered_map, toString() can be inconsistent.

* Move ClientKnob::COORDINATOR_HOSTNAME_RESOLVE_DELAY to FlowKnob::HOSTNAME_RESOLVE_DELAY.

* Move Hostname to its own files.

Also, add resolve-related variables and functions in Hostname.
2022-04-04 19:04:51 -07:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Xiaoge Su
99b030c2f6 Allow the TOML file assign knobs during test
In this patch, for a given test, it is possible to override the knob
values, e.g.

[[test]]

    [[test.knobs]]
    watch_timeout = 999

will set the client knob WATCH_TIMEOUT to 999 during the test. The
original value will be recovered after the test is over.
2022-03-17 16:53:10 -07:00
sfc-gh-tclinkenbeard
8dcac2f76d Fix typos 2022-03-13 10:02:11 -03:00
sfc-gh-tclinkenbeard
a32547cbbe Merge remote-tracking branch 'origin/main' into flow-enhancements 2022-02-15 15:56:42 -08:00
Ray Jenkins
226236600e Preserving original LISTENER_PORT knob and adding new Knob TRACING_UDB_LISTENER_ADDR 2022-02-07 15:49:35 -08:00
Ray Jenkins
1bd084810c Refactor TRACING_UDP_LISTENER_PORT to TRACING_UDP_LISTENER_ADDR. 2022-02-07 15:49:35 -08:00
Steven Li
36c6e23700
Add TraceEvent severity knob (#6326)
* add knob for trace event severity

* add knob for TraceEvent severity

* fix format

* fix switch format

* moved intToSeverity call inside __test initialization

* updated knob name

* fix line length format

* fix format

* git clang-format
2022-02-02 11:36:17 -08:00
sfc-gh-tclinkenbeard
8076806e76 Add ITERATIONS_PER_REACTOR_CHECK 2022-01-01 20:14:57 -08:00
Steve Atherton
035e0d6e52
Merge branch 'master' into bit-flipping-workload 2021-11-16 14:42:22 -08:00
Steve Atherton
508429f30d
Redwood chunked file growth and low priority IO starvation prevention (#5936)
* Redwood files now growth in large page chunks controlled by a knob to reduce truncate() calls for expansion.   PriorityMultiLock has limit on consecutive same-priority lock release.  Increased Redwood max priority level to 3 for more separation at higher BTree levels.

* Simulation fix, don't mark certain IO timeout errors as injected unless the simulated process has been set to have an unreliable disk.

* Pager writes now truncate gradually upward, one chunk at a time, in response to writes, which wait on only the necessary truncate operations.   Increased buggified chunk size because truncate can be very slow in simulation.

* In simulation, ioTimeoutError() and ioDegradedOrTimeoutError() will wait until at least the target timeout interval past the point when simulation is sped up.

* PriorityMultiLock::toString() prints more info and is now public.

* Added queued time to PriorityMultiLock.

* Bug fix to handle when speedUpSimulation changes later than the configured time.

* Refactored mutation application in leaf nodes to do fewer comparisons and do in place value updates if the new value is the same size as the old value.

* Renamed updatingInPlace to updatingDeltaTree for clarity.  Inlined switchToLinearMerge() since it is only used in one place.

* Updated extendToCover to be more clear by passing in the old extension future as a parameter.  Fixed initialization warning.
2021-11-12 13:47:07 -08:00
negoyal
1e7338b6c3 Merge branch 'master' into bit-flipping-workload 2021-10-28 14:24:49 -07:00
A.J. Beamon
abab45760d Add some additional logging if the network thread finishes, fails with an error, gets stopped, or is blocked. 2021-10-19 10:05:29 -07:00
negoyal
f913dfed97 Merge branch 'master' into bit-flipping-workload 2021-10-11 16:34:57 -07:00
Lukas Joswiak
e034f66fc2 Add knob to control trace recording percentage 2021-10-01 16:30:46 -07:00
negoyal
3b34423248 Merge branch 'master' into bit-flipping-workload 2021-08-31 12:14:51 -07:00
Lukas Joswiak
d554205db3 Move common knob initialization macros to header 2021-08-23 11:41:01 -07:00
Lukas Joswiak
e2d897a2c2 Clean up types 2021-08-23 11:41:01 -07:00
Lukas Joswiak
7998ec4232 Add atomicity to knobs 2021-08-23 11:41:00 -07:00
sfc-gh-tclinkenbeard
82546853c0 Rename UseConfigDB to ConfigDBType 2021-08-09 10:04:35 -07:00
sfc-gh-tclinkenbeard
cdbcb69d86 Add configuration database type to ISimulator 2021-08-09 10:04:35 -07:00
negoyal
40b4f3b2f1 Merge branch 'master' into bit-flipping-workload 2021-07-28 18:06:07 -07:00
negoyal
050c218502 New Disk Delay Logic and ChaosMetrics. 2021-07-28 16:03:37 -07:00
Josh Slocum
02f243c821
Merge pull request #5110 from sfc-gh-jslocum/tss_streaming_master
Added Streaming Read comparison for TSS
2021-07-12 10:44:15 -07:00
sfc-gh-tclinkenbeard
bbce95da5f Undo changes unrelated to BOOLEAN_PARAM macro 2021-07-09 05:24:42 -07:00
sfc-gh-tclinkenbeard
020371a78f Merge remote-tracking branch 'origin/master' into add-boolean-param 2021-07-07 16:50:51 -07:00
Trevor Clinkenbeard
f5ade03538
Merge pull request #4233 from sfc-gh-tclinkenbeard/encrypt-backup-files
Added AsyncFileEncrypted
2021-07-07 13:28:28 -07:00
sfc-gh-tclinkenbeard
8cc40e3a2b Expand use of BOOLEAN_PARAM 2021-07-02 21:41:50 -07:00
Josh Slocum
9f1afbb709 Added Streaming Read comparison for TSS 2021-07-01 19:00:05 -05:00
negoyal
df39c5a44e Implement Disk Throttling Chaos workload. 2021-06-30 17:05:04 -07:00
Josh Slocum
ab0ab765c8
Merge pull request #4965 from sfc-gh-jslocum/tss_mapping_changes
TSS Mismatch Changes
2021-06-29 08:24:25 -07:00
sfc-gh-tclinkenbeard
81b8292094 Merge remote-tracking branch 'origin' into encrypt-backup-files 2021-06-25 12:21:12 -07:00
Evan Tschannen
fcb8bd6475
Revert "Make the sim2 run loop match the behavior of the net2 run loop." 2021-06-22 14:50:01 -07:00
Evan Tschannen
154332a94b Merge branch 'master' of https://github.com/apple/foundationdb into feature-sim-time-batching
# Conflicts:
#	fdbserver/VersionedBTree.actor.cpp
2021-06-22 09:37:40 -07:00
Josh Slocum
56dadaa428 TSS Mismatch Changes 2021-06-11 23:13:16 +00:00
sfc-gh-tclinkenbeard
669ca0bb14 Add comments to configuration database code 2021-06-10 17:14:37 -07:00
sfc-gh-tclinkenbeard
c272304e60 Manage global flow knobs with global knob collection 2021-06-09 22:33:00 -07:00
sfc-gh-tclinkenbeard
83a0e473e8 Refactor IKnobCollection code 2021-06-09 20:50:00 -07:00
Evan Tschannen
08a5f17660 Merge branch 'master' of https://github.com/apple/foundationdb into feature-sim-time-batching
# Conflicts:
#	fdbserver/DataDistribution.actor.cpp
2021-06-08 10:04:06 -07:00
sfc-gh-tclinkenbeard
4c059670a0 Parse configuration knob values on the client 2021-06-02 10:04:46 -07:00
sfc-gh-tclinkenbeard
a775f92fca Merge remote-tracking branch 'origin/master' into config-db 2021-06-01 15:39:34 -07:00
Josh Slocum
4257ac2b4d More TSS Changes/Fixes 2021-05-25 20:37:48 +00:00
Evan Tschannen
f57f0d64f4 Merge branch 'master' into feature-sim-time-batching
# Conflicts:
#	fdbserver/DataDistribution.actor.cpp
2021-05-20 09:09:35 -07:00
sfc-gh-tclinkenbeard
01eab20fc0 Remove duplicate Knobs::reset code 2021-05-14 23:02:40 -07:00