405 Commits

Author SHA1 Message Date
Markus Pilman
ccf97eb187 Make transport work 2022-04-22 17:06:05 -06:00
Markus Pilman
436d15ea98 Implemented token management and testing
Code is still untested at this point, but testing code was written

Implemented token management and testing

Code is still untested at this point, but testing code was written
2022-04-16 11:29:13 -06:00
Markus Pilman
bf956f5630 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-07 13:29:27 -06:00
Markus Pilman
e2d7d4075d multiple bug fixes 2022-04-07 11:08:07 -06:00
Chaoguang Lin
7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
He Liu
dd15489605 rename ssd-rocksdb-experimental as ssd-rocksdb-v1. 2022-03-29 10:53:38 -07:00
sfc-gh-tclinkenbeard
77786f4fc6 Merge remote-tracking branch 'origin/main' into change-data-hall 2022-03-27 12:44:05 -07:00
Josh Slocum
f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Markus Pilman
35f7843d84 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-21 12:33:53 +01:00
Josh Slocum
bf9bf19797 Disabling tenants for BlobGranule tests for now 2022-03-18 10:30:54 -05:00
Josh Slocum
37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
A.J. Beamon
ee70837970 Remove some debugging lines, change simulated default cluster frequency, and use a different mechanism to disable tenants for the change feed test. 2022-03-17 12:10:19 -07:00
A.J. Beamon
9e39ebd069 Use the new enum name OPTIONAL_TENANT 2022-03-17 12:10:19 -07:00
A.J. Beamon
d0dc756c6d Allow disabling tenant mode in simulation. Fix a few bugs. 2022-03-17 12:10:18 -07:00
A.J. Beamon
81e8c7c362 Various test fixes to work with tenants. 2022-03-17 12:10:18 -07:00
A.J. Beamon
4b521b38cb Update various workloads to specify that they are expecting raw access. A couple other raw access related fixes. Disable tenant tests in backup and DR tests for now. 2022-03-17 12:10:18 -07:00
A.J. Beamon
05495908b8 Implement some tenant tests 2022-03-17 12:10:18 -07:00
Markus Pilman
118b53b7cf Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-17 12:06:44 +01:00
Steve Atherton
59762fc784
Merge pull request #6423 from sfc-gh-satherton/redwood-bug-fixes-and-memory-leak
Redwood memory usage and small memory leak fixes
2022-03-15 16:52:28 -07:00
Markus Pilman
117ee637db Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-15 17:17:47 +01:00
Markus Pilman
bed799220a Addressed review comments, added test 2022-03-15 16:57:26 +01:00
Yao Xiao
09df9d83fd
Enable RocksDB with Valgrind. (#6595) 2022-03-14 21:11:44 -07:00
Josh Slocum
4d909abbd1 re-enabling all storage engine types 2022-03-09 09:21:45 -06:00
Josh Slocum
e71b3533f9 Merge branch 'main' into blob_integration 2022-03-09 08:59:56 -06:00
A.J. Beamon
5fa9d3e1b7 Add a tenant parameter to read and commit requests. Store a map of all tenants on commit proxy and storage servers. Add an option to require tenant mode. 2022-03-06 21:54:21 -08:00
Steve Atherton
f03c0b8c3c Added ISimulated::restarted for detecting a restarted simulation test. 2022-03-04 17:19:46 -08:00
sfc-gh-tclinkenbeard
59794ffcb2 Add randomlyRenameZoneId test parameter 2022-03-01 00:42:24 -08:00
sfc-gh-tclinkenbeard
db8def68db Use std::unique_ptr for ISimulator::extraDB 2022-02-28 13:12:31 -08:00
A.J. Beamon
250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Renxuan Wang
481587a8c6 Turn on hostname logic. 2022-02-22 16:29:59 -08:00
Josh Slocum
38a75a8b89 Merge branch 'main' into blob_integration 2022-02-17 17:47:38 -06:00
sfc-gh-tclinkenbeard
5b8f4d9be1 Move SimpleIni.h to fdbclient 2022-02-07 13:31:04 -08:00
Josh Slocum
4d34ac9545 disabling redwood and memory storage engines for tests for now 2022-02-07 08:15:45 -06:00
Renxuan Wang
334ba66348 Add the support for using hostname in simulation.
This PR does NOT turn the feature on. Also, disable hostname in downgrade restarting tests.
2022-01-21 10:48:33 -08:00
Suraj Gupta
968a4f9f50 Don't rely on database config to be updated. 2021-12-10 14:00:34 -06:00
Suraj Gupta
fc3376fe8f Move client knob to database config for blob granules. 2021-12-10 14:00:34 -06:00
Josh Slocum
efb21ca6a1 Merge branch 'master' into blob_integration 2021-11-24 13:17:54 -06:00
sfc-gh-tclinkenbeard
766a05d33c Merge remote-tracking branch 'origin/master' into add-format-warning 2021-11-17 12:14:01 -08:00
Renxuan Wang
725d31e264 Fix parameters passes to simulatedMachine(). 2021-11-16 17:50:12 -08:00
Steve Atherton
c53f5aa110 Renamed redwood to redwood-1-experimental and file extension to .redwood-v1. 2021-11-16 02:15:22 -08:00
Josh Slocum
f140028086 Merge branch 'master' into blob_integration 2021-11-15 15:09:40 -06:00
sfc-gh-tclinkenbeard
62efeb6812 Merge remote-tracking branch 'origin/master' into add-format-warning 2021-11-12 11:50:36 -08:00
Josh Slocum
ba53045fcd Fixing BW count for multi-process machines and having BW monitor its own removal since it doesn't get private mutations 2021-11-11 12:33:08 -06:00
sfc-gh-tclinkenbeard
25257f6f87 Enable unused-function warning for clang 2021-11-01 14:18:31 -07:00
sfc-gh-tclinkenbeard
13bb7838aa Enable clang -Wformat warning 2021-10-30 21:07:38 -07:00
Josh Slocum
19076ad4d2 Changed blob manager checkin to by async and to have blob worker speculatively consume change feed ahead of re-snapshot 2021-10-27 16:23:51 -05:00
Josh Slocum
7d25dab96b Re-enabling blob granules for testing 2021-10-26 12:11:37 -05:00
Josh Slocum
0ff8ddc2b6 Merge branch 'master' into blob_full_clean 2021-10-25 13:38:48 -05:00
A.J. Beamon
e882eb33fc Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem. 2021-10-22 11:05:18 -07:00