416 Commits

Author SHA1 Message Date
He Liu
bc5bfaffda
Shard based move (#6981)
* Shard based move.

* Clean up.

* Clear results on retry in getInitialDataDistribution.

* Remove assertion on SHARD_ENCODE_LOCATION_METADATA for compatibility.

* Resolved comments.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-07 20:49:16 -07:00
Markus Pilman
a47ed89018 Linux fixes and addressed review comments 2022-06-23 20:52:13 -06:00
Markus Pilman
38e100ebc5 flow bindings are compiling 2022-06-23 19:06:05 -06:00
Markus Pilman
ffaf15c12a moved wellknownendpoints and fixed some includes 2022-06-23 17:03:53 -06:00
Josh Slocum
ad54be5d0b Enabling tenant tests for blob granules and fixing bug 2022-05-25 17:16:56 -05:00
A.J. Beamon
7ba2be6f30 Update simulator to sometimes create tenants it doesn't use. Fix bug in mapped range where it would fail if any tenants existed and transactions were run not using the tenants. 2022-05-05 10:03:30 -07:00
Steve Atherton
338d2304d7
Bug fix: Killing a machine process would not wait for AsyncFileNonDurable close operations to finish, causing a later reopen of the same file in a new process to hang forever. Renamed AsyncFileNonDurable::deleteFile to closeFile for clarity. Renamed Machine deletingFiles to deletingOrClosingFiles for clarity. (#7007) 2022-04-29 14:01:18 -07:00
Renxuan Wang
c69a07a858
Check in the new Hostname logic. (#6926)
* Revert #6655.

20220407-031010-renxuan-c101052c21da8346           compressed=True data_size=31004844 duration=4310801 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=1:04:15 sanity=False started=100047 stopped=20220407-041425 submitted=20220407-031010 timeout=5400 username=renxuan

* Revert #6271.

20220407-051532-renxuan-470f0fe6aac1c217           compressed=True data_size=30982370 duration=3491067 ended=100002 fail_fast=10 max_runs=100000 pass=100002 priority=100 remaining=0 runtime=0:59:57 sanity=False started=100141 stopped=20220407-061529 submitted=20220407-051532 timeout=5400 username=renxuan

* Revert #6266.

Remove resolving-related functionalities in connection string. Connection string will be used for storing purpose only, and non-mutable.

20220407-175119-renxuan-55d30ee1a4b42c2f           compressed=True data_size=30970443 duration=5437659 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:59:31 sanity=False started=100154 stopped=20220407-185050 submitted=20220407-175119 timeout=5400 username=renxuan

* Add hostname to coordinator interfaces.

* Turn on the new hostname logic.

* Add the corresponding change in config txns.

The most notable change is before calling basicLoadBalance(), we need to call tryInitializeRequestStream() to initialize request streams first.

Passed correctness tests.

* Return error when hostnames cannot be resolved in coordinators command.

* Minor fixes.
2022-04-27 21:54:13 -07:00
Renxuan Wang
ec95fb7e74 Disable hostname in simulations.
Hostnames are causing simulation failures. Turn on again in #6926.
2022-04-25 11:05:25 -07:00
Bharadwaj V.R
ed08cfbf52
Merge branch 'apple:main' into block-down 2022-04-22 06:19:38 -07:00
Zhe Wang
6c9ff6ee5e
Add sharded rocksdb type (#6862)
* add-sharded-rocksdb-type

* address comments

Co-authored-by: Zhe Wang <zhewang@Zhes-MacBook-Pro.local>
2022-04-21 22:53:14 -04:00
Bharadwaj V.R
e4f22e5394
Merge branch 'apple:main' into block-down 2022-04-12 10:50:02 -07:00
Markus Pilman
bf956f5630 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-07 13:29:27 -06:00
Markus Pilman
e2d7d4075d multiple bug fixes 2022-04-07 11:08:07 -06:00
Bharadwaj V.R
58a6442d1f Add tracing and fix the version write and update logic 2022-04-06 21:06:01 -07:00
Chaoguang Lin
7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
He Liu
dd15489605 rename ssd-rocksdb-experimental as ssd-rocksdb-v1. 2022-03-29 10:53:38 -07:00
sfc-gh-tclinkenbeard
77786f4fc6 Merge remote-tracking branch 'origin/main' into change-data-hall 2022-03-27 12:44:05 -07:00
Josh Slocum
f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Markus Pilman
35f7843d84 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-21 12:33:53 +01:00
Josh Slocum
bf9bf19797 Disabling tenants for BlobGranule tests for now 2022-03-18 10:30:54 -05:00
Josh Slocum
37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
A.J. Beamon
ee70837970 Remove some debugging lines, change simulated default cluster frequency, and use a different mechanism to disable tenants for the change feed test. 2022-03-17 12:10:19 -07:00
A.J. Beamon
9e39ebd069 Use the new enum name OPTIONAL_TENANT 2022-03-17 12:10:19 -07:00
A.J. Beamon
d0dc756c6d Allow disabling tenant mode in simulation. Fix a few bugs. 2022-03-17 12:10:18 -07:00
A.J. Beamon
81e8c7c362 Various test fixes to work with tenants. 2022-03-17 12:10:18 -07:00
A.J. Beamon
4b521b38cb Update various workloads to specify that they are expecting raw access. A couple other raw access related fixes. Disable tenant tests in backup and DR tests for now. 2022-03-17 12:10:18 -07:00
A.J. Beamon
05495908b8 Implement some tenant tests 2022-03-17 12:10:18 -07:00
Markus Pilman
118b53b7cf Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-17 12:06:44 +01:00
Steve Atherton
59762fc784
Merge pull request #6423 from sfc-gh-satherton/redwood-bug-fixes-and-memory-leak
Redwood memory usage and small memory leak fixes
2022-03-15 16:52:28 -07:00
Markus Pilman
117ee637db Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-15 17:17:47 +01:00
Markus Pilman
bed799220a Addressed review comments, added test 2022-03-15 16:57:26 +01:00
Yao Xiao
09df9d83fd
Enable RocksDB with Valgrind. (#6595) 2022-03-14 21:11:44 -07:00
Josh Slocum
4d909abbd1 re-enabling all storage engine types 2022-03-09 09:21:45 -06:00
Josh Slocum
e71b3533f9 Merge branch 'main' into blob_integration 2022-03-09 08:59:56 -06:00
A.J. Beamon
5fa9d3e1b7 Add a tenant parameter to read and commit requests. Store a map of all tenants on commit proxy and storage servers. Add an option to require tenant mode. 2022-03-06 21:54:21 -08:00
Steve Atherton
f03c0b8c3c Added ISimulated::restarted for detecting a restarted simulation test. 2022-03-04 17:19:46 -08:00
sfc-gh-tclinkenbeard
59794ffcb2 Add randomlyRenameZoneId test parameter 2022-03-01 00:42:24 -08:00
sfc-gh-tclinkenbeard
db8def68db Use std::unique_ptr for ISimulator::extraDB 2022-02-28 13:12:31 -08:00
A.J. Beamon
250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Renxuan Wang
481587a8c6 Turn on hostname logic. 2022-02-22 16:29:59 -08:00
Josh Slocum
38a75a8b89 Merge branch 'main' into blob_integration 2022-02-17 17:47:38 -06:00
sfc-gh-tclinkenbeard
5b8f4d9be1 Move SimpleIni.h to fdbclient 2022-02-07 13:31:04 -08:00
Josh Slocum
4d34ac9545 disabling redwood and memory storage engines for tests for now 2022-02-07 08:15:45 -06:00
Renxuan Wang
334ba66348 Add the support for using hostname in simulation.
This PR does NOT turn the feature on. Also, disable hostname in downgrade restarting tests.
2022-01-21 10:48:33 -08:00
Suraj Gupta
968a4f9f50 Don't rely on database config to be updated. 2021-12-10 14:00:34 -06:00
Suraj Gupta
fc3376fe8f Move client knob to database config for blob granules. 2021-12-10 14:00:34 -06:00
Josh Slocum
efb21ca6a1 Merge branch 'master' into blob_integration 2021-11-24 13:17:54 -06:00
sfc-gh-tclinkenbeard
766a05d33c Merge remote-tracking branch 'origin/master' into add-format-warning 2021-11-17 12:14:01 -08:00