339 Commits

Author SHA1 Message Date
Chaoguang Lin
7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
Josh Slocum
f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Josh Slocum
37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
Evan Tschannen
7908ea54f3 fix: NetNotifiedQueueWithAcknowledgements could miss disconnects that happen during the delay(0) in deliver() 2022-03-17 09:57:46 -07:00
sfc-gh-tclinkenbeard
baec03090e Fix "guarantee" misspelling 2022-03-14 16:33:09 -07:00
sfc-gh-tclinkenbeard
8dcac2f76d Fix typos 2022-03-13 10:02:11 -03:00
Josh Slocum
a2bbb188c3 Fixing delayed health updater to mark address failed when too many connections triggers before new connection starts, instead of after 2022-03-11 16:50:38 -06:00
A.J. Beamon
250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Trevor Clinkenbeard
3b4cbded96
Merge pull request #6374 from sfc-gh-ajbeamon/protocol-version-pretty-print
Log protocol version as a hex string rather than a uint64_t
2022-02-14 13:51:00 -08:00
A.J. Beamon
1b7e790211 When inserting multiple streams into an EndpointMap, don't assign them in the well-known endpoint space 2022-02-11 12:59:12 -08:00
A.J. Beamon
0a1bcf7a29 Log protocol version as a hex string rather than a uint64_t. 2022-02-10 13:57:48 -08:00
Renxuan Wang
46d17d748f Change member variable fromHostname to type bool.
So that it can be serialized.
2021-11-23 14:25:02 -10:00
Steve Atherton
e752fdd69f
Optimization in FlowTransport's use of XXHash to avoid a malloc() when using the stream API and to not use the stream API when a message being sent is contiguous within a single packet buffer block. (#5970) 2021-11-12 11:45:21 -08:00
Renxuan Wang
f15ceb5489 Add Hostname struct, and fromHostname in NetworkAddress struct. 2021-11-04 11:42:28 -07:00
Steve Atherton
dc3f46c2ae Change asserts to throw internal_error as these function calls should never be compiled out. 2021-11-03 11:00:12 -07:00
Steve Atherton
f3d62ce002 Assert success on XXHash stream functions. 2021-11-02 23:40:53 -07:00
Steve Atherton
313a3dcd63 Change FlowTransport to use 64 bit XXHash instead of CRC32. 2021-11-02 23:16:03 -07:00
Vaidas Gasiunas
16171d8252 Refactoring well-known endpoint registration
- List all well-known endpoints of FDB in a single enum
- Identify well-known endpoints by plain IDs
2021-09-21 11:05:31 -06:00
Lukas Joswiak
d6a7a469ef Remove fdbclient dependency 2021-08-23 10:25:07 -07:00
sfc-gh-tclinkenbeard
658233d0e0 Merge remote-tracking branch 'origin/master' into paxos-config-db 2021-08-19 16:18:59 -07:00
sfc-gh-tclinkenbeard
7d87aa8203 Increase well known endpoint index for WLTOKEN_PROCESS 2021-08-16 12:09:56 -07:00
Xiaoxi Wang
2df0474fec merge master 2021-08-02 11:58:35 -07:00
Trevor Clinkenbeard
5e1639ad32
Merge pull request #5222 from sfc-gh-tclinkenbeard/paxos-config-db
Simple (non-fault tolerant) implementation of replicated configuration database
2021-08-02 10:03:24 -07:00
Xiaoxi Wang
10c82b422f merge master branch 2021-07-28 14:19:46 -07:00
sfc-gh-tclinkenbeard
59ddfd78bd Merge remote-tracking branch 'origin/master' into paxos-config-db 2021-07-28 12:54:02 -07:00
sfc-gh-tclinkenbeard
c74047c665 Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings 2021-07-28 11:51:02 -07:00
Evan Tschannen
81f93d794f Merge branch 'master' of https://github.com/apple/foundationdb into fix-ordered-delay 2021-07-27 13:56:35 -07:00
Evan Tschannen
256a18e43b Flow transport uses an ordered delay to avoid out of order reply promise stream messages 2021-07-27 12:01:32 -07:00
Steve Atherton
507c1f11e3 Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use. 2021-07-26 19:55:10 -07:00
Xiaoxi Wang
1587192331 temperary change to fix local out-of-order delivery 2021-07-26 09:36:23 -07:00
sfc-gh-tclinkenbeard
b20e02ca25 Fix more -Wreorder-ctor warnings across several files 2021-07-24 11:43:19 -07:00
sfc-gh-tclinkenbeard
3442ebd3b7 Fix more -Wreorder-ctor warnings across many files 2021-07-24 11:20:51 -07:00
sfc-gh-tclinkenbeard
6f81155784 Merge remote-tracking branch 'origin/master' into const-serverdbinfo 2021-07-20 10:18:40 -07:00
sfc-gh-tclinkenbeard
9cfd6ed955 Add simple implementation to PaxosConfigConsumer 2021-07-18 17:07:10 -07:00
Evan Tschannen
a5e0e88df8 fix: reply promise streams could receive messages out of order if the sender and receiver were the same process 2021-07-15 15:27:49 -07:00
sfc-gh-tclinkenbeard
4f853b19a6 More const-correctness improvements for Reference<AsyncVar<T>> objects 2021-07-12 21:28:38 -07:00
Evan Tschannen
244e8f3527 Merge branch 'master' of https://github.com/apple/foundationdb into feature-streaming-reply-master
# Conflicts:
#	fdbclient/CMakeLists.txt
#	fdbclient/Knobs.h
#	fdbserver/Knobs.h
2021-06-21 16:35:21 -07:00
Evan Tschannen
1cec223434 cherry pick streaming replies to master 2021-06-19 09:47:13 -07:00
sfc-gh-tclinkenbeard
6619545c30 Merge remote-tracking branch 'origin/master' into config-db 2021-06-18 19:59:56 -07:00
Zhe Wu
07ee4029f2 Addressing comments. 2021-06-17 14:53:15 -07:00
Zhe Wu
4e3e2b0392 Create health monitor in FDB workers to monitor network condition. This change is only inside the worker. 2021-06-16 14:50:44 -07:00
sfc-gh-tclinkenbeard
f10dd70c37 Remove configuration_database from status when disabled 2021-06-06 08:51:18 -07:00
sfc-gh-tclinkenbeard
a775f92fca Merge remote-tracking branch 'origin/master' into config-db 2021-06-01 15:39:34 -07:00
sfc-gh-tclinkenbeard
ca0893571c Move server knobs into fdbclient 2021-06-01 03:12:47 -07:00
sfc-gh-tclinkenbeard
6665f5cc4d Support and test restricted range reads 2021-05-29 03:58:18 -07:00
Dan Lambright
10289ef8f1 Respond to AJs comments 2021-05-27 09:14:32 -04:00
Dan Lambright
64c10d3625 fix joshua failures, formatting 2021-05-27 08:08:07 -04:00
Dan Lambright
fcfb78162c misc cleanup for publishing 2021-05-27 08:08:07 -04:00
Dan Lambright
742c22cef2 Don't allow changing desriptor if knob is set 2021-05-27 08:08:07 -04:00