297 Commits

Author SHA1 Message Date
Chaoguang Lin
7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
sfc-gh-tclinkenbeard
a32547cbbe Merge remote-tracking branch 'origin/main' into flow-enhancements 2022-02-15 15:56:42 -08:00
Ray Jenkins
dd45805312
Merge branch 'apple:main' into threadname-issue-6064 2022-02-01 17:40:07 -06:00
Renxuan Wang
2ea4146e1f Add resolveTCPEndpointBlocking() to resolve hostnames where async resolving is impossible. 2022-01-28 12:20:41 -08:00
Ray Jenkins
ff49bfaeb5 Add thread name for SSLHandshakerThread. 2022-01-24 20:00:00 -06:00
Renxuan Wang
28832a99d6 Address comment. 2022-01-18 14:34:18 -08:00
Renxuan Wang
b8bab06e16 Add the functions to set and get mock DNS.
These functions will be used in restarting tests, where mock DNS needs to be saved to and read from files.
2022-01-18 14:34:18 -08:00
sfc-gh-tclinkenbeard
8076806e76 Add ITERATIONS_PER_REACTOR_CHECK 2022-01-01 20:14:57 -08:00
sfc-gh-tclinkenbeard
2ad8955dd3 Cleanup for Flow Enhancements PR 2022-01-01 20:04:00 -08:00
Jon Fu
476aaa73c6 modify benchmark to avoid threadsleep and change runloop to process IO properly 2021-12-16 16:20:22 -05:00
Jon Fu
169838cec3 stub changes for IO queue and attempt at new benchmark 2021-12-16 13:28:38 -05:00
Andrew Noyes
def41697bf
Merge pull request #6083 from sfc-gh-tclinkenbeard/remove-temporaries
Avoid creating unnecessary temporary objects
2021-12-06 13:24:56 -08:00
sfc-gh-tclinkenbeard
3d36dfe5e9 Fix compilation error in resolveTCPEndpoint_impl 2021-12-02 12:51:00 -08:00
sfc-gh-tclinkenbeard
70c8f98eb9 Apply clang-format to Net2.actor.cpp 2021-12-02 10:22:22 -08:00
sfc-gh-tclinkenbeard
d01a363e29 Avoid creating unnecessary temporary objects 2021-12-01 23:48:34 -08:00
sfc-gh-tclinkenbeard
90ced244eb Fix -Wunused-but-set-variable warnings 2021-12-01 18:15:53 -08:00
Renxuan Wang
22e34bd6b9 Replace <boost/bind.hpp> with <boost/bind/bind.hpp>.
This eliminates many useless warnings when compiling.
`#pragma message: The practice of declaring the Bind placeholders (_1, _2, ...) in the global namespace is deprecated. Please use <boost/bind/bind.hpp> + using namespace boost::placeholders, or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior.`
2021-11-18 14:00:13 -08:00
Lukas Joswiak
18243351e7 Fix possible data race
Transactions (created on a separate thread) can read the `globals` field
at the same time as `setGlobal` is called on the main thread, causing a
potential race. TSAN surfaced this issue.
2021-11-18 10:16:20 -08:00
Steve Atherton
035e0d6e52
Merge branch 'master' into bit-flipping-workload 2021-11-16 14:42:22 -08:00
Renxuan Wang
4630b0ccea Move DNS mock from SimExternalConnection to Sim2.
This is a revise PR of #5934. In simulation, we don't have direct access to SimExternalConnection.
2021-11-15 17:02:51 -08:00
negoyal
1e7338b6c3 Merge branch 'master' into bit-flipping-workload 2021-10-28 14:24:49 -07:00
sfc-gh-tclinkenbeard
49a667c29b Improve const-correctness of INetwork 2021-10-25 14:42:31 -07:00
negoyal
f913dfed97 Merge branch 'master' into bit-flipping-workload 2021-10-11 16:34:57 -07:00
Xiaoge Su
abf73047ca Enforce std:: specifier rather than using namespace 2021-09-16 19:40:28 -07:00
negoyal
3b34423248 Merge branch 'master' into bit-flipping-workload 2021-08-31 12:14:51 -07:00
Lukas Joswiak
5dc9a97230 Merge branch 'master' into fixes/alp6 2021-08-01 20:42:52 -07:00
Andrew Noyes
353efe7db2
Merge pull request #5264 from sfc-gh-tclinkenbeard/fix-more-clang-warnings
Enable more warnings for `clang`
2021-07-29 15:43:54 -07:00
negoyal
40b4f3b2f1 Merge branch 'master' into bit-flipping-workload 2021-07-28 18:06:07 -07:00
negoyal
050c218502 New Disk Delay Logic and ChaosMetrics. 2021-07-28 16:03:37 -07:00
sfc-gh-tclinkenbeard
94a65865d9 Merge remote-tracking branch 'origin/master' into fix-clang-warnings 2021-07-28 12:29:27 -07:00
sfc-gh-tclinkenbeard
c74047c665 Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings 2021-07-28 11:51:02 -07:00
Evan Tschannen
81f93d794f Merge branch 'master' of https://github.com/apple/foundationdb into fix-ordered-delay 2021-07-27 13:56:35 -07:00
Evan Tschannen
256a18e43b Flow transport uses an ordered delay to avoid out of order reply promise stream messages 2021-07-27 12:01:32 -07:00
Lukas Joswiak
3eed4084e2 Merge branch 'master' into fixes/alp6 2021-07-27 11:26:53 -07:00
Lukas Joswiak
59d535149e Merge branch 'master' into fixes/alp6 2021-07-27 10:07:18 -07:00
Lukas Joswiak
e9a1679467 Disable sampling everywhere except fdbserver 2021-07-27 09:53:23 -07:00
Steve Atherton
507c1f11e3 Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use. 2021-07-26 19:55:10 -07:00
sfc-gh-tclinkenbeard
da50e13f3e Fix more -Wreorder-ctor warnings in DataDistribution.actor.cpp, OldTLogServer_4_6.actor.cpp, and Net2.actor.cpp 2021-07-24 17:33:11 -07:00
sfc-gh-tclinkenbeard
3442ebd3b7 Fix more -Wreorder-ctor warnings across many files 2021-07-24 11:20:51 -07:00
sfc-gh-tclinkenbeard
e62e6503ac Fix most delete-non-virtual-dtor clang warnings 2021-07-21 23:32:44 -07:00
Daniel Smith
3fbd6b6143 Enable IThreadPool in simulation 2021-07-08 18:51:01 -04:00
Lukas Joswiak
250d20e7bf Even more cleanup 2021-06-15 15:17:54 -07:00
Lukas Joswiak
0301072690 Refactor 2021-06-11 13:01:32 -07:00
Lukas Joswiak
e9fdbb9c86 Enable sampling 2021-06-07 18:01:14 -07:00
Lukas Joswiak
486a04659f Lazy inititialization 2021-06-04 15:01:18 -07:00
Lukas Joswiak
ca79b8eaab Various ALP fixes 2021-06-04 15:01:18 -07:00
Lukas Joswiak
153de33f57 Revert "Merge pull request #4802 from sfc-gh-ljoswiak/revert/actor-lineage"
This reverts commit 6499fa178e8f65a22105c2cd062a67209b562973, reversing
changes made to 15126319577f915f28aa6308bbf066dc7ec992a2.
2021-06-04 13:31:55 -07:00
A.J. Beamon
a162682d6d Fix accounting for time spent in run loop after breaking due to yield 2021-05-14 11:12:47 -07:00
Lukas Joswiak
4ea760b2a9 Revert "Merge pull request #4136 from sfc-gh-mpilman/features/actor-lineage"
This reverts commit da41534618a2a1edbf6b0b760635175372a66294, reversing
changes made to e6300905d6f294c52ebd166f4714541b084f37b4.
2021-05-10 20:26:12 -07:00