19458 Commits

Author SHA1 Message Date
Chaoguang Lin
7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
Xiaoxi Wang
c7d2f5fee2
Merge pull request #6739 from sfc-gh-jslocum/ddq_assert
fix destination limiting and cancelling logic in move_to_removed_serv…
2022-03-31 14:22:09 -07:00
Tao Lin
001909be08
Fixes for when getMappedRange cannot parse as tuple (#6665) 2022-03-31 14:06:45 -07:00
He Liu
966caadb3e
Merge pull request #6706 from kakaiu/Fix-block-cache-recreation-issue
Fix RocksDB Block Cache Recreation Problem
2022-03-31 13:50:15 -07:00
A.J. Beamon
dee1293da2
Merge pull request #6731 from sfc-gh-ajbeamon/fix-tenant-memory-leak
Use a TenantState object in the MVC implementation to help manage tenant lifetime
2022-03-31 13:35:20 -07:00
A.J. Beamon
b6a0eeda11
Merge pull request #6740 from sfc-gh-ajbeamon/docs-tenant-open-note
Add a note that opening a tenant does not check whether that tenant exists in the cluster
2022-03-31 13:35:04 -07:00
Josh Slocum
9e06881673 fix destination limiting and cancelling logic in move_to_removed_server case 2022-03-31 14:05:15 -05:00
A.J. Beamon
5469b57a2b Add a note that opening a tenant does not check whether that tenant exists in the cluster 2022-03-31 11:39:50 -07:00
A.J. Beamon
68f15650a1 Make sure closed and tenantUpdater are read/written in the same critical section. 2022-03-31 11:12:01 -07:00
A.J. Beamon
15b3465b49
Merge pull request #6647 from sfc-gh-ajbeamon/fdb-tenant-documentation
Add some basic documentation for tenants
2022-03-31 10:33:01 -07:00
A.J. Beamon
d813caff10
Merge pull request #6644 from sfc-gh-ajbeamon/fdb-tenant-java
Java support for tenants
2022-03-31 08:32:18 -07:00
A.J. Beamon
b6350a2535 Add a note that use of special keys may implicitly enable raw access on a transaction. 2022-03-31 08:29:31 -07:00
A.J. Beamon
860ede0c2e Add some basic documentation for tenants 2022-03-31 08:29:31 -07:00
Steve Atherton
4b4082fe62
Merge pull request #6738 from sfc-gh-xwang/cmake-fix
fix cmake bug when no link or compile options
2022-03-31 00:51:59 -07:00
Xiaoxi Wang
61e374f301 fix cmake bug when use gcc 2022-03-30 23:21:22 -07:00
Xiaoxi Wang
d6d4596c35
Merge pull request #6709 from sfc-gh-ajbeamon/fix-fdbcli-bugs
Fix issues with command completion for exclude and storage_migration_type
2022-03-30 19:53:49 -07:00
Steve Atherton
26e34a947f
Merge pull request #6724 from sfc-gh-yiwu/file_latency
AsyncFileKAIO: add latency histograms
2022-03-30 19:17:34 -07:00
A.J. Beamon
be7315473a Move tenant creation/deletion into a TenantManagement class 2022-03-30 16:31:28 -07:00
A.J. Beamon
6570ae44c6 Use special keys to create/delete tenants 2022-03-30 16:31:28 -07:00
Jon Fu
995f1b3601 Support tuples in Java tenants 2022-03-30 16:31:28 -07:00
A.J. Beamon
a2a97e7176 Add tenant support to the Java bindings 2022-03-30 16:31:28 -07:00
Andrew Noyes
1b919f52e9
Combine vector_like_traits::{insert,reserve} (#6689)
* Combine vector_like_traits::{insert,reserve}

and explain semantics better. This should make it more clear what
implementers need to do when implementing the vector_like_traits
concept.

* Update std::unordered_set vector_like_traits impl
2022-03-30 16:29:35 -07:00
Steve Atherton
1d90f17c20
Merge pull request #6721 from sfc-gh-satherton/threadmulticallback-refactor
Remove unordered_map of pointers from ThreadMultiCallback
2022-03-30 14:51:51 -07:00
Andrew Noyes
88a439e156
Fix typo from pull request #6698 (#6729) 2022-03-30 13:40:57 -07:00
Steve Atherton
75247affa3 Renamed member for better readability. 2022-03-30 13:39:33 -07:00
A.J. Beamon
c7d53b31ee Use a TenantState object in the MVC implementation to help manage tenant lifetime. 2022-03-30 12:52:27 -07:00
Steve Atherton
5d74e4d091 Added comments to explain some invariants with ThreadMultiCallback and ThreadCallback and how they are enforced. 2022-03-30 12:38:57 -07:00
Steve Atherton
637a6f654b Merge commit 'fc68cdf45f194d198beb8848c49e3081eff6c396' into threadmulticallback-refactor
# Conflicts:
#	fdbclient/BackupAgent.actor.h
2022-03-30 12:27:03 -07:00
A.J. Beamon
fc68cdf45f
Merge pull request #6693 from sfc-gh-ajbeamon/python-tenant-management-module
Move Python tenant management to its own module
2022-03-30 12:03:08 -07:00
Steve Atherton
e6457b1656 A few changes for clarity / readability. 2022-03-30 11:31:46 -07:00
A.J. Beamon
16cc74e910 Remove documenation for Database.delete_tenant. 2022-03-30 11:01:54 -07:00
Yi Wu
acfee48894 AsyncFileKAIO: add latency histograms 2022-03-30 10:45:04 -07:00
Xiaoxi Wang
e2c66de28f
Merge pull request #6722 from sfc-gh-satherton/fix-backup-backward-compatibility
Fixed Codec<Reference<IBackupContainer>> backward compatibility bug from #6705
2022-03-30 08:38:48 -07:00
Steve Atherton
c5995f2e27
Can't enforce size limit or downgrades from future versions could break. 2022-03-29 23:53:45 -07:00
Steve Atherton
bf505ed816
Simplify conditional.
Co-authored-by: Renxuan Wang <renxuan@apple.com>
2022-03-29 23:44:41 -07:00
Steve Atherton
a1eca85b66
Simplify conditional.
Co-authored-by: Renxuan Wang <renxuan@apple.com>
2022-03-29 23:43:07 -07:00
Steve Atherton
b50886b10a
Fix size check
Co-authored-by: Renxuan Wang <renxuan@apple.com>
2022-03-29 23:42:25 -07:00
Steve Atherton
c00ae2fe85 Fixed Codec<Reference<IBackupContainer>> backward compatibility bug recently introduced. 2022-03-29 23:11:15 -07:00
Steve Atherton
5a0274db65 Fixed Codec<Reference<IBackupContainer>> backward compatibility bug recently introduced. 2022-03-29 23:03:24 -07:00
Steve Atherton
01facc8dfa Refactored callback tracking in ThreadCallback and ThreadMultiCallback to not use an unordered_map of pointers to prevent it from falsely triggering the DEBUG_DETERMINISM check, plus it is lower overhead, saving about 6% CPU in the AbortableSingleAssignmentVar unit test. 2022-03-29 21:13:13 -07:00
He Liu
210726a2db
Merge pull request #6712 from liquid-helium/rename-rocks-engine
rename ssd-rocksdb-experimental as ssd-rocksdb-v1.
2022-03-29 19:02:31 -07:00
He Liu
ca4bfb55d6 Merge branch 'main' of https://github.com/apple/foundationdb into rename-rocks-engine 2022-03-29 16:24:50 -07:00
Xiaoxi Wang
e644c6da49
Merge pull request #6716 from sfc-gh-xwang/memory-test-ide
fix cmake error after adding memory test when OPEN_FOR_IDE=ON
2022-03-29 14:23:16 -07:00
A.J. Beamon
8a68781150
Merge pull request #6711 from sfc-gh-anoyes/anoyes/tenant-test-fixes
Fix a few memory issues found by ASAN
2022-03-29 13:43:35 -07:00
Josh Slocum
7fc6dfa6c5 Adding useful debugging trace events 2022-03-29 14:48:28 -05:00
Andrew Noyes
d727e7648e Fix a few memory issues found by ASAN 2022-03-29 12:39:12 -07:00
Xiaoxi Wang
bc3e5cdaa1 fix cmake error when OPEN_FOR_IDE=ON 2022-03-29 12:02:56 -07:00
Jingyu Zhou
da0673ccce
Merge pull request #6705 from RenxuanW/another
Add proxy option to backup and restore params.
2022-03-29 11:36:13 -07:00
He Liu
dd15489605 rename ssd-rocksdb-experimental as ssd-rocksdb-v1. 2022-03-29 10:53:38 -07:00
Renxuan Wang
e1775627ab Add a check on proxy format. 2022-03-29 10:17:14 -07:00