21327 Commits

Author SHA1 Message Date
Chaoguang Lin
5045844ea1
Add -r changeclusterkey tool for snapshot restore (#7687)
* Debug version of fdbserver -r changeclusterkey tool

* Remove debug symbols; move function to Coordinaton.actor.cpp

* Format and add traces

* fix comments
2022-07-25 22:12:28 -07:00
Yao Xiao
98bff116a4
Disabled unsupported tests. (#7693) 2022-07-25 21:57:47 -07:00
Zhe Wu
410ad5ff18 TLog track unpopped recovery tag 2022-07-25 20:10:18 -07:00
Renxuan Wang
dc9599f2e9
Get network interfaces ready for https proxy. (#7556)
* Move HTTP from fdbclient/ to fdbrpc/.

* Move md5 and libb64 to contrib/.

* Get network interfaces ready for https proxy.

* Rebase
2022-07-25 17:08:32 -07:00
A.J. Beamon
6719e5a85b
Merge pull request #7676 from liquid-helium/ss-shard-management
Fixed compiler warnings.
2022-07-25 13:28:54 -07:00
He Liu
118c2ae806
Fixed memory leak for RocksDb and CheckpointReader. (#7665)
* Fixed memory leak for RocksDb and CheckpointReader.

* Close kvStore at the end of StorageServerCheckpointRestoreTest.
2022-07-25 12:36:29 -07:00
Jingyu Zhou
88cfaf8793
Merge pull request #7681 from jzhou77/main
Add timeout during Status generation
2022-07-25 11:04:00 -07:00
Kevin Hoxha
058276493f
ddsketch_utility: Add utility scripts for ddsketch (#7602)
* Adds ddsketch_calc.py which implements a class for DDSketch related calculations
* Adds ddsketch_conversion.py for quickly computing a bucket index to a value or vice-versa
* Adds ddsketch_compare.py to compute how similar two ddsketch distributions are
* Adds export_graph.py to graph the ddsketch distribution outputted from mako

The arguments for ddsketch_conversion.py are:
-b, --bucket: the bucket index that we need to calculate the value from (optional)
-v, --value: the value that we need to calculate the bucket index from (optional)
-e, --error_guarantee: the error guarantee for ddsketch (optional, default is 0.005)

The arguments for ddsketch_compare.py are:
--file1: Path to first ddsketch json
--file2: Path to second ddsketch json
--txn1: The transaction type for the first file
--txn2: The transaction type for the second file
--op: The operation name (ex: GRV, GET ...)

The arguments for export_graph.py:
--file: path to ddsketch distribution
--txn, -t: Transaction type from file
--title: title for graph (optional, otherwise "Title" is used)
--savefig: Path to save the image plot (optional)
--op: Which operation to plot
2022-07-25 10:29:33 -07:00
Markus Pilman
80d4c85a2f
Merge pull request #7686 from sfc-gh-jshim/wingdi-name-collision
Fix WinGDI name collision by defining macro NOGDI
2022-07-25 11:26:40 -06:00
Jingyu Zhou
72e2933763
Merge pull request #7678 from xis19/clang
fixup! Let clang accepts crc32_wrapper.c
2022-07-25 09:31:07 -07:00
Zhe Wu
2fdd9d7407 Add testing mechanism 2022-07-25 09:18:02 -07:00
Zhe Wu
98fbc98274 Error in FetchKey GRV shouldn't fail storageserver 2022-07-25 09:18:02 -07:00
Josh Slocum
01b7e5395e
Merge pull request #7530 from sfc-gh-jslocum/forgot_java_bindings
Java bindings for BG Purge (#6816)
2022-07-25 08:56:51 -05:00
Bharadwaj V.R
dcf09fd691
Move DD queue code over to using movement-reasons rather than priority (#7614)
* Use enum variables to invoke Priority checking

* add an explicit isDataMovementForReadBalancing function

* Set up RelocateShard in terms of data movement reason instead of priority

* Remove isMountainChopperPriority

* Remove isDiskRebalancePriority

* Fix formatting

* Fix misnamed DataMovementReason::TEAM_HEALTHY

Co-authored-by: Zhongxing Zhang <zhongxing.zhang@snowflake.com>
2022-07-25 00:50:37 -07:00
Junhyun Shim
2a172e1586 Fix WinGDI name collision by defining macro NOGDI
TenantState::ERROR collides with WinGDI's #define ERROR 0
2022-07-25 09:11:34 +02:00
Jingyu Zhou
16519a9e5f Update status json doc with fetch_storage_wiggler_stats_timeout error 2022-07-24 15:24:21 -07:00
Jingyu Zhou
ed8c64f1d3 Reduce a DNS lookup in Status generation 2022-07-24 15:17:46 -07:00
Jingyu Zhou
da1865d6ac Add timeout during Status generation
Resolving hostname might be blocking.
2022-07-24 13:49:34 -07:00
Xiaoxi Wang
7b6f1ca712
Merge pull request #7621 from sfc-gh-xwang/feature/dd-refactor-incremental
[DD Testability] Add unittest for resuming infligh relocations
2022-07-23 22:14:45 -07:00
He Liu
03755d2489 Fixed compiler warnings. 2022-07-23 20:06:26 -07:00
Xiaoge Su
8ea3ef5639 fixup! Let clang accepts crc32_wrapper.c 2022-07-23 16:38:10 -07:00
Andrew Noyes
83a317ef88
Disable guard pages for valgrind build. Closes #7610 (#7672) 2022-07-22 16:51:05 -07:00
A.J. Beamon
9459d0c6ca
Merge pull request #7661 from sfc-gh-ajbeamon/tenant-metadata-key-backed-types
Use key-backed types for tenants
2022-07-22 14:01:45 -07:00
Andrew Noyes
79f60691e9
Don't pass non-pod to variadic function (#7668)
fdbserver/storageserver.actor.cpp:654:24: error: cannot pass object of non-trivial type 'const std::string' (aka 'const basic_string<char, char_traits<char>, allocator<char>>') through variadic function; call will abort at runtime [-Wnon-pod-varargs]

Just use fmt instead
2022-07-22 12:51:44 -07:00
A.J. Beamon
b735bf8eb0 Add comments to deleteTenantTransaction. Fetch the tenant ID from the first transaction in deleteTransaction to use on subsequent retries. 2022-07-22 12:34:27 -07:00
Lukas Joswiak
7fb427f4ec Update fdbserver/GrvProxyServer.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-07-22 10:37:29 -07:00
Lukas Joswiak
a26344b877 Update fdbserver/GrvProxyServer.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-07-22 10:37:29 -07:00
Lukas Joswiak
8395c6cf3f Fix Tuple error 2022-07-22 10:37:29 -07:00
Lukas Joswiak
5fa8e3f7d0 Simplify refresh condition 2022-07-22 10:37:29 -07:00
Lukas Joswiak
703aa1d279 Mess with timeout values 2022-07-22 10:37:29 -07:00
Lukas Joswiak
6c17ca038a Add delay to allow for global config sync time 2022-07-22 10:37:29 -07:00
Lukas Joswiak
40d403ed5f Reduce global configuration system key reads from proxy
Clients now poll the proxy for the latest global config for a specific
version. The proxy now periodically requests the latest global
configuration data and stores it in memory, enabling it to respond
immediately to clients with the appropriate version.
2022-07-22 10:37:29 -07:00
Lukas Joswiak
56dfdbda83 Add migration timeout 2022-07-22 10:37:29 -07:00
Lukas Joswiak
c32f3f8746 Update fdbclient/include/fdbclient/CommitProxyInterface.h
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-07-22 10:37:29 -07:00
Lukas Joswiak
e024db85a0 Update fdbserver/GrvProxyServer.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-07-22 10:37:29 -07:00
Lukas Joswiak
7a48a53778 Perform migration automatically on proxy boot 2022-07-22 10:37:29 -07:00
Lukas Joswiak
2e99d5f6cc Batch global config refresh requests 2022-07-22 10:37:29 -07:00
Lukas Joswiak
c33e44b0f4 Proxy GlobalConfig reads through GRV proxies
Clients should avoid reading system keys unless authorized. Under global
config, each client reads from the system keyspace to check for new
global config keys. This commit moves these reads to a server role (the
GRV proxies) and sends the results back to GlobalConfig for an in-memory
update.
2022-07-22 10:37:29 -07:00
He Liu
7a8be255cd
Ss shard management (#7340)
* Storage server shard management with physical shards.

* Cleanup.

* Resolved comments.

* Added `UnlimintedCommitBytes`.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-22 09:30:44 -07:00
Jingyu Zhou
c759357687
Merge pull request #7662 from xis19/databaseContext
Add TraceEvent to the construction/destruction of DatabaseContext
2022-07-22 09:29:56 -07:00
Xiaoge Su
25d9fa856c fixup! Update code per comments 2022-07-21 23:14:31 -07:00
Zhe Wu
5441eb4610 Add CommitDebug event after Commit Proxy reporting committed version to master 2022-07-21 21:29:53 -07:00
A.J. Beamon
17146c484b Use key-backed types for tenants. Add a tenant state field that will be used in upcoming work. Some other tenant related refactoring. 2022-07-21 20:33:28 -07:00
Xiaoge Su
32b4968835 Add TraceEvent to the construction/destruction of DatabaseContext
This is to figure out why huge databaseLogger related TraceEvents
generated after 7.1 released.
2022-07-21 19:38:59 -07:00
Andrew Noyes
89141d4b3a
Validate subrange reads in simulation (#7597)
* Add extra validation to special key space reads in simulation

* Fix bugs turned up by validating subrange reads

* Change to validateSpecialSubrangeRead

It is in general not safe to expect that a read from the special key
space returns the same results if performed again, since the
transaction may be being modified concurrently.

* Add comment

* Add comment
2022-07-21 14:42:08 -07:00
He Liu
257bab7311
Allow read range on null shard (#7654)
* Fixed ShardedRocks unit test.

* Removed `UNIT` from noSim/ShardedRocksDBTest.toml.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-21 13:59:49 -07:00
Zhe Wu
bd99f4fa3b Log tlog initialization 2022-07-21 13:54:44 -07:00
Andrew Noyes
45c8a4e09d
Build fixes/improvements for sanitizers (#7657)
* Don't build fdb c shim with ubsan

This avoids duplicate symbols when linking. It doesn't really make sense
to assemble files with -fsanitize=undefined anyway, since it won't
insert instrumentation.

* Consolidate boost_asan with boost_target
2022-07-21 12:43:38 -07:00
Trevor Clinkenbeard
9d63a859b0
Merge pull request #7443 from sfc-gh-xwang/readaware
[Read-aware DD] Use KeyRangeStorageMetrics instead of StorageMetrics
2022-07-21 12:37:51 -07:00
Xiaoxi Wang
01569963da update comments 2022-07-21 09:26:20 -07:00