21551 Commits

Author SHA1 Message Date
He Liu
013b9e3baa
Fixed ChangeServerKeysContext name issue. (#7761)
* Fixed ChangeServerKeysContext name issue.

* Update fdbserver/storageserver.actor.cpp

Co-authored-by: Andrew Noyes <andrew.noyes@snowflake.com>

Co-authored-by: He Liu <heliu@apple.com>
Co-authored-by: Andrew Noyes <andrew.noyes@snowflake.com>
2022-08-02 13:54:27 -07:00
Xiaoxi Wang
07eafcec93
Merge pull request #7763 from sfc-gh-xwang/feature/main/unittest
move waitForMost into generic actors
2022-08-02 13:30:30 -07:00
Andrew Noyes
5dbb6f1dd3
Make Tuple::pack return a Standalone<StringRef> (#7764)
This makes it less error-prone and more like other similar functions
like BinaryWriter::toValue

Closes #7748
2022-08-02 12:45:56 -07:00
Chaoguang Lin
48e46cbc81
Add test coverage for SpecialKeyRangeAsyncImpl::getRange (#7671)
* Add getRange test coverage for SpecialKeyRangeAsyncImpl

* Fix the bug in SpecialKeyRangeAsyncImpl found by the test

* Refactor ConflictingKeysImpl::getRange to use containedRanges to simplify the code

* Fix file format

* Initialize SpecialKeyRangeAsyncImpl cache with correct end key

* Add release notes

* Revert "Refactor ConflictingKeysImpl::getRange to use containedRanges to simplify the code"

This reverts commit fdd298f469bf73b107e5ca29072c49774820d3d6.
2022-08-02 12:04:40 -07:00
Xiaoxi Wang
3c76ad9e72 move waitForMost into generic actors 2022-08-02 10:38:11 -07:00
Josh Slocum
4b66645d80
Granule file performance benchmark and improvements (#7742)
* added cpu microbenchmark for blob granule files

* Added edge case read benchmarks, and sorting memory deltas

* Sorted merge for granule files

* key block comparison optimization in granule files

* More performance improvements to granule file read

* fixing zlib not supported build

* fixing formatting

* Added debug macro for new debugging prints

* review comments

* more strict compression size validation assert
2022-08-02 11:36:44 -05:00
Josh Slocum
b1ff8b8340
Merge pull request #7753 from sfc-gh-jslocum/cf_op_chaos
Add Chaos to Change Feed Operations test
2022-08-02 09:04:38 -05:00
Josh Slocum
7b9a9c0176
Merge pull request #7752 from sfc-gh-jslocum/rearrange_granule_tests
disable movekeys granule test and shift other granule tests around
2022-08-02 09:03:36 -05:00
Hao Fu
9db19b9a7c
Retain debug id in prefetch server-server call (#7754) 2022-08-01 19:46:45 -07:00
Ata E Husain Bohra
ef6012c1d1
Encrypt BlobGranule delta files (#7735)
* Encrypt  BlobGranule delta files

Description

 diff-1: Address review comments

Major changes proposed by the patch are:
1. Refactor code to allow caching of 'encryption key ctx' as part of
BlobFilePointerRef. The refactoring allows snapshot and/or delta files
to store their own file encryption context.
2. Enable BlobGranule delta file encryption/decryption semantics.

Testing

BlobGranuleCorrrectness  
BlobGranuleCorrectnessClean
BlobGranuleFileUnitTestToml

Description

Testing
2022-08-01 16:34:44 -07:00
Josh Slocum
c5c423f2b7 Disabling update/clear post-stop for now in change feed ops 2022-08-01 17:42:32 -05:00
Josh Slocum
83806b4a31 disable movekeys granule test and shift other granule tests around 2022-08-01 14:48:38 -05:00
Renxuan Wang
51b92d59b9
Merge pull request #7733 from RenxuanW/hostname-backup
Prefer IPv6 in hostname resolving.
2022-08-01 12:34:02 -07:00
Renxuan Wang
4a0bea2230 Document for pickOneAddress(). 2022-08-01 10:54:04 -07:00
Markus Pilman
9caece6fc2
Merge pull request #7743 from sfc-gh-ajbeamon/fix-dummy-transaction
Fix: Transactions could change tenant ID part way through
2022-08-01 11:28:32 -06:00
Markus Pilman
204ef3a933
Merge pull request #7734 from sfc-gh-anoyes/anoyes/require-python3
Disallow python2 for cmake scripts
2022-07-31 22:18:56 -06:00
A.J. Beamon
ea76eb2beb Transactions could change tenant ID part way through. The dummy transaction could set a new tenant ID but use the old tenant prefix. 2022-07-30 16:16:18 -07:00
A.J. Beamon
eeefb12f14
Merge pull request #7744 from sfc-gh-ajbeamon/fix-token-unit-test
Fix bug in token cache unit test where the expiration time was underflowing
2022-07-30 15:40:14 -07:00
A.J. Beamon
5a071e1b19 Fix bug in token cache unit test where the expiration time was underflowing. 2022-07-30 14:22:24 -07:00
Xiaoxi Wang
85f2200d6b
Merge pull request #7741 from sfc-gh-xwang/refactor/main/enableState
create ShardsAffectedByTeamFailure.h
2022-07-29 16:20:08 -07:00
Xiaoxi Wang
dd902fc772 create ShardsAffectedByTeamFailure.h 2022-07-29 15:12:15 -07:00
Andrew Noyes
1558edce8c Python_EXECUTABLE to Python3_EXECUTABLE 2022-07-29 14:57:29 -07:00
Andrew Noyes
db348598a1 Update Python target name to Python3 2022-07-29 14:57:29 -07:00
Andrew Noyes
52e9990d32 Python_Interpreter_FOUND -> Python3_Interpreter_FOUND 2022-07-29 14:57:29 -07:00
Andrew Noyes
efc3045834 Disallow python2 for cmake scripts
The fdb c shim build requires python3. Let's not allow python2 to get
used there.

Otherwise we can see something like this:
```
FAILED: bindings/c/libfdb_c.so.init.c bindings/c/libfdb_c.so.tramp.S
cd /home/anoyes/workspace/foundationdb/build/bindings/c && /usr/bin/python2.7 /home/anoyes/workspace/foundationdb/contrib/Implib.so/implib-gen.py --target aarch64 --outdir /home/anoyes/workspace/foundationdb/build/bindings/c --dlopen-callback=fdb_shim_dlopen_callback /home/anoyes/workspace/foundationdb/build/lib/libfdb_c.so
  File "/home/anoyes/workspace/foundationdb/contrib/Implib.so/implib-gen.py", line 27
    sys.stderr.write(f'{me}: warning: {msg}\n')
```
2022-07-29 14:57:29 -07:00
Markus Pilman
50fd0f11dd
Merge pull request #7738 from sfc-gh-mpilman/bugfixes/investigate-token-cache-segfaults
fix token cache unit test
2022-07-29 13:55:51 -06:00
Markus Pilman
b600966cf9 fix token cache unit test 2022-07-29 12:47:55 -06:00
Nim Wijetunga
3e45b6657c
Merge pull request #7714 from sfc-gh-nwijetunga/nim/tenant-encryption-property
Encryption as a tenant property
2022-07-29 11:30:19 -07:00
Xiaoxi Wang
c21aa193e6
Merge pull request #7736 from sfc-gh-xwang/refactor/main/teamrequest
[Read-aware DD] remove duplicate code in BgDDLoadRebalance
2022-07-29 11:24:52 -07:00
Xiaoxi Wang
f0caaad93c
Update fdbserver/DataDistributionQueue.actor.cpp 2022-07-29 10:22:21 -07:00
Xiaoxi Wang
e3cf40002d format code 2022-07-29 10:11:10 -07:00
Xiaoxi Wang
d8323820f1 remove duplicate code 2022-07-29 10:04:14 -07:00
Josh Slocum
cfc13e7018
Adding more blobrange cli commands and a couple other tweaks (#7727) 2022-07-29 08:20:45 -07:00
Josh Slocum
ddad5b189e fixing change feed ops test race with live reader and destroying 2022-07-29 07:31:22 -05:00
Vaidas Gasiunas
d99d0370b1
C shim library: API for setting client library path; additional tests (#7702)
* Adding sources of the Implib.so project

* Run C unit tests and API tests with the shim library

* Reuse compilation of C test binaries with and without shim library

* Resolve client library path from an environment variable

* Refactoring: Reusable module for downloading FDB binaries

* Testing client shim library with current version and last release version

* Tests for specifying client library over an environment variable

* Enable C shim library tests on ARM

* Restore the original path for including fdb_api.hpp

* Improve cmake dependencies on doctest

* Remove unnecessary Implib.so files

* Shim library: API to specify path to the library, additional tests

* Adding shim library to Linux distribution packages

* Update link dependencies of fdb_c_shim_lib_tester

* Add dependencies on Implib.so sources

* Abort on fatal errors of shim library

* fix typo
2022-07-29 11:45:45 +02:00
Renxuan Wang
316ea60232 Prefer IPv6 in hostname resolving. 2022-07-28 20:07:16 -07:00
Hao Fu
62bee7ac9a
Add g_traceBatch for prefetch query (#7730) 2022-07-28 17:54:31 -07:00
Trevor Clinkenbeard
8491c2530f
Merge pull request #7416 from sfc-gh-ljoswiak/fixes/change-coordinators-commit-unknown-result
Ensure change coordinators request came from the same generation proxy
2022-07-28 17:29:24 -07:00
Dan Lambright
83d23512d2
Track when version not found in version vector. (#7653)
* Track when version not found in version vector.

* Respond to comments

* include vv in event

* Respond to review comments.

* Fix bug where we logged too many lines
2022-07-28 18:26:15 -04:00
He Liu
35a4cb91d5
Disable ShardedRocksDB in simulation when shard_encode_location_metadata is disabled (#7726)
* Disabled tests for ShardedRocks.

Cleaned up ShardedRocks TraceEvent.

Added assertion in ShardManager::validate().

* Added test trace.

* Make sure TraceEvent contains `ShardedRocks`.

* Exclude ShardedRocksDB when SHARD_ENCODE_LOCATION_METADATA is disabled.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-28 13:54:29 -07:00
Markus Pilman
ade9d20780
Merge pull request #7628 from sfc-gh-mpilman/features/code-coverage-test-harness
Change test harness to include comments
2022-07-28 13:54:21 -06:00
Junhyun Shim
83e94781b4
Merge pull request #6842 from sfc-gh-mpilman/features/authz
Authorization
2022-07-28 19:50:24 +02:00
Markus Pilman
92b1d1f647 Don't fail TestHarness if comment doesn't exist 2022-07-28 11:03:09 -06:00
Markus Pilman
5469e58562 Merge remote-tracking branch 'origin/main' into features/code-coverage-test-harness 2022-07-28 10:01:20 -06:00
Nim Wijetunga
f07a38b75a use default values 2022-07-27 21:17:28 -07:00
Nim Wijetunga
ed0b1ed030 default init 2022-07-27 19:53:49 -07:00
Nim Wijetunga
dd8bf5076f initalize bool 2022-07-27 18:44:42 -07:00
Nim Wijetunga
7d287edc30 initalize values 2022-07-27 17:29:10 -07:00
Xiaoxi Wang
4091a567ab
Merge pull request #7725 from sfc-gh-xwang/fix/main/status
fix storage_metadata disappear because other attribution is missed
2022-07-27 17:14:08 -07:00
Nim Wijetunga
cc1b8d27c9 fix tests 2022-07-27 16:26:59 -07:00