20038 Commits

Author SHA1 Message Date
Bharadwaj V.R
8a0ce5bfc7 Rename isInvalidMagic and fix formatting 2022-04-20 09:17:28 -07:00
Bharadwaj V.R
96ab7c0254
Update fdbserver/Status.actor.cpp
Co-authored-by: Trevor Clinkenbeard <trevor.clinkenbeard@snowflake.com>
2022-04-20 08:52:14 -07:00
Bharadwaj V.R
a2449041ea Fix formatting of ProtocolVersion.h 2022-04-20 08:45:53 -07:00
Bharadwaj V.R
4d6f4ecd9c
Merge branch 'main' into dbcorever 2022-04-20 08:23:34 -07:00
Ray Jenkins
5df3bac110
Migrate to OpenTelemetry tracing. (#6855) 2022-04-20 09:26:37 -05:00
Bharadwaj V.R
a711c55061
Merge branch 'apple:main' into dbcorever 2022-04-20 06:16:27 -07:00
Evan Tschannen
442d2b34c7
fix: pops which were ignored during a snapshot would not be replayed on the proper tlogs within a shared tlog (#6892) 2022-04-19 16:57:41 -07:00
Andrew Noyes
297d831192
Put guard pages next to fast alloc memory (#6885)
* Put guard pages next to fast alloc memory

I verified that we can now detect #6753 without creating tons of
threads.

* Use pageSize instead of 4096

* Don't include mmapInternal for windows
2022-04-19 11:22:35 -07:00
Vaidas Gasiunas
eedac50bb5
Merge pull request #6888 from sfc-gh-vgasiunas/vgasiunas-upgrade-test-72
Regression tests for upgrades to 7.2
2022-04-19 17:42:01 +02:00
Vaidas Gasiunas
3372b3a6eb Upgrade Tests: Check logs for warnings and dump them 2022-04-19 13:37:21 +02:00
Vaidas Gasiunas
79b2ebeb4b Upgrade Tests: Test upgrade from 7.0 to 7.2 2022-04-19 11:53:33 +02:00
Vaidas Gasiunas
43b3ac23f8
Merge pull request #6876 from sfc-gh-ljoswiak/fixes/upgrade-tests-arm-support
Disable end-to-end upgrade tests for non-x86_64 platforms
2022-04-19 10:55:35 +02:00
Ata E Husain Bohra
a38318a6ac
Update 'salt' details for EncryptHeader AuthToken details (#6881)
* Update 'salt' details for EncryptHeader AuthToken details

Description

Major changes:
1. Add 'salt' to BlobCipherEncryptHeader::cipherHeaderDetails.
2. During decryption it is possible that BlobKeyCacheId doesn't
    contain required baseCipherDetails. Add API to KeyCache to
    allowing re-populating of CipherDetails with a given 'salt'
3. Update BaseCipherKeyIdCache indexing using {BaseCipherKeyId, salt}
    tuple. FDB processes leverage BlobCipherKeyCache to implement
    in-memory caching of cipherKeys, given EncryptKeyProxy supplies
    BaseCipher details, each encryption participant service would
    generate its derived key by using different 'salt'. Further,
    it is possible to cache multiple {baseCipherKeyId, salt} tuples;
    for instance: CP encrypted mutations being deciphered by
    StorageServer etc.

Testing

1. Update EncyrptionOps simulation test to simulate KeyCache miss
2. Update BlobCipher unit tests to validate above mentioned changes
2022-04-18 22:01:56 -07:00
Jingyu Zhou
8003f4207f
Merge pull request #6843 from sbodagala/main
Do not invoke version vector related code when the feature is disabled
2022-04-18 21:35:23 -07:00
Ray Jenkins
d45d6f9f1f
Merge pull request #6882 from sfc-gh-rjenkins/typo-client-testing
Fix typo, expose a function...
2022-04-18 16:43:42 -05:00
Sreenath Bodagala
abd3d5a3d7 Merge remote-tracking branch 'apple-upstream/main' 2022-04-18 20:55:16 +00:00
Russell Sears
8819534a5e
Improve YCSB image logging; remove logic to support partial restarts (#6830) 2022-04-18 13:32:04 -07:00
Ray Jenkins
865b5f1ff8 Substitute other with author. 2022-04-18 14:50:30 -05:00
Ray Jenkins
28ee6fd7b2 Fix typo, expose a function... 2022-04-18 14:45:03 -05:00
Lukas Joswiak
01807502ca Remove Arm support 2022-04-18 12:39:39 -07:00
Lukas Joswiak
07d11ec2e1 Fix failing upgrades due to non-persisted initial cluster version 2022-04-18 10:59:17 -07:00
Markus Pilman
3cbba4bea4
Don't test requests that don't initialize properly (#6880)
* Don't test requests that don't initialize properly

Some request objects don't initialize their members
properly when being constructed using the default
constructor. This makes valgrind unhappy. Don't test
these endpoints for now.

* fixed code formatting
2022-04-18 10:44:56 -07:00
Jingyu Zhou
17dc1a61f3 ClientDBInfo may be unintentionally not set
The ClientDBInfo's comparison is through an internal UID and shrinkProxyList()
can change proxies inside ClientDBInfo. Since the UID is not changed by that
function, subsequent set can be unintentionally skipped.

This was not a big issue before. However, VV introduces a change that the
client side compares the returned proxy ID with its known set of GRV proxies
and will retry GRV if the returned proxy ID is not in the set. Due the above
bug, GRV returned by a proxy is not within the client set, and results in
indefinite retrying GRVs.
2022-04-18 09:09:14 -07:00
Markus Pilman
1f26943099
Merge pull request #6859 from sfc-gh-ajbeamon/check-tenant-clear-range
When clearing the database between tests, check that the normal key-space is empty
2022-04-16 11:24:41 -06:00
Lukas Joswiak
7b737e91a1 Disable upgrade tests an non-x86_64 architectures 2022-04-15 16:28:11 -07:00
Lukas Joswiak
86a4e778d6 Enable Arm architecture in end-to-end upgrade tests 2022-04-15 16:24:49 -07:00
Bharadwaj V.R
47a1a282e9
Merge branch 'apple:main' into dbcorever 2022-04-15 15:47:14 -07:00
A.J. Beamon
6151f9c858
Merge pull request #6873 from sfc-gh-ajbeamon/tenant-test-fix
The tenant deletion test now deletes multiple tenants concurrently rather than serially
2022-04-15 14:29:47 -07:00
Jingyu Zhou
0a03b190da Fix multiple PeekStream requests to log routers
There is a bug in how a log router handles streaming read:
* Log router has a `logRouterPeekStream` actor A running.
* Remote tlog detects some problem and starts another streaming connection (maybe just reuse the connection?)
* Log router now has a new `logRouterPeekStream` actor B running.
* B runs and found that popped version > reqBegin, so `LogRouterPeekPopped` . This is because A is still running and changed the popped version.
* A ends with `TLogPeekStreamEnd operation_obsolete`
* B become stuck at `wait(req.reply.onReady() && store(reply.rep, future)`, because the future was sent `Never()`.

As a result, the remote tlog can no longer retrieve data from this log router.

Fix by killing the `logRouterPeekStream` B.
2022-04-15 14:11:52 -07:00
Bharadwaj V.R
11f2827c82 Make names even across change and add lowest-compatible-version to status json 2022-04-15 13:57:58 -07:00
Vaidas Gasiunas
eb436ec14b
Merge pull request #6868 from sfc-gh-vgasiunas/vgasiunas-disable-upgrade-tests-sanitizer
Disable upgrade tests in sanitizer builds
2022-04-15 22:55:15 +02:00
A.J. Beamon
e2222355dc The tenant deletion test now deletes multiple tenants concurrently rather than serially. Fix some variable shadowing in the delete test. 2022-04-15 13:17:19 -07:00
neethuhaneesha
5ffd9e0172
Merge pull request #6867 from neethuhaneesha/rocksdb-aggrProperty
RocksDb using aggr property metrics for pendingCompactionBytes.
2022-04-15 11:02:29 -07:00
Bharadwaj V.R
ce5bcc0261 Remove system key tracking newest server version 2022-04-15 10:57:11 -07:00
Bharadwaj V.R
576957d335 Fix version status reporting 2022-04-15 09:52:42 -07:00
Bharadwaj V.R
3787ddae89 Add trace-based status reporting for server version 2022-04-15 09:04:52 -07:00
Vaidas Gasiunas
5abdb1e655 Upgrade Tests: Disable upgrade tests in sanitizer builds 2022-04-15 16:33:48 +02:00
Neethu Haneesha Bingi
6543bce8ae RocksDb using aggr property metrics for pendingCompactionBytes. 2022-04-14 18:08:42 -07:00
Andrew Noyes
29cf5f1fbf
Fix an ASSERT when an fdbcli command times out (#6857)
* Re-throw operation_cancelled

There's a few places in fdbcli where we don't rethrow operation
cancelled but wait on a future. It's very unusual that you don't want to
rethrow operation_cancelled.

* Update ASSERT

It's possible to get error_code_broken_promise here if the network has
already shutdown.
2022-04-14 12:09:25 -07:00
A.J. Beamon
cf5d3c83a1 Fix formatting issues. 2022-04-14 12:03:39 -07:00
A.J. Beamon
19d78cf2a3 When clearing the database between tests, check that clearing the tenant left the entire normal key-space empty. Update the configuration of some tests. Disable a special key-space test that is invoking broken behavior. 2022-04-14 11:39:02 -07:00
Bharadwaj V.R
adfc4ca379 Revert accidental commit of a ProtocolVersion.h 2022-04-14 11:12:26 -07:00
Bharadwaj V.R
78c4771f9d Fix ProtocolVersion.h formatting diffs 2022-04-14 11:10:39 -07:00
Bharadwaj V.R
3c080c6382
Merge branch 'apple:main' into dbcorever 2022-04-14 10:39:32 -07:00
A.J. Beamon
218ab6377c
Merge pull request #6651 from sfc-gh-vgasiunas/vgasiunas-upgrade-test
Automated end-to-end upgrade tests
2022-04-14 09:36:52 -07:00
Bharadwaj V.R
d9f7b97315 Add software version to system key for status reports 2022-04-14 07:19:41 -07:00
Vaidas Gasiunas
0a6b74ad3f Upgrade Tests: Remove 6.2 from regression testing, because of sporadic failures 2022-04-14 14:30:55 +02:00
Markus Pilman
3598c6b56b
Merge pull request #6675 from sfc-gh-jshim/tenant-token-sign
Sign and verify auth tokens for multi-tenant FDB
2022-04-13 16:55:20 -06:00
Zhe Wang
2f75a4bd78
Use actor collection for rocksdb histogram actors. (#6805)
Co-authored-by: Zhe Wang <zhewang@Zhes-MacBook-Pro.local>
2022-04-13 14:41:54 -07:00
Junhyun Shim
edc659d339 Use camelCase & move error code to 6xxx 2022-04-13 21:11:52 +02:00