2432 Commits

Author SHA1 Message Date
Hui Liu
c9b4fc5761 disallow MoveKeysWorkload running in parallel 2022-09-22 17:00:01 -07:00
Josh Slocum
430f6e9670
Fix purge at latest racing with other checking threads at the end of BlobGranuleVerify (#8281) 2022-09-22 16:32:33 -07:00
A.J. Beamon
bd006526d6
Merge pull request #8251 from sfc-gh-ajbeamon/fdbcli-tenant-group-metadata
Fdbcli command to get tenant group metadata
2022-09-22 14:17:08 -07:00
A.J. Beamon
97a325adab Add an fdbcli command to get tenant group metadata 2022-09-22 13:24:21 -07:00
Steve Atherton
7831f6b2f1
Merge pull request #7578 from sfc-gh-fzhao/RedwoodIOLaunchLimit
[DRAFT] Redwood PriorityMultiLock enable different launch limits to be specified based on different priority level.
2022-09-22 12:29:17 -07:00
Nim Wijetunga
37b93a6232
fix tests (#8286) 2022-09-22 11:34:06 -07:00
Steve Atherton
04b4960786 Merge branch 'main' into fzhao/RedwoodIOLaunchLimit
# Conflicts:
#	fdbserver/VersionedBTree.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/ReadWrite.actor.cpp
2022-09-22 00:39:51 -07:00
Jon Fu
7a09b701cc
Merge pull request #8141 from sfc-gh-jfu/network-disable-bypass
Introduce network option for disabling mvc bypass
2022-09-21 17:33:48 -07:00
Hui Liu
bf943bfd57
Merge pull request #8273 from sfc-gh-huliu/testfix
disable MoveKeysWorkload when running SpecialKeySpaceCorrectness test
2022-09-21 17:32:44 -07:00
Jon Fu
937fb38dcc address review comments 2022-09-21 16:23:12 -07:00
Hui Liu
f115599a67 disable MoveKeysWorkload when running SpecialKeySpaceCorrectness test 2022-09-21 16:12:20 -07:00
Josh Slocum
4794ebabcc
Merge pull request #8262 from sfc-gh-dzhou/purge-fixes
blob: java api fixes (purge + verify)
2022-09-21 17:25:54 -05:00
Dennis Zhou
4ea4546cb6 blob/java: verifyBlobRange() with latestVersion 2022-09-21 14:07:16 -07:00
Dennis Zhou
e353169a50 blob: teach purge about latestVersion
This teaches purgeBlobGranules about latestVersion and rejects
versions <= 0.
2022-09-21 14:04:58 -07:00
Josh Slocum
6270016bed
Seq insert perf fixes main (#8264)
* Force flushing granules post-split to guarantee parent feeds get cleaned up

* fixing bug and cleaning up split finalize code
2022-09-21 12:36:02 -07:00
Jon Fu
0343ca9c53 Merge branch 'main' of github.com:apple/foundationdb into network-disable-bypass 2022-09-21 11:33:27 -07:00
Hui Liu
c237186ec3
Merge pull request #8252 from sfc-gh-huliu/fixtest
Fix correctness test: skip lock check in MoveKeysWorkload
2022-09-20 16:34:22 -07:00
Trevor Clinkenbeard
3b5117ca92
Merge pull request #8233 from sfc-gh-mpilman/features/disable-failure-injection-for-workload
All workloads to disable some failures
2022-09-20 16:17:35 -07:00
Nim Wijetunga
eadb769cfa
Encrypt Backup Mutation Log (#8159)
* encrypt backup mutation log

* format

* address pr comments

* format

* fix bug

* revert knobs

* address pr comments
2022-09-20 15:43:39 -07:00
Chaoguang Lin
9628561235
Add DataDistributionMetrics workload into correctness packages, (#8237)
which makes the code probe hit in nightly tests.
2022-09-20 15:33:15 -07:00
Hui Liu
52f1bef8ec Fix correctness test: skip lock check in MoveKeysWorkload 2022-09-20 15:16:56 -07:00
Jon Fu
1f778f9d76 Merge branch 'main' of github.com:apple/foundationdb into network-disable-bypass 2022-09-20 09:30:53 -07:00
Andrew Noyes
4ac5498f2e Fix a UBSAN diagnostic caused by use of uninitialized memory
The diagnostic is

fdbserver/workloads/DiskFailureInjection.actor.cpp:57:64: runtime error: load of value 208, which is not a valid value for type 'bool'

The fix is simply to initialize the default value of verificationMode
2022-09-20 08:54:45 -07:00
Markus Pilman
62ac01b481 All workloads to disable some failures 2022-09-19 16:42:55 -06:00
Chaoguang Lin
125137b987
Change the special key space correctness workload to hit code probe (#8214) 2022-09-19 15:01:21 -07:00
Josh Slocum
0f3f493c28
Merge pull request #8218 from sfc-gh-jslocum/aligned_purge
fixes for non-aligned blob range calls
2022-09-19 15:00:02 -05:00
A.J. Beamon
4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
Markus Pilman
e1627e0a78 Merge remote-tracking branch 'origin/main' into features/always-inject-faults 2022-09-19 09:38:55 -06:00
Josh Slocum
88f88707f5 fixes for non-aligned blob range calls 2022-09-16 19:06:15 -05:00
Markus Pilman
ef2d301305 Merge remote-tracking branch 'origin/main' into features/always-inject-faults 2022-09-16 16:41:16 -06:00
Jon Fu
1abac8ea9f check on shared state ptr in native api and add to test spec in api tester 2022-09-16 15:11:33 -07:00
Markus Pilman
9441795c8e address review comments 2022-09-16 14:40:10 -06:00
sfc-gh-ngoyal
1bd97fe628
Recruit new singleton for consistency checker. (#5804)
* Recruit new singleton for consistency checker.

* Recruit the consistency checker only if enabled.

* Add a yield in monitorConsistencyChecker().

* Minor fixes.

* Consistency check workload enhancements.

* Minor fixes and clarifications.

* clang format

* Clang format.

* Minor fixes, cleanup, debug tracing.

* Misc.

* Move the consistency scan information from dbconfig to a key backed object.

* Move consistency scan config out of db cofig to a state object and feature rename.

* ConsistencyCheck workload refactor.

* devFormat

* Update fdbcli/ConsistencyScanCommand.actor.cpp

* Review Comments.

Co-authored-by: negoyal <neelam.goyal@gmail.com>
Co-authored-by: Ata E Husain Bohra <ata.husain@snowflake.com>
2022-09-16 09:03:06 -07:00
Fuheng Zhao
ac65c3f569 merge upstream main 2022-09-15 14:19:19 -07:00
sfc-gh-tclinkenbeard
b9fbd8c130 Fix -Wlogical-op-parentheses warning 2022-09-15 09:00:33 -07:00
sfc-gh-tclinkenbeard
82adc1e856 Make g_simulator a pointer 2022-09-15 09:00:33 -07:00
Lukas Joswiak
cfbd04ae78 Add explicit check for a simulated network 2022-09-14 14:14:49 -07:00
Ata E Husain Bohra
d2b82d2c46
Introduce "default encryption domain" (#8139)
* Introduce "default encryption domain"

Description

In current FDB native encryption data at-rest implementation,
an entity getting encrypted (mutation, KV and/or file) is categorized
into one of following encryption domains:
1. Tenant domain, where, Encryption domain == Tenant boundaries
2. FDB system keyspace - FDB metadata encryption domain
3. FDB Encryption Header domain - used to generate digest for
plaintext EncryptionHeader.

The scheme doesn't support encryption if an entity can't be categorized
into any of above mentioned encryption domains, for instance, non-tenant
mutations are NOT supported.

Patch extend the encryption support for mutations for which corresponding
Tenant information can't be obtained (Key length shorter than TenantPrefix)
and/or mutations do not belong to any valid Tenant
(FDB management cluster data) by mapping such mutations to a
"default encryption domain".

TODO

CommitProxy driven TLog encryption implementation requires every transaction
mutation to contain 1 KV, not crossing Tenant-boundaries. Only exception to
this rule is ClearRange mutations. For now ClearRange mutations are mapped
to 'default encryption domain', in subsequent patch appropriate handling
for ClearRange mutations shall be proposed.

Testing

devRunCorrectness - 100k
2022-09-14 10:58:32 -07:00
Lukas Joswiak
424bb87f3e Apply feedback 2022-09-13 16:53:54 -07:00
Lukas Joswiak
2a26777d64 Disable coordinator change when using simple configuration database 2022-09-13 16:53:54 -07:00
Lukas Joswiak
74ac617a34 Add support for changing coordinators to the configuration database
Configuration database data lives on the coordinators. When a change
coordinators command is issued, the data must be sent to the new
coordinators to keep the database consistent.
2022-09-13 16:53:54 -07:00
Trevor Clinkenbeard
b641bd6c04
Merge pull request #8168 from sfc-gh-tclinkenbeard/add-doappendiffits-unittest
Add `/Atomic/DoAppendIfFits` unit test
2022-09-13 15:18:59 -07:00
Xiaoxi Wang
2ae01bdf2d
Merge pull request #8162 from sfc-gh-xwang/feature/main/moveKey
Implement txnProcessor->moveKeys(const MoveKeysParams& params)
2022-09-13 14:11:49 -07:00
Markus Pilman
cebf49798e Merge remote-tracking branch 'origin/main' into features/always-inject-faults 2022-09-13 12:36:12 -06:00
sfc-gh-tclinkenbeard
2bea5b88bf Add /Atomic/DoAppendIfFits unit test 2022-09-13 11:35:39 -07:00
Trevor Clinkenbeard
9f1e04cf9f
Merge pull request #8147 from sfc-gh-tclinkenbeard/improve-backup-code-coverage
Check backup agent activity in BackupAndRestoreCorrectnessWorkload
2022-09-13 09:32:40 -07:00
Markus Pilman
3e31820078 fix compilation issues 2022-09-13 10:29:29 -06:00
Markus Pilman
6e97bfb15a Added RandomMoveKeys 2022-09-12 17:20:57 -06:00
Markus Pilman
389ea4c952 added additional workloads 2022-09-12 17:16:32 -06:00
Markus Pilman
acd24d6c81 Merge remote-tracking branch 'origin/main' into features/always-inject-faults 2022-09-12 16:44:16 -06:00